Prompt drift — the slow degradation of a once-reliable prompt as vendors update models underneath it and you edit it in small unrecorded steps — is catchable with a habit borrowed from software: keep every prompt as a numbered version with a changelog line and a date, hold a small fixed pack of test inputs with their accepted outputs, and rerun the pack whenever the prompt or the model changes. The entire system fits in a folder and takes minutes per run. Its payoff is answering the question that otherwise eats evenings: did my prompt break, or did the model change under it, or was today's input just weird?
RechargeMe publishes information, not advice. This workflow uses documented behavior of AI products — silent model updates and API deprecations are vendor-documented facts — and standard file discipline anyone can replicate.
Where does drift come from?
Three documented sources. Vendor updates: chat products change default models silently, and API versions retire on published schedules, so a prompt tuned against one model generation meets a different one without asking you. Your own edits: five one-word tweaks over three months, none recorded, and the version that worked exists nowhere. And input drift: the material you feed the prompt shifts — longer documents, different jargon — until it falls outside the envelope the prompt was shaped for. The habit below separates the three, because each has a different fix: re-tune, revert, or resize.
What are the three files?
One: the prompt file itself — plain text, numbered versions stacked or in an archive folder, each with a two-line header recording date, model and version it was tuned on, and status (active, retired, testing). Two: the test pack — five to ten fixed inputs representative of your real work, including one edge case and one input that should trigger the prompt's refusal or gap-flagging rule. Three: the acceptance log — for each test input, the output you approved, recorded once and then pasted alongside on every rerun. The comparison of fresh output against accepted output is the whole detection mechanism; no tooling required.
Related stories: How to build reusable prompt templates you'll still trust in six months · Voice dictation as a serious input: a workflow, not a party trick.
How does a rerun actually go?
Minutes, not hours. Paste each test input through the current prompt on the current model, put the outputs next to the accepted ones, and mark each pair same, acceptable, or broken. "Acceptable" covers harmless variation in phrasing; "broken" means format collapsed, the gap-flagging rule stopped firing, or quality visibly dropped. All broken — or a majority — points at the model or prompt version: check the version header against the model actually running, and if the vendor moved underneath you, re-tune from the current version, increment it, log the change. One broken on one input points at input drift: that input has left the envelope; fix the prompt for it or route such inputs elsewhere. The pattern of breakage is the diagnosis.
| Signal | Likely cause | Fix |
|---|---|---|
| Most tests broken at once | Model changed under the prompt | Re-tune, increment version, record new model |
| Broke right after your edit | Your edit | Revert to prior version, redo smaller |
| One input newly broken | Input drift | Extend prompt bounds or reroute input |
| Slow fade over months | Accumulated micro-edits | Reset to a known-good archived version |
Why not just use version-control software?
Git works, and for teams sharing prompts it is the right answer — diffs, history, and pull requests are exactly this discipline, industrialized, and the NIST AI Risk Management Framework's emphasis on documented, versioned AI processes points the same direction for organizations. For one person with ten prompts, a dated text file with stacked versions delivers ninety percent of the value at five percent of the ceremony. The failure mode to avoid is the middle: a git repository set up with good intentions, updated twice, abandoned — the habit matters more than the tool, and the tool should be whatever you will actually maintain.
How do you know which model you're testing on?
The underrated part of the system. In chat products, check the model selector and release notes — vendors document default-model changes there, though not always loudly. Via API, record the exact model version string in the prompt file's header; versioned endpoints are a documented feature of serious API usage and pin behavior until you deliberately move. This metadata is what turns "the output got worse" into "the output got worse when the vendor moved from version A to B in March," which is a fact you can act on and a fact your future self will thank you for.
FAQ
- How many test inputs do I need? Five to ten: typical cases, one edge case, one that should trigger the refusal rule. Enough to localize breakage, few enough that reruns actually happen.
- How often should I rerun the pack? On every prompt edit, on every known model change, and on a calendar tick — monthly for load-bearing prompts, quarterly for the rest.
- Do I need git? For teams, yes; for one person, dated text versions are enough. Choose the tool you will maintain, not the one that sounds rigorous.