In a lot of production systems the prompt is the most behaviour-critical string in the codebase and the least governed. It gets edited in a dashboard, tweaked to fix one complaint, and nobody can say afterwards what it used to be or why it changed.
Prompt versioning is the fix, and it is mostly a matter of applying practices the team already uses for everything else.
Put the prompt where the code is
Prompts stored in a vendor dashboard or a database row sit outside review, outside history and outside your deployment process. Somebody can change production behaviour without a pull request, a test run or a record.
Keeping them in the repository gives you diffs, review, history and rollback for free. The objection is usually that non-engineers need to edit them, which is real and better solved with a small editing interface that commits than by moving the source of truth out of version control.
A prompt change is a deployment
Changing a prompt changes what the system does, so it deserves the same treatment as a code change: reviewed, tested against the evaluation set, released deliberately, and reversible.
- Require a second pair of eyes, because prompt edits are unusually easy to get subtly wrong.
- Run the evaluation set before merging and record the scores in the change.
- Tag the version and stamp it onto every log entry the prompt produces.
- Keep rollback to one action, because you will need it under time pressure.
None of this is heavy. It is the difference between a bad change being reverted in two minutes and a day spent working out which of four edits caused the problem.
The worst incidents are the ones where a prompt was edited to fix one customer complaint and quietly broke a case nobody thought to check.
David Kwon, Head of Automation, Engineered With AI
Version the whole configuration together
Output depends on the prompt, the schema, the model and version, the temperature and other parameters, and any retrieval settings. Changing any of them changes behaviour, so versioning the prompt alone leaves most of the surface unmanaged.
Treat the set as one unit with a single version identifier. That is what makes a log entry reproducible months later, and reproducibility is the whole point of the exercise.
Test the change, not the intention
A prompt edit is made to fix a specific case. The risk is entirely in the cases nobody was thinking about, so the evaluation set has to cover the ordinary path as well as the problem.
Add the case that prompted the change to the set, with its correct answer, before making the change. It then becomes a permanent guard, and the set grows toward the places the system actually fails rather than the places somebody imagined it might.
Watch for the fix that only moves the problem
Instructions added to suppress one behaviour frequently suppress a legitimate neighbouring one. A rule stopping the model speculating can also stop it giving a reasonable answer where speculation was appropriate, and that trade is worth seeing in the evaluation output rather than discovering later.
Roll changes out gradually where you can
Where volume allows, running a new prompt against a small share of traffic first surfaces problems on real inputs at a survivable scale. Compare the operational signals rather than waiting for complaints: length distribution, validation failures, refusal rate, escalations.
Where a gradual rollout is not practical, release early in the day rather than at five o'clock, and watch it. The instrumentation for this overlaps entirely with observability, which is worth having in place first.
Write down why, not just what
A diff shows what changed. It rarely shows why, and six months later the reason is the thing you need, usually because someone is about to reverse the change for a reason that was already considered.
One line in the commit message covering the case that prompted it and the evaluation result is enough. It costs nothing at the time and it is the difference between a prompt that improves over time and one that oscillates.
Prompts changing without a trail?
We will put your prompts under proper version control with tests, so changes are safe to make.