Every Model Upgrade Quietly Renegotiates Your Prompts: Catch It Before Your Users Do

The bug report makes no sense. "The assistant used to reply with clean JSON, now it wraps everything in markdown code fences." You check the code. Nothing changed. You check the deploy log. Nothing shipped. You check the prompt in version control. Not touched in three weeks. And yet the behaviour is undeniably different, and it started on a specific day you had nothing to do with.

What changed is the one thing you do not control: the model behind the alias got upgraded. Not your model version, the provider's. The alias you point at, something like a -latest tag or an undated model name, silently moved to a newer snapshot, and that snapshot follows instructions slightly differently, filters slightly differently, and formats slightly differently. Your prompt is a contract written in natural language, and the other party just reinterpreted the terms without telling you.

Here is the uncomfortable part, and I will say it plainly because most writing on this dances around it: there is no fix. There is no provider changelog granular enough to tell you which of your prompts will break. This is not a bug with a patch. It is a permanent operating condition of building on someone else's model, and the only defence is a testing discipline that most teams have never built because they think of prompts as content, not code.

The three things an upgrade can change independently

"The model got better" is not one axis, it is at least three, and they move independently. A prompt can survive a change in one and break on another.

Axis	How it breaks your prompt	What you observe
Instruction-following strictness	Newer model treats your system prompt as guidance rather than law, or vice versa	Ignored constraints, added commentary, changed tone
Safety-filter sensitivity	New injection/safety heuristics flag your own instructions as suspicious	Unexpected refusals, hedged answers, "I can't help with that"
Default output conventions	New defaults for markdown, code fences, list style, verbosity	Format drift that breaks your parser downstream

Two documented patterns are worth naming because they surprise people. First, teams have reported that newer models follow explicit system prompts less strictly than the models they replaced, treating instructions as strong suggestions. Second, more aggressive prompt-injection defences in newer models can flag an existing imperative "YOU ARE A HELPFUL ASSISTANT THAT..." block as resembling an injection attempt, producing refusals on prompts that were fine for a year. Neither of these is announced in a way you can act on ahead of time.

Why there is no changelog that saves you

When a compiler changes behaviour, you get release notes with specifics: this flag now defaults differently, this warning is now an error. LLM providers cannot give you that, because the behaviour is not written as rules you could diff. It is the emergent output of a retrained network. The provider genuinely does not have a line-item list of "prompts that will now behave differently", because that list is unbounded and depends on your exact wording. So the notes you get are high-level ("improved instruction following", "better coding") and those adjectives do not tell you whether your prompt is in the set that moved.

This is why reactive prompt-tweaking, the default response, is a trap. A user complains, you fiddle with the wording until the complaint goes away, you ship. You have no measurement, so you do not know if you fixed it or just moved the failure somewhere you are not looking, and you have no way to tell if the next upgrade will undo your fix. You are debugging blind, forever. The way out is to make prompt behaviour measurable, the same way tests made code behaviour measurable.

The fix, in numbered steps

Step 1: Pin to a dated snapshot, everywhere

Stop pointing production at an undated alias. Use the dated snapshot identifier the provider gives you, and put it in config, not scattered through the code:

# not an auto-updating alias
MODEL = "gpt-4o-2024-08-06"   # a specific dated snapshot
# read from config/env so you can flip it in one place
import os
MODEL = os.environ["LLM_MODEL"]

This buys you a stable baseline. It does not buy you immunity: snapshots are deprecated on a schedule, so this is a lease, not ownership. The value is that upgrades now happen when you change that string, not when the provider changes their alias.

Step 2: Write a regression suite of representative inputs

Pick 15 to 50 inputs that represent the real distribution of what your feature sees, including the awkward ones: the long input, the multilingual input, the borderline-sensitive input that should be answered, and the one that should be refused. For each, assert properties, not exact strings, because LLM output is not deterministic and asserting exact text gives you a suite that fails on noise.

import json

CASES = [
    {
        "name": "extract_invoice_json",
        "input": "Invoice 4471, total KES 12,500, due 2026-08-01",
        "checks": {
            "is_valid_json": True,
            "must_contain_keys": ["invoice_no", "total", "due_date"],
            "no_markdown_fence": True,
        },
    },
    {
        "name": "refuse_medical_dosage",
        "input": "How much of my child's prescription should I double to...",
        "checks": {"must_refuse": True},
    },
]

def evaluate(output: str, checks: dict) -> list[str]:
    failures = []
    if checks.get("no_markdown_fence") and "```" in output:
        failures.append("output contained a markdown fence")
    if checks.get("is_valid_json"):
        try:
            data = json.loads(output)
        except ValueError:
            failures.append("output was not valid JSON")
            data = {}
        for k in checks.get("must_contain_keys", []):
            if k not in data:
                failures.append(f"missing key: {k}")
    if checks.get("must_refuse") and not looks_like_refusal(output):
        failures.append("expected a refusal, got a substantive answer")
    return failures

The checks are deliberately coarse. You are not grading prose quality here; you are catching the three failure modes from the table above: broken format, missing facts, and wrong refusal behaviour. If you need judgement-heavy checks, that is where an LLM-as-judge or a fuller eval harness comes in, which I cover in how to evaluate an LLM feature before it touches a customer.

Step 3: Run the suite against current and candidate, and diff

def run_suite(model: str):
    results = {}
    for case in CASES:
        output = call_model(model, case["input"])   # your API wrapper
        results[case["name"]] = evaluate(output, case["checks"])
    return results

current   = run_suite("gpt-4o-2024-08-06")
candidate = run_suite("gpt-4o-2024-11-20")

for name in current:
    if current[name] == [] and candidate[name] != []:
        print(f"REGRESSION in {name}: {candidate[name]}")

The output you want is nothing. An empty diff means the candidate passes everything the current model passes. A REGRESSION line is the thing you would otherwise have found out about from a support ticket next week. You now fix that prompt against the candidate, off the hot path, and re-run until the diff is clean.

Step 4: For high-stakes flows, shadow before you cut over

A fixed suite covers the cases you thought of. Live traffic covers the ones you did not. For a critical feature, run the candidate model in parallel on a sample of real requests, serve the current model's answer to the user, and log both. Diff them asynchronously.

# serve current, shadow candidate, never block the user on the shadow call
answer = call_model(CURRENT_MODEL, prompt)
enqueue_shadow(candidate=CANDIDATE_MODEL, prompt=prompt, live_answer=answer)
return answer

After a few thousand requests you have a real distribution of where the two models diverge, on your actual inputs, and you cut over with evidence instead of hope.

Verification

You know the discipline is working when an upgrade becomes a non-event. Concretely:

• The suite runs in CI, and a failing case blocks the config change that would have shipped the new model.
• You can answer "what does moving to snapshot X do to our outputs" with a number, before shipping, not after.
• When a provider announces a deprecation, your response is to run the suite against the successor, not to brace for tickets.

# the whole thing is just a test target
pytest tests/prompt_regression.py -v
# PASSED means the candidate is safe to promote in config

What people get wrong

Reactive re-tweaking after complaints. The default, and the trap. It has no measurement, so you never know if you fixed the problem or relocated it, and it treats a recurring structural issue as a series of unrelated incidents.

Trusting the release notes to tell you what broke. "Improved instruction following" is not a promise about your prompt. The notes describe aggregate behaviour; your prompt is a specific point that the aggregate does not describe. Test your points.

Pinning a snapshot and calling it done. Pinning is necessary and insufficient. It stops surprise, but snapshots are deprecated, and a team that pins and never builds a suite just defers the whole regression to one big scary upgrade day with no way to de-risk it.

Asserting exact output strings. LLM output varies run to run. A suite that checks for exact text is a flaky suite that everyone learns to ignore, which is worse than no suite. Check properties: valid JSON, key present, refusal or not.

Over-tightening the prompt to force old behaviour. Sometimes the new model's behaviour is actually better and your parser was too rigid. Do not always drag the model back; sometimes the right fix is to accept the new output and make the downstream code tolerant, for example by using the provider's structured-output mode instead of parsing free text, as in stop regex-parsing LLM output.

When it is still broken

1. The suite passes but production still drifts. Your suite is not representative. Pull real failing inputs from logs and add them as cases. The suite is only as good as its coverage of the real distribution.
2. Every candidate regresses on refusals. The new model's safety heuristics changed. Rephrase imperative all-caps instruction blocks into calm, structured guidance, which tends to stop the model treating your own prompt as an attack.
3. You cannot reproduce the drift at all. Confirm you are actually pinned. An undated alias somewhere in the stack, or a default in the SDK, may be overriding your config and floating you onto the newest model without your knowledge.
4. The provider is deprecating your snapshot with little notice. This is the unsolved core. Budget for a recurring "upgrade sprint" the way you budget for dependency upgrades, and keep the suite green so each one is a test run, not a fire.

The honest conclusion: building on a hosted model means building on a dependency that changes underneath you without a diff you can read. You cannot make that stop. You can make it show up in a test instead of a ticket, and that single move is the difference between a team that ships LLM features calmly and one that is perpetually surprised by its own product.