Models Get Retired: Planning for the Next Version
The model you build on will be replaced. Designing for that change, with tests and swappable components, is part of the work.
On this page
The model you build on today will be replaced. The replacement is usually newer and better on average, sometimes cheaper, and occasionally different in ways your product depends on. Retirement is not an emergency if you planned for it. It becomes one when prompts, tests and integrations were written around a single model name with no record of what that model actually did.
How models change under you
Providers release new versions, deprecate old ones and publish retirement dates. Some offer aliases that point to the latest version of a model family. An alias is convenient, but its target moves, so your product may change without any deliberate decision on your part. Other providers offer dated or numbered versions that stay fixed until they are retired.
Notices usually come with a stated period before a version stops being available. The length of that period and the channels used for notices are described in the provider's documentation. Read it when you start a project and again when a notice arrives. Do not rely on memory of it. Also remember that an announced retirement date is not necessarily the date your traffic changes. If you use an alias, the change may arrive earlier, when the alias is repointed.
Retirement is also a moment to ask whether the model is still the right choice, not only whether a replacement exists. Newer options may suit the task better, cost less for the same quality, or behave worse in ways your evaluation would catch. The scheduled change is an opportunity to recheck the original decision.
Pinning and portability
Pin to a specific version wherever the provider supports it, and record the identifier in one place, such as a configuration file. Keep prompts out of application code where practical. Store them as versioned files that can be changed without a full redeploy, and keep the evaluation set in the same repository so the two travel together. Store evaluation results with the model identifier as well, so the history shows how each prompt performed on each version. That history lets you compare a candidate against the version you are leaving, rather than against memory.
Record the settings each prompt was tuned for, such as sampling values and output limits. A new model may need different values to behave the same way. Write a thin interface around model calls, so the rest of the product does not depend on one provider's response shape. Avoid provider-specific features unless the benefit is clear, and keep a list of the ones you do use. That list becomes the first input to any future migration.
A migration checklist
When a replacement is needed, a numbered sequence keeps the work visible:
- Confirm the new model's version, its retirement date and its terms in the provider documentation.
- Run the full evaluation set against the current and candidate models, with identical settings.
- Compare results by category rather than overall, and list each regression by name.
- Adjust prompts for the regressions, rerunning the evaluation after every change.
- Check every piece of code that depends on output format, including parsers, validators and downstream consumers.
- Roll out to a small share of traffic, watching the logs and the fallback rate, and keep the old version available until the new one has run cleanly for a set period.
Cost and latency belong in the same review, measured under realistic load rather than from a quiet test. Decide in advance what would trigger a rollback, and make rolling back a configuration change rather than a new deployment. A rollback that takes an afternoon of work is not much of a plan.
The real costs of switching
Switching is more than swapping a name. Outputs change in wording, length and structure, and those changes can break parsers and the systems that read them. Behaviors that the old model handled through quirks, such as its treatment of ambiguous instructions, may change in ways nobody wrote down. Prompts tuned over months may need retuning. Cost and latency can shift, and the new model may need different settings to perform the same way.
Users may notice differences in tone even when accuracy is unchanged. Estimate the effort honestly. Include evaluation time, prompt work, changes to validation, and the rollout period. The first migration usually takes longer than expected. Later ones are faster, because the evaluation set, the interface layer and the logs already exist. Staff and users also carry habits built around the old model. Support teams have learned its failures and built workarounds, and people have adjusted how they phrase requests. A migration removes some of those workarounds and creates unknowns. Write those down before the switch, so they are not discovered later through support tickets.
Practice: run a migration dry run
Pick one feature in a product you work on, or imagine one that you know well. Write the migration checklist for it using the steps above, then fill in what each step would require. Name the evaluation set, the parsers that would need checking, the rollout share and the logs you would watch.
Estimate the effort in working days, as a range, and state the assumptions behind it. Finally, identify one dependency that would be hard to move, such as a provider-specific feature or an unwritten assumption about output format, and write down what you would do to reduce it. The exercise shows whether your system can be moved or only assumed to be movable.
