migration

Model migrations break things that benchmarks never show, format quirks, refusal shifts, tool-calling drift. The eval-replay and shadow-traffic playbook for migrating without surprises.

The cheap swap is not a model swap. The prompt becomes atomic typed questions, the parsed label becomes a distribution with a confidence, and the release gate becomes a per-version threshold sweep on the customer's own labeled tickets.

Vectors from two versions of an embedding model are not comparable, and nothing in the stack will tell you so. The cosine between the same document's old and new vector is roughly zero, which is why a half-finished migration returns confident nonsense at full speed.

Everyone computes the stale fraction as a share of the corpus, gets 0.03%, and moves on. Weighted by queries rather than documents it is 1.4%, roughly fifty times larger, because the documents that changed during the rebuild are the ones people are asking about.

The obvious engineering answer is to rewrite the notebooks properly and schedule them. Do that and you will have built something correct that the people who own the work can no longer maintain.

torch_xla makes the demo run in an afternoon; making it fast is where migrations die. The lazy-tensor mental model, the recompilation traps, and the honest fork between porting and rewriting in JAX.