migration

Model migrations break things that benchmarks never show, format quirks, refusal shifts, tool-calling drift. The eval-replay and shadow-traffic playbook for migrating without surprises.

Vectors from two versions of an embedding model are not comparable, and nothing in the stack will tell you so. The cosine between the same document's old and new vector is roughly zero, which is why a half-finished migration returns confident nonsense at full speed.

Everyone computes the stale fraction as a share of the corpus, gets 0.03%, and moves on. Weighted by queries rather than documents it is 1.4%, roughly fifty times larger, because the documents that changed during the rebuild are the ones people are asking about.

The obvious engineering answer is to rewrite the notebooks properly and schedule them. Do that and you will have built something correct that the people who own the work can no longer maintain.

torch_xla makes the demo run in an afternoon; making it fast is where migrations die. The lazy-tensor mental model, the recompilation traps, and the honest fork between porting and rewriting in JAX.