simulated users

Word error rate is the number every vendor quotes and the one that predicts the least. The offline suite is built from confirmed real calls with noise and codecs applied, scored on entity accuracy, tail latency and false interruptions. Production adds what a recording cannot simulate: real callers and the reopen rate.

Nobody deployed anything is true of your repository and false of the system. The model provider, a tool behind an API, the knowledge base and the traffic mix all change without a commit. Segmentation comes before theory: a broken intent and a mix shift produce the same headline and need different fixes.

A nightly suite of fifty simulated conversations per intent catches a collapsed tool with certainty and a five-point drift almost never. Knowing which is which, and sizing each layer for what it can actually see, is the whole design. The grader checks outcomes, not transcripts, or the agent learns to sound resolved.