58Your voice agent talks over callers and mishears account numbers, yet word error rate is 6%. Diagnose both, and say what to measure instead.▼hardNewSierraDecagonCresta2 replies◆ premiumA 6% word error rate is a good number that measures the wrong thing. Per-digit accuracy of 98% loses 18% of ten-digit account numbers, and no aggregate metric contains that. Talking over callers is a turn-taking problem, not a model problem, and it has its own metrics too.Open full answer →
59Design the eval for a voice agent taking inbound support calls. What do you measure offline, and what can only production tell you?▼hardNewSierraDecagonCresta2 replies◆ premiumWord error rate is the number every vendor quotes and the one that predicts the least. The offline suite is built from confirmed real calls with noise and codecs applied, scored on entity accuracy, tail latency and false interruptions. Production adds what a recording cannot simulate: real callers and the reopen rate.Open full answer →
