Measure throughput and user latency before choosing a batching policy

Use a scoped roofline estimate and a three-request timing fixture to distinguish queueing, prefill and decode. Calculate aggregate throughput over a common window, compare goodput against latency limits, and test batching rather than assuming a universal sweet spot.

30 MIN · PLUS