13Design a job queue for long-running LLM tasks, priorities, cancellation, progress, and poison messages.▼mediumOpenAIAnthropicRetool1 replies○ sign inA 'simple' queue question with four traps wired in: minutes-long tasks break every default timeout, cancellation has to actually stop spend, progress needs a contract, and one bad job must never wedge the lane.Open full answer →
55When a user closes the browser mid-inference, how do you propagate cancellation and actually stop the GPU work?▼hardOpenAIAnthropicVercel1 replies◆ premiumA 2-second inference and a user who closed the tab. Naive stacks keep the GPU burning to completion. Real cancellation means propagating disconnect all the way to the batch scheduler.Open full answer →
