VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling
Summary
This work stabilizes first-token and inter-token latency for interactive LLM services through tail-calibrated scheduling.
Why this matters
Covers LLM and inference · Relevant to this topic
Source: arXiv cs.AI · arxiv.org