AI Tools & Models HOTFeaturedAll
arXiv cs.AIScore70

VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling

Summary

This work stabilizes first-token and inter-token latency for interactive LLM services through tail-calibrated scheduling.

Why this matters

Covers LLM and inference · Relevant to this topic

Open original ↗

Source: arXiv cs.AI · arxiv.org

Topics

Related items