AI Tools & Models HOTFeaturedAll
arXiv cs.AIScore60

FlashSampling: Fast and Memory-Efficient Exact Sampling

Summary

FlashSampling removes the extra memory traffic that sampling causes in large-vocabulary decoding, making categorical sampling faster and more memory-efficient during inference.

Why this matters

Relevant to this topic

Open original ↗

Source: arXiv cs.AI · arxiv.org

Related items