FlashSampling: Fast and Memory-Efficient Exact Sampling
Summary
FlashSampling removes the extra memory traffic that sampling causes in large-vocabulary decoding, making categorical sampling faster and more memory-efficient during inference.
Why this matters
Relevant to this topic
Source: arXiv cs.AI · arxiv.org