LACCO: Local Accumulation with Computation--Communication Overlap
Summary
This work cuts synchronization overhead when training language models over low-bandwidth networks by overlapping local computation with communication.
Why this matters
Covers model · Relevant to this topic
Source: arXiv cs.AI · arxiv.org