Learning to Read Out: Unembedding Dynamics in Language Model Pretraining
Summary
This study tracks when a language model acquires a capability during pretraining, finding the key is when its internal readout mechanism forms, not when it first answers correctly.
Why this matters
Covers model · Relevant to this topic
Source: arXiv cs.AI · arxiv.org