Stage 02 · 02 · LLM core · skill 1 of 2
Transformers & LLMs
Attention, tokens, and why next-token prediction scales.
Recommended first: Neural networks
After this you will be able to
- Explain self-attention in one paragraph
- Tokenize a sentence and estimate context length
- Train a tiny GPT on a toy corpus (Karpathy)
Andrej Karpathy · why we picked it
Completing this unlocks: Fine-tuning basics