python · intermediate · ~6h

Mini tokenizer

Implement a character-level tokenizer first, then merge the most frequent adjacent pairs a few times. Encode and decode must round-trip on the training text.

Requirements

• encode(text) -> list[int]
• decode(ids) -> text
• round-trip the training corpus

Skills: Transformers & LLMs · Lists, dicts, sets