Understanding the Llama 3 Tokenizer | Llama for Developers
AI at Meta · 24:50
Llama 3’s jump over Llama 2 is not just “more data and longer context”: Meta switched the tokenizer from SentencePiece to tiktoken, grew the vocabulary from 32k to 128k, and raised compression so the same text uses fe...