/tags/bpe/Recent content in BPE on Hitesh PattanayakHugoen-usWed, 02 Oct 2024 00:00:00 +0000/posts/tokenization/Wed, 02 Oct 2024 00:00:00 +0000/posts/tokenization/Take a detour before diving into transformers and explore sub-word tokenization techniques like Byte-Pair Encoding, WordPiece, and Unigram models. Learn how they handle rare words, reduce vocabulary size, and make models more efficient!