Blog tagged as Tokenization

In this article, we'll dive into a recent study that uncovers substantial disparities in the tokenization process used by language models across different languages.

When designing BloombergGPT, a new large language model optimized for financial data, the researchers at Bloomberg made an important decision around tokenization.

When designing BloombergGPT, a new large language model optimized for financial data, the researchers at Bloomberg made an important decision around tokenization.

What is Tokenization? Let's Explore, Using Novel AI's New Tokenizer as a Use Case