🧩 Tokenizer: how AI sees text
AI doesn't read letters and words. It sees 'tokens' — chunks of text. Paste anything and watch your text turn into tokens.
Клубника·—·strawberry.·Сколько·в·ней·букв·«r»?·🍓
17 tokens49 characters2.9 chars/token
🍓 Why AI can't count letters
To AI, 'strawberry' is 2-3 tokens, not 10 letters. It doesn't see the individual 'r's inside a token — that's why it often miscounts letters. Now you know why.
💸 Your language can be 'more expensive'
The same phrase in different languages = different token counts. Many languages cost more tokens than English — meaning more expensive and slower with AI. Look:
EN 🇬🇧
8
RU 🇷🇺
14
中文 🇨🇳
8
🧠 Context window: why AI 'forgets'
AI has limited memory — a 'context window'. When there are too many tokens, the oldest ones fall out and AI 'forgets' the start of the conversation.
17 / 8,000 tokens used