Why the Count Is Exact, Not Estimated
OpenAI publishes the byte-pair-encoding vocabularies its models use — o200k_base for GPT-5 and GPT-4o, cl100k_base for GPT-4 and GPT-3.5. This tool loads those exact vocabularies and runs the same merge rules, so the count matches what the tokenizer inside the model produces. That's the difference between this and word-count-times-1.3 approximations: for billing and context-limit decisions, the exact number is the one that matters.
How GPT Models Split Text Into Tokens
Tokens are subword chunks, not words. Common English words like 'the' or 'hello' are single tokens, while rarer words split into pieces — 'tokenization' becomes two or three. Spaces attach to the following word, punctuation often stands alone, and numbers split digit by digit in surprising ways. Code is token-dense because of symbols and indentation, and non-Latin scripts can use several tokens per character.
o200k_base vs cl100k_base
The o200k_base vocabulary (GPT-5, GPT-4o) has roughly 200k entries against cl100k_base's 100k, so it compresses text into fewer tokens — the same paragraph typically counts a few percent smaller, with bigger savings on non-English text. If you're migrating prompts from GPT-4 to a newer model, recounting with the right encoding keeps your budget math honest.
