LLM Token Counter

Before a language model reads a prompt, the text becomes a sequence of numbered pieces called tokens. A token may represent a word, part of a word, punctuation, whitespace, or part of a character's encoded bytes. Counting these pieces helps establish how much text can fit in a request and contributes to cost calculations when a service charges by token.

Visible length can be misleading. A paragraph, a block of code, and a Japanese sentence can have very different token counts relative to their word or character counts. Code identifiers, punctuation, spaces, and line breaks all participate in tokenization. A fixed characters-per-token estimate cannot establish the exact size of an unfamiliar document.

Byte pair encoding, or BPE, builds tokens from recurring sequences of bytes. Its learned vocabulary allows common fragments to form larger pieces while other text is represented by smaller pieces. An encoding specifies the vocabulary and splitting rules. The same text can therefore produce different tokens under cl100k_base and o200k_base, even when its spelling and meaning remain unchanged.

These exact examples contain no added quotation marks or trailing newline. They show why neither word count nor character count provides a universal conversion to tokens.

A Unicode code point is a numbered element of Unicode text. In these examples, each visible character occupies one code point, but that relationship does not hold for every accented character or emoji sequence. Token boundaries are different again. The long English word uses six tokens in both encodings, yet the pieces and their numerical IDs differ. A smaller count from another encoding is useful only when that encoding matches the intended model.

The context window is the token capacity available for a model request. Instructions, conversation history, reference material, and the generated response compete for that capacity. A document that fits on its own may leave too little space once earlier messages and a substantial answer are included. Reserving room for the reply makes that tradeoff explicit. A larger reply allowance leaves less room for input, while a smaller allowance can constrain the answer.

An exact count of ordinary text covers the supplied text under one encoding. A complete application programming interface (API) request can also contain message formatting, special control tokens, tool definitions, and non-text content. Those additions require separate accounting. Context capacity and any separate output limit also depend on the model, so an encoding name alone does not establish either limit.

How to Use This Tool:

Start with the exact text you plan to send and the encoding documented for your model. Counts and budget results update after edits.

1. Paste your text into Text, replacing the sample, or choose Browse file to load one UTF-8 TXT or Markdown file. Markdown punctuation and code are counted as written.
   If a file is rejected, correct its encoding or size before loading it again. The previous text stays in place.
2. Select o200k_base or cl100k_base under Encoding according to the model's documentation.
3. Enter the model's actual Context window in tokens. The initial 128,000-token value is an editable example.
4. Set Reply reserve to the generation allowance you want to keep available. Use the number field when you need an exact allowance.
5. Add any separately measured tokens for other request content through Advanced and Extra request overhead. Its initial value is zero, and no automatic overhead estimate is added.
6. Check Exact text tokens and the Window result, then compare Reply reserves to choose an allowance that fits.
   If Check the inputs appears, correct the field named in the message. Invalid settings hide results until the values are valid again.

Interpreting Results:

Exact text tokens counts the complete accepted text under the selected encoding. Changing the context window, reply reserve, or extra overhead changes the budget without changing that text count. Remaining tokens are calculated after both the reply reserve and overhead have been deducted.

Fits confirms the arithmetic for the values entered. Before sending a complete request, verify its model encoding and context limit and account for content outside the pasted text. A provider's complete-request token count can check those additions. Increasing the context setting only represents more capacity when the chosen model actually supports it.

Reply reserves compares several allowances, including your current choice and the calculated maximum context allowance. The Context headroom chart plots remaining tokens against reply reserve, with a zero line marking the fit boundary. Both describe available space without predicting response length or answer quality.

A positive Input price under Advanced displays Text input cost. A zero price hides this figure; it does not establish that a service has no charge.

Token boundaries previews the first 400 tokens. The summary still counts all of them, and copying or downloading the token trace includes every token. Display markers make spaces, newlines, and tabs visible as ·, ↵, and →. A hexadecimal byte fragment can be a valid part of a character split across tokens; it does not by itself indicate damaged text.

Technical Details:

Transformation Core

Tokenization first divides text into groups using the encoding's rules for letters, numbers, whitespace, and punctuation. Each group becomes UTF-8 bytes. Within a group, byte pair encoding repeatedly merges the eligible adjacent pair with the highest learned priority, stopping when no permitted merge remains. Each resulting piece maps to an integer token ID. These rules and priorities are fixed, so the same complete text and encoding produce the same sequence.

For antidisestablishmentarianism, the transformation produces the following pieces and IDs. Each row contributes one token, so both sequences contain six tokens even though their middle pieces differ.

The complete byte sequence reconstructs the original text. Individual tokens need not end at UTF-8 character boundaries. In the Japanese example above, the character 誕 has the UTF-8 bytes e8 aa 95, split between token fragments e8 aa and 95 in both encodings. Those fragments become a complete character when joined in order.

Capitalization, whitespace, line endings, and any byte order mark remain part of the counted text. No Unicode normalization is applied. Literal text such as <|endoftext|> is counted as ordinary characters. Special tokens inserted as part of a model's request format need separate accounting. Comparisons should therefore keep the exact text and its encoding fixed, including leading spaces and trailing newlines.

Encoding and model compatibility

o200k_base has a larger vocabulary than cl100k_base and different text-splitting rules. Some text uses fewer tokens with it, as the Japanese example demonstrates, but a larger vocabulary does not promise a smaller count for every string. The encoding names describe token vocabularies; they do not specify context-window capacity.

These are representative mappings documented by OpenAI. Confirm the encoding for the exact deployed model, particularly when using a newer or differently named variant.

Exact counting here is limited to the two listed ordinary-text encodings. An embedding model accepts input for producing a numerical representation of text, so a generated-reply reserve is unnecessary for that use. Other tokenizer families require their own matching tokenizer.

Formula Core

Let N be Exact text tokens, C the Context window, O Extra request overhead, and R the Reply reserve. All four quantities are measured in tokens. Headroom H is the context left after adding the text, overhead, and planned reply.

A budget fits when H is greater than or equal to zero. Each additional token reserved for the reply reduces headroom by exactly one token. Extra overhead has the same effect on space, while the original text token count remains unchanged.

The context available for a reply before choosing a reserve is M. Negative available capacity is reported as zero.

If text plus overhead already exceeds the window, a zero reply reserve still leaves the request over budget. This context allowance also remains subject to the model's separate output limit. For reasoning models, generation capacity may need to cover reasoning tokens as well as the visible answer.

For optional text input cost K, let P be the entered Input price in USD per million text-input tokens. Multiplying the counted text by that rate and dividing by one million gives the cost in USD.

At a hypothetical price of USD 2.50 per million tokens, the seven tokens in 2 + 2 = 4 cost USD 0.00001750. Only the counted text contributes to this figure. Extra overhead, generated output, cached-input rates, and other provider charges are excluded.

The Chars/token ratio divides the text's Unicode code-point count, U, by its token count. It describes this particular text and encoding.

Empty text has zero tokens and displays a ratio of zero. Token counts and budget arithmetic use whole numbers without rounding. The ratio is displayed to at most two decimal places, and a displayed input cost has eight decimal places. Decimal cost arithmetic uses floating-point precision; display rounding does not change the context calculation.

Text length and accepted limits

Tokens, Unicode code points, and encoded text length are different measurements. The text limit uses UTF-16 code units, so many emoji consume two units even when the displayed character count increases by one. The file limit measures UTF-8 bytes, and loaded text must also fit the text-length limit.

These bounds are inclusive and describe accepted entries, without asserting any model's actual capacity. Blank numeric fields are invalid. Invalid UTF-8 files, text beyond the length limit, and text that cannot be encoded and decoded without change are rejected.

Privacy Notes:

Text and loaded file contents are tokenized locally in your browser. They are not uploaded to a token-counting service.

• Settings links preserve the encoding and numeric settings. They exclude the entered text and file contents. Reloading restores the sample, so your own text must be supplied again.
• The tokenizer and optional chart require external resources to load. If a connection or loading problem prevents tokenization, Retry tokenizer can try again while retaining the current text.
• Copied or downloaded token traces contain text fragments and token IDs. Sharing a trace can disclose the original content, even though a settings link does not contain it.

Worked Examples:

References:

• Byte pair encoding algorithm, OpenAI.
• How to count tokens with Tiktoken, OpenAI Cookbook.
• Token encoding definitions, OpenAI.
• Model-to-encoding mappings, OpenAI.
• Managing the context window, OpenAI.
• Counting tokens for complete requests, OpenAI.
• Vector embeddings, OpenAI.
• UTF-8, UTF-16, UTF-32 & BOM, Unicode Consortium.