Why Your Token Counter Thinks a 2MB Audio Clip Is 3 Million Tokens

You wire up a multimodal chat feature. A user uploads a short voice note, your app base64-encodes it into the message, and before the request even leaves your server your token counter screams that this conversation is 3.1 million tokens. Your context-management code panics, truncates the entire chat history to make room, and the model answers a question it can no longer see.

The audio clip was 2 MB. The model would have billed it as a few thousand tokens at most. Your counter was off by roughly three orders of magnitude, and it was off in the most damaging possible direction: it threw away real context to make room for a cost that does not exist.

This is not a rare edge case. Every char-based token approximation does this the moment you hand it binary content, and most teams do not notice until history starts vanishing or their cost dashboard shows a "conversation" that supposedly cost more than the monthly bill.

The symptom in code

A multimodal message is not a string. It is a list of typed content blocks, something like:

message.content = [
    {"type": "text", "text": "What is in this recording?"},
    {"type": "input_audio", "input_audio": {"data": "<~2.7 million base64 chars>", "format": "wav"}},
]

Now watch what a generic approximator does. Most of them are built for text and, when handed a non-string, fall back to stringifying the whole thing:

# The naive approximation, roughly what many helpers do internally
def count_tokens_approximately(content) -> int:
    text = content if isinstance(content, str) else repr(content)
    return len(text) // 4   # "4 chars per token"

For the message above, repr(content) is dominated by the 2.7 million base64 characters, so this returns about 685,000 tokens. The model, given the same audio, bills it in the low thousands. The helper did not measure tokens. It measured the length of an encoding that the model never sees as text at all.

Why char-counting collapses for binary content

The chars / 4 rule works for English prose because a subword tokenizer really does average around four characters per token on natural language. It is a fine back-of-envelope guess for text. It has no meaning for base64 for two independent reasons.

First, base64 is not text the model tokenizes. When you send an image or audio block, the provider decodes the base64 back into bytes, processes those bytes through a vision or audio encoder, and charges you for the output of that encoder, not for the transport string. The base64 characters are packaging. Counting them is like weighing a parcel by weighing the shipping label after photocopying it a thousand times.

Second, base64 actively inflates. It encodes 3 bytes of binary into 4 ASCII characters, so the text form is about 1.37 times the raw byte size before you even apply the bogus / 4. Chain the two together and a binary attachment of B bytes produces roughly B * 1.37 / 4 phantom tokens. That is where the 10x to 100x inflation comes from, and it scales linearly with file size, so bigger uploads look catastrophically expensive.

What the providers actually bill (where it is documented)

Multimodal token accounting is provider-specific and, frankly, under-documented, which is exactly why a single cross-provider "just count the chars" shortcut exists in the first place. Here is what is publicly documented and stable enough to build on.

Modality	What it is billed on	Usable heuristic
Image (OpenAI, detail high)	Resized dimensions split into 512px tiles	base tokens + per-tile cost from the resized width and height
Image (OpenAI, detail low)	Flat	a small fixed cost regardless of size
Image (Anthropic)	Pixel area	approximately (width x height) / 750 tokens
Audio	Playback duration	seconds x provider per-second token rate
Video	Sampled frames + audio track	frames sampled x per-frame image cost + audio estimate
Text / PDF text layer	Extracted text	the normal subword estimate on the extracted text

The single thing every row has in common: none of them is a function of the base64 length or the file size in bytes. An image is billed on pixels, audio on seconds, video on frames. If your estimator's answer changes when you re-encode the same image at a different compression level, it is measuring the wrong thing.

A token estimator that special-cases binary blocks

The correct shape is: walk the content blocks, char-count only the text ones, and hand every binary block to a per-modality estimator that never looks at the payload string. This is deliberately an estimate for a client-side guard; when you need the real number, see the next section.

import base64, math

def estimate_text_tokens(text: str) -> int:
    return max(1, len(text) // 4)

def estimate_image_tokens(width: int, height: int, detail: str = "high") -> int:
    # OpenAI-style tiling. Adjust constants to your provider's published values.
    if detail == "low":
        return 85
    # scale so the long side <= 2048, then the short side <= 768
    scale = min(2048 / max(width, height), 1.0)
    width, height = width * scale, height * scale
    scale = min(768 / min(width, height), 1.0)
    width, height = width * scale, height * scale
    tiles = math.ceil(width / 512) * math.ceil(height / 512)
    return 85 + 170 * tiles

def estimate_audio_tokens(seconds: float, tokens_per_second: float) -> int:
    # tokens_per_second is provider-specific; read it from their docs.
    return max(1, int(seconds * tokens_per_second))

def estimate_message_tokens(content) -> int:
    if isinstance(content, str):
        return estimate_text_tokens(content)

    total = 0
    for block in content:
        btype = block.get("type")
        if btype == "text":
            total += estimate_text_tokens(block["text"])
        elif btype in ("image", "image_url", "input_image"):
            # Use known dimensions if you have them; do NOT read the base64.
            w = block.get("width", 1024)
            h = block.get("height", 1024)
            total += estimate_image_tokens(w, h, block.get("detail", "high"))
        elif btype in ("input_audio", "audio"):
            total += estimate_audio_tokens(block.get("seconds", 0.0),
                                           tokens_per_second=block.get("tps", 10.0))
        else:
            # Unknown binary block: a small fixed floor beats char-counting it.
            total += 128
    return total

The load-bearing line is the else branch. When you meet a block type you do not recognise, a small fixed floor is a far better estimate than len(repr(block)) // 4, because you already know the wrong answer is "the base64 length", and any constant is closer to the truth than that. The one thing you must never do is fall through to stringifying the block.

Verification: compare against the real counter

Do not trust your heuristic on faith. Every major provider exposes an authoritative token count, either a dedicated endpoint or a usage object on the response. Send one representative multimodal message both ways and compare.

# After a real call, the response carries the ground truth:
resp = client.messages.create(model="...", messages=messages, max_tokens=64)
print("real input tokens:", resp.usage.input_tokens)
print("my estimate:", estimate_message_tokens(messages[-1]["content"]))

Your estimate should land within a sane band of the real number, say plus or minus 20 percent, and crucially it must not scale with file size. Re-encode the same image larger on disk and rerun: the real count barely moves and your estimate must barely move too. If your estimate jumps, you are still char-counting something. For the pre-flight guard, prefer the provider's count-tokens endpoint where one exists, so you are gating on the same number they will bill.

What people get wrong

"Just raise the context limit / truncation threshold." This treats the symptom. The count is wrong, so no threshold is safe: set it high and you will overflow the real window on genuinely large text; set it low and you keep truncating on phantom tokens. Fix the measurement, not the limit.

Trusting a generic framework helper for multimodal. Fast char-based approximators were built for text, and counting the repr of a content list, base64 and all, has been reported as an open framework bug rather than intended behaviour. Even after a framework special-cases image blocks with a fixed penalty, it may still mishandle audio, video and arbitrary file blocks. Treat any generic approximator as text-only until you have personally verified its multimodal path against a real usage number.

Stripping attachments to "save tokens" based on the bad count. I have watched a team drop the one image that held the answer because their estimator said it cost 40,000 tokens. It cost about 800. They optimised away the payload to fit a budget that was never real.

When it is still broken

1. Log the per-block breakdown. Print each block's estimated tokens next to its type. The offending block will be obvious: it will be the one reporting six figures. This is the same "read the actual number, not the summary" discipline I use for evaluating an LLM feature before it ships.
2. Confirm your dimensions and durations are real. A heuristic keyed on image size is only as good as the width and height you feed it. If you default everything to 1024x1024, resize-heavy inputs will drift. Read the real metadata from the file header.
3. Pin to the provider endpoint for anything user-facing. If a wrong estimate can drop context in production, do not estimate at all for that path. Call the count-tokens API and cache the result per attachment.
4. Check whether the framework version changed the multimodal path. This area is actively being fixed upstream. A version bump can move you from "counts base64" to "fixed penalty per image", which changes your numbers overnight. Re-run the verification after any upgrade of the framework that owns your counter.

The theme here is the same one that runs through most LLM cost surprises: the number on your dashboard is only as trustworthy as the thing that produced it. If you are also sizing infrastructure around token throughput, the real cost math for self-hosting open-weight models depends on getting these counts right too. Measure what the model bills, never what your transport encoding weighs.