Your LLM Cost Dashboard Says $0 and It's Lying: get_openai_callback

Your cost dashboard says you spent nothing today. Zero dollars. Zero tokens. That would be lovely if it were true, but your OpenAI billing page tells a different story, and the two numbers are drifting further apart every week. Somewhere between "the LLM call happened" and "the counter went up", the accounting broke, and it broke without raising a single exception.

If you are tracking cost with LangChain's get_openai_callback(), this is almost certainly what happened: you wrapped your chain in an AgentExecutor, or moved to streaming, or upgraded LangChain, and the callback silently detached. It still returns a valid object. That object just reports total_tokens=0 and total_cost=0.0. No error, no warning, no log line. The most dangerous kind of monitoring failure is the one that keeps reporting confidently while measuring nothing.

The symptom: get_openai_callback returns 0 with no exception

The reproduction is depressingly simple. This pattern used to work and now returns zeros once an agent is involved:

from langchain_community.callbacks import get_openai_callback

with get_openai_callback() as cb:
    result = agent_executor.invoke({"input": "..."})

print(cb.total_tokens)   # 0
print(cb.total_cost)     # 0.0

No traceback. cb is a real object with real attributes. They are just all zero. This exact breakage is tracked in multiple LangChain issues (the callback returning zero tokens after wrapping in AgentExecutor or after a version upgrade), and the common thread is that no error is ever raised. Your logs look healthy. Your dashboard looks healthy. Your bill does not.

Why the callback silently detaches

get_openai_callback() works by pushing a callback handler onto LangChain's callback stack. When an LLM call completes through the specific code path the handler expects, the handler reads the token usage out of the response and adds it up. The critical word is specific. The handler is coupled to one internal execution path.

The moment the framework routes the call differently, the handler is never invoked and simply adds nothing. Three things reliably trigger this:

• Agents. AgentExecutor and the newer create_openai_tools_agent / create_openai_functions_agent constructions run the model through a different path, and the usage never reaches the callback.
• Streaming. When you stream tokens (including astream_events), the usage block often is not present on the streamed chunks at all unless you explicitly request it, so there is nothing for the handler to count.
• Version upgrades. Because the coupling is to an internal path, a LangChain release that refactors that path detaches the handler with no change on your side. You upgrade, the tests pass, the tokens quietly go to zero.

This is the structural problem: framework-level token counting is coupled to framework internals, and those internals are not a stable contract. The usage data itself is right there on the provider's HTTP response the whole time. The framework just stopped handing it to the counter.

Step 1: Read usage off the raw response instead

The provider tells you exactly how many tokens it charged, on the response object itself. When you call the OpenAI SDK directly, it is response.usage:

from openai import OpenAI
client = OpenAI()

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "..."}],
)
usage = resp.usage
print(usage.prompt_tokens, usage.completion_tokens, usage.total_tokens)

This number is path-independent. It does not care whether an agent, a chain, or your own loop made the call, because it comes from the provider, not the framework. If you are inside LangChain and want the same data, the model response carries it in response_metadata / usage_metadata on the returned message; read it there rather than relying on the ambient callback.

Step 2: Wrap it in a version-agnostic logger

Centralise the counting so every call goes through one place you control, not a callback you hope is attached:

import logging
logger = logging.getLogger("llm.cost")

# Rough USD per 1K tokens - keep these in config, verify against the pricing page
PRICING = {
    "gpt-4o-mini": {"in": 0.00015, "out": 0.00060},
}

def log_usage(model: str, usage) -> float:
    p = PRICING.get(model, {"in": 0.0, "out": 0.0})
    cost = (usage.prompt_tokens / 1000) * p["in"] + \
           (usage.completion_tokens / 1000) * p["out"]
    logger.info("llm_usage model=%s prompt=%d completion=%d cost_usd=%.6f",
                model, usage.prompt_tokens, usage.completion_tokens, cost)
    return cost

Call log_usage(model, resp.usage) right after every completion. Because it reads the provider's own counts, it keeps working across LangChain upgrades, agent changes and refactors. Verify the per-token prices against the current pricing page rather than trusting numbers baked into a library; providers change them and a stale table silently under-reports cost.

Step 3: Force usage onto streamed responses

If you stream, the usage block is not on the chunks by default. Request it explicitly so you still get a count at the end of the stream:

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[...],
    stream=True,
    stream_options={"include_usage": True},   # adds a final usage-only chunk
)
for chunk in stream:
    if chunk.usage:            # only the final chunk carries usage
        log_usage("gpt-4o-mini", chunk.usage)

Without stream_options={"include_usage": True}, streamed responses give you no usage at all, which is a second, independent reason a token counter reads zero. This is worth checking whenever multimodal or streaming is involved; token accounting has other sharp edges too, like over-counting base64 attachments.

Step 4: Treat the provider's usage API as the source of truth

Anything counted in your own process is a secondary check, not the ledger. The authoritative number is the provider's billing and usage export. Reconcile your in-app counter against the provider's usage API on a schedule (daily is plenty). When they disagree by more than a small margin, trust the provider and go find what your counter missed.

Verification

Prove the counter is alive, not just non-erroring. Run one known call and check the number is non-zero and roughly right:

# tail your logs while sending one request through the real path (agent, streaming, whatever prod uses)
grep llm_usage app.log | tail -1

llm_usage model=gpt-4o-mini prompt=812 completion=143 cost_usd=0.000208

A non-zero prompt and completion through the exact code path production uses is the only proof that matters. Then reconcile a full day against the provider's usage dashboard: the two should be within a few percent.

What people get wrong

"The callback returns an object, so it is working." Returning an object proves nothing. The failure mode is precisely a valid object full of zeros. "No exception" is not "correct".

"We'll trust the framework counter and skip the reconciliation." The framework counter is the thing that broke. Reconciling against the provider's billing is the only way you would ever have noticed. Skip it and you find out at the end of the month.

"Just downgrade LangChain to the version where it worked." That freezes you on old code and the same breakage returns the next time you must upgrade. Decouple the counting from the framework instead, then upgrades stop touching your cost visibility. For deciding whether the feature is even worth its token bill, see evaluating LLM features before you ship.

When it is still broken

• Non-streamed calls still show zero. You are probably reading the wrong field. Confirm you are looking at resp.usage on the raw SDK response, not a LangChain wrapper attribute that may be empty.
• Counts are present but cost is wrong. Your pricing table is stale or keyed by the wrong model name. Log the model string the provider echoes back and price against that exact id.
• Agent makes several model calls per invoke and you only see one. Each underlying LLM call has its own usage. Sum them; a single agent invoke can be five model round-trips, which is also why agent cost surprises people.
• You need a hard guardrail, not just visibility. Add an alert that fires when tracked cost is exactly $0 for any hour in which you know requests were served. A confident zero on an active workload is the signal that the counter has detached again.

The principle worth keeping: your cost number should come from the party that charges you, not from the framework that happens to sit in the middle. Framework callbacks are a convenience that breaks quietly across versions. The provider's usage on the response, reconciled against its billing export, is the only figure that will not lie to you at the end of the month.