Your Multi-Model Gateway Is Not OpenAI-Compatible for Every Model Behind It

You built your app against gpt-4o through OpenRouter. Tool calling worked, structured output worked, streaming worked. Then someone in a standup said a different model was cheaper and "it is the same OpenAI API, just change the model string." You changed one line. Now tool calls come back as a blob of text inside content instead of a parsed tool_calls array, your JSON-mode parser throws, and half your streamed responses arrive as one giant final chunk. Nothing in your code changed except a model name.

Here is the uncomfortable truth about every multi-provider gateway, OpenRouter, LiteLLM, Together, and the rest: they expose an OpenAI-shaped API, but "OpenAI-shaped" is a promise about the request and response envelope, not about behaviour. The envelope is identical. What the model does inside it is not. A gateway can translate fields it knows how to translate. It cannot invent a capability the upstream model does not have, and it cannot perfectly normalise every provider's quirks in streaming, tool-call encoding and stop handling.

So the bug that looks like "the gateway is broken" is almost always a real, untranslatable feature gap between two backends that happen to share a URL shape. The gateway did its job. The two models are just not interchangeable for the feature you are using.

The symptom: same code, same gateway, different model, broken output

There is no single error string here, which is exactly why it is hard. The failure mode depends on which feature the new model does not really support behind the compatible facade. The four you will actually hit:

 class="language-text"

All four pass their type checks. The response object is valid OpenAI shape. The content of that shape is wrong. This is why static typing and "it returns a 200" give you false confidence.

Why a compatible API cannot guarantee compatible behaviour

Think of the gateway as a translator standing between your OpenAI-dialect request and whatever dialect the upstream provider speaks. Translation works when both sides have the same concept. It fails silently when they do not.

The concrete gaps, in order of how often they bite:

Feature	How the gap shows up	Why the gateway cannot fully fix it
Tool / function calling	Some backends have native structured tool calls; others get tools injected into the system prompt and reply with tool JSON as plain content.	The gateway can sometimes re-parse that text back into tool_calls, but the parsing is heuristic and breaks on multi-tool or malformed output.
Structured output / JSON mode	response_format is a hard grammar constraint on some models and a polite suggestion (or ignored) on others.	Constrained decoding happens inside the inference engine. A gateway cannot bolt a grammar onto a backend that does not expose one.
Streaming event shape	Delta chunks vs one final chunk; different finish_reason timing; tool-call deltas split differently across chunks.	If the upstream does not stream token deltas, the gateway can only forward what it gets, which may be a single block.
Stop sequences	stop honoured, ignored, or applied after the stop token leaks into output.	Stop handling is implemented per inference engine; the gateway forwards the parameter and hopes.
Token usage / cost	usage present, zero, or estimated.	Some backends do not return token counts, so the gateway reports zeros or its own estimate.

If your cost dashboard is the thing reading zero, that is its own well-worn trap, covered in your LLM cost dashboard says $0 and it is lying. For the tool-call encoding specifically, the newest open-weight models have their own 400-level surprises, dissected in why tool calls that work on every other model return a 400.

The fix: a compatibility smoke test you run per model+gateway pair

You do not fix this by hoping. You fix it by treating every model+gateway combination as an untrusted integration and probing the three features that break, before it sees production traffic. Point the standard OpenAI client at the gateway's base_url and assert on the content of the response, not just its shape.

Step 1: Probe tool calling for a real structured result

 class="language-python"

The key assertion is msg.tool_calls is populated. A model that emulates tools returns tool_calls = None and stuffs the JSON into content. That is your first and most common failure.

Step 2: Probe structured output for an actual constraint

 class="language-python"

A model that treats response_format as a hint will wrap the JSON in prose ("Sure, here is the object: ...") and json.loads throws on the leading text. If you rely on JSON mode, that is a hard failure, not a warning. The deeper reasons JSON mode is not the same as a schema guarantee are in stop regex-parsing LLM output.

Step 3: Probe streaming for real deltas

 class="language-python"

If a "streaming" response yields one content chunk, your UI's typing effect is a lie and any code that assumes incremental delivery (backpressure, early cancellation) will not behave as designed.

Step 4: Run the matrix and fail the build on a regression

 class="language-python"

 class="language-text"

Wire this into CI as a nightly job against the models you actually route to. It is the same discipline as any pre-ship evaluation, described in how to evaluate an LLM feature before it touches a customer: you assert on behaviour, not on the fact that a call returned.

Verification: what a clean run proves

A model+gateway pair that passes all three probes is safe for the features you tested, and only those. If you later add a fourth feature (parallel tool calls, vision, a specific stop sequence), add a fourth probe. The green checkmark is scoped to what you actually asserted, never to "the model works."

What people get wrong

"It is the same OpenAI API, so any model is a drop-in swap." This is the whole bug in one sentence. The API surface is identical; the capabilities behind it are not. Interchangeability is a property you verify per pair, not a property of the gateway.

Trusting a 200 and a well-typed response. Every failure mode here returns a valid, well-typed OpenAI object. Your type checker is happy. The tool_calls field is simply None, which is legal. You have to assert on content.

Reproducing "the gateway is broken" and filing it upstream. Usually the gateway is faithfully forwarding a model that does not support the feature. Reproduce the same request against the provider's own API before you blame the middle layer. Most of these are feature gaps, not gateway bugs.

Testing against one model and shipping the config for all of them. You validated gpt-4o and left the model configurable in an env var so ops can "tune cost." The moment someone changes it to an untested model, your smoke test is the only thing between you and a silent production regression.

When it is still broken

• Tool calls parse sometimes, not always. The gateway's heuristic re-parsing works for single, well-formed tool JSON and breaks on multi-tool or truncated output. Set the model to one with native tool calling for anything that depends on tools, and keep the emulated ones for plain chat.
• Structured output passes the probe but fails on complex schemas. JSON mode guarantees valid JSON, not adherence to your schema. If you need schema conformance, use a model that supports a real JSON Schema / grammar constraint, and validate the parsed object against your schema anyway.
• Streaming works but tool-call deltas are malformed. Tool calls streamed as deltas are assembled across chunks differently by different backends. Buffer the full tool-call arguments before parsing rather than parsing per delta.
• Usage is zero. The backend does not return token counts through the gateway. Fall back to counting tokens yourself with the model's tokenizer, or price from the gateway's own billing export rather than the response usage field.

The rule that saves you is dull and absolute: pin the specific model+gateway combinations you have tested, treat any change to that pairing as a code change that must pass the smoke test, and never let "it is OpenAI-compatible" stand in for "I checked."