Metering, model routing and the regression suite that stops your agent rotting

Two questions decide whether an AI feature is still running a year after launch, and neither of them is about the model.

Does each request make money or lose money? And when somebody edits a prompt on a Thursday afternoon, will you find out if it broke something?

Both have unglamorous answers. Getting them wrong is how AI features die - not dramatically, but by quietly costing more than they earn while slowly becoming less correct than they were at launch.

Why the usual pricing instinct fails

Ordinary SaaS has a marginal cost near zero. One more user is some rows and a little CPU, so per-seat pricing works: revenue scales, cost barely does.

An LLM feature does not behave like that. Every request costs real money, and the distribution of usage is heavily skewed. Charge a flat fee and a handful of enthusiastic customers consume the margin of everyone else. Charge purely per request and, in this market, the customer stops using it - which is the worst outcome available, because usage is what creates the value in the first place.

So: a flat monthly add-on with an included credit bucket, then metered overage. Predictable for the shop, capped for us. The credit sizing is derived from measured cost, which requires actually measuring it.

Meter where the customer cannot reach

Rai ships a connector to the client's own server. The metering does not live there.

This is not about distrust. It is that a counter running on someone else's machine is a counter they can edit, and once one client discovers that, the economics of the product are a rumour. Keeping the ledger server-side means nobody has to think about it again - including us.

The ledger is append-only in Postgres. Every request writes a debit row; balance is a sum, never a mutable column. That makes disputes answerable ("show me the requests") and makes double-charging a bug you can detect rather than a possibility you argue about.

CREATE TABLE credit_ledger (
    id           bigserial PRIMARY KEY,
    tenant_id    uuid        NOT NULL REFERENCES tenant(id),
    request_id   uuid        NOT NULL UNIQUE,
    delta        integer     NOT NULL,        -- negative = debit
    reason       text        NOT NULL,        -- 'report' | 'advice' | 'topup'
    model        text,
    input_tokens integer,
    output_tokens integer,
    created_at   timestamptz NOT NULL DEFAULT now()
);

The UNIQUE on request_id is the whole idempotency story. A retried request debits once.

The three levers on margin

Route by difficulty. Generating a routine report - the metric is defined, the parameters are extracted, the query is known - is a small job. Advice is not: "what should I do about my slowest stock" needs real reasoning over several retrieved figures. A cheap fast model handles the first, a stronger one handles the second. This single decision moves unit economics more than everything else in the system combined.

Cache identical work. Three people in one shop asking for the same report on the same morning is one computation. The cache key is the metric, the resolved parameters, and a data-version token - not the raw question string, which varies pointlessly.

Size credits from measurements, not estimates. Which requires knowing what each request actually cost. Which brings us to the second half.

The failure mode ordinary software does not have

Here is the scenario that should worry anyone shipping an agent.

You change a sentence in a prompt to fix a formatting complaint. Everything still works. Nothing errors. But a class of questions that used to resolve to gross_margin_by_category now resolves to revenue_by_category, and for three weeks a set of customers get answers that are wrong in a way nobody notices.

There is no stack trace for that. No test failed, because in most codebases there was no test that could fail.

So write the test

Rai has a fixed suite of cases in the form when a user asks X, the correct metric is Y. It runs on every deploy.

CASES = [
    ("what did we sell last month",            "revenue_by_period"),
    ("am I making money on power tools",       "gross_margin_by_category"),
    ("ni nini haijauzwa kwa muda mrefu",       "dead_stock"),
    ("which supplier is charging me more",     "supplier_price_variance"),
    ("nani ananidai",                          "receivables_aging"),
    ("stock ya kaa gani inaisha",              "days_of_stock"),
]

@pytest.mark.parametrize("question,expected", CASES)
async def test_metric_resolution(question, expected, agent):
    assert (await agent.resolve_metric(question)).name == expected

Real phrasings, including the awkward ones, including Swahili and Sheng, because those are exactly the cases a prompt tweak quietly breaks. A change that re-routes a question now fails the build instead of reaching a shop.

It is a boring artefact and it is the highest-leverage thing in the repository. It converts the scariest class of agent bug - the silent behavioural regression - into an ordinary red build. Nothing else in the toolchain does that.

It resolves the metric rather than asserting on generated prose, deliberately. Asserting on output text produces a suite that breaks on every wording change and gets deleted within a quarter. Asserting on the routing decision is stable, and the routing decision is what actually determines correctness.

Traces, errors, staging

Every agent run is traced through Langfuse - prompts, tokens, latency, path taken. It does double duty: it is how you debug a strange answer, and it is the source of truth for what a request actually cost, which is what credit sizing depends on. Errors go to Sentry.

Both are cloud free tiers on purpose. There is a version of this where you self-host the observability stack on day one and spend a fortnight on it. That fortnight belongs in the metric catalog. Move to self-hosted if volume ever makes it worth it, and not before.

And prompt and catalog changes go to a staging brain before production, always. Those are precisely the changes that break things without failing anything, and they are the ones people are most tempted to push directly because they "are only text".

What I would tell anyone building one of these

The interesting engineering is not the agent loop. It is:

• knowing what each request costs, from real traces rather than estimates;
• routing cheap work to cheap models, and being disciplined about which is which;
• metering somewhere the customer cannot reach;
• and a regression suite that catches the day a prompt tweak silently changed what a question means.

None of that appears in a demo. All of it decides whether the thing is still running, and still correct, twelve months later.