Hosted brain, dumb connector: splitting an AI agent so the data never leaves the client

Every AI product that touches a client's live business database has the same architectural fork in front of it, and most teams pick a branch by accident rather than on purpose.

Either you pull the client's data into your platform and reason over it there, or you ship your reasoning to the client's environment and run it next to their data. The first is convenient and makes you the custodian of somebody else's business records. The second protects the data and hands your entire product to whoever runs the server.

Rai does neither, and the third option is worth writing up because it solves the privacy problem and the IP problem with one seam.

The two deployables

The hosted brain is FastAPI, on our infrastructure. It holds the agent loop and prompts, the metric catalog and the catalog-first hybrid engine, the advice, forecasting and anomaly logic, metering and rate limiting, and the channel orchestration that pushes briefs out over WhatsApp with SMS fallback. Everything of value is here.

We build the agent on the raw Anthropic SDK rather than a framework. That is a deliberate call: this thing has to embed cleanly and stay dependable for years, and every abstraction layer between us and the model API is a layer we would eventually be debugging through. Two-tier model routing - cheap and fast for routine report generation, stronger for advice - is where the token margin lives, and that is much easier to reason about without a framework mediating it.

The connector is a small Frappe custom app installed on the shop's own server. It can do exactly four things:

• authenticate to the brain;
• receive a query plan;
• execute that plan locally against the shop's own MariaDB, through Frappe's permission model;
• return results and aggregates.

It contains none of the reasoning. No prompts, no catalog logic, no advice generation. This is not an oversight to be corrected in v2 - it is the product's most important property.

The direction of travel

Owner question
      |
      v
[Thin connector] --> [Rai brain]  decides WHAT to fetch (a query plan) +
      ^                            phrases the answer / advice
      |                                   |
      +---- executes plan locally <-------+
            against Frappe DB,
            returns aggregates only

Read that loop carefully, because the arrows are the whole design.

Up to the brain: the question, and a computed result such as "revenue by SKU, last month" - a small table of totals.

Never up to the brain: the invoices those totals came from, the customer list, individual transactions, serial numbers, or any raw row of the business.

The brain decides what it wants to know. The shop's own server does the knowing.

Why aggregates are actually sufficient

The objection I expected was that you cannot do good analysis on aggregates. In practice almost every question a shop owner asks is a question about totals, trends and rankings - what sold, what did not, what is tied up, who charges more. Those are aggregates by nature. The individual rows are the mechanism, not the answer.

Where it does bite is serial-level lookup: "where did this unit come from, is it in warranty". That returns a specific record, and the honest answer is that the record travels. The mitigation is that it is one record, requested explicitly by the owner, scoped to their own company, and logged - not a bulk export.

Why this survives commercial pressure

Any vendor can promise not to look at your data. The promise is worth what the company is worth, and companies get acquired, run out of money, and revise their terms.

This is structural instead. The rows do not leave because the code on the client's own machine has no path that sends them, and they can read it.

The part I find more interesting is that the same split protects us. If we shipped the whole system, the metric catalog and agent design would be sitting on every client server we do not control, and the metering counter would be a number on someone else's machine. Because the brain stays hosted, the IP cannot be copied and the metering cannot be tampered with.

That symmetry is what makes the arrangement durable. Privacy guarantees that only cost the vendor get renegotiated the first time revenue is tight. Guarantees the vendor also needs for themselves do not.

The exception, and why it is priced differently

There is a client type that demands a genuinely air-gapped deployment, and pretending otherwise is not useful. For those, the code ships - protected with compiled binaries via Cython or Nuitka, obfuscation, and a machine-bound expiring licence through PyArmor, backed by a licence agreement.

I want to be honest about what that is: it deters, it does not prevent. Anyone determined enough will get through compiled Python eventually. So it is a higher-priced, contract-bound tier rather than the default, and the pricing reflects the risk being taken rather than the work being done.

The seam that protects the future

One more piece, and it is the cheapest discipline in the system.

Frappe is the only data source in v1. But the agent never calls Frappe. There is a single thin data-access module between them, and it is the only code in the entire system that knows what a doctype is or how tabGL Entry is laid out. The agent asks for canonical things - "sales by SKU last month", "dead stock" - and that module translates.

class DataSource(Protocol):
    async def run_metric(
        self, metric: str, params: dict, tenant: Tenant
    ) -> MetricResult: ...

    async def run_guarded_sql(
        self, plan: QueryPlan, tenant: Tenant
    ) -> MetricResult: ...

That is the whole seam. Two methods. The Frappe adapter implements them by posting a plan to the connector; a future spreadsheet adapter implements them by querying a per-tenant canonical store in Postgres; a QuickBooks adapter implements them by calling an API and normalising.

None of the agent code changes. The catalog does not change. The advice logic does not change. Going standalone later becomes new adapters, not a rewrite, which is the difference between a roadmap item and a second company.

It costs almost nothing to put that Protocol in on day one. It costs a year to add it in year two.