Your PII Filter Leaks in Streaming Mode: The Guardrail Gap Nobody Tests

You shipped a chatbot with a guardrail. There is a moderation step, or a PII redactor, that inspects the model's answer and blocks anything it should not say before it reaches the user. You tested it, it caught the bad completions, everyone signed off. Then you turned on streaming, because a response that types itself out feels faster, and nobody thought that changed anything.

It changed everything. Your guardrail inspects the final completion. Streaming sends tokens to the user as they are generated. So the tokens leave your server, reach the browser, and render on screen before the completion your filter is waiting for even exists. By the time the guardrail runs, the customer has already read the phone number, the diagnosis, the slur, the other user's email address. The filter runs, sees the violation, and blocks a response that is already on the screen.

This is not a rare edge case. It is the default outcome of bolting a whole-response filter onto a send-as-you-generate transport, and almost nobody tests for it because the non-streaming path, the one the guardrail was written against, still passes every test. This is one of those problems where the usual advice does not exist yet, because most middleware in major frameworks does not handle it correctly. So let me be plain about the mechanism and the shape of a real fix.

The bug, stated as an issue title

It has been filed, more than once, in the exact words teams reach for:

Streaming bypasses guardrails/middleware

The reports all describe the same thing: a moderation or redaction hook that fires on the complete response works perfectly in blocking mode and does nothing useful in streaming mode, because the transport has already emitted the tokens. It is not that the guardrail is misconfigured. It is that it is architecturally in the wrong place for a stream.

Why this happens: two architectures that disagree about time

Streaming and post-hoc filtering make opposite assumptions about when a response is "done".

• Streaming is "emit as you generate". The point of it is that token N reaches the user without waiting for token N+1. There is no moment where the whole response sits on your server waiting to be looked at. Each token is forwarded and forgotten.
• Post-hoc filtering is "inspect the whole thing, then decide". A PII regex, a moderation model, a policy classifier: they all need the complete text, or at least a meaningful span of it, to make a call. A phone number split across two token boundaries is invisible to a regex that only ever sees one token at a time.

Put those together and the filter is structurally too late. Worse, the split-token problem means even a naive "check each token as it streams" does not work: models tokenise mid-word and mid-number, so +2547 and 0812345 can arrive as separate tokens that individually match nothing. You cannot catch what only becomes a violation once assembled unless you hold some assembled context.

The real fix: a buffered, sliding-window moderation wrapper

The principle: never emit a token until enough context around it has been checked. You keep a small buffer of the most recent tokens, run your check against it, and only release tokens off the front of the buffer once they are far enough back that no in-progress violation could still be forming around them.

Step 1: Buffer a sliding window instead of forwarding raw tokens

Wrap the model's token stream. Accumulate into a string. Keep a tail of the last few tokens unflushed, because a violation might be mid-formation at the current boundary. Flush everything before the tail once it has passed the check.

import re

# crude example: a phone / email screen. In production use a proper
# PII model or your moderation endpoint, not just regex.
PII = re.compile(r"(\+?\d[\d\s-]{7,}\d)|([\w.]+@[\w.]+\.\w+)")
TAIL_HOLD = 24          # chars kept unflushed so a boundary match can still form

async def guarded_stream(token_iter):
    buffer = ""
    async for token in token_iter:
        buffer += token
        # only consider flushing the part that is safely behind the tail
        safe, tail = buffer[:-TAIL_HOLD], buffer[-TAIL_HOLD:]
        if safe:
            if PII.search(safe):
                yield "[redacted]"        # block, replace, and stop
                return
            yield safe
            buffer = tail
    # end of stream: check whatever is left in the tail
    if PII.search(buffer):
        yield "[redacted]"
    else:
        yield buffer

The TAIL_HOLD is the load-bearing idea. Without it, a phone number that straddles the current flush point escapes in two clean halves. Holding a tail of characters means any pattern shorter than the tail is fully assembled before it can be released.

Step 2: Use a real check, and decide block-vs-redact behaviour

A regex catches structured PII. For toxicity, jailbreak output, or free-text policy, you need a moderation model or classifier call against the buffered span. That call has latency, so you do not want to make it on every single token; run it on the safe span each time you are about to flush, or on sentence boundaries. When it trips, you have a design choice: replace the offending span with a placeholder and continue, or hard-stop the stream and send a canned refusal. Hard-stop is safer for anything you truly cannot show; span replacement is friendlier for redaction.

Step 3: Fail closed at the transport, not just in the handler

If the moderation call errors or times out, the stream must not fall through to raw tokens. Fail closed: hold the buffer, and if you cannot verify it, do not emit it. The one thing you cannot do is treat a check failure as "probably fine, send it". That is how a filter that works in testing leaks in production the first time the moderation endpoint has a bad minute.

The cost: buffer size versus perceived latency

This is a real trade and you should size it deliberately, not by accident.

Buffer / hold size	Safety	Perceived latency
None (raw streaming)	Leaks; no protection	Lowest, feels instant
Small tail (tens of chars)	Catches structured PII across boundaries	Barely noticeable
Sentence / clause window	Catches free-text policy violations	Streams in visible chunks, not token-by-token
Full response (no streaming)	Strongest	User waits for the whole answer

Most products land on a small tail for PII plus a sentence-level moderation call, which still feels like streaming to the user but no longer emits token-by-token. If you find yourself needing the full response before you can clear anything, accept that this response should not stream at all, and be honest about it rather than faking a stream that is actually buffered end to end.

Verification: prove the leak, then prove the fix

You cannot verify this by eye on a fast connection. Force the model to produce something your filter should block, and inspect what the client actually receives on the wire, not what the final state is.

1. Craft a prompt that reliably makes the model output a fake phone number or a test PII string.
2. Capture the raw streamed chunks the client receives, in order, with timestamps. In a browser, log every ReadableStream chunk; server-side, log every yielded chunk.
3. Before the fix: the PII string appears in the captured chunks, and only later does a block event arrive. That ordering is the proof of the leak.
4. After the fix: the PII string never appears in any chunk; the client sees [redacted] or a refusal, and nothing else.

Automate this as a test that asserts the forbidden string never appears in the concatenation of streamed chunks. That assertion is the one your original test suite was missing.

What people get wrong

"We check each token as it streams, so we are covered." Per-token checks miss anything that only becomes a violation when assembled, which is most structured PII, because models split numbers and emails across token boundaries. Without a buffered window you are checking fragments that individually match nothing.

"The frontend hides it until moderation returns." Hiding rendered text on the client is not redaction. The bytes still reached the browser; anyone watching the network tab, or a logging proxy, or a screen reader, already has them. Redaction has to happen before the token leaves your server.

"Our framework has a guardrails integration, so streaming is handled." Check it, do not assume it. Many framework guardrail hooks run on the aggregated final output and silently do nothing on the streaming path. Treat this as an open problem: no major framework middleware reliably does buffered streaming moderation for you yet, so you own the buffering wrapper.

When it is still broken

• Tool-call and function-call streams. If the model streams a tool call whose arguments contain PII, and you forward that to another service, the same leak exists on a different channel. Buffer and screen tool-call arguments too, not just the visible message.
• Multi-modal or markdown rendering. A violation split across a code fence or a formatting token can slip a naive text check. Screen the decoded text, not the wire format.
• Latency now unacceptable. If sentence-level buffering makes the product feel slow, that is a signal the response should not stream, or that moderation should run on a faster/cheaper model on the hot path with a slower model as an async audit. Decide the trade explicitly.

If you are building serious LLM features, this belongs next to your other production controls: constraining the model's output shape, defending against prompt injection, and evaluating the feature before it touches a customer. A guardrail that only works when streaming is off is not a guardrail; it is a test-environment prop.