# Streaming redaction: test the split as well as the sentence

> A streaming redactor must recognise supported sensitive values across chunk boundaries. Test the same input under many partitions, including splits inside the value and splits inside the token on restoration.

Published 2026-11-08T09:00:00+01:00, updated 2026-11-08T09:00:00+01:00 by Godfrey Lebo.

## TL;DR

- Test splits inside originals and inside issued tokens.
- Finite streaming buffers require workload-specific length tests.
- Collect JSON-mode content fragments before restoring the inner document.

A complete string can pass a redaction test while its streamed equivalent leaks part of an address. The difference is timing. The detector can only inspect bytes it has received, and a stream may release an early prefix before the rest of a pattern arrives.

Imagine two fictional chunks: `Contact alice@` and `example.com about the delivery`. A detector applied independently to each chunk does not see the complete address in either call. Concatenating the redacted results afterward does not undo what has already left the process.

## Preserve enough unfinished input

Sether's streaming redactor retains a tail instead of immediately emitting every received byte. Its default `safeDistanceBytes` is 256. It also retains an unfinished whitespace-free run, bounded by `max(safeDistanceBytes × 4, 8192)` bytes. The [streaming documentation](https://github.com/raeven-co/sether#streaming-safety) explains both mechanisms and their limits.

That limit matters. It is inaccurate to describe a finite buffer as a guarantee for every possible value length. If your inputs include unusually long secrets or identifiers, measure them, select a suitable configuration and test the actual pattern. Complete-payload redaction can be the clearer choice when streaming is unnecessary.

The documented Node interface is a transform:

```ts
import { Readable } from 'node:stream';
import { Sether } from '@raeven-co/sether';

const sether = new Sether();
const source = Readable.from(['Contact alice@', 'example.com']);
const redacted = source.pipe(sether.redact());
```

The example establishes the input path; the application must consume the resulting stream and handle errors. Node's [stream documentation](https://nodejs.org/api/stream.html) is the primary reference for consumption, errors and backpressure. Redaction does not replace those responsibilities.

## Partition an example systematically

Begin with a fictional sentence whose expected matches you know. Run it as one chunk, then split it at every position around the sensitive value. Include a split before the delimiter, after it and immediately before the end of the value. Repeat with several values in one sentence so overlap decisions are exercised too.

Check the emitted output, not only the final restored sentence. A perfect redact-restore round trip can coexist with an outgoing disclosure if part of the original was allowed through. Assert that the supported original is absent from the complete outgoing stream and that restoration with the same vault returns the expected text.

Next split the tokenised output at different positions before restoration. The response stream can divide a token just as the request stream divided the original. Test cancellation and upstream errors so retained text is not accidentally logged or sent through a fallback path.

## Distinguish transport JSON from content JSON

An SSE envelope may be valid JSON while its `delta.content` fragments collectively describe a second JSON document. Handling the envelope safely does not automatically restore that inner document safely.

For JSON-mode output streamed over SSE, Sether documents that the application should collect tokenised content fragments and call `restoreJson` on the complete inner document. This can trade immediate display for a clear structure boundary. Decide that trade before showing partially restored tool arguments or attempting to execute them.

## Test the workload you actually ship

A short ASCII address does not exercise long credentials, national-format phone configuration or multilingual prose. Add examples drawn from the expected format classes, using invented values rather than production records. Include negative examples to catch damage to ordinary numbers and code.

The result should be a small reproducible corpus and a partition strategy, not a screenshot saying streaming worked once. Keep the [documented limitations](https://github.com/raeven-co/sether#honest-limitations) beside the acceptance criteria. If a new data format changes those criteria, update the tests before making a broader promise about coverage.

## Frequently asked questions

### Does a passing complete-string test prove streaming safety?

No. Test partitions inside supported values and tokens, plus the buffering limits of the actual configuration.

### Can SSE JSON handling restore an inner JSON-mode document automatically?

A transport event and a document assembled from content fragments are separate layers. Collect tokenised inner fragments and restore the complete inner JSON as documented.


Canonical: https://sether.ai/blog/streaming-boundary-tests