tokenization · architecture · pii redaction

Reversible redaction needs a lifecycle for the token map

Plan vault ownership, expiry, isolation and missing-token behaviour before depending on reversible redaction in production.

Godfrey Lebo Founder, Sether November 6, 2026, 09:00 WAT · 3 min read

Sether™ / Field notes tokenization A token needs its map. Keep originals in the right vault.

Replacing an email with a token is the visible step. The less visible step is keeping a mapping from that token to the original. Lose the mapping and restoration cannot happen. Give the wrong request access to it and a placeholder can become a route to data that request should never receive.

SetherThe idea, illustrated Your map. Your side.

Sether separates replacement from the model. The model receives tokenised text. Your application retains the mapping and decides when to restore. The vault documentation explains this boundary and the synchronous vault interface.

Start with one complete round trip

This example uses fictional data and the documented synchronous TypeScript functions:

import { basicDetectors, MemoryVault, redactSync, restoreSync } from '@raeven-co/sether';

const vault = new MemoryVault();
const outbound = redactSync('Contact alice@example.com', {
  detectors: basicDetectors,
  vault,
});
const restored = restoreSync(outbound, { vault, strict: true });

The important relationship is the shared vault. A fresh vault in the final line does not contain the original mapping. This example restores plain text. For a serialised JSON document, use the JSON-aware restoration functions instead.

For application tests, capture the issued token from actual redaction. Do not invent a token such as <EMAIL_1> and expect the library to know the original. Short placeholders in explanatory diagrams are schematic; real engine tokens carry their own format.

Choose how long restoration must remain possible

The supplied MemoryVault defaults to 10,000 entries with a one-hour time to live. An entry can disappear through expiry or capacity eviction. A process restart also discards in-memory state. These are documented storage behaviours, not rare production surprises.

Ask how long the workflow takes. A single streamed answer and a job resumed tomorrow require different storage decisions. If a delayed job needs restoration, either keep the relevant state in a storage implementation you own or redesign the job to avoid depending on yesterday's in-memory mapping. The default vault is not a persistent encrypted database.

Do not increase retention indefinitely to make failures disappear. The map contains originals, so retaining it creates a data handling obligation of its own. Define when it can be discarded and how a cancelled or abandoned task reaches that point.

Make missing mappings an explicit result

By default, unresolved token-shaped text can remain in output. Strict restoration raises an unresolved-token error instead. Choose deliberately between showing an unresolved draft, requesting a retry and stopping an action. A visible placeholder in a preview and an unresolved recipient in a send-email tool call deserve different treatment.

Strict mode checks resolution. It does not decide whether an otherwise valid action is authorised. Validate the restored action against the application schema and the current user's permissions before executing it. A token that resolves successfully may still belong to an operation the user is not allowed to perform.

Keep maps out of observability shortcuts

Debugging is an easy way to rebuild the disclosure path. Logging an entire vault to diagnose one failed reply defeats the point of keeping the originals out of the model request. Prefer a request identifier, detector type, result status and controlled error category. Inspect originals only through an authorised investigation path when genuinely necessary.

Test a wrong vault, an expired entry and an evicted entry. Confirm that each produces the intended user experience and that none silently substitutes another person's value. These tests are more useful than asserting that a placeholder appeared once.

Reversibility is a product capability supported by state. The implementation becomes dependable when the lifetime and permissions of that state are as deliberate as the replacement itself. The restore implementation is available for teams that need to inspect the exact unresolved-token behaviour.

Written by Godfrey Lebo

Founder of Sether. Ships production software for a living and reads every comment below. More about the team.