2026-07-21T00:35:03Z by Showboat 0.6.1
How the open mart tier proves it carries no patron PII — by refusing anything a manifest didn’t explicitly declare.
chimpy-lake exposes gold marts to staff through Datasette. Some marts
are safe for a broad audience; some carry patron PII (names, barcodes,
holds). The governing decision (design spec §Topic) was to stop running
one Datasette per mart and instead serve a small number of
sensitivity tiers — open and
restricted — so PII is enforced at one governed
choke point instead of N drifting per-instance auth
configs.
This walkthrough follows a single mart through that choke point. The claim it proves is narrow and load-bearing: on the open tier, “no PII” is not a promise a scanner keeps — it is a structural consequence of default-deny. A mart reaches the open tier only if every one of its columns is named in a manifest; silence is refusal. We then follow the same mart to the machinery that will carry it to a new reporting server — code that is merged and tested but deliberately not yet live — and end on an honest map of what still stands between “merged” and “serving.”
Every command below is real and reproducible
(showboat verify re-runs it) and drives the actual modules
over small in-code fixtures. No patron data appears anywhere.
Two versions. This is the full technical walkthrough. A plain-language retelling in Simple English — shorter sentences, every term explained — sits alongside it: read the Simple English version →.
What. The house pattern was one
build.timer + serve.service per mart, each on
its own port. As marts proliferate that becomes dozens of Datasette
instances — and, more to the point, dozens of independent auth
configurations to keep consistent.
Why it matters. Datasette’s allow
blocks, auth plugins, and column masking are configured per
instance. N instances is N places for one mistyped or drifted rule
to silently expose PII — the hand-maintained-parallel-list anti-pattern.
The threat model of record (spec §1) is a
staff-but-not-restricted tailnet identity, plus the build
mistake that publishes PII to the open surface by accident.
The move. Consolidate serving into two sensitivity tiers. The tier a mart belongs to is declared in its manifest and derived fail-closed — an unrecognised sensitivity never resolves to the open directory. Everything that follows is what “declared, fail-closed” buys you at the open-tier door.
What. A consume tenant’s manifest carries a
[marts] section. Its allowlist names every
column that may appear on the open tier. This list — not a value scan —
is the confidentiality boundary.
Why it matters. The boundary is a thing you write down and review, not a heuristic that has to recognise every shape PII can take. You declare your way in.
How we do it — the real contract validator parses one tenant manifest:
uv run python docs/demos/declare-to-publish/_driver.py declare
The declaration — the whole boundary is this list
=================================================
sensitivity open
declared columns ['work_key', 'title', 'author', 'format', 'call_number']
justified PII cols {'call_number': 'shelf location, not a patron barcode'}
content exceptions {'title': ['phone']}
Parsed by the real contract validator. Every column a mart may expose on
the open tier is named above. A column absent from this list cannot be
published — you declare your way IN, you never scan your way out.What. Point the guard at a mart carrying a column
the manifest never mentioned (patron_email). Nothing else
is wrong with the file.
Why it matters. This is the load-bearing property. The guard doesn’t need to notice that the column holds email addresses — it refuses the column for the sole reason that it was not declared. A new PII column added upstream is refused the day it appears, before anyone writes a rule about it.
How we do it — the same
scan_open_mart() the publisher calls
(publish.py:96):
uv run python docs/demos/declare-to-publish/_driver.py default-deny
Default-deny — an undeclared column is refused before a value is read
=====================================================================
fixture table `works` columns: work_key, title, patron_email
`patron_email` is not in the manifest allowlist.
RESULT: REFUSED — the open tier will not accept this mart:
[works] undeclared column 'patron_email' (default-deny)
The leak dies at silence-in-the-manifest. The guard never had to
recognise an email address — the column was simply never declared.What. Even a declared column is refused if
its name looks like PII
(barcode/patron/email/…) and the
manifest doesn’t carry a written justification for it.
Why it matters. Declaration is a deliberate act, but a reviewer can rubber-stamp a long allowlist. The name-token lint forces an explicit, reviewable reason to sit next to any column whose very name should give a reviewer pause.
How we do it — patron_barcode is
allowlisted, but unjustified:
uv run python docs/demos/declare-to-publish/_driver.py name-lint
Second gate — an allowlisted but PII-named column needs written justification
=============================================================================
`patron_barcode` IS in the allowlist, but has no pii_justifications entry.
RESULT: REFUSED — the open tier will not accept this mart:
[works] allowlisted PII-shaped column 'patron_barcode' lacks justification
The column NAME (barcode/patron/email/...) trips a lint even once declared;
clearing it takes an explicit, reviewable reason in the manifest — not silence.What. Only now does the guard read values: a best-effort regex sweep for email/phone/barcode shapes in declared free-text columns.
Why it matters — and the honest limitation. This is explicitly not the boundary. It catches a shaped value that slipped into an allowed column, but it misses names, DOBs, and unseparated numbers by construction (it says so in its own docstring). If you ever find yourself leaning on this layer to stop a leak, the declaration above already failed. It is a net behind the wall, not the wall.
How we do it — a barcode-shaped value hiding in an
allowlisted author column:
uv run python docs/demos/declare-to-publish/_driver.py content-net
The content scan — a best-effort net, NOT the boundary
======================================================
`author` is allowlisted and benignly named, but one value is barcode-shaped.
RESULT: REFUSED — the open tier will not accept this mart:
[works] barcode-shaped content in 'author'
This secondary regex sweep catches shaped values in free text. It is
explicitly best-effort: it misses names, DOBs, and unseparated numbers by
construction. The wall is the allowlist above; this is a backstop behind it.What. Default-deny is not a blanket “no.” A mart
that declares every column, justifies its PII-named ones, and admits its
irreducible content shapes (a hotline phone number in a title) through a
narrow content_exceptions entry, passes cleanly.
Why it matters. The boundary rewards honest
declaration; it doesn’t punish publishing. This is the same gate that
runs for real on the build server: at go-live, a rogue PII file planted
in the open tier was refused with the reason
scan: OPEN TIER REFUSED … default-deny — the
allowlist caught it, not a content regex, exactly as the beats
above show.
How we do it — every column declared, justified, or excepted:
uv run python docs/demos/declare-to-publish/_driver.py clean
A fully-declared, clean mart — the guard gets out of the way
============================================================
Every column declared; `call_number` justified; a phone-shaped title
admitted only because `title=["phone"]` is declared in content_exceptions.
RESULT: clean — no offenders. This mart may publish to the open tier.
Default-deny is not a blanket 'no' — a mart that declares itself honestly
passes. The boundary rewards declaration, it doesn't punish publishing.The open tier is live on the build server today. The next step — moving the serving role to a new reporting server — is built, reviewed, and merged, but deliberately gated: no push has ever run against that server. The rest of this walkthrough shows that machinery running where it can run safely, and is honest about where it can’t yet.
What. The publisher writes a telemetry run-record around each push so the fleet nurse can see it. But telemetry is a monitor — and a monitor must never be able to take down the thing it monitors.
Why it matters.
TelemetryClient.from_env() is strict (a fleet-wide
misconfig should be loud). But on the serving path that strictness is
inverted danger: a telemetry typo would kill the push. So
push/cli.py catches the raise and fails
open to a no-op buffer. The push serves marts regardless.
How we do it — the same env a misconfigured unit would have:
uv run python docs/demos/declare-to-publish/_driver.py telemetry
Telemetry fails OPEN — a monitor can never kill the thing it monitors
=====================================================================
TelemetryClient.from_env() with no config -> raises RuntimeError
push/cli._telemetry_client() same config -> returns a client, buffer disabled = True
from_env() is strict on purpose (one typo shouldn't run a whole fleet
un-telemetered). But the PUSH must serve marts even when telemetry is
misconfigured, so cli catches the raise and degrades to a no-op buffer.
The house stance is fail-open: never let the monitor take down the mart.What. The cutover ladder’s safe first rung.
push open reporting-server --dry-run compiles the tier
manifest and renders the staff landing page, then returns before
deliver() — no ssh, no deploy key, no remote
write.
Why it matters. It exercises the real compile/render path end-to-end while being provably incapable of shipping anything. Here it runs against a fixture tier so the output is reproducible:
uv run python docs/demos/declare-to-publish/_driver.py dryrun
Gate 1a — the real dry run compiles and stops before delivery
=============================================================
dry-run: would deliver 1 mart(s) to reporting-server
compiled manifest — files: ['holdings.sqlite']
That one line is the whole of what a dry run does to the world: it
compiles the manifest, renders the staff landing page, and returns — the
code path stops at push/cli.py:84, before deliver() ever opens an ssh
connection. It is the safe first rung of the cutover ladder.The same command, run for real against the live 370 MB open tier on
the build host (captured once, 2026-07-20 — not re-run by
verify; the destination name is generalized here), returns
just as safely:
$ python -m chimpy_lake.marts push open <server> --dry-run
dry-run: would deliver 1 mart(s) to <server>and leaves behind exactly the compiled manifest it would have shipped — nothing more:
{
"compiled_at": "2026-07-20T23:32:51+00:00",
"files": [
{ "name": "catalog_holdings.sqlite", "bytes": 370196480,
"sha256": "d8c49ec6…d467bc2f", "built_at": "2026-07-20T10:05:44+00:00" }
],
"instance": "<server>",
"registry_commit": "d07f250",
"tier": "open"
}registry_commit: d07f250 is the very commit this
walkthrough’s code is built from — the manifest carries its own
provenance to the serve host. No bytes left the build host.
Every command above re-runs under showboat verify.
Underneath, the governance and push machinery ship green — the guard,
the push seam tests, and the CLI’s dry-run/fail-open paths:
uv run pytest tests/marts/test_pii_guard.py tests/marts/push/test_push_seam.py tests/marts/push/test_push_cli.py -q 2>&1 | tail -1 | sed -E 's/ in [0-9.]+s//'38 passedThe full branch at merge ran 1243 passed with a mutation gate of 46 killed / 0 survived (run alone, serially). That discipline earned its keep: this branch shipped eight seam defects, every one the same shape — an artifact named by one task and created by another, each half green against a stub. None was found by reading a diff; every one came from gutting the code and watching a test fail to fail.
Everything above is finished and reproducible. Going live on the new reporting server is not done, and this is the map — posed as the gated checklist it is, not a claim:
authorized_keys forced-command line by hand; verify the
rrsync allowlist variant.1a dry-run (shown
above, blast radius none) → 1b telemetry-unset run (first
to touch the live tier) → 2 a manual fire with the timer
disabled → 3 enable the timer.The through line holds all the way down: the open tier is safe to expose because a mart gets in only by declaring itself, column by column — and the machinery that will carry it to the new reporting server is built to the same standard, waiting on gates a human still has to walk.
← all walkthroughs · Rendered from a8057af on 2026-07-21 · showboat verify: reproduces. A living artifact — the version ledger is git.