Our Lake, Our Governance

1. The shape of a lakehouse architecture
2. The shape of our lake (and how faithfully we follow it)
3. Examples and what it has answered

Please hold all big questions for the examples / showcase at the end.

The shape of a lakehouse architecture

Six common patterns around data lakes

1. Land data raw / medallion architecture: bronze is append-only. Silver and gold are for transformations and report consumers.
2. Open data formats: the Parquet file format — a non-proprietary storage container that allows fast reads, compact storage, and flexibility in what is stored and how that data is described.
3. Cheap: storage and computing decoupled. Runs on a single machine or many — it scales easily, on few resources.

The shape of a lakehouse architecture (cont.)

The other three are about what it gives back.

4. It remembers: every version of every record, as of any date you like.
5. One copy, many uses: everything reads the same files.
6. New sources land easily: no rebuild to add the next one.

(Typically sold alongside these: live streaming data, and machine-learning features.)

The system forgets. The lake remembers.

It updates records in place, and vendors do not keep the past for us.

• Records change in place: transit legs, item status and location, the card numbers behind e-resource use. The new value overwrites the old, and nothing wrote down that it changed.
• Many small sources, not one firehose — all of it on a single server.
• No subscription, no license — the running cost is close to zero.
• Strong data protection — the data never leaves our control unless we send it.

What we took, what we skipped, what we added

• Took, faithfully: land raw data, remember every version, add new sources cheaply.
• Took by being small: one machine we already own.
• Declined on purpose: warehouse-scale, many engines, streaming, a place to train models later.
• Added: privacy, which is not on their list and is a very big point of ours.

What we kept is what matters here: it remembers, and it protects.

What it costs, and who owns it

• It runs on one machine we already own. There is no cloud bill for the data, and it is easier to secure and monitor.
• The files are a plain, open format. Any tool can read them — there is no vendor to leave, because there is no vendor.
• 215 GB today, across 24 registered sources, each one reporting what it did, every run.
• Encrypted at rest, backed up for disaster recovery, and destroyable on purpose — part of the design, not bolted on.

What is actually in each layer

Three stages, and the same rule at every step: never edit what came before.

Bronze is captured hourly to daily; silver and gold rebuild overnight. Every run reports what it did — and this morning they all ran.

You can always walk backwards.

It remembers, and it protects.

The raw copy is encrypted; the readable copy has no borrower attached.

A transit note is written by a person. The readable copy keeps their login; the published answer keeps a place, not a person.

The reason it does not rot

The standing failure of these projects is a pile nobody trusts.

• Every run reports what it did — what it read, what it wrote, and whether it actually worked.
• Every source passes the same checks before it is allowed to run at all.
• A test that always passes is not a test. Ours can fail — and sometimes do.

The least interesting part of the system, and the reason the rest of it is still true in three years.

What it has answered

All of these are real, all of them are dated, and most of them are from the last few weeks. Pick one and we will open it.

Every one of these is the same machinery, pointed at a different question.