Ending, partly over eBook costs. How much eBook use is Kentucky’s?
OverDrive has the barcode. Sierra has the patron and address.
Barcodes change: 4,854 patrons got new ones in eight weeks.
So: whose barcode was it on the day of the checkout? An as-of query.
Joining two vendors’ data as of the day it happened.
Since August 2023, our circulation history covers
53.4Mcirculation transactions2.99Mdistinct items544kof those items are no longer in the catalog: nearly 1 in 5
All of it in about 16 GB, on one machine we already own.
Sierra is built to run the library right now. It keeps the current state and doesn’t track changes.
That is correct behaviour for running a library. Keeping the history is a different job.
Our Lake, Our Governance
The shape of a lakehouse architecture
The shape of our lake (and how faithfully we follow it)
Two questions Sierra cannot answer: how long transit really takes, and where collection pressure is building
Everything else it has answered
A lakekeeps everything as it arrived.A warehousekeeps it tidied up.A lakehousewants both.
So what’s it actually supposed to give you?
The shape of a lakehouse architecture
Six common patterns around data lakes
Land data raw / medallion architecture: bronze is append-only. Silver and gold are for transformations and report consumers.
Open data formats: the Parquet file format — a non-proprietary storage container that allows fast reads, compact storage, and flexibility in what is stored and how that data is described.
Cheap: storage and computing decoupled. Runs on a single machine or many — it scales easily, on few resources.
The shape of a lakehouse architecture (cont.)
The other three are about what it gives back.
It remembers: every version of every record, as of any date you like.
One copy, many uses: everything reads the same files.
New sources land easily: no rebuild to add the next one.
(Typically sold alongside these: live streaming data, and machine-learning features.)
The lake helps in many practical ways
Sierra keeps the current state, and vendors do not keep the past for us.
Records change in place: transit notes, item status and location, the logins behind a transit, the card numbers behind e-resource use. The new value overwrites the old, leaving no record it changed.
Small data, many sources: the hard part is joining them, not storing them.
No subscription, no license — the running cost is close to zero.
Strong data protection — the data never leaves our control unless we send it.
What we took, what we skipped, what we added
Took, faithfully: land raw data, remember every version, add new sources cheaply.
Took by being small: one machine we already own.
Declined on purpose: not warehouse-scale, no streaming, no AI integration (yet).
Added: privacy, which is not on their list and is a very big point of ours.
What we kept is what matters here: it remembers, and it protects.
Everything so far was what we chose.
Now: what it actually is.
Where our data comes from: Sierra
Records saved as snapshots on a regular schedule.
EVERY HOUR
item records: status, location, transit notes
holds, including the ones Sierra will purge
volume records
patron records, encrypted the moment they land
EVERY NIGHT
circulation transactions
bib records and the full MARC
orders: placed, paid, received
record links, and staff logins
EVERY WEEK
the code tables: locations, item types, patron types, statuses
so last year’s report can still use last year’s names
Each one is its own scheduled job: a systemd timer, Linux’s standard way to schedule and run tasks.
…and beyond Sierra
Vendor data only makes sense next to our own. The lake is where they meet.
VENDORS
OverDrive / Libby: e-checkouts, twice a day, in the only windows OverDrive publishes
SenSource: the door counters (visits, occupancy, time spent), nightly
Purchase suggestions from patrons, every five minutes
OPEN DATA
Open Library and Wikidata: which editions are the same work
Reading levels, gathered from the record and from open sources
Award and bestseller lists: Newbery, the New York Times
Started: grouping every edition, eBook and audiobook of a book into one work (FRBR). 2.19 million records, 1.94 million works so far.
A new source is a new job, not a new system.
What is actually in each layer
Three stages, and the same rule at every step: never edit what came before.
BRONZE · raw
what we were handed, encrypted, never edited
every item record, versioned
53.4 million circulation transactions
2.19 million catalog records
staff work locations, daily
SILVER · readable
the borrower removed on the way in
items, with no patron attached
transit notes parsed: from, to, when
branches and item types, current
GOLD · finished
the answers, rebuilt on a schedule
last copies, by branch
transit journeys, door to door
collection pressure by location
Captured hourly to daily, rebuilt overnight. Every run reports what it did, and this morning they all ran.
You can always walk backwards.
It remembers, and it protects.
The raw copy is encrypted; the readable copy has no borrower attached.
Bronze is append-only. Silver is a projection
of it, from any one version or many, as of any date.
The reason it does not rot
The standing failure of these projects is a pile nobody trusts.
Provenance for every number. Which run made it, from what, and when.
The same checks for every source. Freshness, counts, keys. Inherited, not rewritten.
Checks that can fail. A test that always passes is not a test. Ours can, and sometimes do.
Someone gets told. Stale, failed, or an odd count: it sends a message.
The least interesting part of the system, and the reason the rest of it is still true in three years.
What a run writes down
Two nights of the people-counter harvest, exactly as the record has them.
FRI NIGHT · FAILED
run_id 2f661e18…
source sensource
max_age 24 hours
triggered_by timer
started 2026-10-03 00:40:09
finished 2026-10-03 00:41:53
window 09-29 → 10-03
status failed
error ReadTimeout
records —
Each night asks for the last four days, so a missed night is covered by the next. Reported at 3 a.m. Repaired by the next night.
Two questions the live catalog structurally cannot answer.
Not won’t. Can’t.
How long are items really in transit?
Sierra knows an item is in transit. It does not know the journey.
816,162 journeys rebuilt since July 2. Median 40 hours door to door.
Reconstructed, not recorded.
Where is collection pressure building?
Sierra knows where every item is today. Direction needs a past.
Avondale’s collection went to the Distribution Center.
Visible only because last month was kept.
Questions?
Ask anything.
What it has answered sits below the last slide.
Choose Boring Technology
Almost nothing in here is new — and that is the achievement.
“Let’s say every company gets about three innovation tokens. You can
spend these however you want, but the supply is fixed for a long while.”
Dan McKinley, Choose Boring Technology (2015) mcfunley.com/choose-boring-technology
Everything here is ordinary, durable technology, pointed at questions
that are ours.
Oldest first. Every date is a public fact you can check.
What it is
Since
Where it stands
Debian — the operating system
1993 · 33 years
we run 12 “bookworm”; security support to June 2028
Postgres — the database
1996 · 30 years
#1 database; 55.6% of professional developers
systemd — starts things on a schedule
2010 · 16 years
the default on every major Linux
Parquet — the file format
2013 · Apache 2015
the interchange default
Datasette — the data browser
2017
still 1.0-alpha, and we run it in production on purpose
Podman — runs things in containers
2018 · Quadlet 2023
ships natively in Red Hat 9.1 and 10
dbt — the transformation tool
Dec 2021
the de-facto standard
DuckDB — the query engine
June 2024
◀ the one bet
DuckLake — the table format
April 2026
◀ the one bet · v1.1 due Sept 2026
Three of these are younger than the problem they solve — DuckLake, Quadlet
and Datasette. That is the bet, named.
What it costs, and who owns it
One machine we already own. No cloud bill. Easier to secure and watch.
Plain, open files. Any tool can read them. No vendor to leave.
About 50 GB exists nowhere else. Growing about 10 GB a month, before any retention policy.
Encrypted, backed up, destroyable on purpose. By design, not bolted on.
What it has answered
All of these are real and all of them are dated.
A branch closing. Delhi Township, August: 751 titles where it held every active copy, 733 the only one we owned. This week: 224 of its boxed items have patrons waiting.
A branch moving its collection. Avondale, September: 596 boxed items worth retrieving, 37 true last copies.
A branch asking about its shelves. Forest Park: twelve months of world-language and teen graphic-novel circulation, with the system median beside it.
A borrowing agreement ending. Kentucky’s eBook use: about 200,000 OverDrive checkouts a year from roughly 4,500 cardholders, joined by today’s barcode, so a floor. And who to tell: 9,719 by address, sent in August.
Temporary cards. 5,276 on file in August, 2,394 already expired. Whether they convert needs a past. Now there is one.
Loan rules vs. published policy. Where what we tell the public and what Sierra does disagree.
Reading lists on the public web. Live, rebuilt nightly.
Holdings in WorldCat. Refreshed by the lake every day; the catalog-against-WorldCat cleanup written up in August.
Each of these was once a project: exports stitched by hand, one time. Now each is a query against the same machinery.
