Programming model
Puffgres chooses what WAL events to route and what to send to turbopuffer based on user-defined"configs" and corresponding transformations. These are based on two principles:
- Be simple. Transformations should simply take in a set of Postgres rows, and output a set of turbopuffer rows. Batching, backfills, streaming, retries, routing, interrupts, etc are handled by the core puffgres process.
- Be maximally expressive. We found that we needed lots of flexibility in what we would transform to turbopuffer; for example, we'd sometimes need to tokenize before embedding, combine fields into the text we entered, filter, etc. Transforms are defined as Typescript files that can access any Node package and make any arbitrary set of transformations. We also autogenerate a TS schema based on the underlying Postgres schema that lets us have some typing on our input rows.
Each config is three files that fit together like this:
╔═ Postgres ═══╗ ┌─ config.toml ─────┐ ┌─ transform.ts ────┐ ╔═ turbopuffer ═══╗
║ ║░ │ │ │ │ ║ ║░
║ page ║░────▶│ which table │─────▶│ rows ──▶ docs │─────▶║ page_chunks ║░
║ ║░ │ ──▶ namespace │ │ any TypeScript │ ║ ║░
╚══════════════╝░ │ │ │ │ ╚═════════════════╝░
░░░░░░░░░░░░░░░░ └───────────────────┘ └─────────┬─────────┘ ░░░░░░░░░░░░░░░░░░░
│
input rows typed by
schema.ts (autogenerated)
Config
A config is a one-to-one mapping from a Postgres table to a turbopuffer namespace:
┌─ config.toml ──────────┐
╔═ Postgres ══╗ │ │ ╔═ turbopuffer ══╗
║ ║░ │ source.table │ ║ ║░
║ page ║░───────▶│ = "page" │───────▶║ page_chunks ║░
║ ║░ │ namespace │ ║ ║░
╚═════════════╝░ │ = "page_chunks" │ ╚════════════════╝░
░░░░░░░░░░░░░░░ │ │ ░░░░░░░░░░░░░░░░░░
└────────────────────────┘
Schema
A schema is an autogenerated TypeScript representation of the Postgres table, so transforms get typed input rows:
┌─ schema.ts ──────────────────────┐
╔═ Postgres ═══════════╗ │ │
║ ┌─ page ─────────┐ ║░ │ // autogenerated — do not edit │
║ │ id uuid │ ║░ puffgres generate │ columns = [ │
║ │ title text │ ║░─────────────────────▶│ { name: "id", "string" }, │
║ │ body text │ ║░ │ { name: "title", "string" }, │
║ └────────────────┘ ║░ │ { name: "body", "string" }, │
╚══════════════════════╝░ │ ] │
░░░░░░░░░░░░░░░░░░░░░░░░ └──────────────────────────────────┘
Transform
A transform is defined in TypeScript and run by a long-running process; puffgres streams events in and reads actions back over JSONL:
╔═ puffgres core ══╗ ╔═ transform.ts ════════════════╗
║ ║░ Event[] JSONL ║ ║░
║ batching ║░──────────stdin──────────▶║ long-running TS process ║░
║ retries ║░ ║ Event[] ──▶ Action[] ║░
║ routing ║░◀─────────stdout──────────║ any TypeScript, any package ║░
║ ║░ Action[] JSONL ║ ║░
╚══════════════════╝░ ╚═══════════════════════════════╝░
░░░░░░░░░░░░░░░░░░░░ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
We build configs to be immutable, sort of like database migrations. This is because we didn't want a transform to change in the middle of a backfill, leaving a namespace with some rows transformed with old logic and some transforms transformed with new ones.
As such, the correct way to change a transform is to "tombstone" the old one and create another one.
Tombstoning and recreating
┌─ 0701_page ─────────┐ ┌─ 0701_page ─────────┐ ┌─ 0715_page ─────────┐
│ page │ tombstone │▞▞▞▞▞▞▞▞▞▞▞▞▞▞▞▞▞▞▞▞▞│ recreate │ page │
│ ──▶ page_chunks │ ─────────▶│▞▞▞▞ tombstoned ▞▞▞▞▞│ ─────────▶│ ──▶ page_chunks_v2 │
└──────────┬──────────┘ └─────────────────────┘ └──────────┬──────────┘
│ ✗ │ backfill
▼ ▼
╔═ page_chunks ═══════╗ ╔═ page_chunks ═══════╗ ╔═ page_chunks_v2 ════╗
║ ■■■■■■■■■■■■■■■■■■■ ║░ ║ ■■■■■■■■■■■■■■■■■■■ ║░ ║ ■■■■■■■░░░░░░░░░░░░ ║░
╚═════════════════════╝░ ╚═════════════════════╝░ ╚═════════════════════╝░
░░░░░░░░░░░░░░░░░░░░░░░ ░░░░░░░░░░░░░░░░░░░░░░░ ░░░░░░░░░░░░░░░░░░░░░░░
■ synced ░ pending backfill ▞ tombstoned — no more writes
Batching and retries
Changes stream out of Postgres continuously; rather than make a network call per change, puffgres buffers them and writes once per batch. puffgres will also batch really large operations into smaller ones.
changes from Postgres
● ● ● ● ● ● ● ● ● ●
└────────────┬─────────────┘
collect until batch_size (100)
│
▼
┌───────────┐ one write
│ batch │ ──────────────▶ turbopuffer ✓
└───────────┘ │
▲ │ fail
└────────── retry ◀───────────┘
up to max_retries (5),
then ──▶ dead letter queue
Dead letter queue
When a batch fails after max_retries, it goes to the dead letter queue (DLQ) so the stream isn't blocked. The DLQ is replayed on a timer, off to the side of the live stream:
every dlq_replay_interval (10s)
│
▼
pull up to dlq_replay_batch_size (50) entries
│
▼
retry write ──── ok ──▶ turbopuffer ✓ (removed from DLQ)
│
│ fail × dlq_max_retries (5)
▼
marked permanent — kept dlq_permanent_max_age_hours (72h)
for inspection, then pruned
All of these knobs are configurable — see advanced options.