joey.hersh

systems / throughline

Throughline

A self-hosted personal data hub with a plugin contract any language can implement

Make the integration contract the product.

Period
November 2026
My role
Creator. Designed the data model, the HTTP plugin contract, the security boundary, and the contract test kit; built the core, CLI, SDK, and connectors.
State
Core API, CLI, Python SDK, contract kit, seven connectors, and a synthetic public twin. Apache-2.0.
  • 500 msg/sIngest throughput floorasserted on 20,000 sequential and 60,000 random-order messages
  • 9Connectors and templatesBeeper, GBrain, Plex, Spotify, YouTube and more, plus Python and Node templates
  • 3Postgres rolesowner, app, read-only reader
  • Python
  • FastAPI
  • Typer
  • PostgreSQL
  • pgvector
  • Docker Compose
  • Node.js
  • GitHub Actions

Source on GitHub ↗

an isometric database tower fed by three gated cables from a chat bubble, a record, and a map pin, with one shielded port beaming into a read-only window
Three sources, three gates, one guarded way out.
throughline.ingest: architecture flowcertifiesHTTPSource pluginsPython or Node · manifest · cursorPython or Node · manifest…Contract kitcrash points · replays · secret leakscrash points · replays ·…FastAPI /v1hashed tokens · owner · agent · pluginhashed tokens · owner · a…Ingest and versioningimmutable · payload-hashedimmutable · payload-hashedIdentity · time · pla…merge · split · visitsmerge · split · visitsPostgreSQLapp · reader · owner rolesapp · reader · owner rolesRead-only SQLAST-validated · audit-loggedAST-validated · audit-log…People and agentsCLI · SDK · fused searchCLI · SDK · fused search

Diagram nodes, in flow order: Source plugins (Python or Node · manifest · cursor); Contract kit (crash points · replays · secret leaks); FastAPI /v1 (hashed tokens · owner · agent · plugin); Ingest and versioning (immutable · payload-hashed); Identity · time · place (merge · split · visits); PostgreSQL (app · reader · owner roles); Read-only SQL (AST-validated · audit-logged); People and agents (CLI · SDK · fused search).

deterministic code model judgment human durable state tool / API

The problem

Every personal-data project I built kept re-solving the same three things: getting data out of a source reliably, working out that four accounts are the same person, and letting an agent query the result without handing it the keys to the database. Throughline solves them once, as a platform, so each new source is an integration and not a project.

The integration contract

A plugin is any program that answers three commands, manifest, backfill, and sync, and posts records to the HTTP API. Ingest takes up to 5,000 items per call, supports dry runs, and reports inserted, new-version, unchanged, and rejected counts. Records are immutable and versioned by payload hash, so replaying a batch is always safe. Cursors are written only after a batch is acknowledged, and transport errors, 429s, and 5xx responses get bounded exponential backoff.

The playbook for writing one is six steps: write a manifest, subclass the plugin, yield records and checkpoints, test against recorded responses, then run the contract kit. The kit runs the plugin against disposable databases and checks crash points, reversed delivery, duplicate replay, dry-run counts, stale health, and secret leaks. A plugin that passes is ready for real data.

The security boundary

Three database roles carry three levels of trust. Only the one-shot migrate step holds the superuser connection. The API runs as an app role with no delete grants. SQL runs as a reader role on a read-only connection with a 30-second timeout, behind an AST validator that admits a single SELECT over the published api schema and an allowlist of functions, with a row cap and an audit log. Tokens are shown once and stored as SHA-256 hashes.

Deploying it

docker compose up starts pgvector Postgres, runs migrations and role setup in a one-shot service, and brings the API up once that succeeds. Backups are throughline dump and throughline restore. A seeded synthetic twin, twelve invented people across two years, lets anyone run the whole thing without a single real record. Meaning search turns on with three environment variables pointing at any OpenAI-compatible embedding endpoint, and reports its own token spend.