systems / throughline
Throughline
A self-hosted personal data hub with a plugin contract any language can implement
Make the integration contract the product.
- Period
- November 2026
- My role
- Creator. Designed the data model, the HTTP plugin contract, the security boundary, and the contract test kit; built the core, CLI, SDK, and connectors.
- State
- Core API, CLI, Python SDK, contract kit, seven connectors, and a synthetic public twin. Apache-2.0.
- 500 msg/sIngest throughput floorasserted on 20,000 sequential and 60,000 random-order messages
- 9Connectors and templatesBeeper, GBrain, Plex, Spotify, YouTube and more, plus Python and Node templates
- 3Postgres rolesowner, app, read-only reader
- Python
- FastAPI
- Typer
- PostgreSQL
- pgvector
- Docker Compose
- Node.js
- GitHub Actions
Diagram nodes, in flow order: Source plugins (Python or Node · manifest · cursor); Contract kit (crash points · replays · secret leaks); FastAPI /v1 (hashed tokens · owner · agent · plugin); Ingest and versioning (immutable · payload-hashed); Identity · time · place (merge · split · visits); PostgreSQL (app · reader · owner roles); Read-only SQL (AST-validated · audit-logged); People and agents (CLI · SDK · fused search).
The problem
Every personal-data project I built kept re-solving the same three things: getting data out of a source reliably, working out that four accounts are the same person, and letting an agent query the result without handing it the keys to the database. Throughline solves them once, as a platform, so each new source is an integration and not a project.
The integration contract
A plugin is any program that answers three commands, manifest, backfill, and sync, and posts records to the HTTP API. Ingest takes up to 5,000 items per call, supports dry runs, and reports inserted, new-version, unchanged, and rejected counts. Records are immutable and versioned by payload hash, so replaying a batch is always safe. Cursors are written only after a batch is acknowledged, and transport errors, 429s, and 5xx responses get bounded exponential backoff.
The playbook for writing one is six steps: write a manifest, subclass the plugin, yield records and checkpoints, test against recorded responses, then run the contract kit. The kit runs the plugin against disposable databases and checks crash points, reversed delivery, duplicate replay, dry-run counts, stale health, and secret leaks. A plugin that passes is ready for real data.
The security boundary
Three database roles carry three levels of trust. Only the one-shot migrate step holds the superuser connection. The API runs as an app role with no delete grants. SQL runs as a reader role on a read-only connection with a 30-second timeout, behind an AST validator that admits a single SELECT over the published api schema and an allowlist of functions, with a row cap and an audit log. Tokens are shown once and stored as SHA-256 hashes.
Deploying it
docker compose up starts pgvector Postgres, runs migrations and role setup in a one-shot service, and brings the API up once that succeeds. Backups are throughline dump and throughline restore. A seeded synthetic twin, twelve invented people across two years, lets anyone run the whole thing without a single real record. Meaning search turns on with three environment variables pointing at any OpenAI-compatible embedding endpoint, and reports its own token spend.