Garage

Engineering plans / Proposed

Simplification and robustness plan

Findings from reading main at db351e8 on 2026-10-04, with line numbers from that commit. Three themes, each one a chain of PR-sized steps that leaves the app working after every step:

Theme One line
One facade Every reader and every writer of the corpus goes through GarageService; the app stops parsing psql output and the ingest worker stops silently falling back to direct Postgres.
Extraction contract Extractors are registered by media type (image/* → Tesseract), not by a hand-kept list of suffixes, with one result shape, one error hierarchy and one name per extractor.
Schema Drop what nothing writes, remember every ingest outcome in one place, bound the bookkeeping tables, apply migrations once, and tune the local-first Postgres for what it actually does.

Two defects found on the way are being fixed separately on main (see §5): extractor_revision() raises KeyError for .eml/.emlx, and a PlaceholderFile raised inside extract() is counted as an ingest error instead of a placeholder.

0. Where we start

What exists and shapes the design. Each item is one thing the plan changes.

The facade has leaks.

Extraction is keyed on suffix tables that drift.

The schema carries dead weight and forgets things.

1. One facade: GarageService

Goal: the only process that opens a Postgres connection is the one hosting GarageService (plus the CLI and tests, which are the same code in-process). Everything else, the app, the ingest helper, the embed helper, the MCP helper, speaks protobuf to it. Then the server can own the connection pool, the session settings, the migrations and the invariants, and nothing can bypass them.

1.1 The app reads only through the RPCs

1.2 The ingest worker persists only through the facade

1.3 Server internals: handlers translate, modules decide

1.4 Connection lifecycle the app can trust

1.5 Helper configuration is one struct

Order: replay fix (1.2 bullet 3) → explicit gateway (1.2) → app reads over RPC (1.1) → ops extraction and convert layer (1.3) → oneof and enums (1.3) → lifecycle (1.4) → helper config (1.5). Each is one PR; 1.1 and 1.2 are independent of each other.

2. Extraction contract: media types in, one result shape out

Goal: adding a format means registering one extractor under the media types it handles; the walker, the revision table, documents.mime, the chunker’s language choice and the scanner all read the same registry. image/* goes to Tesseract because the registry says so, not because eleven suffixes are listed in the right set.

2.1 Identify the file once

extract/media.py:

@dataclass(frozen=True)
class MediaType:
    type: str           # "image/png", "application/pdf", "text/x-python", "message/rfc822"
    source: str         # "suffix" | "name" | "magic" | "uttype"

def identify(path: Path, *, size: int) -> MediaType | None

2.2 One registry, one protocol

class Extractor(Protocol):
    name: str                       # the one name: "image", "pdf", "docx", "email", "code", ...
    version: str                    # bumps retry remembered outcomes
    media_types: tuple[str, ...]    # "image/*", "application/pdf", "text/markdown", ...
    def extract(self, path: Path, media: MediaType, ctx: ExtractContext) -> ExtractResult: ...

REGISTRY: list[Extractor]          # first match on media type wins; "text/*" is the fallback

def extractor_for(media: MediaType) -> Extractor
def extractor_revision(media: MediaType) -> str   # f"{e.name}:{e.version}", derived, cannot be missing

2.3 Every other suffix table reads the registry

2.4 Sources produce documents through one pipeline

The Messages path is a second pipeline because its input is not a file. Make the unit of work a DocumentCandidate (uri, media type, a way to get bytes or text, stat) produced by a Producer per source kind: the walker yields file candidates; the sqlite producer yields one candidate per thread with media_type = "application/x-garage-thread" and the rendered text already in hand. ingest_one then runs the same steps for both: stat skip, extract (a no-op extractor for pre-rendered text), quality gate, chunk, classify, attribute, persist. The thread-specific parts (one chunk per message, direction/sender) become the CONVERSATION chunker, which is where they belong. Then communications get the chunk cap and attribution evidence like everything else, and _ingest_source’s if kind == "sqlite" branch (pipeline.py:546-566) goes.

Order: identify + documents.mime (2.1) → registry and derived revision (2.2, replaces the hand-kept table and lands the one-name rule; a migration rewrites documents.extractor from the old engine names) → error hierarchy (2.2) → the other tables (2.3) → delegation for scanned PDFs (2.2, with a budget setting and the schema regen) → producers (2.4). The two bug fixes in §5 land first and the registry step deletes the table they patch.

3. Schema: resilient, bounded, tuned

Goal: the schema says what the code does, every ingest outcome has one home, bookkeeping cannot grow without bound, migrations apply once, and the server is configured for a single-user local corpus. Migrations keep the repo’s rules: idempotent, re-applicable, data/sql is truth and db/models.py mirrors it, and docs/schema.md is updated with each.

3.1 Drop what nothing writes (015_drop_unused.sql)

3.2 Migrations apply once (db/migrate.py)

3.3 One home for ingest outcomes (016_ingest_outcomes.sql)

Today a file’s last outcome lives in two places with different rules: documents.state/error for a document that exists, ingest_outcomes for one that does not, and nowhere for rejected.

3.4 Bound the bookkeeping (017_last_seen.sql)

3.5 Search-side columns

3.6 Postgres session and cluster settings

Set by the server (§1.4) per session role, and by the app in postgresql.conf for the bundled cluster:

Setting Where Why
synchronous_commit = off ingest and backfill sessions one commit per document or batch; losing the last one on a crash is a re-ingest the stat skip handles
statement_timeout = 30s read sessions (search, lists) a runaway query cannot wedge the UI; the app’s 30 s client deadline becomes a server-side fact
lock_timeout = 5s migrations, DROP TABLE emb_* fail fast behind a backfill instead of queuing and blocking every reader
hnsw.ef_search search sessions (exists) unchanged
maintenance_work_mem = 512MB backfill session before CREATE INDEX HNSW builds are memory-bound; the index on a new model table is built after the first backfill, not before (create_embedding_table builds it empty today, so every insert is an index insert)
jit = off cluster short OLTP queries lose to JIT warm-up
shared_buffers, effective_cache_size cluster, from physical RAM the bundled cluster ships Postgres defaults sized for a shared host

The index-after-backfill change is the one with a visible payoff: build the per-model table without its HNSW index, backfill, then CREATE INDEX once (emb_tables.create_embedding_table, registry.index_ddl). embedding_models gets index_built boolean so search falls back to exact KNN (SET LOCAL enable_indexscan, or no ORDER BY operator hint needed; pgvector does exact scans without an index) until it is built, and GetStats reports it.

3.7 Constraints that catch bugs

3.8 Messages as rows, keyed to their chunks (018_messages.sql)

The question “what did this person write, and how” needs each message as a row with its author, its time and its chunks. A message is the unit of authorship; a chunk is the unit of embedding; one message has one or many chunks (a text has one, a long mail has several), so the link runs from chunk to message, as chunks.fact_id (007) runs from chunk to fact. Threads are a relation between messages, since mail threads are trees and span files, while a Messages thread is one file.

CREATE TABLE messages (
    id            bigserial   PRIMARY KEY,
    document_id   bigint      NOT NULL REFERENCES documents(id) ON DELETE CASCADE, -- the thread file, or the .eml
    author_id     bigint      REFERENCES authors(id) ON DELETE SET NULL,           -- resolved sender
    direction     text        NOT NULL CHECK (direction IN ('sent', 'received')),
    sender        text,                       -- the raw handle or address
    sent_at       timestamptz,                -- NULL when the source has no usable time (a mail with no Date)
    external_id   text,                       -- chat.db guid, mail Message-ID
    in_reply_to   text,                       -- mail: the parent's Message-ID, as written
    thread_key    text        NOT NULL,       -- chat: the chat guid; mail: the root Message-ID, else 'doc:<document_id>'
    subject       text,
    meta          jsonb       NOT NULL DEFAULT '{}'::jsonb,
    CONSTRAINT messages_external_unique UNIQUE (document_id, external_id)
);
CREATE INDEX messages_author_sent ON messages (author_id, sent_at DESC NULLS LAST);
CREATE INDEX messages_thread      ON messages (thread_key, sent_at);
CREATE INDEX messages_external    ON messages (external_id);

ALTER TABLE chunks ADD COLUMN message_id bigint REFERENCES messages(id) ON DELETE CASCADE;
CREATE INDEX chunks_message ON chunks (message_id) WHERE message_id IS NOT NULL;

3.9 Versions, comments and shares (019_versions.sql, 020_shares.sql)

Three more things a document is besides its current text. Each is a side table keyed to documents, messages or authors, with a chunk link only where its text should be searchable, so they add to §3.3 and §3.8 without reshaping them. Each is built when a source produces it; the tables are designed now so the ones that land first do not have to move.

Versions. Today replace_document overwrites: the previous text is gone, ingested_at is the only history, and chunk reuse by hash (gateway.py:619-633) is the one version-aware thing, since an unchanged chunk keeps its id and vectors.

CREATE TABLE document_versions (
    id                bigserial   PRIMARY KEY,
    document_id       bigint      NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
    version_no        int         NOT NULL,                 -- 1.. per document, head is max
    content_sha256    bytea       NOT NULL,
    source_sha256     bytea,
    byte_size         bigint,
    mtime             timestamptz,
    content           text        NOT NULL,                 -- TOAST-compressed by Postgres
    extractor         text        NOT NULL,
    extractor_version text        NOT NULL,
    author_id         bigint      REFERENCES authors(id) ON DELETE SET NULL,   -- who made this version, when known
    external_id       text,                                 -- provider revision id, git commit, ...
    observed_at       timestamptz NOT NULL DEFAULT now(),
    meta              jsonb       NOT NULL DEFAULT '{}'::jsonb,
    CONSTRAINT document_versions_no_unique UNIQUE (document_id, version_no)
);
CREATE INDEX document_versions_sha ON document_versions (document_id, content_sha256);

Comments. A comment is a message about a document, anchored to part of it: it has an author, a time, a body, a parent and a resolution. Rather than a third table of authored text, it is a messages row with kind = 'comment' and an anchor, so “everything X wrote” stays one query across texts, mail and comments, and the chunk link gives it search and vectors for free.

ALTER TABLE messages ADD COLUMN kind text NOT NULL DEFAULT 'message'
    CHECK (kind IN ('message', 'comment', 'reaction'));
ALTER TABLE messages ADD COLUMN anchor jsonb;          -- {"page": 3}, {"paragraph": 12, "quote": "..."}, {"cell": "B7"}, {"char_start":..,"char_end":..}
ALTER TABLE messages ADD COLUMN resolved boolean;      -- comments only; NULL for messages
-- in_reply_to / thread_key already give replies and the comment thread; document_id is the commented document.

Shares. Who a document went to, or came from, and by what channel. This is the relation that turns a received trust tier from a path rule into evidence, and answers “what have I sent X” and “what did X send me”.

CREATE TABLE document_shares (
    id            bigserial   PRIMARY KEY,
    document_id   bigint      NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
    direction     text        NOT NULL CHECK (direction IN ('sent', 'received', 'shared')),  -- shared: a live shared folder or link
    author_id     bigint      REFERENCES authors(id) ON DELETE SET NULL,     -- the counterparty
    channel       text        NOT NULL CHECK (channel IN ('mail', 'messages', 'airdrop', 'download', 'dropbox', 'icloud', 'drive', 'link')),
    message_id    bigint      REFERENCES messages(id) ON DELETE SET NULL,    -- the mail or text that carried it
    external_id   text,                                                      -- share link id, provider share id
    shared_at     timestamptz,
    meta          jsonb       NOT NULL DEFAULT '{}'::jsonb,
    -- One row per share event. NULLS NOT DISTINCT (Postgres 15+; the bundled server is 18, CI runs
    -- pg18) so a download or AirDrop with no provider id and no resolved person still collapses to
    -- one row on every rescan, while two mails carrying the same file stay two rows (message_id).
    CONSTRAINT document_shares_unique
        UNIQUE NULLS NOT DISTINCT (document_id, channel, message_id, external_id, author_id)
);
CREATE INDEX document_shares_author ON document_shares (author_id, shared_at DESC);
CREATE INDEX document_shares_message ON document_shares (message_id);

Order within §3: versions can land any time after §3.3 (it only touches replace_document); comments need §3.8 and the extractor side-outputs of §2.2; shares need §3.8 for message_id and the attachment matching in the mail and Messages readers.

Order: migrations-apply-once (3.2, no schema change, unblocks safe edits) → drop unused (3.1) → outcomes (3.3, with the gateway oneof from §1.3 so the wire change and the table change are one PR) → last-seen (3.4) → tsv config and lang (3.5) → session roles and index-after-backfill (3.6) → messages (3.8) → versions, comments, shares (3.9) → constraints (3.7, last, once the code guarantees them).

4. Cross-cutting: one source for each truth

Truth Today After
Corpus class, trust tier, source kind, author role SQL enums, Python enums, three Swift string lists proto enums (§1.3); SQL and Python checked by test; Swift uses the generated enums
Progress phases strings compared in six Swift sites and three Python ops proto Phase enum (§1.3)
What is indexable and how eleven suffix sets plus six satellites extract/media.py + the registry (§2)
Extractor name documents.extractor vs _EXTRACTOR_MODULES Extractor.name (§2.2)
Model catalog models.json, GarageConfigLoader with three decode shapes and name heuristics models.json read by Python; Swift gets ListModels plus a ListCatalog RPC that returns the catalog with is_embedding, dims, provider resolved (closes the catalog item of #27)
Config defaults config/__init__.py and GarageConfigLoader.swift:439-487 Swift reads settings through GetSetting; the defaults exist once, in Python
Schema version Python applies everything each time; Swift skips applied Python applies once with checksums (§3.2); Swift calls InitDb
Database URL for helpers three routes to ingest one HelperConfiguration (§1.5), and only two helpers get the URL at all

5. Already underway

Two defects confirmed while reading, fixed on main in their own session and PR, independent of this plan:

6. Sequencing

Each row is one PR, with the tests that guard it. Rows in the same group are independent of each other.

# Change Guard
1 §5 defects new unit tests
2 Replay helper configuration when Python becomes ready (§1.2) GarageXPCServiceBase unit test with a stubbed runtime
3 Migrations apply once, with checksums and per-file transactions (§3.2) test_migrate.py, test_postgres.py
4 Explicit storage choice for the ingest worker, delete the env sniffing (§1.2) test_ingest_gateway.py, test_ingest_xpc.py
5 App reads models, sources, stats and runs InitDb over gRPC; delete the psql reads and the Swift migrator (§1.1) Swift unit tests; test_grpc_operations.py
6 Move inline handler queries into ops/; service/convert.py with round-trip tests (§1.3) test_grpc_documents.py, test_grpc_serialization.py, test_cli_commands.py
7 identify() + documents.mime populated (§2.1) test_fixture_corpus.py asserts a mime per document
8 Extractor registry, derived revision, one name, migration rewriting old names (§2.2) test_image_extract.py, test_mail_extract.py, test_postgres.py for the rename
9 One error hierarchy; one except in the pipeline (§2.2) pipeline tests over a mocked gateway
10 Drop the old messages and conversations, unused enum values and indexes (§3.1) test_postgres.py migration re-apply
11 PersistDocument oneof + ingest_outcomes as the one outcome record, documents.state dropped (§1.3, §3.3) test_ingest_gateway.py, test_postgres.py, test_grpc_serialization.py
12 ingest_seen → run_id on outcomes; prune ingest_runs (§3.4) reconcile tests, test_postgres.py
13 Proto enums and Phase; ingest progress in proto; Swift switches on them (§1.3) enum-parity test; Swift presentation tests
14 Connection watchdog, one call() path, server sized for streams, unreachable vs needs_migration (§1.4) test_grpc_server.py; Swift unit tests for the watchdog state machine
15 Session roles and settings; index after first backfill (§3.6) test_postgres.py builds a model table, backfills, builds the index, searches
16 Remaining suffix tables read the registry; scanner and walker agree (§2.3) test_scanner.py counts == walker counts over the fixture corpus
17 Delegation: scanned PDF pages and embedded images to image/* (§2.2) fixture PDF with one scanned page; budget setting documented and in the schema
18 tsv configuration per chunk kind; drop documents.lang (§3.5) test_postgres.py keyword search over an identifier
19 Producers: Messages through the one pipeline (§2.4) test_messages.py, test_fake_messages.py
20 messages rebuilt with chunks.message_id, author, time and thread key; mail quoted-text split; back-fill; rag_list_authors per-person counts (§3.8) test_postgres.py back-fill over fixture threads and mail; test_messages.py one row per message with the resolved author; test_mail_extract.py a reply’s quoted chunks carry no message_id and a three-mail thread orders by thread_key
21 document_versions written by replace_document, retention setting, rag_get_document(version=) (§3.9) test_ingest_gateway.py two replaces give two versions and head unchanged chunks keep ids; test_postgres.py migration copies the head
22 Comments as messages rows with kind/anchor; .docx/.pdf/.xlsx extractors report them; tapbacks as reactions (§3.9, §2.2) fixture docx with two comments and a tracked change; test_messages.py a tapback is one row and no chunk
23 document_shares from mail and Messages attachments and kMDItemWhereFroms; attribution evidence share:… (§3.9) test_mail_extract.py an attachment matches a fixture document by hash; test_attribution.py a received share sets received with evidence
24 Constraints (§3.7); ListCatalog and Swift config through GetSetting (§4) test_postgres.py; Swift tests
25 HelperConfiguration as one struct; server stops chdir (§1.5) Swift tests; test_grpc_server.py

Groups: {1, 2, 3} → {4, 5, 6, 7} → {8, 9, 10} → {11, 13} → {12, 14, 15, 16} → {17, 18, 19} → {20, 21} → {22, 23} → {24, 25}. Step 12 follows 11 because it updates the outcome row that 11 starts keeping for indexed files; today success deletes that row, so landing 12 first would make record_seen update nothing. Nothing here changes the egress guard; every step that touches embed/, enrich/ or net/ keeps test_egress_block.py and test_embed_egress.py green, and the deletion of the worker embed path (§1.2) removes one caller from CALLERS rather than adding one.