Schema reference

DDL lives in data/sql/00*.sql at the repository root, which is the source of truth (it holds the CHECK constraints and the generated tsvector). db/models.py mirrors it for typed reads and writes, not for schema creation.

Migrations are idempotent — IF NOT EXISTS plus duplicate_object guards for enum types — so applying them repeatedly is the migration story. Sufficient for a single-user local corpus, and it avoids a migration framework.

The files run in order, 001_extensions.sql through 014_chunk_direction.sql. 001 creates vector and pg_trgm, and Apache AGE (age, graph queries in openCypher) only where the server has it installed: the app’s bundled Postgres always does, Homebrew’s and the CI image usually do not, and nothing in the schema depends on it yet.

The two axes

The design decision worth understanding: what a thing is and how trusted it is are independent.

corpus_class — the primary partition:

trust_tier — provenance:

The pairing is what makes the corpus queryable:

A single trust_tier with a communication value would have conflated these — “is this a private conversation” is a property of what the content is, not of how much you trust it. The embedding guard that keeps communications off any provider not on this machine is keyed on corpus_class for that reason.

Tables

sources

Registered roots. kind ∈ filesystem | git | sqlite | maildir | feed — feed is the extension point for social media connectors.

expected_elements is the item count of the last scan, which the app’s progress bars measure ingest against. The rest of that scan lives beside it (008_source_scan.sql): scan_item_type (files, messages, …), scan_details (the scanner’s per-kind breakdown) and scanned_at. config holds only the source’s own settings, such as include_code, as written by sync from garage.json.

authors / author_identities

An author owns many identities (git_email, email, phone, imessage_handle, handle). Resolution is lookup-by-identity, create-on-miss, so the same person arriving first via a git email and later via a PDF byline collapses onto one row.

A partial unique index enforces at most one is_self author.

documents

One row per logical document, unique on (source_id, uri).

placeholder is no longer written: a file whose bytes are not on this machine gets no document, and each run counts such files in ingest_runs.placeholder_count instead. 011_drop_placeholder_documents.sql removes the empty rows older builds wrote and returns indexed rows they had marked placeholder to ok.

document_authors

M:N with role (author, committer, sender, recipient, cc), confidence, and evidence — git-log:self-commits:4/4, path-rule:Reference, document-metadata. Evidence makes a misattribution diagnosable instead of mysterious.

chunks

The unit of embedding, and deliberately model-agnostic — every model references these same rows, which is what makes re-indexing a backfill rather than a re-ingest.

tsv tsvector GENERATED ALWAYS AS (to_tsvector('english', text)) STORED

Postgres maintains the keyword index itself; no application bookkeeping.

char_start / char_end are the chunk’s span of documents.content: its exact text, or for markdown (whose header splitter drops blank lines) the span from its first line to its last. They are NULL when the chunk cannot be found in the content, and on chunks built before offsets were recorded; those gain them the next time the document is re-chunked.

Re-chunking a document keeps every row whose ord, chunk_sha256, text and chunker are unchanged, updating only its offsets, heading, direction and sender, and replaces the rest. A kept row keeps its id, so its vectors in the emb_* tables survive and backfill embeds only the chunks that actually changed. Fact chunks are always replaced.

chunks.fact_id (007_chunk_fact_link.sql) is a nullable REFERENCES facts(id) ON DELETE CASCADE column with a partial unique index (WHERE fact_id IS NOT NULL), so a fact has at most one chunk. It marks a chunk as distilled from a fact rather than cut from documents.content; such chunks carry chunker = 'facts:langextract:<model>'. Nothing else about the row is special — the backfill’s anti-join finds it like any other chunk, which is what gets facts embedded under every model without a fact-specific path. Deleting a fact cascades into its chunk and, through chunk_id, into every emb_* table.

chunks.direction and chunks.sender (014_chunk_direction.sql) say which way a message went. A Messages thread is one document holding both sides, and trust is per document, so each message chunk carries direction (sent when the owner wrote it, received otherwise; a CHECK allows only those) and sender (the handle that wrote it, or me). Both are NULL on every chunk that is not a message, so search’s direction filter (rag_search, garage search --direction) keeps to messages. They are plain columns, not jsonb, because both search engines filter on direction in their WHERE clauses.

facts

Atomic, self-contained claims distilled out of a document’s text by enrich/facts.py (006_facts.sql, 013_fact_prompts.sql). Same shape as chunks: ordered rows scoped to a document_id (ON DELETE CASCADE), and within it to the prompt that produced them (unique on (document_id, prompt_name, ord)). Running one prompt over a document again replaces that prompt’s facts and leaves every other prompt’s alone.

These rows are potential facts: what one prompt extracted from one document. The same claim is often extracted many times, so garage cluster-facts links each to one distilled fact (below).

distilled_facts

The distinct claims of the corpus (015_distilled_facts.sql, 016_fact_dedup.sql), each backed by the potential facts that state it (facts.distilled_fact_id). garage cluster-facts builds them in two levels (enrich/clusters.py):

• Within each document, restatements point at the document’s representative of the claim (facts.restates_fact_id) and back whatever it backs.
• Across the corpus, representatives (the nodes) are grouped by growing each group from a tight seed under the default model, stopping at the first neighbour too far from the centroid (facts.cluster_threshold) or pulling the centroid too far from the seed (facts.cluster_max_drift).
• A group whose members differ in numbers, dates or negation is never formed.
• A loose group is shown to the local chat model, which confirms it and states its claim once; a tight one (facts.cluster_tight_similarity) is kept as is.

A node nothing restates gets a distilled fact of its own. Potential facts are never changed. A re-run places only new facts unless it is --full or a grouping setting changed, and a full run keeps each row whose nodes are unchanged, with its id, statement and vectors.

A mail’s sender, recipients and subject and a Messages thread’s participants are written as potential facts with no model (facts.extractor = 'metadata', prompt_name = 'metadata', the key in attributes->>'key'). Each distinct value is one anchored distilled fact: its metadata facts link to it by the key, and its centroid is its representative’s vector, fixed. Inferred facts of the same class join it when close enough without moving it.

Each embedding model has a fact_emb_<slug> table of distilled-fact vectors beside its emb_<slug> table, keyed on distilled_fact_id (ON DELETE CASCADE). It is created and dropped with the model, not by a migration, because its width is the model’s. Under the clustering model, cluster-facts writes each row: the normalized centroid of its nodes, and in seed the seed it grew from (a single fact’s own vector for both), which later facts are attached against. Under every other model the backfill writes the normalized mean of the nodes’ vectors, once each node has one; nothing is embedded for distilled facts.

fact_cluster_state holds the one row saying which model and grouping settings the last corpus pass ran with; a difference makes the next run a full one.

fact_runs

The last extraction of each prompt over each document (013_fact_prompts.sql), keyed on (document_id, prompt_name) with document_id ON DELETE CASCADE: the prompt’s hash (prompt_sha256), the document’s content_sha256 and the extractor_model it ran with, how many facts it found, and extracted_at. A run that found nothing still leaves a row. garage enrich-facts --stale-only compares all three inputs with the current ones and skips a prompt whose run is up to date, so changing a prompt’s text, a document’s content or the model re-extracts exactly what it affects.

conversations / messages

Structured storage for communication sources (sms/imessage chat.db, mail) prior to synthesis into a document.

A conversations row is one thread between the corpus owner and a single other participant (other_author_id), scoped to the source_id it came from and keyed on the source’s native thread identifier (external_id) so re-ingest finds the same conversation rather than duplicating it.

messages rows accumulate under a conversation as they are ingested, each attributed to its sender via author_id (the owner or the conversation’s other_author_id) and deduplicated on (conversation_id, external_id).

A synthesis step concatenates a conversation’s messages in sent_at order into one synthetic documents row — conversations.document_id — which is then chunked and embedded exactly like any other document, using the same comms_window_minutes / comms_window_messages grouping as the chunker. Deleting that document (e.g. to force a rebuild) clears document_id back to null rather than deleting the raw messages, so re-synthesis has the full history to work from.

These tables are not written yet. Messages ingest (ingest/conversations.py) reads chat.db directly on every run and stores each thread as one documents row with one chunk per message, keyed on <chat.db path>#<chat GUID>; who wrote each message is on its chunk (chunks.direction / chunks.sender).

embedding_models and the emb_* tables

One table per model, because vector(1024) and vector(2560) cannot share a column, and per-model nullable columns would make HNSW indexes and backfills painful.

CREATE TABLE emb_<slug> (
  chunk_id  bigint PRIMARY KEY REFERENCES chunks(id) ON DELETE CASCADE,
  embedding <type>(<dims>) NOT NULL,
  embedded_at timestamptz NOT NULL DEFAULT now()
);

chunk_id as both primary key and cascading foreign key is the load-bearing detail: deleting a chunk removes its vectors from every model table at once, so stale vectors cannot outlive the text they came from.

Each model has two more tables of the same shape, each with its own vector index built the same way:

• potential_fact_emb_<slug>, keyed on fact_id (REFERENCES facts ON DELETE CASCADE), holds the potential facts’ vectors. A fact’s text is embedded through its chunk (chunks.fact_id) like any other, but the backfill stores the vector here rather than in emb_<slug>, so fact-to-fact neighbour queries (cluster-facts) walk an index of facts only and passage search an index of passages only. Vector search takes its nearest neighbours from both tables and ranks them together. 017_fact_vectors.sql created these for models registered earlier, copying the chunk table’s index, and moved their fact vectors across.
• fact_emb_<slug>, keyed on distilled_fact_id, holds the distilled facts’ vectors (see distilled_facts).

Names longer than Postgres’s 63-byte identifier limit are cut to it.

Storage selection

pgvector 0.8 HNSW ceilings are hard limits — vector ≤ 2000 dims, halfvec ≤ 4000:

Modality

embedding_models.modality (014_model_modality.sql) is text (the default, and what every row from before the column reads as) or image. A text model embeds every chunk but the image chunks (chunks.chunker = 'image', the one chunk a picture gets: its text is the title or caption full-text search sees). An image model embeds only those, from the image file at documents.uri, in the app’s GarageImageEmbedXPCService (provider image_xpc), and embeds search queries through the model’s text tower, so one query ranks pictures beside text. Both kinds share chunks and the emb_* layout; only the pending query differs.

Distance

embedding_models.distance (009_model_distance.sql) is the similarity the model was trained for: cosine, l2 or inner_product. It is declared per model in data/models/models.json (or register-model --distance for a model the catalog does not list) and fixes two things that must agree: the HNSW operator class (vector_cosine_ops, halfvec_l2_ops, vector_ip_ops, …) and the operator search orders by (<=>, <->, <#>). An index built for one metric is not used by a query on another, so the metric is chosen once, at registration. Models registered before the column existed were indexed for cosine, which is its default.

models.json is the one model catalog: the app reads it for presets and downloads, and garage_rag.db.catalog reads the same file for widths, supports_mrl, distance and per-provider names (provider_refs, e.g. an Ollama tag), found through GARAGE_MODEL_MANIFEST or in the repository (data/models/models.json). The site serves a copy of that file, which its build puts in place (site_files in docs/BUILD.bazel, and the Pages workflow), as https://garagerag.app/.data/models.json; the app fetches it at launch, keeps the copy in its data folder when it decodes as a catalog with presets, and points GARAGE_MODEL_MANIFEST at that copy, else at the one in its bundle. A catalog change therefore reaches installed apps without a release. Its lists are text_embedding, image_embedding (below), and inference_models for the generative models, each tagged inference (chat, rag_ask), distillation (enrich-facts) or both; the app still reads the list’s older name, fact_distil, as distillation models. Every entry names its maker (e.g. IBM), country_of_origin (an ISO 3166-1 alpha-2 code, e.g. DE) and origin_zone (US, EU, CN, UK, CH or OTHER, e.g. EU for DE), which the app shows as a badge and a test holds to the country; tool_calling: true marks a model trained to call tools, which can drive Garage’s MCP tools. Reference embeddings of fixed inputs are kept apart from it, keyed by slug, in docs/.data/model_test_vectors.json, so the catalog the app fetches stays small.

The catalog has three sections. text_embedding and inference_models entries name one GGUF (download_model_id, download_file, sha256). image_embedding entries (modality: image, provider image_xpc) describe a CLIP-style model as Core ML packages: image_model and text_model (the .mlpackage folder names under models/<slug>/), tokenizer (a Hugging Face tokenizer.json), image_size (the square the image tower takes), text_length (tokens the text tower takes), image_mean / image_std (per-channel normalization) and text_lowercase; download_files lists every file to fetch, each with its path under the repository and its sha256 (Manifest.json, a small file whose formatting is not pinned, may carry none).

Truncation is only sound for MRL-trained models, so supports_mrl is declared per model rather than assumed. A CHECK constraint refuses to register an hnsw-indexed model above its type’s ceiling, so the mistake cannot reach a failing CREATE INDEX thousands of documents into an ingest.

Truncated vectors are renormalized: a prefix of a unit vector is not itself unit length, and pgvector’s cosine operator does not normalize for you.

ingest_runs / ingest_seen

Coverage bookkeeping that makes deletion safe. completed is true only for a walk that ran to exhaustion (not one cut short by --limit or cancellation). ingest_seen holds one row per observed URI per run, including files that were stat-skipped without being opened; nothing prunes old runs yet, so it grows with every ingest.

ingest_outcomes

One row per file that made no document because it holds no text (no_text: an empty file, an icon, a photo) or its extraction failed (failed, with error). It keeps the file’s byte_size, mtime and source_sha256, and extractor_revision, the extractor and its VERSION at the time (image:1). While the stat or the raw hash still matches, the next run skips the file instead of reading it again. A row whose revision differs from the current extractor’s is ignored, so bumping an extractor’s VERSION retries every file it gave up on. Indexing the file deletes its row. DDL in 012_ingest_outcomes.sql.