Production RAG Infrastructure: Ingestion to Gateway

This lab documents a production baseline for RAG infrastructure. The goal is not a demo notebook that embeds a folder of PDFs once. It is a platform pattern: repeatable ingestion, versioned embeddings, a durable vector store, and a gateway that applications can call safely.

Sailbird stays on the infrastructure side of the line. Prompting, agent graphs, and product UX belong to the application team. We build the layer underneath: pipelines, indexes, auth, rate limits, evaluation hooks, and cost control.

Many RAG projects look healthy in a prototype and fail under operational load. Documents change, embedding models get upgraded, indexes drift, and every team scripts its own way into the vector store. When retrieval quality drops, nobody can tell whether the cause is stale data, a bad chunking change, or gateway overload.

This baseline focuses on the platform questions: how content enters the system, how embeddings are promoted, how indexes are operated, and how retrieval is exposed without becoming an unauthenticated free-for-all.

The design is a pipeline into a store behind a gateway. Source systems feed ingestion workers. Embedding jobs write to collections with explicit metadata. Applications never talk to the vector database with admin credentials; they call a retrieval gateway that enforces identity, quotas, and logging.

GitOps covers deployment of pipeline workers, store configuration, and gateway services. Index contents are data, not Git objects, but the jobs and policies that produce those indexes are versioned and reviewable.

Ingestion is where most RAG systems quietly fail. Sources move, formats vary, and partial failures leave indexes in an unknown state. A production pipeline treats documents as work items: detect change, chunk consistently, attach metadata, and write results idempotently.

Dead-letter queues and replay matter more than clever parsers. Operators need to reprocess a single source, a date range, or a failed batch without rebuilding the entire corpus by hand.

Embedding models are not free configuration. Changing them without a migration plan breaks retrieval quality in ways that look like application bugs. The baseline pins embedding versions in pipeline images and treats upgrades as a promotion event.

New embedding versions get their own collection or alias strategy, evaluation against a held-out query set, and a cutover plan. CI builds the pipeline image; staging proves the re-index; production switches only after quality gates pass.

Qdrant, Weaviate, and pgvector are all viable. The choice depends on filtering needs, operational familiarity, and how tightly you want vectors coupled to an existing Postgres estate. The baseline cares more about backup, capacity, and tenancy than about brand preference.

Collections should map to clear ownership boundaries: tenant, product domain, or security zone. Capacity planning includes disk growth, memory for indexes, and rebuild time after failure.

A gateway is the production boundary. It authenticates callers, applies rate limits, enforces tenant filters, and emits the logs you need for later evaluation. Without it, every application becomes a privileged client of the store.

The gateway should stay thin: retrieval and related controls, not prompt orchestration. Agent frameworks call the gateway. That separation keeps infrastructure stable while product teams iterate on agent behavior.

RAG platforms need more than process uptime. Operators need pipeline lag, embedding job failures, retrieval latency, empty-result rates, and cost per million tokens or vectors written.

Evaluation hooks let product and platform teams share a contract: sample queries, expected document IDs, and quality scores can run against staging indexes before cutover. The platform provides the plumbing; product owns the judgment of answer quality.

Document corpora are often more sensitive than model weights. Access control belongs at the gateway and in collection metadata, not only in application code that might be bypassed.

Secrets for source systems and embedding providers stay in a proper secrets backend. Pipeline workers and gateway services use short-lived identity. Admin paths to the vector store are limited to platform operators.

This reference shows how Sailbird approaches RAG and agent infrastructure: production ingestion, durable indexes, and a gateway that applications can trust while product teams keep ownership of agents and UX.