RAG and vector search: building useful, testable knowledge systems

Retrieval-augmented generation (RAG) is often described as “putting documents into an AI.” That description hides the engineering work. A useful system must decide which documents are authoritative, represent them at a retrievable granularity, find evidence for a particular question, preserve access rules, and make the final answer traceable. The language model is only one component, and it cannot repair a corpus that is stale, badly segmented, or retrieved without authorization.

The original RAG paper framed the pattern as a combination of a model’s parametric memory—information stored in its weights—and non-parametric memory, an external index consulted at generation time. Its experimental system used a neural retriever over a dense vector index of Wikipedia, and the paper identified provenance and updating world knowledge as motivations for external memory (Lewis et al., 2020). Modern production systems vary widely, but the central separation remains: search selects evidence; generation turns selected evidence into an answer.

Start with the information need, not the vector database

A RAG project should begin with a defined information need. “Chat with our documents” is not a testable requirement. A better requirement says who asks what kind of question, which sources are in scope, how current the answer must be, what a user is allowed to see, and what the system should do when evidence is missing. A support analyst may need a short answer plus the exact policy paragraph. A researcher may need a broad set of passages and competing interpretations. A workflow agent may need no free-form answer at all, only a structured result that a program can validate.

RAG is a good fit when the answer depends on a changing or private corpus and the user benefits from synthesis across several passages. It is less compelling when a precise keyword query, a database query, a faceted catalogue, or a link to a known document already solves the problem. Adding an LLM does not automatically make retrieval better. It adds latency, cost, a new failure surface, and the possibility that a fluent response will conceal weak evidence.

A simple architecture has five data flows:

1. Ingestion extracts records from source systems and records identity, ownership, timestamps, and provenance.
2. Indexing transforms records into searchable chunks, lexical fields, embeddings, and metadata.
3. Retrieval turns a user request into candidate passages using one or more search methods.
4. Answering supplies selected passages and instructions to a language model, then validates the result.
5. Operations monitors freshness, access decisions, quality, cost, latency, and security incidents.

Keeping these boundaries explicit makes it possible to test retrieval without a model and generation without pretending that generation metrics prove search quality.

Open the full-size diagram

Ingestion is a data-governance problem

Ingestion is the first quality gate. Treat each source as a dataset with an owner, a connector, a collection schedule, a deletion mechanism, and a documented trust level. Capture the canonical source identifier, title, section path, author or owning team where relevant, publication and modification times, source URL or repository path, content type, language, and an immutable revision identifier. Store the raw object or a reproducible pointer to it before applying transformations. Without that record, a citation can point to a page that has changed and an operator may be unable to explain why a passage appeared.

Extraction must preserve structure. A PDF is not necessarily a sequence of paragraphs: headings, footnotes, tables, captions, columns, and page boundaries may carry meaning. A slide deck, spreadsheet, ticket, source file, and web page each need different parsers. If a parser loses a table’s row labels or interleaves two columns, the embedding can faithfully represent corrupted text. Measure extraction errors on representative samples rather than assuming that a successful parse is a correct parse.

Deduplicate at more than one level. Exact hashes catch unchanged files; normalized hashes catch formatting-only copies; near-duplicate detection can catch a policy copied into several repositories. Duplicates can crowd the top results and give a language model false confidence through apparent agreement. When copies have different owners or dates, do not blindly merge them. Represent the relationship and apply an explicit authority or recency policy.

Make ingestion idempotent. Re-running a job for the same source revision should not create a second set of chunks. Use a stable document ID, a revision ID, and a deterministic chunk ID derived from the revision and chunk position. A change should create a new revision and retire the old one according to a documented policy. Deletions need equal priority: a removed document must disappear from every lexical index, vector index, cache, backup used for serving, and citation lookup that can expose it.

Chunking defines what can be found

Start with document structure. Split at headings, paragraphs, list boundaries, table rows, and code blocks where those boundaries represent a coherent claim. Keep a heading path with every child chunk. A passage headed “Refunds > Exceptions > Annual plans” is more useful than the same sentences without that context. For long sections, use a size budget as a ceiling, then split at the nearest semantic boundary. Use overlap selectively when a definition or procedure commonly crosses a boundary; overlap everywhere increases index size and duplicate results without guaranteeing continuity.

Chunking should be evaluated like a retrieval model. Build a small set of real questions and mark the spans that contain sufficient evidence. Compare strategies on whether the correct evidence appears in the top _k_, whether it arrives with enough context to be understood, and how many irrelevant tokens must be sent to the generator. A chunking change can improve recall while making citations ambiguous, or improve answer fluency by adding context while reducing precision. Keep those measures separate.

Parent-child retrieval is a useful compromise. Index smaller child passages for recall, but retain a pointer to a larger parent section that can be fetched after ranking. Another option is sentence or passage retrieval followed by adjacent-context expansion. Both approaches make context assembly an explicit step instead of hoping that fixed overlap will preserve every relationship. When expanding context, deduplicate and cap the final token budget so one long document cannot crowd out independent evidence.

Embeddings and vector indexes: useful geometry, not understanding

An embedding is a vector of numbers produced from text by a model. Similarity between vectors is used as a proxy for relatedness. The model does not place a legal obligation, product code, date, or permission into a guaranteed symbolic field. It maps patterns into a geometry learned from its training and fine-tuning. OpenAI’s current embedding documentation, for example, describes embeddings as floating-point vectors used for search and related tasks, documents model-specific dimensions and input limits, and recommends cosine similarity for its models (OpenAI embeddings guide). Those values are vendor-specific behavior, not properties of embeddings in general.

Choose the embedding model with the query and corpus in mind. Check language coverage, domain vocabulary, maximum input, licensing, data-handling requirements, dimensionality, and whether the model was trained for symmetric similarity or for query-to-document retrieval. Embed documents and queries with the compatible model and preprocessing. Changing the model, tokenizer, normalization, or prompt format generally requires a new index or an explicitly supported migration; vectors from incompatible spaces cannot be meaningfully compared just because they have the same length.

Vector search returns nearest candidates according to a distance or similarity function. FAISS, an open-source library from Meta, documents Euclidean and maximum-inner-product search, batch queries, disk-backed indexes, ID predicates, and approximate methods that trade some recall for speed or memory (FAISS documentation). A vector database may add persistence, replication, metadata filters, updates, and operational controls, but its index algorithm still makes a quality and latency trade-off. Approximate nearest-neighbor search is not “wrong” by definition; it should be measured against an exact or high-recall reference on the workload that matters.

Dense retrieval is good at paraphrase and conceptual similarity, but it can blur exact distinctions. Product IDs, error codes, names, version strings, quoted phrases, negations, and dates are often better served by lexical matching. The answer to “What is error E-417?” should not depend on a semantically similar E-471 being nearby in vector space.

Hybrid retrieval covers complementary failure modes

Hybrid retrieval combines lexical and dense retrieval. Lexical search matches terms using an inverted index and a scoring model such as BM25. In Apache Lucene’s implementation, BM25’s k1 controls term-frequency saturation and b controls document-length normalization, with documented defaults of 1.2 and 0.75 (Lucene BM25Similarity). BM25 is a well-known algorithm, but a particular analyzer, stemming policy, field boost, synonym set, and implementation version are product configuration, not a standard guarantee.

Dense retrieval can find “cancel my subscription” when the document says “terminate a plan.” Lexical retrieval can find an exact identifier that an embedding treats as noise. Run both against the same permission-filtered corpus, collect enough candidates from each, merge duplicates by stable chunk or parent-document ID, and retain each method’s rank and score for observability.

Combining raw scores requires calibration because BM25 and vector similarity have different distributions. One product may implement min-max or another score normalization and weighted combination; OpenSearch documents both score-based normalization and rank-based reciprocal rank fusion (RRF) as options in its hybrid-search tooling (OpenSearch hybrid search). This is an implementation feature, not a universal hybrid-search protocol.

RRF is attractive when labeled data or reliable score calibration is unavailable. For each result list, it adds a contribution that decreases with rank; the original RRF paper gives the commonly used form as the sum of 1 / (k + rank) and reports experiments across TREC and LETOR collections (Cormack, Clarke, and Büttcher, 2009). Do not turn that result into a guarantee that RRF wins on every corpus. Tune candidate depth, the rank constant, duplicate handling, and query-specific weighting on held-out queries.

Reranking spends computation where it matters

First-stage retrieval is usually optimized for recall and throughput. A reranker examines a smaller candidate set and estimates which passages best satisfy the query. A cross-encoder reads the query and candidate together, allowing richer interactions than two independently embedded vectors, but it costs more per candidate. A late-interaction model such as ColBERT independently encodes query and document tokens and performs a cheaper fine-grained interaction later; its original paper describes both reranking and direct retrieval uses (Khattab and Zaharia, 2020).

A practical pipeline might retrieve 50–200 candidates, remove duplicates and unauthorized items, rerank the remainder, then pass 5–12 passages to the generator. The right numbers depend on corpus size, model cost, and question type. Reranking cannot recover a passage that first-stage retrieval never returned, and it can amplify a spurious candidate if its relevance model is biased toward lexical or stylistic cues. Log the pre-rerank and post-rerank positions so a missed answer can be localized.

Citations are evidence links, not decoration

A citation system should answer three questions: which source supports this claim, which revision was retrieved, and can the user open or inspect the cited span? Store source identity, revision, section path, page or line offsets where available, and the exact text used in the prompt. Pass a stable citation token alongside each context block. Ask the model for structured claim-to-citation associations, then validate that every citation token exists and that the cited passage is actually relevant.

Do not accept a source list generated after the answer as proof of grounding. The model can produce plausible URLs that were never retrieved, cite a document that supports a neighboring claim rather than the stated one, or cite a whole page when only a sentence is relevant. A deterministic renderer should map citation tokens to stored source records. If the system cannot support a claim, it should say that evidence was not found or ask a clarifying question.

There is no single citation format mandated by RAG. Google’s Gemini Search documentation illustrates one vendor-specific design: its API returns inline URL-citation annotations with character offsets and source URLs in grounding metadata (Google Gemini grounding documentation). A private-document RAG system needs a different mapping, usually to internal document IDs and permission-checked links. Treat citation rendering as an application contract and test it independently from the model.

Grounding is also a wording discipline. Distinguish “the policy states” from “the system inferred.” Require dates when the corpus contains revisions. If two authoritative sources disagree, show the disagreement and their effective dates rather than blending them into a synthetic sentence. A concise refusal with a traceable reason is more useful than an uncited answer that sounds complete.

Access control must precede exposure

Vector similarity is not authorization. A user’s embedding can be close to a confidential chunk even when the user has no right to read it. Filtering only after retrieval can leak through result counts, snippets, cache keys, reranker inputs, model prompts, logs, or citations. The safe default is to enforce identity and authorization before a passage enters the candidate set, and to repeat the check before prompt construction and citation display.

At ingestion, attach access metadata from the source of truth: tenant, owner, groups, roles, classification, and document or field-level policy identifiers. At query time, derive an authorization filter from the authenticated principal rather than accepting arbitrary group IDs from the user’s text. Use a security-aware index, separate indexes or partitions, or a retrieval service that can evaluate the policy. Metadata filtering in a vector store is useful only if its identity binding, filter semantics, and failure behavior are understood and tested.

The boundary is product-specific. OpenSearch’s document-level security documentation says DLS determines which documents a role can retrieve during read operations such as search and get, but does not restrict writes; a role with write permission can still modify or delete documents hidden by DLS (OpenSearch DLS). That example shows why “the database has security” is not enough: indexing credentials, administrative APIs, replicas, exports, and deletion paths need separate controls.

Use least privilege for every component. The connector should read only its permitted source scope. The embedding worker should not automatically gain serving access to all tenants. The answer service should retrieve through a narrow API, not receive broad database credentials. Keep authorization decisions and denied-access events auditable, but do not log sensitive query text or retrieved passages without a documented retention and redaction policy. Test cross-tenant and revoked-user cases as adversarial cases, not as a final checklist item.

Freshness is a lifecycle, not a timestamp field

A RAG answer can be factually correct for an old revision and still be operationally wrong. Define a freshness service-level objective by source: minutes for incident status, hours for internal procedures, days for a reference manual, or a change-triggered update for controlled policies. Measure time from source change to searchable revision, not merely the time an embedding job last ran.

Use change detection and event-driven updates where source systems provide reliable events, with periodic reconciliation to catch missed events. Separate content freshness from index freshness: a current document may be blocked by a failed parser, a stale embedding queue, a lagging replica, or a cache. Keep old revisions for audit where required, but exclude them from serving unless the query explicitly asks for history.

When a document changes, reprocess the affected parent and any derived chunks. A model change may require re-embedding the full corpus or maintaining two indexes during migration. Version prompts, chunkers, embedding models, rerankers, analyzers, and fusion parameters. A response log should make it possible to reproduce which versions produced an answer.

Freshness does not mean “always use the newest text.” Effective dates, publication status, supersession, and authority matter. A future policy should not be served as current. A new draft may be less authoritative than a signed older release. Encode those rules as metadata and test them with temporal queries.

Evaluation must separate retrieval from generation

Build a test set from real or carefully authored questions. Include answerable questions, unanswerable questions, ambiguous questions, exact-identifier queries, multilingual cases if relevant, long documents, conflicting revisions, permission boundaries, and adversarial text. For each query, record the expected evidence, acceptable alternatives, answer constraints, user identity, and the date at which the judgment is valid. Protect a held-out set from tuning.

Evaluate retrieval first. Recall@k asks whether an acceptable evidence item appears in the first _k_ results. Precision@k asks how much of that result set is relevant. MRR rewards the first relevant result. MAP averages precision at the ranks where relevant results occur; NDCG handles graded relevance and discounts lower ranks. The Stanford Information Retrieval textbook explains these ranked-retrieval measures and their trade-offs (Manning, Raghavan, and Schütze). Also track context sufficiency: a result can be top-ranked yet omit the exception needed to answer safely.

Evaluate multiple retrievers and ablations: lexical only, dense only, hybrid, with and without reranking, and several chunking strategies. The BEIR benchmark found BM25 to be a robust baseline across heterogeneous tasks, while reranking and late-interaction systems performed strongly at higher computational cost (Thakur et al., 2021). That is a useful warning against selecting a method from one internal demo, not a promise about your corpus.

Then evaluate answer quality. Measure citation precision (does the cited passage support the claim?), citation completeness (are material claims supported?), groundedness (does the answer stay within the evidence?), answer relevance, refusal quality, and factual accuracy. Use expert judgments for domain questions, with written rubrics and inter-rater checks. Automated model grading can help triage at scale, but it is not a substitute for a domain expert deciding whether a safety exception was omitted.

Report operational measures beside relevance: p50 and p95 latency for each stage, embedding and reranking cost, index size, ingestion lag, cache hit rate, refusal rate, and access-denied correctness. Watch for metric gaming. A system can raise citation coverage by attaching a citation to every sentence while citing weak evidence, or raise answer relevance by guessing instead of refusing. Release only when quality, security, freshness, and latency gates all pass.

Failure modes and security controls

The most common failure is retrieval miss: the answer exists but the parser, chunker, analyzer, embedding model, filter, or candidate depth prevents it from being found. Diagnose with a retrieval trace, not by asking the model to try again. The next is retrieval noise: the prompt contains plausible but irrelevant passages, so the model blends them. Deduplication, reranking, source authority, and smaller context budgets help, but they cannot substitute for better indexing.

A third failure is context conflict. Two revisions or two owners state different rules. Use effective dates and authority metadata, expose the conflict, and require the application to choose or escalate. A fourth is answer overreach: the model supplies an unstated conclusion or fills a missing number from general knowledge. Use answer schemas, explicit “insufficient evidence” behavior, claim-level citations, and deterministic validation for fields that matter.

Security changes the threat model because retrieved text is also model input. A malicious document can contain instructions aimed at the model, invisible markup, poisoned metadata, or a payload intended to cause data exfiltration. NIST’s Generative AI Profile distinguishes direct and indirect prompt injection and describes how an attacker can place instructions in data likely to be retrieved; it also recommends risk measurement, provenance, red-teaming, and ongoing verification (NIST AI 600-1). OWASP’s LLM01:2025 guidance likewise notes that RAG and fine-tuning do not fully mitigate prompt injection and recommends external-content separation, output validation, least privilege, and human approval for high-risk actions (OWASP LLM01:2025).

Treat retrieved content as untrusted data, not as a higher-priority instruction. Delimit it, label its origin, and tell the model that it may contain instructions that are not commands. Keep tools and credentials outside the model’s control; enforce permissions in application code. Validate tool arguments and output against a schema. Require a human or an independent policy engine before irreversible actions such as sending messages, changing records, or granting access.

Data poisoning can enter through a connector, a compromised source, a careless upload, or an index rebuild. Use source ownership, signed or authenticated change paths where available, anomaly detection for bulk changes, review for high-impact content, and rollbackable index versions. Protect embedding and search infrastructure against unauthorized reads because embeddings may reveal information about the source even when raw text is not directly exposed. Encrypt data in transit and at rest, limit retention, and document whether an external embedding or generation service receives the content.

Prompt-injection defenses are not a proof of safety. Exercise the whole path with hidden instructions in documents, malicious links, cross-tenant requests, stale revocations, oversized chunks, Unicode confusables, and attempts to make the model disclose system prompts or retrieved text. Log enough structured evidence to investigate without retaining unnecessary secrets.

A concrete implementation and decision framework

A small team can make the design reviewable with a sequence of gates:

1. Define the task. Write five to ten representative questions, including at least one unanswerable and one permission-sensitive case. State the freshness target, acceptable latency, and consequence of an incorrect answer.
2. Map sources. Name the authoritative systems, owners, revision identifiers, access policies, deletion paths, and expected change frequency. Reject sources that cannot provide a trustworthy identity or removal mechanism.
3. Build a lexical baseline. Implement exact and BM25 search with fields for title, headings, body, identifiers, dates, and authority. Save the top results and explanations for every test query.
4. Create a chunking benchmark. Compare structure-aware chunks, bounded sections, and parent-child expansion. Measure evidence recall, context sufficiency, duplicate rate, and prompt size.
5. Add dense retrieval. Select and version an embedding model. Measure exact-neighbor recall against the approximate index and test domain terms, languages, identifiers, and adversarial near-matches.
6. Fuse and rerank. Start with a documented hybrid rule such as RRF, then test score normalization or learned weights only when labels justify them. Add a reranker after measuring whether it improves top-k evidence enough to pay for its latency.
7. Enforce authorization. Apply identity-bound filters before candidate exposure and again before prompt assembly. Test denial, revocation, cross-tenant isolation, and connector overreach.
8. Make evidence visible. Store revisioned source records and stable citation tokens. Reject fabricated or unmapped citations and provide a clear no-evidence response.
9. Operate for change. Monitor ingestion lag, parse failures, embedding queues, index versions, deletion completion, cache invalidation, and freshness by source.
10. Release by gates. Require retrieval, answer, citation, security, freshness, and latency thresholds on held-out tests. Re-run them after changing a parser, chunker, model, index, prompt, or policy.

This framework deliberately postpones model selection. A larger generator cannot compensate for an authorization filter that runs too late, and a more expensive embedding model cannot repair a parser that dropped the table headers.

When simpler search is better

Use ordinary search when users want a list of documents, exact phrases, facets, filters, sorting, or a known record. Use SQL or an application query when the answer is a calculation over structured fields. Use a knowledge graph or an entity-relation query when correctness depends on explicit relationships rather than textual similarity. Use a deterministic rules engine when the policy can be expressed as tested conditions. These are not failures to adopt RAG; they are appropriate boundaries.

A useful product can combine modes. Let a search result page show exact matches and facets, then offer a grounded synthesis over the selected results. Route identifier-heavy questions to lexical search, numerical questions to a calculator or database, and broad explanatory questions to hybrid retrieval. Tell the user which mode answered and preserve links to the underlying records.

The decision test is simple: does synthesis create material value after accounting for evidence quality, permissions, freshness, latency, and operational burden? If the answer is no, ship search. If the answer is yes, keep the retrieval and generation contracts separate, test every boundary, and make uncertainty visible.

Conclusion

RAG is best understood as a knowledge-system pattern, not a prompt trick. Ingestion determines whether the corpus is trustworthy. Chunking determines what can be found. Embeddings provide a useful but imperfect geometry. Hybrid retrieval protects against complementary lexical and semantic misses. Reranking spends computation on a smaller set. Citations, access control, freshness, and evaluation turn a demo into an accountable service.

The practical standard is not that an answer sounds intelligent. It is that the system can show what it searched, why those passages were eligible, which revision supported each material claim, what it did when evidence was missing, and how its quality changes when the corpus or model changes. Where a simpler search or database query is more precise, use it. Where synthesis is valuable, build it as a measured, permission-aware retrieval pipeline with generation at the end—not as a substitute for one.

References and further reading

• Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”
• Thakur et al., “BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models”
• Khattab and Zaharia, “ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT”
• FAISS documentation
• Apache Lucene `BM25Similarity` API documentation
• Cormack, Clarke, and Büttcher, “Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods”
• Manning, Raghavan, and Schütze, “Evaluation of ranked retrieval results”
• NIST, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile”
• OWASP GenAI Security Project, “LLM01:2025 Prompt Injection”
• OpenSearch, “Building effective hybrid search in OpenSearch”
• OpenSearch, “Document-level security”
• Google Gemini API, “Grounding with Google Search”