How do we evaluate RAG? A regression suite built from 30 questions and 8 follow-up sets

While comparing Amazon Bedrock Managed Knowledge Base with our pgvector pipeline, we misdiagnosed the same failure twice. First we blamed the parser for missing a table. Then we blamed chunking for splitting a row. Direct inspection showed that the chickenpox (varicella) exclusion row was present in the index. Natural-language queries failed to surface it near the top.

The final answer could not reveal that distinction. A parsing failure, a chunk-boundary failure, and a ranking failure all look like “I could not find the answer” to the user. RAG evaluation begins by locating the broken stage, not by assigning a single score to the final answer.

Do not replace the model because an answer is wrong

Evaluate at least three layers separately.

Changing the model cannot repair a missing source chunk. Tuning retrieval cannot repair a response that distorts evidence already in context. First isolate the layer that failed.

A test set is a decision contract, not a question list

We selected 30 single-turn questions from categories that appeared in the actual workflow: fever-response procedures, disease-specific school exclusion periods, actions by alert level, and reporting chains. We kept 8 follow-up sets separate.

The school-health RAG benchmark exposes the actual responses and evidence from all three configurations, prompt by prompt.

A question alone does not make a reproducible test case. Each case should retain:

• an anonymized case ID and either a prompt or conversation turns
• the evidence document and version used for evaluation
• the source chunk or table row that supports the answer
• the expected behavior: answer, refuse, or escalate to a person
• the consequence if the system gets it wrong

The following public example shows our application-harness case format, not the Bedrock JSONL schema. It uses a question and evidence text already disclosed in the comparison article. It is not a raw user log or an internal source ID; a production case also records the evidence-document version.

{"id":"exclusion-followup-example",
 "turns":["What is the influenza exclusion period?","What about varicella?"],
 "expectedSourceIds":["disease-table:varicella"],
 "referenceAnswer":"Until all blisters have crusted, for at least five days after rash onset",
 "requiredBehaviors":["resolve_previous_intent","cite_source"],
 "risk":"high"}

The Korea Disease Control and Prevention Agency publishes the same chickenpox return-to-school criterion.¹

The suite should grow from failures found in production. A set of common happy-path questions can pass a demo and still fail on unsupported questions, stale documents, or ambiguous phrasing. Separate cases the system must answer from cases it must refuse. That distinction measures correct refusal, not refusal volume.

Evaluate retrieval before generation

When one source chunk is authoritative, the first metric can stay simple. The case passes Hit@k if that chunk appears in the top-k results. Record its rank as well. A chunk slipping from rank 2 to rank 18 is a regression signal even if a large k still lets the case pass. Calculate recall only when you have labeled multiple pieces of required evidence.

Tables need one more check. The title, column header, and target row must survive together. A cell value without its unit or condition can mislead the generator. In our comparison, the varicella rule existed in the index but did not rank reliably for queries such as “How many days should a student with varicella stay home?” Increasing the result count to 20 and enabling the reranker still did not surface that row near the top. A reranker cannot reorder a candidate that retrieval never returned.

Amazon Bedrock RAG Evaluation separates retrieve-only from retrieve-and-generate jobs and offers context relevance and context coverage for retrieve-only runs.²³ Context Coverage, however, checks whether retrieved context contains the information needed for referenceResponses; it does not compare against a required chunk list. In BYOI data, referenceContexts is optional and the built-in metrics do not use it.⁴ If a service must retrieve one authoritative row, keep source-chunk IDs and ranks in your own deterministic regression layer.

Evaluate generation only after retrieval passes

Run generation checks on cases where retrieval returned the expected evidence. We held Claude and the prompt constant when comparing the managed and custom paths. That let us attribute answer differences to retrieval rather than to the generator.

We separated generation quality into four questions:

• Does the response use only information present in the evidence?
• Does it retain every material duration and exception?
• Does each citation support the claim attached to it?
• Does the system refuse instead of inventing an answer when evidence is absent?

Bedrock retrieve-and-generate evaluations include correctness, completeness, helpfulness, logical coherence, faithfulness, citation precision and coverage, and refusal, among other metrics.³ These scores help compare many outputs, but they are not Boolean unit tests. The Refusal metric scores how evasive a response is; it does not determine whether refusing was the correct behavior. The dataset must separate answerable and unanswerable cases first.

An LLM judge is still a model. When using built-in metrics, record the evaluator model ID and the date on which you checked the published judge prompt.⁵ If the prompt itself must be version-controlled, use a custom metric your team owns.⁶ Calibrate Korean high-risk cases against a small human-labeled set. If the average improves while one mandatory case starts answering without evidence, block the release.

Keep follow-up questions in their own partition

Single-turn accuracy does not prove conversational accuracy. After “What is the influenza exclusion period?”, the question “What about varicella?” omits the phrase “exclusion period.” The system must recover that intent from the previous turn before retrieval.

Both paths resolved the context in all 8 follow-up sets. This did not make their overall RAG quality equal. It told us that conversation-history delivery was not the bottleneck in that comparison. Table retrieval and ranking still differed.

Bedrock retrieve-and-generate datasets support up to five turns per conversation, while retrieve-only evaluations support one turn.⁷ Preserve follow-up regressions either in retrieve-and-generate jobs or in the application harness.

Do not collapse every failure into one wrong answer

Assign each failure to a stage.

The last class fooled us once. When the fully managed Knowledge Base path failed all 30 prompts, we initially treated it as product quality. The test code had called RetrieveAndGenerate, which that managed path does not support.⁸ Without a separate harness-failure class, an evaluation can produce a confident but false product conclusion. The full investigation is in our pgvector versus managed Knowledge Base comparison.

Change one variable, then rerun the same suite

A regression run freezes more than the questions:

• corpus and document version
• parser and chunker versions
• embedding model and index version
• search mode, filters, k, and reranker
• generator model and prompt version
• evaluation set and judge version

After we re-ingested the same corpus with Smart Parsing explicitly selected, 6 of 30 answers changed from “not found” to correct. Because Smart Parsing had already been the default, this run alone did not establish the cause. In a later run, increasing the result count and enabling reranking still did not surface the critical varicella row near the top. Reusing the same questions showed which change repaired which failure.

If a run changes several variables, a higher score has no clear cause. Change one setting at a time, run the fixed suite, and retain a case-level diff. That result becomes the baseline for the next change.

Protect critical cases before optimizing the average

One average is easy to report and easy to misuse. An unsupported claim in one high-risk case should not be offset by minor wording improvements in several low-risk answers.

Release conditions can be designed this way:

• block when a mandatory evidence case misses its source chunk
• block when an unsupported case invents an answer
• block when a candidate version breaks a high-risk case that the baseline passed
• review the remaining quality scores, latency, and per-request cost with case-level diffs

Each team must set thresholds for its own domain and risk. The essential decision comes first: which failures stop a release?

Bedrock Evaluation is one layer of the regression pipeline

Bedrock accepts JSONL datasets in S3 and evaluates either a Knowledge Base or results from an external RAG system. BYOI lets a common evaluator configuration inspect outputs from pgvector, another cloud, or an on-premises pipeline.⁹ A job accepts up to 1,000 prompts and writes its results to S3. CreateEvaluationJob runs asynchronously, so CI must poll GetEvaluationJob, read the completed output, and compare it with the baseline.¹⁰¹¹

The managed layer and the application layer own different checks.

RAG Evaluation charges for the judge model's input and output tokens at standard on-demand rates. Evaluating a Bedrock Knowledge Base also incurs the applicable Knowledge Bases usage charges, and retrieve-and-generate jobs add generator-model inference.¹² Track evaluation cost by version so a growing regression suite stays within budget.

The minimum viable CI gate

Run the fixed suite when the parser, chunker, index, retrieval settings, model, or prompt changes. Compare baseline and candidate results case by case. Block a merge or deployment when a critical case regresses. Store retrieved chunks, the response, citations, failure class, and every component version, not just the judge average.

Do not tune against the entire suite. Keep a development partition for frequent runs and a holdout partition for final checks. Otherwise the system becomes good at 30 questions rather than at the job. When production reveals a new failure, anonymize it into a reproducible case and do not delete old cases without a reason.

This process makes the quality and automated-regression gates from our AI PoC production-readiness framework concrete for RAG. The evaluation set is not a benchmark for a report. It is an executable product specification that can stop a change.

What evaluation actually gave us

Without separating retrieval from generation, we would have kept treating parser, chunking, ranking, and API integration failures as the same problem. The greatest value of the 30 single-turn questions and 8 follow-up sets was not one score. They made failures reproducible, classifiable, and comparable under the same conditions.

A useful RAG evaluation does not ask whether the model is smart. It asks: Did the system retrieve the required evidence, answer only from that evidence, and preserve every critical case that passed yesterday?

We design domain-specific evaluation sets and RAG regression pipelines through Data & ML Engineering, and connect those results to release and approval decisions through AX consulting.

References

We checked the Bedrock evaluation behavior and limits below against the official documentation on July 16, 2026. The 30 single-turn questions, 8 follow-up sets, and configuration-change observations came from our own corpus.