A grounded retrieval agent, a two-layer verification gate, and an evaluation harness that answers the question most applied-LLM projects skip — not “does it work?” but “on exactly which cases can you trust it, and on which can’t you?” Validated across a real, adversarial 320-case ground-truth set.
Grounded RAG · LLM-as-judge · agentic self-correction · a stub → local (8B/14B) → Bedrock provider ladder behind one interface. Python stdlib-only, deterministic-first, 175 tests at $0. The test domain: NIST 800-171, the DoD contractor-security standard — chosen because its ground truth is genuinely hard, not because compliance is the point.
An LLM will fluently assert anything. The hard part of an applied system isn’t generation — it’s making the output trustworthy, and then knowing where to trust it. I built the machinery for both: retrieval that only exposes hand-checked facts, a gate that refuses any claim it can’t trace to a source, and an eval that scores every case against a human answer key across a ladder of models.
The test corpus is NIST 800-171 — a real compliance standard with 320 assessment objectives — and it was chosen precisely because the ground truth is adversarial: “the system provides evidence” and “the objective is satisfied” are frequently different things, a confident plausible answer is often wrong, and roughly half the objectives turn on human judgment no amount of evidence-matching can settle. That’s exactly the setting where a naive accuracy number lies and a careful eval earns its keep.
Same harness, three generators, all 14 families. The stub is the floor a model must beat — “has evidence → met” — and it cannot emit customer or inherited. Sorting families by how far the local qwen2.5:14b beats that stub traces a clean line from +100 to −18.
Read top-to-bottom, this is the deliverable. Where ground truth is mostly human judgment the heuristic can’t express (3.2 is customer 9/9 → stub 0, model 100), the model wins by the maximum margin. Where it aligns with an evidence-presence heuristic (3.4 is met-heavy), the model loses — by refusing to rubber-stamp borderline met calls the stub gets for free.
Why this is the whole point, not a footnote. A single accuracy number, averaged across the corpus, would rank a reasoning model below a one-line heuristic — because half the cases are exactly where the heuristic looks good. Aggregate rank is a property of the data’s shape, not of the models. So the harness scores the per-item diff, not a headline number, and the curve above is the proof it has to. That is the transferable lesson: an eval’s job is to tell you where capability lives, not to emit a leaderboard figure that averages the answer away.
Which model tier can actually do each job — and could a self-hosted model do it at all? I ran the identical harness — same prompts, same strict parser, same gate → judge flow — changing only the transport (a stdlib HTTP call to a local Ollama model vs. the AWS SDK to Bedrock). So a difference in the numbers is a difference in the model, not the scaffolding: a controlled ablation across four capability tiers. (Temperature 0, reproducible.)
| Role · metric | stub $0 | llama 8B | qwen 14B | Haiku (Bedrock) |
|---|---|---|---|---|
| judge · fabrications caught | 40% | 70% | 80% | 90% |
| generator · status-correct | 66% | 38% | 69% | 76% |
The 8B is a usable judge but an unusable generator — below the trivial stub, failing to emit parseable output for a third of objectives (a multi-part structured-output ceiling that recurred, near-identically, on all 14 families — evidence it’s a model property). Constrained decoding fixed the formatting and left correctness pinned; a bigger local model crossed to usable (38%→69%, zero unassessed). Recommendation, with a number behind it: Bedrock/GovCloud is the default, a 14B is a defensible self-hosted alternative, an 8B is a $0 judge pre-filter and never the sole gate.
You can’t catch a fluent fabrication by scoring it — a confident hallucination scores well. You catch it by refusing any sentence that doesn’t resolve to a hand-checked source. The gate is two layers: a deterministic citation gate (no model) rejects uncited or unresolvable sentences outright; an LLM entailment judge then answers the question the gate can’t — does this specific source actually support this sentence? When the agent met a real AWS account, a rule’s citable text said “CloudTrail is enabled” while its recorded verdict was NON_COMPLIANT: a grounded, cited, false finding that a lexical judge waves through. The reasoning judge reads the verdict and catches it — while still supporting the accurate reading.
An eval earns its keep by catching your own errors. Across the build it surfaced:
The eval needed real infrastructure to reconcile claims against, so the agent runs on a production-grade AWS platform I designed and built: a dual-partition Landing Zone Accelerator (commercial + GovCloud), seven compliant-by-default Terraform modules (KMS, VPC, RDS, ECS/Fargate, endpoints, Inspector, patch management), Service Control Policy guardrails, an AWS Config conformance pack for continuous evidence, and a CI gate (fmt / validate / Checkov / policy-lint) on every change. The GenAI system is the differentiated work; this is the substrate that made its evidence real — and the platform / DevOps / IaC engineering it took to ship.
Built a grounded RAG + LLM-as-judge system with a two-layer verification gate (deterministic citation check + LLM entailment judge), agentic self-correction, and an MCP tool server, then designed the evaluation harness that measures where it can be trusted — hand-labelling a 320-case ground-truth set and running a controlled stub → local (8B/14B) → Bedrock ablation across four capability tiers. Produced an eval-validated per-item map of model trustworthiness, established a monotone relationship between a reasoning model’s advantage and how much of a task needs human judgment (+100 to −18), and proved that a naive aggregate accuracy number ranks the models backwards. The harness also caught and corrected a systematic defect in its own upstream ground truth. Runs on a production AWS platform (Terraform / Landing Zone Accelerator / SCPs / CI) I built.
“An eval that catches your own ground-truth errors is doing its job.”