A grounded retrieval agent that drafts citation-checked NIST 800-171 assessment findings for every one of the standard’s 320 objectives — and an evaluation harness that maps exactly where a language model can be trusted on the task, and where it can’t.
Built on my own CMMC L2 / NIST 800-171 AWS landing zone. Python stdlib-only (boto3 the one optional dep). SQLite control graph · two-layer grounding gate · MCP server · a stub → local → Bedrock provider ladder behind one interface.
CMMC Level 2 makes a contractor prove, per assessment objective, that a control is implemented — and keep the SSP honest about what is actually evidenced versus merely claimed. In practice the implementation matrix over-claims: a row says “implemented” while the defining artifact is a TODO, or the mapped check tests something adjacent to what the objective asks. Reconciling claim against evidence, at objective granularity, across a whole standard, is exactly the work a grounded LLM should help with — if it can be trusted not to invent the evidence, and if you can measure where its judgment holds.
Same harness, three generators, all 14 families. The stub is the floor a model must beat — “has evidence → met” — and it cannot emit customer or inherited. Sorting families by how far the local qwen2.5:14b beats that stub traces a clean line from +100 to −18.
Read top-to-bottom, this is the deliverable. Where ground truth is mostly human judgment the heuristic can’t express (3.2 is customer 9/9 → stub 0, model 100), the model wins by the maximum margin. Where it aligns with an evidence-presence heuristic (3.4 is met-heavy), the model loses — by refusing to rubber-stamp borderline met calls the stub gets for free.
Why this is the point, not a footnote. A single accuracy number, averaged across families, would rank a reasoning model below a one-line heuristic — because half the standard is exactly where the heuristic looks good. Aggregate rank is a property of the family’s shape, not of the models. That is why the harness scores the per-objective diff, and the curve above is the proof it has to.
Could a self-hosted model keep CUI on infrastructure you control? I ran the identical harness — same prompts, same parser, same gate → judge flow — differing only in the transport. A difference in the numbers is a difference in the model, not the scaffolding. (Family 3.3, temperature 0.)
| Role · metric | stub $0 | llama 8B | qwen 14B | Haiku (Bedrock) |
|---|---|---|---|---|
| judge · fabrications caught | 40% | 70% | 80% | 90% |
| generator · status-correct | 66% | 38% | 69% | 76% |
The 8B is a usable judge but an unusable generator — below the trivial stub, failing to emit parseable output for a third of objectives (a multi-part structured-output ceiling that recurred, near-identically, on all 14 families — evidence it’s a model property). Constrained decoding fixed the formatting and left correctness pinned; a bigger local model crossed to usable (38%→69%, zero unassessed). Recommendation, with a number behind it: Bedrock/GovCloud is the default, a 14B is a defensible self-hosted alternative, an 8B is a $0 judge pre-filter and never the sole gate.
You can’t catch a fluent fabrication by scoring it — a confident hallucination scores well. You catch it by refusing any sentence that doesn’t resolve to a hand-checked source. The gate is two layers: a deterministic citation gate (no model) rejects uncited or unresolvable sentences outright; an LLM entailment judge then answers the question the gate can’t — does this specific source actually support this sentence? When the agent met a real AWS account, a rule’s citable text said “CloudTrail is enabled” while its recorded verdict was NON_COMPLIANT: a grounded, cited, false finding that a lexical judge waves through. The reasoning judge reads the verdict and catches it — while still supporting the accurate reading.
An eval earns its keep by catching your own errors. Across the build it surfaced:
Built a retrieval-augmented compliance agent that drafts citation-grounded NIST 800-171 findings for all 320 objectives, with a two-layer grounding gate (deterministic citation check + LLM entailment judge), an agentic revise loop, an MCP server, and a hand-labelled eval harness. Ran a stub → local (8B/14B) → Bedrock comparison over the complete standard and produced an eval-validated per-objective map of where a self-hosted model is trustworthy — establishing a monotone relationship between a model’s advantage and how much of a family needs human judgment (+100 to −18), and proving a naive cross-family accuracy number ranks the models backwards. The eval also caught and corrected a systematic defect in its own upstream catalog.
“An eval that catches your own ground-truth errors is doing its job.”