Accuracy has several failure points
Test the path, not only the final sentence.
Build questions from real decisions
Collect questions from support escalations, onboarding, sales handoffs, policy lookups, and repeated manager requests. Include short factual questions, multi-source questions, ambiguous entity names, current-versus-historical questions, and “who owns this?” questions.
For each question, record the expected evidence, acceptable answer, unacceptable shortcut, audience, and owner who can update the test.
Separate retrieval and answer grading
First inspect whether the correct permitted evidence was retrieved. Then grade the answer for factual support, completeness, qualification, citation, and action safety. RAG evaluation tools can provide useful metrics, but they do not replace domain review or permission testing.
Make abstention a passing result
If no approved source answers the question, the correct output may be “I cannot verify this” plus an owner or next step. Rewarding every question with a confident answer trains the system to invent certainty.
A review set should contain deliberate missing and conflicting evidence so the system has to demonstrate restraint.
Turn failures into maintenance work
Keep the failed question, retrieved evidence, expected result, explanation, fix, and regression status. Fix the earliest failure you can prove: source authority, extraction, chunking, retrieval, permissions, model instruction, or tool validation. Rerun passing cases after each change.
Build an acceptance set with expected evidence
For every question, record the role asking it, allowed sources, expected answer or evidence, acceptable uncertainty, and owner. Include current and historical versions, ambiguous names, missing evidence, conflicting policies, and questions where the correct answer is to abstain. This prevents a fluent but unsupported answer from passing because it sounds plausible.
Grade retrieval, permission eligibility, grounding, completeness, and operational conclusion separately. A system can retrieve the right paragraph and still omit the exception that changes the decision.
Run regression tests as the system changes
Rerun the acceptance set after source updates, connector changes, chunking changes, model changes, prompt changes, permission updates, or new tools. Keep the old result, new result, evidence, and reviewer decision.
Track useful-answer rate, unsupported-answer rate, abstention quality, stale-source retrieval, unauthorized retrieval, and time to repair. The test suite is a maintenance queue, not a launch badge.
The LeanOrchestr Business Brain connects approved company sources, preserves access boundaries, and turns useful context into a maintained operating layer. Start with one question, one accountable owner, and an evaluation set you can rerun.
Sources and verification
Primary product, architecture, risk, and security sources checked on 10 September 2026. Product behavior can change, so verify current plan and admin documentation before implementation.