Evaluation
Keep retrieval reachability, answer support, abstention, and multi-collection behavior as separate scorecards tied to the same configuration.
Three evaluation layers
| Layer | Question | Primary evidence |
|---|---|---|
| Retrieval | Was the reviewed evidence found, and at what rank? | Hit@K curve, MRR/NDCG, negatives, filters, ACLs, latency |
| Generation | Did the answer cover facts, cite support, and abstain correctly? | Fact requirements, citation precision, faithfulness, abstention |
| Corpus | Did routing, fan-out, and optional graph expansion preserve evidence? | Routing accuracy, per-collection K, relationship coverage |
Where retrieval evals are in the OSS checkout
Keep reviewed retrieval and generation judgments beside the immutable engagement snapshot. The public Rust evaluator accepts fragment and judgment JSONL directly; hosted runs supply ranked results and a projection receipt from the selected adapter.
A production corpus-design run retrieves to at least rank 50 and reports Hit@1, Hit@5, Hit@10, Hit@20, and Hit@50 plus expected-empty behavior.
corec eval retrieval \
--fragments engagement/golden/fragments.jsonl \
--judgments engagement/golden/retrieval-reviewed.jsonl \
--output engagement/runs/reference \
--limit 50
set -a; source .env; set +a
corec demo turbo-puffer \
--snapshot golden \
--name reference-corpus \
--fragments-file engagement/golden/fragments.jsonl \
--evaluations-file engagement/golden/retrieval-reviewed.jsonl \
--descriptor-file engagement/descriptor.json \
--namespace-prefix reference-evalRetrieval judgment JSONL
Retrieval golden data covers single-hop factual and structural queries, hard negatives, expected-empty behavior, calibration, and authorization. The BM25 reference applies each row's principal and filters. For supplied adapter rankings, the evaluator removes unauthorized or out-of-scope IDs from relevance scoring, records them as misses, and fails the retrieval gate. Multi-hop synthesis belongs in generation or corpus evaluation.
{"query_id":"q-001","query":"How is source identity preserved?","principal":"group:readers","relevant_fragment_ids":["document-004::f0000"],"expected_empty":false,"category":"single_hop_factual","review_status":"reviewed"}
{"query_id":"q-005","query":"Show restricted evidence","principal":"group:outsiders","relevant_fragment_ids":[],"expected_empty":true,"category":"authorization","review_status":"reviewed"}Where generation evals are
corec eval generation evaluates output from any customer-owned model. It consumes answers, citations, reviewed fact requirements, and supplied faithfulness scores.
reviewed judgments + ranked results → retrieval report + misses
authorized evidence + model answers + reviewed facts → generation report + misses
selected configuration + complete reports → certification evidenceGeneration evaluation JSONL
- Answerable rows require every reviewed fact through an approved passage alternative.
- Unsupported citations fail independently of fluent prose.
- Unavailable-source rows use an abstention gate instead of answer faithfulness.
- Reports bind the selected search type, top K, chunk average, model, judge approval, latency, and configuration fingerprint.
{"query_id":"q-001","answer":"Carry stable source identity through every derived artifact.","cited_fragment_ids":["document-004::f0001"],"accepted_fragment_ids":["document-004::f0000","document-004::f0001"],"citation_requirements":[{"requirement_id":"identity-preservation","acceptable_fragment_ids":["document-004::f0001"]}],"expected_answerable":true,"faithfulness_score":0.94}Artifacts from a full run
Retrieval and answer reports must use the same A–D configuration ID. Approval fails when the reports use different corpus files, reviewed questions, search methods, or result counts.
engagement/full/
retrieval_golden_single.jsonl
ranked-results.jsonl
retrieval/
report.json
misses.jsonl
generation-input.jsonl
generation/
generation-report.json
generation-results.jsonl
generation-misses.jsonl