Evaluation · Retrieval, generation, corpus

Measure evidence before answers

Keep retrieval reachability, answer support, abstention, and multi-collection behavior as separate scorecards tied to the same configuration.

Three evaluation layers

Layer	Question	Primary evidence
Retrieval	Was the reviewed evidence found, and at what rank?	Hit@K curve, MRR/NDCG, negatives, filters, ACLs, latency
Generation	Did the answer cover facts, cite support, and abstain correctly?	Fact requirements, citation precision, faithfulness, abstention
Corpus	Did routing, fan-out, and optional graph expansion preserve evidence?	Routing accuracy, per-collection K, relationship coverage

Where retrieval evals are in the OSS checkout

Reviewed fixture judgments live in examples/fda-labels/golden/evals.jsonl and examples/support-kb/golden/evals.jsonl. The offline runner is tools/reference_demos.py; the hosted runner is the corec demo turbo-puffer command implemented in the CLI.

The five-record support fixture reports Hit@1, Hit@3, MRR, and expected-empty accuracy as an integration check. A production design-loop run retrieves to at least rank 50 and reports Hit@1, Hit@5, Hit@10, Hit@20, and Hit@50.

shellCoreCortex Core

python3 tools/reference_demos.py run support-kb

set -a; source .env; set +a
corec demo turbo-puffer \
  --snapshot golden \
  --name support-kb \
  --namespace-prefix support-eval

Retrieval judgment JSONL

Retrieval golden data covers single-hop factual and structural queries, hard negatives, expected-empty behavior, calibration, and authorization. Multi-hop synthesis belongs in generation or corpus evaluation.

jsonlCoreCortex Core

{"query_id":"q-001","query":"How do I rotate an API token?","expected_sources":["support-004"],"should_match":true,"category":"single_hop_factual","review_status":"reviewed"}
{"query_id":"q-005","query":"How do I enable a feature that does not exist?","expected_sources":[],"should_match":false,"category":"negative","review_status":"reviewed"}

Where generation evals are

Core publishes the synthesis evidence and response contracts but does not ship a universal generator or model judge. CoreCortex Enterprise implements the production retrieval runner in src/corec_fde/eval/run.py and the generation runner in src/corec_fde/eval/run_generation_eval.py.

The generation runner evaluates output from any customer-owned model. It consumes answers, citations, reviewed fact requirements, and faithfulness scores; synthesis does not need to run inside CoreCortex.

shell · Enterprise CLICoreCortex Core

corec-fde eval-retrieval \
  --fragments engagement/full/fragments.jsonl \
  --judgments engagement/full/retrieval_golden_single.jsonl \
  --retrieval-mode hybrid \
  --ranked-results engagement/full/ranked-results.jsonl \
  --limit 50 \
  --output engagement/full/retrieval

corec-fde eval-generation \
  --input engagement/full/generation-input.jsonl \
  --retrieval-report engagement/full/retrieval/report.json \
  --model customer-model@revision \
  --judge-model approved-judge@revision \
  --faithfulness-target 0.80 \
  --qualified-answer-target 0.80 \
  --output engagement/full/generation

Generation evaluation JSONL

Answerable rows require every reviewed fact through an approved passage alternative.
Unsupported citations fail independently of fluent prose.
Unavailable-source rows use an abstention gate instead of answer faithfulness.
Reports bind the selected search type, top K, chunk average, model, judge approval, latency, and configuration fingerprint.

jsonlCoreCortex Core

{"query_id":"q-001","answer":"Create a replacement token, update clients, then revoke the old token.","cited_fragment_ids":["support-004::f0001"],"accepted_fragment_ids":["support-004::f0000","support-004::f0001"],"citation_requirements":[{"requirement_id":"rotation-order","acceptable_fragment_ids":["support-004::f0001"]}],"expected_answerable":true,"faithfulness_score":0.94}

Artifacts from a full run

Retrieval and generation reports must bind the same A–D configuration fingerprint. Certification fails when full reports use a different corpus, judgment version, search policy, or evidence depth.

textCoreCortex Core

engagement/full/
  retrieval_golden_single.jsonl
  ranked-results.jsonl
  retrieval/
    report.json
    misses.jsonl
  generation-input.jsonl
  generation/
    generation-report.json
    generation-results.jsonl
    generation-misses.jsonl

Continue readingRun the ordered A–E loop→