Rehearse A–E with the public fixtures

What this walkthrough proves

The public fixtures are deliberately small. They prove deterministic normalization, source identity, provenance, access shape, reviewed JSONL, retrieval metrics, expected-empty behavior, and backend-neutral planning. They are a rehearsal of the method, not evidence that their 240-token configuration is optimal for your data.

The OSS checkout does not contain the proprietary A–E controller. You create candidate configurations and retain artifacts manually; CoreCortex Enterprise automates the same ordered process and refuses incomparable runs.

1 — build and validate the checkout

Validation checks the authored corpus contract before state changes. The default demo is local and deterministic: it exercises FDA-shaped records, support articles, maintenance rows, relationship extraction, and adapter discovery without requiring a hosted search service.

git clone <public-corecortex-repository> corecortex
cd corecortex

cargo install --path crates/corec-cli --locked
corec validate examples/support-kb/corec.yaml
corec validate examples/maintenance-work-orders/corec.yaml
make demo

2 — inspect normalization and provenance

The final cmp should be silent. It proves that the CSV and XLSX delivery formats normalize to identical canonical records while preserving typed fields, access policy, source URL, publication time, observed time, and source version.

This is a contract check before Stage A. For raw PDF, Office, or mixed document corpora, Stage A would compare converter representations before normalizing the winning path into candidate fragments.

corec source normalize \
  --config examples/maintenance-work-orders/corec.yaml \
  --dataset work-orders-csv \
  --output /tmp/work-orders-csv.jsonl

corec source normalize \
  --config examples/maintenance-work-orders/corec.yaml \
  --dataset work-orders-xlsx \
  --output /tmp/work-orders-xlsx.jsonl

cmp /tmp/work-orders-csv.jsonl /tmp/work-orders-xlsx.jsonl

3 — inspect the baseline corpus hypothesis

The fixture config proposes 240-token chunks with 30-token overlap, a 384-dimensional embedding, BM25 plus vector candidates, reciprocal-rank fusion, and a final limit of five. In a research loop these are candidate values—not defaults and not yet a winner.

sed -n '1,220p' examples/support-kb/corec.yaml
sed -n '1,20p' examples/support-kb/golden/fragments.jsonl
sed -n '1,20p' examples/support-kb/golden/evals.jsonl

4 — establish the deterministic lexical control

Keep this lexical control visible when testing semantic and hybrid candidates; a vector plan is not automatically superior. The next section shows the exact output from the current checkout.

python3 tools/reference_demos.py run support-kb

4b — read the control result

The perfect score verifies a five-record integration contract: four reviewed positive queries are found, and the one expected-empty query returns no result. It is not evidence that the fixture configuration will generalize to a production corpus.

{
  "demo": "support-kb",
  "records": 5,
  "fragments": 5,
  "metrics": {
    "queries": 5,
    "positive_queries": 4,
    "expected_empty_queries": 1,
    "hit_at_1": 1.0,
    "hit_at_3": 1.0,
    "mrr": 1.0,
    "no_result_accuracy": 1.0
  }
}

5 — materialize Stage B candidates

Copy the configuration into an ignored experiment directory. Keep the representation fixed and vary only the declared chunk candidate. Each candidate needs a unique fragment path, projection identity, configuration fingerprint, and metrics row.

• Audit exact avg, p50, and p95 token counts plus chunks per document.
• Map the same reviewed source spans to each candidate’s fragment IDs.
• Run the same mini-query sample and record Hit@1/5/10/20/50.
• Select one B winner before creating embedding-model candidates.

mkdir -p .corec/experiments/support-loop/{b-200,b-350,b-500}
cp examples/support-kb/corec.yaml .corec/experiments/support-loop/b-200/corec.yaml
cp examples/support-kb/corec.yaml .corec/experiments/support-loop/b-350/corec.yaml
cp examples/support-kb/corec.yaml .corec/experiments/support-loop/b-500/corec.yaml

# Edit only text.chunk.tokens and its declared overlap in each copy.
# Regenerate fragments and embeddings; never copy them between candidates.

6 — run semantic and hybrid candidates on an isolated projection

The remote demo uses the same committed fragments, evaluations, access filters, projection verification, and rebuildability contract as the offline reference. Use a different namespace for every Stage C model candidate and delete or archive candidates only after their receipts and result hashes are retained.

ollama pull all-minilm
set -a; source .env; set +a

corec demo turbo-puffer \
  --snapshot golden \
  --name support-kb \
  --namespace-prefix support-loop-candidate \
  --embedding-model all-minilm:latest

6b — project a customer corpus

Use the custom target with explicit fragment, evaluation, and descriptor paths. Keep credentials in an ignored .env file, and give every candidate a distinct namespace and report file.

corec demo turbo-puffer \
  --snapshot candidate \
  --name custom \
  --namespace-prefix customer-corpus-v1 \
  --fragments-file /path/to/fragments.jsonl \
  --evaluations-file /path/to/evals.jsonl \
  --descriptor-file /path/to/demo.json \
  --report-file /path/to/candidate-report.json

7 — compare Stage D curves, not one top_k

Export or retain ranked results through at least rank 50 for sparse, dense, and hybrid plans. Compute the same Hit@K points for each. After selecting the policy, sweep generation K on a stable 10-query sample.

{
  "experiment_id": "D-hybrid",
  "configuration_fingerprint": "sha256:...",
  "corpus_version": "support-v1",
  "judgments_version": "reviewed-v1",
  "query_sample_sha256": "...",
  "metrics": {
    "hit_at_1": 0.72,
    "hit_at_5": 0.84,
    "hit_at_10": 0.89,
    "hit_at_20": 0.91,
    "hit_at_50": 0.91,
    "plateau_k": 20,
    "p95_latency_ms": 57
  }
}

8 — assemble Stage E reviewed inputs

Promote only human-reviewed judgments into the full golden set. Generated questions are useful integration checks, but remain unreviewed until a domain owner confirms answerability, positive evidence, accepted alternatives, hard negatives, and authorization expectations.

• Retrieval: single-hop factual and structural queries, negatives, expected-empty, calibration, and authorization cases.
• Generation: fact requirements, accepted passage alternatives, redundant passages, hard negatives, and external faithfulness scores.
• Unavailable sources: a separate abstention run with its own target.
• Multi-hop and cross-collection questions: generation or corpus evaluation, not ordinary Hit@K.

9 — write the decision, not just the score

A defensible handoff includes source hashes, reviewed-sample identities, every candidate result, the selected configuration, miss analysis, full-report hashes, known gaps, and exact reproduction commands. If any gate fails, preserve the failure and start the next ordered iteration instead of weakening the target.

{
  "corpus_version": "support-v1",
  "judgments_version": "reviewed-v3",
  "ordered_decisions": ["A", "B", "C", "D", "E"],
  "selected": {
    "converter": "docling",
    "chunk_strategy": "markdown_heading",
    "chunk_tokens": 350,
    "overlap": 35,
    "embedding_model": "model-b@revision",
    "search_type": "hybrid",
    "top_k": 10
  },
  "configuration_fingerprint": "sha256:...",
  "retrieval_report_sha256": "...",
  "generation_report_sha256": "...",
  "rationale": "Selected on reviewed evidence; K=20 was the retrieval plateau, while generation improvement from 10 to 20 stayed below 0.03 and added noise."
}