Design the corpus before you scale it

The rule: measure in dependency order

A retrieval configuration is a hypothesis about one corpus version, not a reusable default. The research loop declares the search space first, compares equal samples, fixes one upstream winner at a time, and runs the full evaluation only after the optimum is known.

Later stages must use the winners selected upstream. Changing the converter changes the text; changing the text changes chunks; changing chunks changes embedding inputs; changing the embedding space can change whether dense, sparse, or hybrid retrieval wins.

Step 0 — declare the search space

Write every free parameter and its candidates before inspecting results. Values used to hold downstream variables constant are experimental controls, not product defaults; record why each control is reasonable.

Use one immutable corpus version, one reviewed judgment version, and one hashed 10–20 query mini-set across comparable runs. If source data, normalization, converter behavior, or reviewed judgments change, start a new iteration.

• Hash the reviewed query sample and retain the reviewer identity.
• Keep reviewed judgment identity separate from configuration-specific fragment IDs.
• Set retrievalSweepTopK to at least 50 and at least twice the largest candidate generation top_k.
• Declare tie-breaks before seeing scores: lower latency, lower cost, simpler strategy, or smaller K.

{
  "corpusVersion": "support-2026-08-12",
  "judgmentsVersion": "reviewed-v3",
  "miniQueries": 20,
  "converters": ["markitdown", "docling"],
  "chunkingStrategies": ["token_window", "markdown_heading"],
  "chunkSizes": [200, 350, 500],
  "overlaps": [20, 50],
  "embeddingModels": [
    {"id": "model-a@revision", "dimensions": 384},
    {"id": "model-b@revision", "dimensions": 768}
  ],
  "searchTypes": ["sparse", "dense", "hybrid"],
  "topK": [5, 10, 20, 50],
  "retrievalSweepTopK": 100,
  "plateauDelta": 0.03
}

Stage A — select the document representation

Compare converters on the same representative files. Inspect headings, reading order, repeated headers, tables, code blocks, lists, and source coverage before trusting retrieval metrics. Conversion sets the quality ceiling for every later stage.

For mixed corpora, an automatic policy may route spreadsheets to table-header-preserving conversion, PDF and Office documents to structure-preserving conversion, and code or Markdown to a lightweight path. Every fragment must record the concrete converter actually used; ‘auto’ describes policy, not provenance.

• Review one or two files for every observed format with a named reviewer.
• Build a fresh, isolated projection for every converter candidate.
• Run the same reviewed mini retrieval set against every candidate.
• Select on relevance; use latency and structural simplicity only as declared tie-breaks.
• If input is already canonical text JSONL, explicitly record that raw-document conversion is outside this corpus boundary instead of inventing a converter result.

{"fragment_id":"policy-17::f0004","source_extension":"pdf","requested_converter":"auto","effective_converter":"docling","processor_version":"...","token_count":318,"section_path":["Eligibility","Exceptions"]}

Stage B — sweep chunk strategy, size, and overlap

Keep the Stage A winner fixed. Recreate fragments for every chunk candidate and measure the chunks actually produced. A 500-token maximum does not imply 500-token chunks: headings, paragraphs, tables, and atomic blocks may create much smaller natural boundaries.

Audit average, p50, and p95 token counts, chunks per document, and structural outliers. Exact token_count must come from the selected tokenizer; a word-count estimate is not interchangeable.

Stage C — select the embedding model

Keep conversion and chunking unchanged. Create an isolated projection for each model candidate, precompute embeddings over the exact selected text, verify the stored vector dimension, and bind the ingestion receipt to the full configuration fingerprint.

Compare both early precision and retrieval ceiling. A model can improve Hit@10 while harming Hit@1; record both, alongside latency, cost, normalization, and model revision.

• Never reuse vectors between model candidates.
• Verify query embeddings use the same model and revision as ingestion.
• Disqualify a candidate that loses provenance, access filters, or source coverage even if relevance rises.
• Use cost as a declared near-tie decision, not as an after-the-fact justification.

pipeline:
  - use: text.chunk
    with: { strategy: markdown_heading, tokens: 350, overlap: 35 }
  - use: embedding.generate
    with:
      provider: your-provider
      model: model-b@revision
indexes:
  - fields:
      vector:
        - { name: content_embedding, dimensions: 768 }

Stage D1 — compare sparse, dense, and hybrid retrieval

Run the formal search-type sweep on the permanent Stage C projections. Retrieve deeply—at least rank 50—and report Hit@1, Hit@5, Hit@10, Hit@20, and Hit@50 together. Search type is query-time policy, but it still depends on the final embedding space.

Interpret the curve before selecting: a low Hit@50 points to representation or judgment problems; a healthy Hit@50 with weak Hit@5 points to ranking or fusion. Label any earlier search-type run preliminary and repeat it after Stage C.

Stage D2 — choose the smallest sufficient top_k

Fix the winning search type. Identify the first retrieval plateau where gain between adjacent K values is below the declared delta, then run a 10-query mini-generation sweep at candidate K values. Retrieval depth measures the ceiling; generation top_k controls how much evidence enters the answer context.

• Do not set top_k from habit or another corpus.
• Record the plateau K and the smaller generation K separately.
• For multi-collection retrieval, retain a per-collection K and bound fan-out.
• If more context lowers faithfulness, treat that as evidence—not a reason to relax the gate.

Stage E1 — full retrieval evaluation

Lock the exact A–D configuration fingerprint and run the full reviewed retrieval set. Retrieval judgments should cover single-hop factual, single-hop structural, hard-negative, expected-empty, calibration, and authorization behavior. Keep multi-hop synthesis in generation evaluation.

• Preserve the full Hit@K curve and miss categories, not only a pass/fail flag.
• Keep lexical, semantic, and hybrid controls visible in the decision evidence.
• Verify authorization independently from relevance.
• Reject a full report produced by a configuration other than the selected optimum.

{"query_id":"q-001","query":"How do I rotate an API token?","expected_sources":["support-004"],"should_match":true,"category":"single_hop_factual","review_status":"reviewed"}
{"query_id":"q-005","query":"How do I enable a feature that does not exist?","expected_sources":[],"should_match":false,"category":"negative","review_status":"reviewed"}

Stage E2 — full generation and abstention evaluation

Evaluate answerable and unavailable-source scenarios separately. An answer-quality pass requires a produced answer, coverage of every reviewed fact requirement through an approved passage alternative, no unsupported citation, and faithfulness at or above the target.

Expected-empty and unavailable-source runs use an abstention gate instead. Do not apply an answer faithfulness gate to a run where the correct outcome is no answer.

• Bind generator model, judge model, reviewer approval, prompt revision, and checkpoint.
• Require a judge at least as capable as the generator for the evaluated task.
• Record selected top_k, search type, chunk average, retrieval Hit@K, and plateau K in the generation report.
• Review plausibly supportive unjudged citations instead of weakening citation precision.

{"query_id":"q-001","answer":"Create a replacement token, update clients, then revoke the old token.","cited_fragment_ids":["support-004::f0001"],"accepted_fragment_ids":["support-004::f0000","support-004::f0001"],"citation_requirements":[{"requirement_id":"rotation-order","acceptable_fragment_ids":["support-004::f0001"]}],"expected_answerable":true,"faithfulness_score":0.94}

Stage E3 — certify multi-collection behavior

For a corpus with multiple collections, report each collection’s selected K and label execution as routed or as an unrouted baseline. Set the global cap from expected collections and their selected depths so irrelevant collections cannot crowd out the evidence that generation needs.

Evaluate routing accuracy separately. Graph-assisted retrieval is another named plan: adopt relationship expansion only when it improves relationship-specific judgments without breaking authorization, provenance, latency, or evidence coverage.

Read misses as instructions for the next iteration

A new converter, chunk strategy, production model, corpus snapshot, or reviewed judgment version starts a new ordered iteration. Do not hide unresolved upstream choices in a backlog while certifying a downstream configuration.

What Core provides—and what Enterprise automates

CoreCortex Core provides the public corpus configuration, canonical record and fragment contracts, provenance, access policy, retrieval plans, adapter interfaces, and evaluation exchange formats needed to implement this method.

CoreCortex Enterprise automates search-space validation, A–E experiment planning, equal-sample enforcement, chunk and converter audits, isolated projection receipts, deterministic stage selection, optimum binding, full-report finalization, and certification manifests. The methodology stays public so teams can implement their own orchestration against the same Core contracts.