Retrieval experiments

Why the order matters

Each stage changes the input to the next stage. A document converter produces text. A chunker splits that text. An embedding model turns each chunk into a vector. The search method ranks those chunks.

Compare one stage at a time on the same files and questions. Keep the best result from that stage while you test the next one. This makes each result easier to explain and reproduce.

Step 0 — write down the experiment

List every setting you plan to compare before you run the first test. Name the corpus, retrieval judgments, generation judgments, fixed mini samples, sample size, and tie-break rule.

CoreCortex resolves each artifact path and computes its SHA-256 identity when the design loop starts. Use the same files and the same 10–20 reviewed questions in every comparable run. Start a new experiment when the files or questions change.

• Store the reviewed files and the reviewer's identity. CoreCortex records each file's resolved path and computed hash.
• Keep the reviewed questions separate from passage IDs created by a particular chunker.
• Set retrievalSweepTopK to at least 50 and at least twice the largest candidate generation top_k.
• Declare tie-breaks before seeing scores: lower latency, lower cost, simpler strategy, or smaller K.

{
  "apiVersion": "corecortex.io/enterprise/v1alpha1",
  "kind": "CorpusDesignSearchSpace",
  "dataset": "reference-documents",
  "collections": ["reference-documents"],
  "artifacts": {
    "corpus": "../prepared/records.jsonl",
    "retrievalJudgments": "../golden/retrieval-reviewed.jsonl",
    "generationJudgments": "../golden/generation-reviewed.jsonl",
    "miniRetrievalJudgments": "../golden/retrieval-smoke-reviewed.jsonl",
    "miniGenerationJudgments": "../golden/generation-smoke-reviewed.jsonl"
  },
  "miniQueries": 20,
  "miniGenerationQueries": 10,
  "retrievalSweepTopK": 100,
  "parameters": {
    "converters": ["markitdown", "docling"],
    "chunkingStrategies": ["token-window", "markdown-aware"],
    "chunkTokens": [200, 350, 500],
    "overlapTokens": [20, 50],
    "embeddingModels": [
      {"id": "model-a@revision", "dimensions": 384, "costPerMillionTokens": 1.0},
      {"id": "model-b@revision", "dimensions": 768, "costPerMillionTokens": 2.0}
    ],
    "searchTypes": ["semantic", "hybrid"],
    "topK": [5, 10, 20, 50]
  },
  "anchors": {
    "converter": "markitdown",
    "chunkingStrategy": "markdown-aware",
    "chunkTokens": 350,
    "overlapTokens": 20,
    "embeddingModel": "model-a@revision",
    "searchType": "semantic",
    "topK": 20
  },
  "anchorRationale": "Controls for the first comparison; these are candidates rather than defaults.",
  "selection": {"metricTieDelta": 0.01, "plateauDelta": 0.03}
}

Stage A — convert source files to text

Run each converter on the same representative files. Check headings, reading order, repeated page headers, tables, code blocks, lists, and missing text before you compare search scores.

A mixed corpus can use different converters for different file types. Record the converter used for every output passage. If a policy selected it automatically, record both the policy and the selected converter.

JSONL and Parquet may already contain the text you want to search. Give that text layout a name, such as canonical:title-content-v1, so later runs can identify it exactly.

• Review one or two files for every observed format with a named reviewer.
• Build a fresh search index for every converter candidate.
• Run the same reviewed mini retrieval set against every candidate.
• Select on relevance; use latency and structural simplicity only as declared tie-breaks.
• If input is already canonical JSONL or Parquet, use the canonical:<strategy> audit path with a named reviewer and representative file.

{"fragment_id":"document-17::f0004","source_extension":"pdf","requested_converter":"auto","effective_converter":"docling","processor_version":"...","token_count":318,"section_path":["Eligibility","Exceptions"]}

Stage B — split documents into searchable passages

Use the converter selected in Stage A. Create passages for every chunking candidate and measure the passages produced. Headings, paragraphs, and tables often create passages shorter than the configured maximum.

Record average, median, and 95th-percentile token counts. Also record passages per document and unusually short or long passages. Use the same tokenizer that the chunker uses.

Stage C — compare embedding models

Keep the selected converter and chunks unchanged. Build a separate index for each embedding model. Check that stored vectors have the dimensions declared by the model.

Record Hit@1 and Hit@10, latency, cost, vector normalization, and the exact model revision. A model can improve one rank while making another rank worse.

• Create new vectors for every model candidate.
• Verify query embeddings use the same model and revision as ingestion.
• Disqualify a candidate that loses provenance, access filters, or source coverage even if relevance rises.
• Use cost as a declared near-tie decision, not as an after-the-fact justification.

pipeline:
  - use: text.chunk
    with: { strategy: markdown_heading, tokens: 350, overlap: 35 }
  - use: embedding.generate
    with:
      provider: your-provider
      model: model-b@revision
indexes:
  - fields:
      vector:
        - { name: content_embedding, dimensions: 768 }

Stage D1 — compare keyword, vector, and hybrid search

Use the index selected in Stage C. Fetch at least 50 results and report Hit@1, Hit@5, Hit@10, Hit@20, and Hit@50 for each search method.

A low Hit@50 usually means the relevant passage is missing, poorly represented, or judged incorrectly. A strong Hit@50 with a weak Hit@5 points to ranking or hybrid-search tuning.

Stage D2 — choose how many passages to return

Use the search method selected in D1. Keep each result count that scores close to the best retrieval result. Then test answer generation on 10 questions at those result counts. Choose a count that supplies the required facts without adding distracting passages.

• Choose top_k from results on this corpus.
• Record the plateau K and the smaller generation K separately.
• For multi-collection retrieval, retain a per-collection K and bound fan-out.
• If more context lowers faithfulness, choose a smaller result count or improve ranking.

Stage E1 — run every reviewed search question

Save the selected A–D settings and run the full set of reviewed search questions. Include factual queries, structural queries, hard negatives, expected-empty queries, score calibration, and access checks.

• Save the full Hit@K curve and a category for every miss.
• Keep lexical, semantic, and hybrid controls visible in the decision evidence.
• Verify authorization independently from relevance.
• Reject a full report produced with settings other than the selected A–D settings.
• Bind the report to the ready projection receipt that produced its rankings.

{"query_id":"q-001","query":"How is source identity preserved?","principal":"group:readers","relevant_fragment_ids":["document-004::f0000"],"expected_empty":false,"category":"single_hop_factual","review_status":"reviewed"}
{"query_id":"q-005","query":"Show information absent from the corpus","principal":"group:readers","relevant_fragment_ids":[],"expected_empty":true,"category":"negative","review_status":"reviewed"}

Stage E2 — test answers and no-answer behavior

For answerable questions, check that the answer covers every required fact, cites an approved passage, contains no unsupported citation, and meets the faithfulness target.

For questions whose answer is absent, check that the system says it cannot answer from the corpus. Score this behavior separately from answer quality.

• Bind generator model, judge model, reviewer approval, prompt revision, and checkpoint.
• Require a judge at least as capable as the generator for the evaluated task.
• Record selected top_k, search type, chunk average, retrieval Hit@K, and plateau K in the generation report.
• Review plausibly supportive unjudged citations instead of weakening citation precision.

{"query_id":"q-001","answer":"Carry stable source identity through every derived artifact.","cited_fragment_ids":["document-004::f0001"],"accepted_fragment_ids":["document-004::f0000","document-004::f0001"],"citation_requirements":[{"requirement_id":"identity-preservation","acceptable_fragment_ids":["document-004::f0001"]}],"expected_answerable":true,"faithfulness_score":0.94}

Stage E3 — test queries that span datasets

For a corpus with several datasets, record the result count used for each dataset. Compare query routing with a baseline that searches every dataset.

Test routing accuracy on its own. Add graph search when reviewed relationship questions improve and access checks, source links, latency, and fact coverage still pass.

Read misses as instructions for the next iteration

Start a new experiment when the converter, chunker, model, corpus snapshot, or reviewed questions change. Resolve earlier-stage problems before approving the final configuration.

Continue with an implementation path

Use the two-dataset tutorial to run the method manually. The Enterprise overview describes the licensed automation around the same method.