Tutorial

The scenario

An information-operations team wants an assistant that helps analysts understand recurring source changes and find the approved operating guide. The source information already exists, though it lives in two collections with different structures and uses.

The tutorial treats both collections as one knowledge-operations corpus. A useful response may need a reference guide from one dataset, exact collection and change history from the other, and the relationship between them.

1 — write the corpus charter

This charter gives reviewers a stable definition of useful retrieval. It also reveals mandatory filters: collection type, guide status, effective date, access scope, and change-record visibility.

2 — inventory the two datasets

Reference guides carry explanatory language and procedural sequence. Change records carry exact identifiers, topic, time, access scope, and outcome data. Their different meanings justify different representations and indexes within the same corpus.

3 — inspect representative records with a domain reviewer

• Confirm which identifier remains stable across corrections and versions.
• Ask whether guide revision 4 supersedes revision 3 or coexists by collection type.
• Clarify whether notes are authoritative observations, author interpretation, or both.
• Identify fields that carry access restrictions or personally identifying information.

{"dataset":"reference-guides","guide_id":"GUIDE-17","title":"Source change review","revision":"4","effective_at":"2026-05-01","status":"approved","body":"Confirm source version... compare canonical identity..."}
{"dataset":"change-records","change_id":"CHANGE-4821","collection_id":"RECORD-104","collection_type":"COLLECTION-A","topic_code":"TOPIC-07","observed_at":"2026-07-08","action":"refreshed canonical snapshot","outcome":"projection rebuilt","notes":"source version mismatch detected"}

4 — collect questions before choosing indexes

These questions establish the art of the possible: passage retrieval, structured analysis, cross-dataset retrieval, temporal filters, authorization, and abstention all belong to the same corpus experience.

5 — define normalized records and passages

For this tutorial, one guide revision becomes one record and its sections become passages. Each change row remains one record with typed fields and one searchable narrative.

6 — discover schema and relationships

7 — choose storage by evidence role

Begin this tutorial with local JSONL, canonical PostgreSQL fragments, and an in-memory vector index. Core refreshes that vector index when a process starts or canonical fragments change. Move it to a persistent adapter after the measured workload requires faster startup, more scale, concurrency, or lower latency.

8 — create corec.yaml

This abbreviated excerpt names both datasets, their processing steps, the in-memory index, and the first retrieval plan. Use corec init and the Configuration page to produce the complete file, including storage, source mappings, access, and each generated mapping fingerprint.

apiVersion: corecortex.io/v1alpha1
kind: Corpus
metadata:
  name: knowledge-operations
spec:
  datasets:
    - name: reference-guides
      # corec init writes mappingFingerprint in the complete file
      source: { type: jsonl, path: ./data/reference-guides.jsonl }
      pipeline:
        - { use: text.normalize }
        - { use: text.chunk, with: { strategy: corec.chunk.markdown-aware, tokens: 350, overlap: 35 } }
        - { use: embedding.generate, with: { provider: local, model: deterministic-token-hash@starter-v1 } }
    - name: change-records
      source: { type: csv, path: ./data/change-records.csv }
      pipeline:
        - { use: tabular.normalize }
        - { use: record.render_text, with: { template: "{collection_id} {topic_code} {action} {outcome} {notes}" } }
        - { use: embedding.generate, with: { provider: local, model: deterministic-token-hash@starter-v1 } }
  indexes:
    - name: evidence-vector
      backend:
        type: extension
        adapter: io.corec.memory.vector
        settings: { endpoint: memory://local, index: knowledge-operations, dimensions: 384, metric: cosine }
      fields:
        lexical: [title, content]
        vector: [{ name: content_embedding, dimensions: 384 }]
        filter: [access_scope, collection_id, collection_type, topic_code, status, effective_at]
  retrievalPlans:
    - name: topic-investigation
      index: evidence-vector
      candidates:
        - { type: bm25, field: content, limit: 50 }
        - { type: vector, field: content_embedding, limit: 50 }
      fusion: { type: rrf, rankConstant: 60 }
      limit: 20

9 — validate and inspect the plan

Validation checks the complete corec.yaml before an index changes. The query-plan demo explains backend capability planning with a built-in fixture. Live retrieval against this corpus begins after the configuration is applied and both datasets are ingested.

corec validate ./knowledge-operations/corec.yaml
corec demo query-plan

10 — register IDs, ingest, and run a live access check

Tenant, corpus, and dataset arguments are PostgreSQL UUIDs. Slugs and names cannot stand in for them. Run source normalize and corec ingest once for each dataset ID before querying.

Repeat the query with a principal outside the records' allow lists. The result must contain no restricted evidence. Core executes this live retrieval without the Enterprise extension.

Inspect explain.candidates in the allowed result. The vector candidate should return evidence. This proves the new CLI process rebuilt the in-memory semantic projection from canonical fragments.

export DATABASE_URL=postgresql://dbuser@localhost/postgres
corec db migrate
corec tenant provision --slug knowledge-ops --display-name "Knowledge Operations"
export TENANT_ID=<tenant-uuid-from-output>

corec apply \
  --tenant-id "$TENANT_ID" \
  --actor analyst@example.com \
  --idempotency-key knowledge-operations-config-v1 \
  ./knowledge-operations/corec.yaml
export CORPUS_ID=<corpus-uuid-from-output>

corec corpus inspect --tenant-id "$TENANT_ID" --corpus-id "$CORPUS_ID"
export GUIDES_DATASET_ID=<reference-guides-uuid>
export CHANGES_DATASET_ID=<change-records-uuid>

corec source normalize --config ./knowledge-operations/corec.yaml --dataset reference-guides --output ./.corec/guides.jsonl
corec ingest --tenant-id "$TENANT_ID" --corpus-id "$CORPUS_ID" --dataset-id "$GUIDES_DATASET_ID" --actor analyst@example.com --idempotency-key knowledge-guides-v1 ./.corec/guides.jsonl

corec source normalize --config ./knowledge-operations/corec.yaml --dataset change-records --output ./.corec/changes.jsonl
corec ingest --tenant-id "$TENANT_ID" --corpus-id "$CORPUS_ID" --dataset-id "$CHANGES_DATASET_ID" --actor analyst@example.com --idempotency-key knowledge-changes-v1 ./.corec/changes.jsonl

corec query retrieve \
  --tenant-id "$TENANT_ID" \
  --corpus-id "$CORPUS_ID" \
  --plan topic-investigation \
  --principal group:analysts \
  --query "What guide applies to the repeated topic on RECORD-104?"

11 — create reviewed golden data

• Review answerability and acceptable evidence alternatives with a domain owner.
• Keep dataset-specific, cross-dataset, negative, and authorization cases visible.
• Retain stable source or record identity across representation candidates.
• Version reviewed questions separately from corec.yaml.

{"query_id":"q-guide-01","query":"What is the approved source-change inspection sequence?","principal":"group:analysts","relevant_fragment_ids":["GUIDE-17@4::f0000"],"expected_empty":false,"category":"guide_factual","review_status":"reviewed"}
{"query_id":"q-work-01","query":"Which COLLECTION-A records had TOPIC-07 twice this quarter?","principal":"group:analysts","relevant_fragment_ids":["CHANGE-4821::f0000","CHANGE-4970::f0000"],"expected_empty":false,"category":"structured","review_status":"reviewed"}
{"query_id":"q-cross-01","query":"What guide applies to the repeated topic on RECORD-104?","principal":"group:analysts","relevant_fragment_ids":["CHANGE-4821::f0000","CHANGE-4970::f0000","GUIDE-17@4::f0000"],"expected_empty":false,"category":"cross_dataset","review_status":"reviewed"}
{"query_id":"q-empty-01","query":"Show the retired guide for this restricted access scope","principal":"group:outsiders","relevant_fragment_ids":[],"expected_empty":true,"category":"authorization","review_status":"reviewed"}

12 — establish the local retrieval baseline

The deterministic BM25 baseline produces a report, per-query results, and misses. Read each miss by dataset and question type. A missing guide passage, a structured aggregation, and a cross-dataset join point to different next actions.

corec eval retrieval \
  --fragments ./knowledge-operations/golden/fragments.jsonl \
  --judgments ./knowledge-operations/golden/evals.jsonl \
  --output ./knowledge-operations/runs/baseline \
  --limit 50

13 — run retrieval experiments

Run these comparisons with the guide and change-record questions from Step 11. Save the selected configuration ID with the resulting reports.

14 — add relationship retrieval when the questions justify it

Create canonical relations such as change_record observed_topic topic:TOPIC-07, guide applies_to topic:TOPIC-07, and change_record occurred_on collection:RECORD-104. A Ladybug projection can traverse from collection to repeated changes to approved guides for relationship-specific queries.

Compare the graph-assisted plan with the routed lexical and semantic baseline on reviewed cross-dataset questions. Promote graph expansion when it improves evidence coverage while preserving authorization, provenance, latency, and comprehensibility.

15 — connect an optional synthesis model

Add a response plan after the cross-dataset query returns GUIDE-17 and the matching change records. Require citations to those passage IDs and test the retired-guide question for no-answer behavior.

16 — approve and operate the selected configuration

• Record the winning representation, segmentation, embedding, retrieval, routing, and evidence-depth decisions.
• Bind source versions, judgment version, configuration fingerprint, projection receipts, and evaluation reports.
• Activate the selected projection through the chosen adapters.
• Monitor source coverage, schema drift, access changes, retrieval quality, latency, and cost.
• Start a new design-loop iteration when corpus meaning, sources, questions, or production behavior changes materially.