Configuration

Create corec.yaml

Use one file to name the datasets, source mappings, processing steps, indexes, access rules, and retrieval plans.

Generate a valid starting contract

Use corec init as the authoring path for your own source data. It inspects the source, writes every required section, validates the result through the same public configuration parser, and protects an existing destination unless --force is explicit.

Raw document and tabular sources require an immutable acquisition timestamp plus either --public or one or more --allow principals. Canonical JSONL preserves access from each record. Source paths are written relative to the configuration when possible. Keep secrets out of the file and bind remote credentials through secret references.

shellCoreCortex Core

corec init \
  --name reference-corpus \
  --dataset reference-documents \
  --source ./data/source-documents \
  --pattern '**/*.xml' \
  --record-type reference_document \
  --title-element title \
  --provider "reference document export" \
  --observed-at 2026-08-29T00:00:00Z \
  --allow group:readers \
  --output .corec/reference-corpus/corec.yaml

corec validate .corec/reference-corpus/corec.yaml

Choose the source authoring path

The outer source object tells CoreCortex how to read the dataset. source.mapping.provenance records the provider, acquisition time, and optional source URL. The explicit provenance name avoids the ambiguous source.mapping.source nesting found in earlier examples.

Input	Generator behavior	Normalization behavior
XML, Markdown, text	One-file-per-record mapping; optional glob and XML title element	Preserves relative identity, access, source digest, and converter provenance
CSV	Reads headers; mapping flags select identity, title, content, and repeatable attribute columns	Builds canonical text plus reviewed filter attributes
XLSX	Requires explicit identity, title, and content columns; accepts repeatable attribute columns	Reads the selected sheet and header row
Canonical JSONL	Declares the existing normalized-record source	Validates every line and reports file plus line on failure

Minimal shape

This is the exact alpha hierarchy: each dataset names its source and processing path; storage, indexes, and retrieval plans describe the CoreCortex model of the corpus. The generated configuration schema is the machine-readable authority for this contract.

yamlCoreCortex Core

apiVersion: corecortex.io/v1alpha1
kind: Corpus
metadata:
  name: reference-corpus
  tenant: demo
  environment: development
spec:
  datasets:
    - name: reference-documents
      mappingFingerprint: <sha256-from-corec-init-or-reviewed-registration>
      source: { type: jsonl, path: ./golden/records.jsonl }
      pipeline:
        - { use: text.normalize }
        - { use: text.chunk, with: { tokens: 240, overlap: 30 } }
        - { use: embedding.generate, with: { provider: local, model: deterministic-token-hash@starter-v1 } }
  storage:
    artifacts: { backend: local, path: ./.corec/artifacts }
    records: { backend: jsonl, path: ./golden/records.jsonl }
  indexes:
    - name: reference
      backend:
        type: extension
        adapter: io.corec.memory.vector
        settings: { endpoint: memory://local, index: reference, dimensions: 384, metric: cosine }
      fields:
        lexical: [title, content]
        vector: [{ name: content_embedding, dimensions: 384 }]
        filter: [category]
  retrievalPlans:
    - name: reference-hybrid
      index: reference
      candidates:
        - { type: bm25, field: content, limit: 50 }
        - { type: vector, field: content_embedding, limit: 50 }
      fusion: { type: rrf, rankConstant: 60 }
      limit: 10

Configuration lifecycle

Validate authored state without writing it.
Apply a new immutable revision with an idempotency key.
Build new index versions against that revision.
Activate a tested version without rewriting source records.
Retain lineage so results can identify the plan and index versions used.

Compatibility

Continue readingTutorial→