Start with a corpus

Name the corpus

Begin with the body of information you want people or applications to search. Give the corpus a name and state the subject or work it covers.

A corpus may include documents, database exports, messages, recordings, images, and other sources. The Concepts page defines corpus and the rest of the CoreCortex vocabulary in one place.

Write down the purpose

Write one page that a subject-matter expert can review. State what data belongs in the corpus, who will search it, what they need to decide, and which mistakes would be costly.

• Domain: the body of knowledge and activity represented by the corpus.
• Users: people or agents that will ask questions and consume evidence.
• Decisions: research, review, action, or synthesis the evidence should support.
• Boundaries: content, time periods, jurisdictions, products, languages, and exclusions.
• Trust: reliable sources, access restrictions, retention, and when the system should say that it found nothing.

Inventory datasets and modalities

List each dataset inside the corpus separately. Record its structure, refresh pattern, authority, access policy, and likely retrieval needs.

Read representative records

Open several items from every dataset before designing search. Include common records, unusually long records, sparse records, restricted records, and records that have changed over time.

Ask a person who knows the subject to explain what one item means, which fields matter, how versions work, and which source should win when two records disagree.

Collect real questions

Ask users for questions they already need answered. Include identifier lookups, paraphrases, filters, comparisons, relationship questions, and questions whose answer is absent. Later, reviewers will mark which passages should answer each question.

Read several real records with the questions beside you. A field profile can count missing and repeated values. A subject-matter expert still needs to explain what those values mean.

Start with local search

CoreCortex includes local in-memory vector search, so the first test needs no vector database account. Canonical records and fragments still use PostgreSQL. When the corpus grows, an adapter can move the vector index to pgvector, Milvus, or TurboPuffer. Ladybug provides the separate graph projection. The source records and test questions stay the same.

During this source checkout phase, the public CLI consumes adapter packages from a sibling corecortex-adapters checkout. Clone both repositories into the same parent directory before installing corec.

mkdir corecortex-workspace && cd corecortex-workspace
git clone https://github.com/corecortex-io/corecortex.io.git
git clone https://github.com/corecortex-io/corecortex-adapters.git
cd corecortex.io
# macOS native prerequisites
brew install openssl@3 pkg-config postgresql@17
cargo install --path crates/corec-cli --locked
corec --version
# After corec init writes the default destination:
corec validate ./corec.yaml

Create corec.yaml from your source

corec init reads a document directory, CSV, XLSX, or JSONL file and writes a valid corec.yaml. For raw sources, you provide the acquisition date and either an allow list or an explicit public choice. Repeat --attribute-column for CSV or XLSX fields that retrieval must filter. The command prints both access and empty-attribute review warnings.

corec init \
  --name reference-corpus \
  --dataset reference-documents \
  --source /absolute/path/to/source-documents \
  --pattern '**/*.xml' \
  --record-type reference_document \
  --title-element title \
  --provider "reference document export" \
  --observed-at 2026-08-29T00:00:00Z \
  --allow group:readers \
  --output .corec/reference-corpus/corec.yaml

corec validate .corec/reference-corpus/corec.yaml
corec source normalize \
  --config .corec/reference-corpus/corec.yaml \
  --dataset reference-documents \
  --output .corec/reference-corpus/records.jsonl

Register, ingest, and check access

Runtime commands use UUIDs returned by PostgreSQL. A tenant slug, corpus name, or dataset name cannot replace one of these IDs. Provision the tenant, apply corec.yaml, inspect the corpus, then copy each returned ID into the next command.

Core ingestion runs the chunker and projections in the active configuration. Core retrieval accepts a principal and applies access before returning evidence. Enterprise query routing is an optional step before this retrieval call.

The local vector index is process-local. Before vector retrieval, Core compares it with the active canonical fragments in PostgreSQL. A new process, external ingestion, or deletion triggers a refresh; an unchanged projection stays in memory. Separate ingest and query commands therefore run semantic retrieval without a vector service. Use a persistent adapter when this refresh work becomes material.

export DATABASE_URL=postgresql://dbuser@localhost/postgres
corec db migrate
corec tenant provision --slug tutorial-team --display-name "Tutorial team"
export TENANT_ID=<tenant-uuid-from-output>

corec apply \
  --tenant-id "$TENANT_ID" \
  --actor analyst@example.com \
  --idempotency-key reference-config-v1 \
  .corec/reference-corpus/corec.yaml
export CORPUS_ID=<corpus-uuid-from-output>

corec corpus inspect --tenant-id "$TENANT_ID" --corpus-id "$CORPUS_ID"
export DATASET_ID=<dataset-uuid-from-output>

corec ingest \
  --tenant-id "$TENANT_ID" \
  --corpus-id "$CORPUS_ID" \
  --dataset-id "$DATASET_ID" \
  --actor analyst@example.com \
  --idempotency-key reference-records-v1 \
  .corec/reference-corpus/records.jsonl

corec query retrieve \
  --tenant-id "$TENANT_ID" \
  --corpus-id "$CORPUS_ID" \
  --plan default \
  --principal group:readers \
  --query "How is source identity preserved?"

What happens next

Normalize the records, create searchable passages, and run a simple keyword baseline. Have reviewers mark the expected passages for a small set of questions. Then compare chunking, embedding models, and search methods. Test generated answers separately from search.

The tutorial shows this process with a prose dataset and a structured dataset.