Indexes are projections; schema preserves meaning
Choose lexical, vector, filter, relational, and graph capabilities deliberately—without confusing the search backend with the corpus data model.
What an index is
An index is a rebuildable searchable projection of canonical records and fragments. It is not source truth. If a projection is deleted, CoreCortex can recreate it from the versioned corpus, pipeline, and canonical records.
One index may contain lexical text, vectors, and exact filter fields. ‘Hybrid’ describes how a retrieval plan uses those capabilities; it does not create a second canonical record format.
| Field role | Meaning | Typical use |
|---|---|---|
| Lexical | Tokenized title or content | Exact terms, citations, identifiers, BM25 |
| Vector | Dense embedding with fixed dimensions | Paraphrases and semantic similarity |
| Filter | Exact scalar metadata and access fields | Scope, dates, categories, tenants, ACLs |
The available retrieval types
| Type | What it does | Use it when |
|---|---|---|
| Lexical / BM25 | Ranks token and term matches | Names, codes, citations, or exact language matter |
| Dense vector | Ranks embedding similarity | Questions paraphrase the source |
| Hybrid | Fuses lexical and vector candidates | The workload mixes exact and semantic queries |
| Filtered | Applies exact metadata and authorization | Results must obey scope, dates, policy, or tenant rules |
| Relational | Uses typed columns, joins, and aggregation | The question is structured or analytical |
| Graph-assisted | Expands evidenced relationships or bounded paths | Reviewed multi-hop judgments show a benefit |
Index type is not backend
TurboPuffer, PostgreSQL, Milvus, pgvector, and Ladybug are execution backends. The portable contract declares required capabilities; the adapter advertises what it can execute; the planner chooses placement.
HNSW, IVFFlat, vector distance, analyzers, and collection tuning are physical adapter settings—not portable logical index types.
| Backend | Role in CoreCortex |
|---|---|
| TurboPuffer | Native lexical, vector, filters, and RRF |
| PostgreSQL | Full-text, exact filters, relational queries, optional pgvector |
| Milvus / pgvector | Vector adapters behind the extension contract |
| Ladybug | Embedded graph projection for neighborhoods and bounded paths |
What schema means
Schema is the agreement about fields, types, identity, and constraints. CoreCortex uses four related schemas and keeps their responsibilities separate.
| Schema | Purpose |
|---|---|
| Configuration schema | Legal corec.yaml structure; generated by corec schema |
| Source schema | Observed input fields, types, nulls, cardinality, and lengths |
| Canonical record schema | Stable identity, title, content, typed attributes, access, and provenance |
| Physical index schema | Derived backend columns, vector dimensions, analyzers, and indexes |
A concrete searchable projection
The projection supports three field roles. The plan chooses two candidate generators, fuses their ranks, and returns ten authorized evidence items. Sparse-only and dense-only plans can reuse the same projection during evaluation.
indexes:
- name: support
backend:
type: turbopuffer
connectionSecret: secret://corec/turbopuffer
fields:
lexical: [title, content]
vector:
- name: content_embedding
dimensions: 384
filter: [tenant_id, dataset_id, category, plan]
retrievalPlans:
- name: support-hybrid
index: support
candidates:
- { type: bm25, field: content, limit: 50 }
- { type: vector, field: content_embedding, limit: 50 }
fusion: { type: rrf, rankConstant: 60 }
limit: 10