Indexes and schema
Start with the questions and data. Then choose keyword, vector, filter, relational, or graph search.
Choose fields for an index
Choose which normalized record and passage fields each index should store. CoreCortex can rebuild that projection from canonical data.
One index can hold tokenized text, embedding vectors, and exact filter fields. Hybrid search combines keyword and vector rankings from the same index.
| Field role | Meaning | Typical use |
|---|---|---|
| Lexical | Tokenized title or content | Exact terms, citations, identifiers, BM25 |
| Vector | Dense embedding with fixed dimensions | Paraphrases and semantic similarity |
| Filter | Exact scalar metadata and access fields | Scope, dates, categories, tenants, ACLs |
The available retrieval types
| Type | What it does | Use it when |
|---|---|---|
| Lexical / BM25 | Ranks token and term matches | Names, codes, citations, or exact language matter |
| Dense vector | Ranks embedding similarity | Questions paraphrase the source |
| Hybrid | Fuses lexical and vector candidates | The workload mixes exact and semantic queries |
| Filtered | Applies exact metadata and authorization | Results must obey scope, dates, policy, or tenant rules |
| Relational | Uses typed columns, joins, and aggregation | The question is structured or analytical |
| Graph-assisted | Expands evidenced relationships or bounded paths | Reviewed multi-hop judgments show a benefit |
Choose a backend after you know the required search features
TurboPuffer, PostgreSQL, Milvus, pgvector, and Ladybug run different kinds of searches. corec.yaml states which features the corpus needs. An adapter checks whether a backend provides them.
Settings such as HNSW, IVFFlat, vector distance, text analyzers, and collection tuning belong to a specific backend.
| Backend | Role in CoreCortex |
|---|---|
| TurboPuffer | Native lexical, vector, filters, and RRF |
| PostgreSQL | Full-text, exact filters, relational queries, optional pgvector |
| Milvus / pgvector | Vector adapters behind the extension contract |
| Ladybug | Embedded graph projection for neighborhoods and bounded paths |
Example: one hybrid search index
This index stores searchable text, a 384-number vector, and exact filter fields. The retrieval plan runs keyword and vector search, combines their rankings with RRF, and returns ten results.
indexes:
- name: reference
backend:
type: turbopuffer
connectionSecret: secret://corec/turbopuffer
fields:
lexical: [title, content]
vector:
- name: content_embedding
dimensions: 384
filter: [tenant_id, dataset_id, category, plan]
retrievalPlans:
- name: reference-hybrid
index: reference
candidates:
- { type: bm25, field: content, limit: 50 }
- { type: vector, field: content_embedding, limit: 50 }
fusion: { type: rrf, rankConstant: 60 }
limit: 10