# Data scanning and retrieval

> A seven-step scan that turns a dataset into query patterns and business rules, four layers of stored knowledge including what agents learn in use, and hybrid retrieval fused by reciprocal rank

Source: https://aipril.dev/docs/harness/knowledge

## Why scanning exists

A model handed a schema infers meaning from column names and invents joins. The failure is quiet: the
query runs, returns rows, and is wrong.

So a dataset is not given to an agent as a schema. It is scanned first, and what agents work from is
the scan output - query shapes known to work against that data, and rules known to hold in it.

## The scan pipeline

Seven LLM steps, run when a dataset arrives and re-runnable on demand from the dataset card or
through a tool:

| Step | Produces |
|---|---|
| 1. Schema extraction | Columns, types, cardinality, what is actually populated |
| 2. Domain classification | Which business domain this is - which also routes questions to the right agent |
| 3. Summary and use cases | What the dataset is, and the questions it can answer |
| 4. Semantic-layer enrichment suggestions | Candidate metrics and dimensions, proposed to a person and not applied |
| 5. SQL patterns | The query shapes that work against this data - the joins, the grains, the filters |
| 6. Business rules | Constraints that hold and are not expressible in column types |
| 7. Vector embeddings | 1536 dimensions, batched |

Steps 5 and 6 carry the weight.

**A query pattern** is a working shape for this dataset, not a template string: which table joins to
which on what key, what the natural grain is, which date column means "when it happened" as opposed
to "when it was loaded", which filter has to be present for a total to be correct.

**A business rule** is a fact no column type expresses: an amount is always net of tax; a status
never moves backwards; this code identifies a customer, not an order; rows with a null close date are
open, not erroneous.

Both are what stop an agent from generating a plausible query. It composes from what is known to
work.

## Four layers

Output is stored as four distinct kinds, not one flattened index:

| Layer | Answers |
|---|---|
| Dataset context | What this data is and what it is for |
| Query patterns | How it has been queried successfully |
| Business rules | What is true of it that the schema does not say |
| Agent learnings | What has been established in use since |

The fourth layer is the one that compounds. A correction made once - the right grain, the filter that
was missing, the column that turned out to mean something else - is written back and retrieved next
time, so the same question is not re-derived from scratch.

Separating the layers is a retrieval-precision decision: "what does this table mean" and "how has
this been queried" are different targets, and a single undifferentiated index answers both worse.

There is also **per-user memory**, held separately from dataset knowledge: preferences and facts about
how a particular person works, rather than about the data.

## Hybrid retrieval

Three retrievers with different failure modes run over that store:

| Retriever | Weight | Finds | Misses |
|---|---|---|---|
| Vector | ~60% | Meaning, paraphrase | Exact identifiers |
| Full-text | ~30% | Exact terms, codes | Paraphrase |
| Trigram | ~10% | Typos, transliteration, near-miss names | Semantics |

Results are combined by **reciprocal rank fusion** with a rank constant of 60, not by summing scores.
Scores from a vector index and a text index are not on a comparable scale; ranks are.

**Optional reranking.** A retrieval can be reranked by an LLM judging relevance, opt-in per call,
since it costs a model call and pays off on hard questions and not on routine ones.

**No silent degradation.** Without a configured embedding model, dataset scanning and contextual
search report themselves unavailable instead of falling back to text-only retrieval.

## How it reaches the model

Retrieved context is one bounded component of the assembled turn, alongside the tool schemas in scope
and the trimmed conversation history - typically 0.5-1K tokens, up to 2-3K when a large template is
pulled in, with a retrieval limit and not an open-ended fetch. See
[Models and providers](https://aipril.dev/docs/harness/models-and-providers.md).

A person can also aim it explicitly by mentioning a dataset by name in the message, which puts that
dataset's context in scope directly.

## Evaluation

- **Golden runs** with a tool-accuracy scorer awarding partial credit - the right tool with two of
  three arguments correct is a different result from the wrong tool.
- **Content-quality scoring** - LLM-judged faithfulness and context relevance - run by a separate
  driver against the live search tool and the live chat endpoint on natural-language cases, not
  against mocks.

## The business view

For what this means without the mechanism - why agents answer about your data rather than about data
in general - see [Agents that act, not guess](https://aipril.dev/docs/harness/agents.md) and
[Company knowledge base](https://aipril.dev/docs/overview/knowledge-base.md).