# Sources and storage

> What can be connected, when a question is answered at the source and when from the platform's own data lake, and how freshness is decided

Source: https://aipril.dev/docs/engine/sources-and-storage

## Sources

A source is registered once and becomes queryable; nothing is copied as a precondition.

- **Relational sources** - PostgreSQL, Oracle, Microsoft SQL Server, MySQL, ClickHouse, Snowflake,
  MongoDB and others. They are queried in place through the engine's own compiler, which renders
  the dialect of the database it is talking to. The list is not closed: connecting one is a
  connection setting, not a connector commissioned for it.
- **REST services** are a first-class relation type, not a special case: register the endpoint, or
  import a Swagger or Postman description and get the relations out of it. Outbound calls are
  SSRF-checked and capped on pagination. A model mixing SQL and HTTP relations can be snapshotted
  instead of joined at query time.
- **Files** arrive by upload, or are collected on a schedule from S3-compatible object storage, an
  SFTP server, an SMB network share, a local folder beside the installation for closed networks, or
  a spreadsheet somebody maintains by hand.
- **The platform's own data** needs no connector at all. A published process model is synced to the
  engine as a live relation, so tasks, statuses and their custom fields become an ordinary queryable
  model. Ledger movements arrive through a dataset bridge.

## The built-in data lake

The platform has storage of its own, and it is a lake, not a warehouse: it writes data as Parquet
into object storage and queries it where it lies, with no proprietary format in between and no
schema to declare before the first file lands. It is part of the installation, not a second product
to procure.

### Three ways in

- **A person loads a file** - the path for data that never lived in a system: a plan, a price list,
  a file from a partner.
- **Files collected on a schedule** from any of the file sources above.
- **An extract from a database or a service** - the query runs against the source on a schedule and
  its *result* is kept in the lake. A snapshot of a computation, not a copy of the source.

### Which fork you are on

```mermaid
flowchart TD
    Q["Where does the data live?"]
    Q -->|"a database or a service"| D{"Must the answer be<br/>current to the second?"}
    Q -->|"files: upload, folder,<br/>S3, SFTP, a sheet"| L
    D -->|"yes"| LIVE["Live - the query runs<br/>against the source"]
    D -->|"no, or it must not<br/>carry the load"| L["The data lake - Parquet in<br/>object storage, refreshed<br/>on a schedule"]
    LIVE --> M["One semantic model,<br/>one query contract"]
    L --> M
```

Live is not the superior mode with extract as its compromise. They answer different questions:

| Extract earns its place when | Live is the right call when |
|---|---|
| The source is a production database that should not carry analytical load beside its real work | The answer has to reflect what happened a minute ago - a stock level, a queue, a shift in progress |
| The source is slow, or across a link that makes interactive drilling painful | The data changes faster than any schedule you would set |
| The figures must hold still: a snapshot does not shift underneath a review while it is being read | The volume or the policy makes keeping a second copy the wrong trade |
| The same expensive query is asked by many people, and computing it once is the whole point | The source is already an analytical store built for exactly this |
| The data has to be joined with files that were never in a database at all | |

Because both modes run the same semantic model, the choice is reversible: moving a report between
them is a setting, not a rewrite, and the two paths are checked against each other in CI - see
[Under the hood](https://aipril.dev/docs/engine/under-the-hood.md).

## Freshness

How often each source is collected and its figures recalculated is a setting per dataset, from
minutes to overnight, with an on-demand refresh for the times it matters. Every dataset carries when
it was last updated, so "is this current?" is answered on the screen instead of by asking around.

That timestamp does more than inform: cached results are keyed by it, so a refresh invalidates the
cache by construction instead of waiting out a timer.

## What the lake enforces

Ingestion is metered at two levels: per organisation - dataset count, storage size, jobs per day -
and per bucket, which is also the unit of access. An upload token is valid only inside its own
bucket, so "upload a file" and "read that file" stay two independent checks, not one.

## Related

- [Data model](https://aipril.dev/docs/engine/data-model.md) - what is built on top of what lands here.
- [Under the hood](https://aipril.dev/docs/engine/under-the-hood.md) - how a query reaches these two paths, and where
  access is applied.
- [Connectors](https://aipril.dev/docs/platform/integrations/connectors.md) - the registry the source connections come
  from.
- [Analytics](https://aipril.dev/docs/analytics.md) - what the numbers get used for once they are here.