Sources
A source is registered once and becomes queryable; nothing is copied as a precondition.
- Relational sources - PostgreSQL, Oracle, Microsoft SQL Server, MySQL, ClickHouse, Snowflake, MongoDB and others. They are queried in place through the engine's own compiler, which renders the dialect of the database it is talking to. The list is not closed: connecting one is a connection setting, not a connector commissioned for it.
- REST services are a first-class relation type, not a special case: register the endpoint, or import a Swagger or Postman description and get the relations out of it. Outbound calls are SSRF-checked and capped on pagination. A model mixing SQL and HTTP relations can be snapshotted instead of joined at query time.
- Files arrive by upload, or are collected on a schedule from S3-compatible object storage, an SFTP server, an SMB network share, a local folder beside the installation for closed networks, or a spreadsheet somebody maintains by hand.
- The platform's own data needs no connector at all. A published process model is synced to the engine as a live relation, so tasks, statuses and their custom fields become an ordinary queryable model. Ledger movements arrive through a dataset bridge.
The built-in data lake
The platform has storage of its own, and it is a lake, not a warehouse: it writes data as Parquet into object storage and queries it where it lies, with no proprietary format in between and no schema to declare before the first file lands. It is part of the installation, not a second product to procure.
Three ways in
- A person loads a file - the path for data that never lived in a system: a plan, a price list, a file from a partner.
- Files collected on a schedule from any of the file sources above.
- An extract from a database or a service - the query runs against the source on a schedule and its result is kept in the lake. A snapshot of a computation, not a copy of the source.
Which fork you are on
Live is not the superior mode with extract as its compromise. They answer different questions:
| Extract earns its place when | Live is the right call when |
|---|---|
| The source is a production database that should not carry analytical load beside its real work | The answer has to reflect what happened a minute ago - a stock level, a queue, a shift in progress |
| The source is slow, or across a link that makes interactive drilling painful | The data changes faster than any schedule you would set |
| The figures must hold still: a snapshot does not shift underneath a review while it is being read | The volume or the policy makes keeping a second copy the wrong trade |
| The same expensive query is asked by many people, and computing it once is the whole point | The source is already an analytical store built for exactly this |
| The data has to be joined with files that were never in a database at all |
Because both modes run the same semantic model, the choice is reversible: moving a report between them is a setting, not a rewrite, and the two paths are checked against each other in CI - see Under the hood.
Freshness
How often each source is collected and its figures recalculated is a setting per dataset, from minutes to overnight, with an on-demand refresh for the times it matters. Every dataset carries when it was last updated, so "is this current?" is answered on the screen instead of by asking around.
That timestamp does more than inform: cached results are keyed by it, so a refresh invalidates the cache by construction instead of waiting out a timer.
What the lake enforces
Ingestion is metered at two levels: per organisation - dataset count, storage size, jobs per day - and per bucket, which is also the unit of access. An upload token is valid only inside its own bucket, so "upload a file" and "read that file" stay two independent checks, not one.
Related
- Data model - what is built on top of what lands here.
- Under the hood - how a query reaches these two paths, and where access is applied.
- Connectors - the registry the source connections come from.
- Analytics - what the numbers get used for once they are here.