# Under the hood

> The engineering of the analytics front end: how a result set becomes a table you can scroll, a chart you can click, and a screen that filters itself

Source: https://aipril.dev/docs/analytics/under-the-hood

This page is about the analytics surface: the exploration table, the widgets, the dashboard and
what leaves it. Where the data comes from, what executes the query and where access is applied is a
separate plane with its own documentation - the [Data engine](https://aipril.dev/docs/engine.md). Everything below starts
from the moment a result set arrives.

```mermaid
flowchart TD
    Q["Result set from the engine"] --> CACHE["Client cache<br/>short for live data,<br/>longer for reference"]
    CACHE --> TABLE["Exploration table<br/>paged fetch, windowed render"]
    CACHE --> DISPATCH["Chart dispatcher<br/>one place that knows what draws what"]
    DISPATCH --> SVG["Own SVG components<br/>d3 as geometry primitives"]
    DISPATCH --> HEAVY["Heavy library, lazily<br/>map and custom-chart lab only"]
    TABLE --> BUS["In-page event bus"]
    SVG --> BUS
    BUS --> GRID["Dashboard grid<br/>every widget re-filters"]
    GRID --> OUT["Excel, image, PDF"]
```

## The table never holds the table

The exploration table fetches in pages and renders a window:

- Rows arrive a page at a time, with the total count coming from the server, so the scrollbar is
  honest about a dataset that has not been fetched.
- Only the visible window is in the DOM, and rows outside it are released instead of accumulating,
  which is what stops a long session of scrolling from turning into a memory problem.
- Scroll handling is throttled to the frame rate. Loading is frozen while a page is in flight and
  the scroll position is locked through the cycle, so the ground does not move under the reader.
- The new page is swapped in atomically once it is ready, against stable row keys, instead of being
  appended row by row.
- Fast flicks are treated differently from slow scrolling: past a velocity threshold there is no
  point rendering everything that flies past.

Results are cached in the browser per query, with two lifetimes because two kinds of data are being
fetched: brief for live dashboard and render-path data, considerably longer for lists that rarely
change and for model definitions. An administrator configures both at runtime; neither is compiled
in.

## Calculations that belong to the front end

Not every calculation belongs in the same place, and the difference is not a matter of taste.

| Level | What runs there | Why there |
|---|---|---|
| **The model** | Metrics and dimensions of the [semantic layer](https://aipril.dev/docs/engine/data-model.md) | Defined once for the whole company, computed next to the data |
| **Pushed down** | A table-level formula that turns out to be expressible server-side | Written where you are working, executed where the data is |
| **The browser** | Formulas that need the rows in front of you | Some questions are *about the current view*, and no server can answer them without being told what the view is |

A calculated field is inspected before it runs: if it uses nothing but references to metrics, the
column total, arithmetic and a conditional, it is translated into the engine's derived-metric form
and computed server-side. If it reaches into the rows themselves - the value of the row above, a
rank, a running total, a moving average - it stays in the browser. The line is drawn by what the
expression needs, not by where it was typed. A family of calendar functions goes the other way on
purpose: working days and holidays are evaluated by the store, because a production calendar has to
mean the same thing for everyone.

Local evaluation is not a fallback. Rank within the current filter, share of the visible total, the
difference from the row above - these change with every click, so computing them server-side means a
round trip per click. And an edited formula is re-evaluated over rows already in memory, which is
what makes tuning one cost no query at all.

:::details[The formula language and its sandbox]

Columns are referenced by the label a human reads, not by a database identifier: `{{Order total}}`
is the field on this row, `{{Cell}}` the current cell, `TOTAL(field)` the column's total,
`prevRow[field]` the row above. The braces are the templating convention people already know from
elsewhere; the semantics are a spreadsheet's - structured references to columns, in the language of
the business rather than of the schema. Field names work in either interface language.

Evaluation is sandboxed and narrow by construction: only the current cell, the row, and the maths
and string helpers are in scope; a blocked-pattern list closes prototype-chain escapes and access to
the page; loops and arrow functions are refused outright, which removes the infinite-loop denial of
service along with them; and a formula gets 100 ms. Fields are typed, so a calculated column reaches
the chart pipeline as a metric or a dimension rather than as an untyped string.

Most people never write any of it. Formulas are offered as templates grouped by purpose: pick
**moving average**, point it at a column, say over how many rows, and the expression is written for
you - likewise a running total, change against the previous row, year-on-year in absolute terms and
in percent, share of total, rank, or "count this only if it is a working day". The function
catalogue behind them carries a signature and a one-line description for each entry, and marks
whether it is evaluated in the browser or by the store.

:::

## Drawing

Most charts are our own SVG components, with d3 as geometry and scale primitives, not as a renderer. The heavy chart library is loaded lazily and only under the map and the custom-chart lab,
so a reader who never opens those does not download it.

Chart-type dispatch lives in exactly one place. Builders return data structures and components draw
them, which keeps "what to show" separate from "how to draw it": a new chart type is a new builder
plus one line in the dispatcher, not an edit to every surface that shows charts. Drill-down on click
lives inside the chart component instead of the wrapper around it, so it behaves the same wherever
the chart is mounted.

The same dispatcher renders charts inside portal pages. A chart on a portal and a chart on a
dashboard are one piece of code, not two that look alike, so a fix to one is not a fix owed to the
other.

## The dashboard

Widgets sit on a grid they can be moved and resized on, and copied between dashboards. What holds
them together is an in-page event bus: a click on one widget reaches the others without a round trip
and without a page reload. A multi-select gathers the picks behind a pending-selection step and
fires one query instead of one per click: the difference between a filter that feels immediate and
one that hammers the engine while somebody makes up their mind.

The wiring is shared, not re-implemented per widget type, which is what keeps cross-filtering
behaving identically on a chart, a table and a task board.

## What leaves the screen

- **Excel.** The table exports as a real workbook, not a comma-separated approximation:
  formatting, number types and column widths survive, because the recipient is going to open it in
  Excel and keep working.
- **An image of a dashboard.** The grid is captured as it stands, at a scale computed from its own
  width, with the capture library loaded only when somebody actually exports.
- **PDF.** The captured canvas is wrapped into a single-page document by hand, a minimal PDF with
  the image embedded, instead of pulling in a PDF library for one page. The dependency it saves is
  not small, and the output is deterministic.

## Where the data comes from

Everything on this page starts at a result set. How that result set is produced - sources, the data
lake, live versus extract, the semantic model, the join traps, and where row-level access is
applied - is the [Data engine](https://aipril.dev/docs/engine.md), and its own engineering is on
[Under the hood](https://aipril.dev/docs/engine/under-the-hood.md).