Analytics

Under the hood

The engineering of the analytics front end: how a result set becomes a table you can scroll, a chart you can click, and a screen that filters itself

This page is about the analytics surface: the exploration table, the widgets, the dashboard and what leaves it. Where the data comes from, what executes the query and where access is applied is a separate plane with its own documentation - the Data engine. Everything below starts from the moment a result set arrives.

The table never holds the table

The exploration table fetches in pages and renders a window:

  • Rows arrive a page at a time, with the total count coming from the server, so the scrollbar is honest about a dataset that has not been fetched.
  • Only the visible window is in the DOM, and rows outside it are released instead of accumulating, which is what stops a long session of scrolling from turning into a memory problem.
  • Scroll handling is throttled to the frame rate. Loading is frozen while a page is in flight and the scroll position is locked through the cycle, so the ground does not move under the reader.
  • The new page is swapped in atomically once it is ready, against stable row keys, instead of being appended row by row.
  • Fast flicks are treated differently from slow scrolling: past a velocity threshold there is no point rendering everything that flies past.

Results are cached in the browser per query, with two lifetimes because two kinds of data are being fetched: brief for live dashboard and render-path data, considerably longer for lists that rarely change and for model definitions. An administrator configures both at runtime; neither is compiled in.

Calculations that belong to the front end

Not every calculation belongs in the same place, and the difference is not a matter of taste.

Level What runs there Why there
The model Metrics and dimensions of the semantic layer Defined once for the whole company, computed next to the data
Pushed down A table-level formula that turns out to be expressible server-side Written where you are working, executed where the data is
The browser Formulas that need the rows in front of you Some questions are about the current view, and no server can answer them without being told what the view is

A calculated field is inspected before it runs: if it uses nothing but references to metrics, the column total, arithmetic and a conditional, it is translated into the engine's derived-metric form and computed server-side. If it reaches into the rows themselves - the value of the row above, a rank, a running total, a moving average - it stays in the browser. The line is drawn by what the expression needs, not by where it was typed. A family of calendar functions goes the other way on purpose: working days and holidays are evaluated by the store, because a production calendar has to mean the same thing for everyone.

Local evaluation is not a fallback. Rank within the current filter, share of the visible total, the difference from the row above - these change with every click, so computing them server-side means a round trip per click. And an edited formula is re-evaluated over rows already in memory, which is what makes tuning one cost no query at all.

The formula language and its sandbox

Columns are referenced by the label a human reads, not by a database identifier: {{Order total}} is the field on this row, {{Cell}} the current cell, TOTAL(field) the column's total, prevRow[field] the row above. The braces are the templating convention people already know from elsewhere; the semantics are a spreadsheet's - structured references to columns, in the language of the business rather than of the schema. Field names work in either interface language.

Evaluation is sandboxed and narrow by construction: only the current cell, the row, and the maths and string helpers are in scope; a blocked-pattern list closes prototype-chain escapes and access to the page; loops and arrow functions are refused outright, which removes the infinite-loop denial of service along with them; and a formula gets 100 ms. Fields are typed, so a calculated column reaches the chart pipeline as a metric or a dimension rather than as an untyped string.

Most people never write any of it. Formulas are offered as templates grouped by purpose: pick moving average, point it at a column, say over how many rows, and the expression is written for you - likewise a running total, change against the previous row, year-on-year in absolute terms and in percent, share of total, rank, or "count this only if it is a working day". The function catalogue behind them carries a signature and a one-line description for each entry, and marks whether it is evaluated in the browser or by the store.

Drawing

Most charts are our own SVG components, with d3 as geometry and scale primitives, not as a renderer. The heavy chart library is loaded lazily and only under the map and the custom-chart lab, so a reader who never opens those does not download it.

Chart-type dispatch lives in exactly one place. Builders return data structures and components draw them, which keeps "what to show" separate from "how to draw it": a new chart type is a new builder plus one line in the dispatcher, not an edit to every surface that shows charts. Drill-down on click lives inside the chart component instead of the wrapper around it, so it behaves the same wherever the chart is mounted.

The same dispatcher renders charts inside portal pages. A chart on a portal and a chart on a dashboard are one piece of code, not two that look alike, so a fix to one is not a fix owed to the other.

The dashboard

Widgets sit on a grid they can be moved and resized on, and copied between dashboards. What holds them together is an in-page event bus: a click on one widget reaches the others without a round trip and without a page reload. A multi-select gathers the picks behind a pending-selection step and fires one query instead of one per click: the difference between a filter that feels immediate and one that hammers the engine while somebody makes up their mind.

The wiring is shared, not re-implemented per widget type, which is what keeps cross-filtering behaving identically on a chart, a table and a task board.

What leaves the screen

  • Excel. The table exports as a real workbook, not a comma-separated approximation: formatting, number types and column widths survive, because the recipient is going to open it in Excel and keep working.
  • An image of a dashboard. The grid is captured as it stands, at a scale computed from its own width, with the capture library loaded only when somebody actually exports.
  • PDF. The captured canvas is wrapped into a single-page document by hand, a minimal PDF with the image embedded, instead of pulling in a PDF library for one page. The dependency it saves is not small, and the output is deterministic.

Where the data comes from

Everything on this page starts at a result set. How that result set is produced - sources, the data lake, live versus extract, the semantic model, the join traps, and where row-level access is applied - is the Data engine, and its own engineering is on Under the hood.