Skip to content

dosi-engine vs MetricFlow: performance benchmark

Apples-to-apples comparison of the dosi CLI against the legacy MetricFlow engine. The numbers below are from the 2026-07-11 run.

Methodology

Both engines process the same semantic model (MetricFlow's own simple_model test model, converted 1:1 to Apache Ossie) and the same queries against the same seeded DuckDB database. The queries are the ones CI already checks for identical result sets between the two engines, so the benchmark measures the speed of equivalent work.

measurement dosi-engine MetricFlow
cold CLI one dosi query process: spawn → load model → compile → print SQL one python subprocess: interpreter start + imports + model parse + MetricFlowClient.explain() — the same work mf query --explain does, minus click/config overhead (a bound favorable to MetricFlow)
warm API same full-process invocation (osi has no daemon mode; the bare process-spawn floor is reported separately) in-process explain() loop after a warmup call — interpreter, imports, and model parse all already paid
execute dosi query --execute --db <file> (cold, full process) in-process query() loop (warm) and cold subprocess
peak RSS max resident set size of the cold subprocess (wait4 rusage) same

Cold runs are interleaved A/B, 5 timed runs after 1 discarded warmup; warm loops are 30 iterations. Medians reported.

Environment

  • 2026-07-11, dosi-engine f850dfd, cargo build --release
  • AWS EC2, 4 vCPU Intel Xeon Platinum 8175M @ 2.50GHz, 15 GiB RAM, Linux (al2023)
  • MetricFlow: Datus fork, editable install, CPython 3.12.12 (uv venv)
  • DuckDB CLI v1.4.4, seeded with the simple_model test data

Results

osi process-spawn floor (dosi --version): 2.0 ms median.

SQL compilation — cold CLI (process spawn → SQL printed)

query dosi MetricFlow speedup
simple_agg_by_dim (bookings ⟨is_instant⟩) 10.6 ms 2,321.9 ms 220x
three_metrics_no_dim 10.1 ms 2,403.5 ms 237x
ratio_metric (booking_fees_per_booker) 10.4 ms 2,309.2 ms 222x
expr_metric (views_times_booking_value) 10.4 ms 2,338.8 ms 226x
simple_agg_by_time (bookings ⟨ds⟩) 10.5 ms 2,289.9 ms 219x

MetricFlow cold-start breakdown: ~1,004 ms imports + ~742 ms model parse + ~161 ms client init before any query compiles — ~1.9 s of fixed overhead per CLI invocation. dosi pays ~2 ms of process spawn and parses the model in the remaining ~8 ms.

SQL compilation — warm API (everything resident)

query dosi (full process) MetricFlow (in-process) speedup
simple_agg_by_dim 10.6 ms 107.6 ms 10.2x
three_metrics_no_dim 10.1 ms 218.9 ms 21.6x
ratio_metric 10.4 ms 105.3 ms 10.1x
expr_metric 10.4 ms 205.0 ms 19.8x
simple_agg_by_time 10.5 ms 111.4 ms 10.6x

This is the most generous comparison for MetricFlow — its number is pure in-process compile time with all imports and model parsing already paid, while dosi still re-reads and re-compiles everything in a fresh process. dosi remains 10–22x faster.

End-to-end execution on DuckDB (compile + run + print rows)

query dosi cold MF cold speedup MF warm (in-process) vs dosi cold
simple_agg_by_dim 34.2 ms 2,321.1 ms 68x 127.0 ms 3.7x
three_metrics_no_dim 36.2 ms 2,428.1 ms 67x 233.1 ms 6.4x
ratio_metric 35.7 ms 2,368.8 ms 66x 122.7 ms 3.4x
expr_metric 33.5 ms 2,441.2 ms 73x 235.9 ms 7.0x
simple_agg_by_time 32.9 ms 2,309.1 ms 70x 132.8 ms 4.0x

With execution, DuckDB query time (~20 ms via the CLI shell-out, identical work for both engines) becomes a shared floor; osi's end-to-end cold run is still 3–7x faster than MetricFlow's warm in-process path.

Peak memory (cold process, max RSS)

mode dosi MetricFlow
compile ~15.5 MB ~157 MB
execute ~26.5 MB ~162 MB

Caveats

  • simple_model is small (~25 datasets/metrics). MetricFlow's ~742 ms model parse scales with model size; osi's full 10 ms budget includes its parse.
  • The MetricFlow "cold" path skips the real mf CLI's click startup and Datus config resolution, so real mf query invocations are somewhat slower than measured here.
  • Group-by spelling differs for the time-dimension case (ds for MF, bookings_source.ds for osi) — same semantic query, engine-native syntax.
  • Numbers are single-machine (shared EC2 box); treat relative ratios, not absolute times, as the finding.

Arrow IPC vs JSON: result serialization

How much the columnar egress path (POST /v1/query/execute with Accept: application/vnd.apache.arrow.stream, see rest-api.md) saves over the default JSON body for the same query.

The benchmark generates a high-cardinality synthetic DuckDB dataset (5M-row fact table, 200k distinct user_id) and a matching Ossie model, launches a release dosi-server against them, and runs each result set through both Accept formats.

Methodology

One running server, one query, two Accept headers — the only thing that changes is the response encoding. For each format we measure three things:

measurement what it captures
wire MB bytes of the HTTP response body (median)
e2e ms wall time from request send to full body received — query exec + server serialize + transfer (median)
parse ms client-side cost to turn the raw body into a usable structure: json.loads vs pyarrow.ipc.open_stream(...).read_all() (median)

Query execution over the 5M-row table is identical work on both paths, so it forms a shared floor in e2e ms (isolated by the SMALL row, where the payload is tiny and serialization is negligible). wire MB and parse ms have no shared floor — they are pure format cost. Three result sizes show the advantage scaling with the result, not the scan. 2 warmup + 7 timed runs per cell, medians reported.

Environment

  • 2026-07-14, dosi-engine 2e23173 (working tree), cargo build --release
  • AWS EC2, 4 vCPU Intel Xeon Platinum 8259CL @ 2.50GHz, 15 GiB RAM, Linux (al2023)
  • DuckDB CLI v1.4.4; client CPython 3.9 with pyarrow 21.0.0

Results

result set format rows wire MB e2e ms parse ms
SMALL — 365 rows × 3 cols JSON 365 0.03 1483.5 0.3
Arrow 365 0.01 1463.1 0.2
LARGE — 200k rows × 3 cols JSON 200,000 10.63 2051.8 198.4
Arrow 200,000 6.50 1684.0 1.4
WIDE — 200k rows × 4 cols JSON 200,000 13.99 2749.3 242.7
Arrow 200,000 8.90 1707.9 2.0

Reading the large result sets (Arrow vs JSON):

LARGE (200k × 3) WIDE (200k × 4)
wire — bytes on the network 1.6x smaller 1.6x smaller
parse — client decode ~140x faster (1.4 vs 198 ms) ~120x faster (2.0 vs 243 ms)
e2e — including the shared exec floor 1.2x faster 1.6x faster
e2e minus the ~1.46 s exec floor (serialize + transfer + parse only) ~2.7x faster (221 vs 589 ms) ~5.2x faster (245 vs 1286 ms)

Interpretation

  • Parse is the headline. JSON makes the client re-parse every value out of a string; Arrow hands over typed column buffers with essentially zero decode (1–2 ms for 200k rows vs 200–240 ms). This gap does not shrink in a release build — it is the client's, and it is structural.
  • Serialization is the server-side win. Once the exec floor is removed, Arrow spends ~2.7–5.2x less time on the format itself: no per-row materialization and no per-value JSON string encoding — record batches stream straight from the executor through the IPC writer into the body. The wider the row, the bigger the gap (WIDE > LARGE).
  • Benefit scales with result size. At 365 rows there is no measurable difference — the payload is a rounding error against query execution. The columnar path is worth opting into precisely when the result is large.
  • Wire size is a steady ~1.6x. Real deployments can layer HTTP compression on top; the parse and serialize wins are independent of it.

Caveats

  • Single-machine, loopback transport (no real network RTT or bandwidth limit), so wire MB understates the transfer-time advantage a remote client would see. Treat relative ratios as the finding.
  • The synthetic dataset is numeric/low-width-string; JSON's disadvantage grows with wider rows and more string columns, shrinks for a few wide numeric columns.
  • e2e ms carries the DuckDB scan of all 5M rows on every request (no result cache); the isolated "minus exec floor" row is the cleaner read of format cost, but the floor is only approximately constant across group-by cardinalities.