Arrow result transfer: enabling it, configuration, and what gets faster¶

dosi-engine can move query results as Apache Arrow — columnar end to end for engines whose wire format is already columnar (DuckDB, ClickHouse, StarRocks/Doris, Databricks). This page is the user guide: what to configure, what to build, and which APIs speed up. The research, support matrix, and adoption plan behind it live in design/arrow-adbc.md; connector setup details are in connectors.md.

0. Locked scope (2026-07-12)¶

Arrow paths this round: DuckDB (first priority — the bundled-crate disk constraint is resolved), ClickHouse, StarRocks, Doris, Databricks. Snowflake + BigQuery are one deferred TODO: both ride a future generic exec-adbc channel (their ADBC drivers are maintained by the same org — the ADBC Driver Foundry — as Go .sos; Snowflake's is Stable today, BigQuery's is still Experimental, so the wait is only on BigQuery). Trino/MySQL/TiDB/ Redshift: no Arrow path, dropped — their wire formats are row-oriented, a client-side pivot gains nothing.

1. Do I need to change my configuration?¶

Mostly no. Existing connection files keep working unchanged; the row path stays the default everywhere and behaves identically.

Engine	Config change to get Arrow
DuckDB	none — in-process Arrow becomes the engine's native path automatically
ClickHouse	none — same HTTP profile; the adapter requests `FORMAT ArrowStream` itself
Databricks	none beyond the normal profile (`host`, `warehouse`, `password:` = PAT) — the executor is Arrow-first from day one
StarRocks	add one key: `arrow_flight_port: 9408` (the FE `arrow_flight_port`; needs SR ≥ 3.5.1). Key present = Flight SQL; key absent = MySQL-wire exactly as today
Doris	add one key: `arrow_flight_port: 8070` (the FE `arrow_flight_sql_port`; needs Doris ≥ 2.1.5 recommended). Same opt-in rule

Example:

datasources:
  prod-sr:
    type: starrocks
    host: sr.internal
    port: 9030                # MySQL-wire port, still used as fallback/seeding
    arrow_flight_port: 9408   # presence opts this profile into Arrow Flight SQL
    username: osi
    password: ${SR_PASSWORD}
    database: analytics

There is deliberately no result_format: knob: each adapter has one native decode, and the egress format is chosen per request (Accept header / CLI flag), not per datasource.

2. How do I enable it at build time?¶

Feature	What it turns on	In default build?
`exec-duckdb`	bundled libduckdb, in-process, Arrow-native	yes (new default; `--no-default-features` = today's lean CLI-shell-out build)
`exec-http-arrow`	ClickHouse `FORMAT ArrowStream`	no (implies `exec-http`)
`exec-flightsql`	StarRocks/Doris Arrow Flight SQL client (tonic/tokio, gated)	no
`exec-databricks`	Databricks Statement Execution API, INLINE JSON_ARRAY rows	no
`exec-databricks-arrow`	Databricks `ARROW_STREAM` + `EXTERNAL_LINKS` (arrow-ipc)	no (implies `exec-databricks`)
`exec-all`	all of the above	—
dosi-server `arrow`	Arrow IPC responses on `/v1/query/execute`	yes (server default)

3. Which APIs get faster, and by how much?¶

POST /v1/query/execute with Accept: application/vnd.apache.arrow.stream — the headline win. Batches stream from the warehouse adapter straight into the HTTP body as Arrow IPC: no per-value JSON encoding on the server, no JSON parsing in the client, and columnar consumers (Polars/pandas/DataFusion/ another dosi) read it zero-parse. The JSON response stays byte-identical for existing clients. Benefit scales with result size: negligible for a 10-row aggregate, decisive for high-cardinality group-bys and detail/export queries (10⁵+ rows).
dosi query --execute --format arrow — Arrow IPC on stdout for piping into DuckDB/Polars/anything, skipping the table/JSON rendering entirely.
StarRocks/Doris large reads — Flight SQL reads bypass FE row forwarding (client pulls columnar results from BEs in parallel); vendors measure 20–160x vs the MySQL protocol on large result sets. Small aggregates: ~no change.
ClickHouse — server-side Arrow serialization (lz4-framed) replaces JSON encode/decode on both ends; also removes float/decimal text round-trips.
DuckDB (default engine) — in-process zero-copy replaces spawn-a-CLI-per-query + JSON parse: lower latency on every --execute, plus in-memory databases finally persist across statements in one session.
What does NOT get faster: compile-only APIs (/v1/query/compile, explain) — they never touch results; small metric queries on row-wire engines (MySQL/TiDB/Postgres/Trino) — unchanged by design.

Every Arrow path passes corpus parity before it ships: the same 45 cases, Arrow results ≡ row results ≡ the DuckDB reference.