the-arrow-ecosystem

A self-hosted lab for the Apache Arrow ecosystem: the claim that the same columnar in-memory layout can be read by multiple engines and languages without a conversion at each hop, run end to end against a real stack rather than described in the abstract.

Companion implementation to a written series covering the same stack, layer by layer - format → persistence → wire protocol → client API → query engine, with object storage as the foundation every other layer reads from.

Why this exists

If you've mostly worked in pandas or a single query engine, Arrow can look like an implementation detail you don't need to know about - and often it is one, until you're moving data between a query engine, a DataFrame library, and a training pipeline, and each hop is doing its own serialization whether you asked it to or not. Arrow is a columnar memory format that several engines and libraries agree on, so a table read by one can be handed to another without that conversion - provided both sides actually implement it the same way.

Architecture

Two things move through this stack:

• Data plane (solid arrows, teal) - the runtime query path: notebook/agent → ADBC → driver manager → Arrow Flight SQL → query engine → object storage.
• Control plane / optional layers (dashed, gray) - things that act once at install time (the ADBC Driver Foundry and the dbc CLI), or that are optional additions (a DBAPI compatibility shim; Apache Iceberg as a table-format layer inserted between the engine and storage when catalog/ACID guarantees are needed).

The written series covers the reasoning behind this diagram in more depth - including why ClickHouse fills the query-engine slot in this repo and how storage stays the source of truth throughout.

Repo layout

the-arrow-ecosystem/
├── assets/          diagrams and images referenced in this README
├── clickhouse/       Flight SQL-enabled ClickHouse config
├── data/             dataset seed script - agent-trace Parquet → object storage
├── notebooks/        containerized Jupyter notebook client (ADBC, Flight, DBAPI, dbc CLI, Examples)
└── docker-compose.yml

Status

Built step by step, alongside the written series. Current state: object storage, the Flight SQL-speaking query engine, and the notebook client are all up and verified working end to end against each other.

• Object storage (MinIO) + dataset seed (Exgentic/agent-llm-traces, pulled as-is, no reshaping) - verified loaded via uv run python data/seed.py, visible in the MinIO console
• ClickHouse, Flight SQL interface enabled, reading Parquet via its S3 table function only - never ingested into MergeTree - [x] ClickHouse added to docker-compose.yml, reads the bucket via s3(...) - [x] Arrow Flight SQL interface enabled (clickhouse/config.d/arrow-flight.xml), confirmed on clickhouse/clickhouse-server:26.8
• Containerized Jupyter notebook client: ADBC via the generic flightsql driver, a DBAPI-wrapped comparison, and dbc CLI driver installation run live from the notebook (notebooks/01_driver_foundry.ipynb) - verified end to end
• Follow-up (separate piece): swap in a DataFusion-based Flight SQL server; add an Iceberg REST catalog over the same bucket

Dataset

Exgentic/agent-llm-traces - 1,781 OpenTelemetry agent execution traces across 6 benchmarks, Parquet-native, CDLA-Permissive-2.0. Its nested list<struct<...>> shape (conversation turns, tool calls, typed timestamps) exercises Arrow's nested type support directly.

Running this

cp .env.example .env
uv sync

docker compose up -d minio
uv run python data/seed.py

docker compose up -d clickhouse
docker compose up -d --build jupyter

seed.py downloads each Parquet file straight from the Hugging Face Hub and uploads it to the agent-traces bucket under the agent-llm-traces/data/ prefix.

Open http://localhost:8888/lab (no auth - local lab only), then notebooks/01_driver_foundry.ipynb. Run top to bottom: dbc installs the Flight SQL driver live, ADBC connects and queries ClickHouse's s3() view of the bucket, a DBAPI cursor does the same query for comparison, and a final SHOW TABLES confirms nothing landed in MergeTree - the bucket stayed the only copy of the data throughout.

Notebook 2 (notebooks/02_arrow_in_practice.ipynb) takes the Arrow table notebook 1 pulled over Flight SQL and runs it through an ordinary data-prep-and-EDA sequence:

1. Setup - re-fetches the table standalone (doesn't assume notebook 1's kernel state).
2. Schema discovery - picks a nested column (spans) and numeric columns programmatically from the real schema.
3. Native Arrow compute - derives a span_count feature with pyarrow.compute.list_value_length() directly on the in-memory buffers, no engine involved.
4. DuckDB - same table queried with SQL via DuckDB's replacement scan (no export/reload step).
5. DataFusion - same table queried again via register_record_batches(), a second engine reading the same buffers.
6. Schema/type-fidelity check - compares the nested column's Arrow type before and after both engines; DuckDB's round-trip changes some schema metadata (list field naming, nullability) even though the data itself is untouched.
7. pandas bridge - plain to_pandas() (falls back to NumPy/object) side-by-side with pandas 2's to_pandas(types_mapper=pd.ArrowDtype) (stays Arrow-backed).
8. Plot - one matplotlib chart off the pandas frame, showing where plotting tools still expect a DataFrame.
9. scikit-learn bridge - .to_numpy() conversion feeding a KMeans clustering on span_count, labeled as a real conversion.
10. PyTorch bridge - a small Dataset wrapping the Arrow column, converting one row to a tensor per __getitem__ call, fed through a DataLoader.
11. Closing table - a plain summary of which steps needed a conversion and which didn't.

The throughline: the same table can be queried by three different engines and read into pandas without conversion, and the only places anything actually had to be converted were the two points where the data crosses into row-oriented tooling.

License

Code in this repo: MIT (see LICENSE). The dataset used is separately licensed - see the dataset card linked above.