An implementation and optimization study of the One Billion Row Challenge (1BRC) in Python, focused on parsing, aggregating, and calculating temperature statistics across 1,000,000,000 records (~14 GB text file) on a single machine.
Given a text file containing 1 billion temperature measurements from weather stations across the globe in the format <station_name>;<temperature>, compute the minimum, mean, and maximum temperature per weather station, sorted alphabetically.
Hamburg;12.0
Bulawayo;8.9
Palembang;38.8
St. Johns;15.2
Cracow;12.6
Bridgetown;26.9
- Memory-Mapped Files (
mmap): Direct virtual memory access to avoid transferring multi-gigabyte files completely into RAM. - Parallel Chunk Processing (
multiprocessing): Splitting byte ranges across available CPU cores for concurrent parsing. - Custom Parsing & Type Conversion: Avoiding heavy object overhead by parsing integer and decimal floats directly from byte arrays.
- Optimized Aggregation: High-speed in-memory hash tables combining local process aggregations into a unified final summary.
| Weather Station | Min Temp (°C) | Mean Temp (°C) | Max Temp (°C) |
|---|---|---|---|
| Abha | -31.1 | 18.0 | 66.5 |
| Abidjan | -25.9 | 26.0 | 74.6 |
| Accra | -24.8 | 26.4 | 76.3 |
| Adelaide | -31.8 | 17.3 | 71.5 |
| Hamburg | -42.1 | 9.4 | 64.2 |
| Tokyo | -38.2 | 15.6 | 72.8 |
| Zürich | -42.0 | 9.3 | 63.6 |
- Language: Python 3.10+
- Libraries: Python Standard Library (
multiprocessing,mmap,collections), Pandas / Polars (for comparison benchmarks) - Architecture: Multi-core parallel compute
# Clone the repository
git clone https://github.com/diegobarbosaa/One-Billion-Row-Challenge-Python.git
cd One-Billion-Row-Challenge-Python
# Generate test dataset (optional)
python generate_measurements.py 1000000000
# Execute optimized baseline
python process_1brc.py