Replace aiobotocore with Native Async Python Bindings to the AWS SDK for C++

[thehesiod]

(note had AI write a bunch of this g)

aiobotocore enables asyncio support by layering async I/O on top of botocore, which is a synchronous, low-level AWS SDK designed for boto3. It does this by patching internal components and substituting key behaviors with async-compatible implementations using aiohttp or httpx.

While this has extended the usefulness of botocore in the async world, it comes with significant downsides:
• Fragility: aiobotocore depends on internal botocore APIs, which change frequently and cause frequent breakages.
• Incomplete Async Behavior: Many operations still run synchronously or with blocking I/O, limiting concurrency.
• Maintenance Overhead: Keeping up with botocore releases and protocol changes is a continuous and complex burden.
• Performance Constraints: Python-level async adds overhead compared to true native async implementations.

A New Direction

Rather than continuing to patch around botocore, this proposal suggests building a native async Python client that wraps the AWS SDK for C++, offering:
• ✅ True async I/O and thread safety backed by native code
• ✅ Higher performance for high-throughput services (e.g., S3, DynamoDB streams)
• ✅ Long-term stability from using officially supported AWS SDKs
• ✅ Elimination of fragile monkey-patching and compatibility layers

Why This Is Feasible Now

Previously, this approach was dismissed as too complex due to the cost of:
• Writing Pybind11 or CFFI wrappers for many AWS services
• Managing cross-platform builds of the AWS SDK for C++
• Designing clean, Pythonic async interfaces on top of C++ code

Today, with the help of AI-assisted tooling, these barriers are lower than ever:
• AI can generate and maintain binding code, docstrings, and type hints
• Glue code and examples can be iterated on rapidly with LLM assistance

Goals
• Build a proof-of-concept async Python wrapper for the AWS C++ SDK (e.g. S3 or STS).
• Benchmark it against aiobotocore in terms of latency, throughput, and memory usage.
• Maintain idiomatic asyncio interfaces (e.g., async for, async with, await client.get_object(...)).
• Explore possible adoption paths for production workloads.

[thehesiod]

@claude could you do deep research on this subject, I think before the major sticking point was that we'd have to release many native bindings, and the size of the library would increase drastically

[claude]

Claude finished @thehesiod's task in 7m 14s —— View job

---

Deep Research: Native Async Python Bindings for AWS SDK

• Gather context and understand request
• Research existing native Python AWS SDK binding projects
• Research AWS SDK for C++ Python binding approaches
• Research AWS SDK for Rust + PyO3 alternatives
• Analyze build size and distribution concerns
• Analyze the "many bindings" problem and modern solutions
• Research smithy-generated SDK approaches
• Synthesize findings and write analysis

---

Research Findings

TL;DR

The original concern — "we'd have to release many native bindings and library size would increase drastically" — is partially valid but solvable with modern tooling. More importantly, the entire direction has shifted: the Rust SDK + PyO3 path is far more tractable than C++, and AWS is already building the official async Python SDK via smithy-python. The right question is whether to build native bindings at all vs. contributing to/adopting AWS's own effort.

---

1. The C++ Path: Still a Dead End

The original proposal to wrap aws-sdk-cpp hits the same walls as before:

• No viable Python bindings exist. A 2018 pybind11 attempt immediately hit memory corruption from Aws::InitAPI/Aws::ShutdownAPI global lifecycle conflicting with Python's object model. This is not a tooling problem — it's a fundamental design mismatch.
• Wheel size is prohibitive. S3 alone compiles to ~4.6–9.3 MB; the full SDK is a multi-GB static archive taking 10–40+ minutes to build. With auditwheel vendoring libcurl, openssl, and aws-crt, a single platform wheel for even a small subset of services would be enormous.
• No per-service decomposition. The C++ SDK is a CMake monorepo where you select services at build time — you can't ship one PyPI package per service the way Rust naturally allows.
• Verdict: C++ is the wrong foundation.

---

2. The Rust Path: Tractable and Already Happening

The AWS SDK for Rust (aws-sdk-rust) is GA since November 2023, covers 300+ services via Smithy codegen, uses Tokio natively, and is officially supported. Combined with PyO3 + maturin, the distribution concerns shrink dramatically:

On the "many bindings" problem:

• With C++: manual wrappers × 300 services = infeasible
• With Rust + PyO3: smithy-rs already auto-generates all 300 Rust crates from Smithy models. The remaining step is adding a Python-binding codegen pass to smithy-rs that emits thin #[pymodule]/#[pyclass] wrappers alongside each service crate. This is a one-time codegen investment, not 300 manual wrappers.
• An experimental aws-smithy-http-server-python crate already exists in smithy-rs that does exactly this for the server side via PyO3. Extending it to clients is a defined (if non-trivial) engineering task.

On wheel size:

• Modern approach: one PyPI package per service, each with a native wheel
• obstore (Rust object_store crate via PyO3) covering S3/GCS/Azure comes in at 3.9–4.4 MB per platform wheel — less than grpcio or cryptography
• PyO3's abi3 stable-ABI feature means one wheel per platform covers all Python ≥ 3.11, dramatically shrinking the build matrix
• manylinux_2_17 + cibuildwheel handles Linux/macOS/Windows in a single CI workflow
• Per-service: an S3 wheel at ~5–8 MB is comparable to cryptography. Even 50 services at 6 MB each = 300 MB total, split across 50 packages, each under PyPI's 100 MB limit. Users pip install aws-sdk-s3 and get a 6 MB download — not a 300 MB monolith.

Proof of concept that it works:
obstore (v0.9.5, released today) wraps the Rust object_store crate for S3/GCS/Azure, ships 4.4 MB manylinux wheels, and benchmarks at 2.8× higher throughput than aioboto3 and 9×+ over fsspec for concurrent S3 GETs. This is the template.

Performance numbers:

Approach	300 S3 uploads
boto3	16.94s
aioboto3	2.44s
Rust sync	10.00s
Rust async (Tokio via PyO3)	1.02s

---

3. AWS Is Already Building the Native Async SDK

The biggest development since this issue was originally raised: AWS has an official experimental async Python SDK in progress.

• awslabs/aws-sdk-python (PyPI: aws-sdk-python v0.6.0, May 2026)
• Built on smithy-python — a Smithy-to-Python code generator that reads AWS's service models and emits typed, async-first Python clients
• AWS acknowledged that async "requires an entire rewrite of botocore" and is doing exactly that in a separate project
• Currently pre-alpha, covering 3 services (bedrock-runtime, sagemaker-runtime-http2, transcribe-streaming) — all chosen because they benefit most from async streaming
• Protocols covered so far: restJson1 only. S3 (restXml), DynamoDB/EC2 (awsQuery/json1.1) are not yet supported
• June 2025: AWS open-sourced aws/api-models-aws — Smithy JSON AST models for all 200+ public services, with daily updates. This is the foundation for community-built SDKs.

This is pure Python (no native extension), not a C++/Rust binding. AWS's bet is that async-first generated Python + the CRT library for HTTP/TLS performance is sufficient.

---

4. The Real Remaining Concerns with aiobotocore

For context on why change is worthwhile:

• Blocking operations persist despite async wrapper: SSL context loading, service model parsing (JSON files), request serialization, and credential loading all run on the event loop thread. These are structural — botocore was never designed to be wrapped this way.
• 2–4 week tracking tax: Most aiobotocore releases are botocore version bound bumps, not features. The SHA1 hash validation in tests/test_patches.py requires manual diff review for ~30 tracked functions every botocore release.
• No async s3transfer: Large file transfers either bypass multipart optimization or call sync s3transfer in a thread pool — losing async scheduling benefits.

---

5. Recommended Paths (Ranked)

Near-term (0–12 months): Service-specific Rust/PyO3 packages for high-throughput services

For S3 specifically, obstore already works and outperforms aiobotocore 2.8×. For DynamoDB, aiodynamo (pure async Python, no botocore) is production-quality. Rather than a monolithic replacement, adopt best-in-class native packages service by service where performance matters most.

Medium-term (12–24 months): Track and contribute to awslabs/aws-sdk-python

AWS is spending real engineering resources on smithy-python. It will eventually cover all protocols and all services, with official support. Contributing to protocol coverage (especially restXml for S3 and json1.1 for DynamoDB) would accelerate the timeline significantly and benefit the entire AWS Python ecosystem.

Long-term: Auto-generated PyO3 bindings from smithy-rs (the native-performance path)

If true native async performance is required across all services (not just S3/DynamoDB), the architecturally correct approach is to add a Python-binding codegen module to smithy-rs — auto-generating #[pymodule] wrappers for all 300 Rust service crates from the same Smithy models. This is a 6–18 month focused engineering effort, but the building blocks (smithy-rs, pyo3-async-runtimes, aws-smithy-http-server-python as a PyO3 template, maturin for wheel packaging) all exist and are production-quality. The obstore project proves the distribution model works.

---

6. Direct Answers to the Original Concerns

"we'd have to release many native bindings"

With auto-codegen from smithy-rs (the path that generates all 300 Rust crates today), the manual binding cost is a one-time codegen investment plus CI infrastructure, not 300 handwritten files. The aws-smithy-http-server-python crate proves AWS has already built the PyO3/smithy-rs integration for one direction.

"the size of the library would increase drastically"

Per-service PyPI packages with Rust + PyO3 + abi3 wheels come in at ~4–8 MB per platform — less than cryptography or grpcio. Users install only the services they need. The monolithic distribution model (all services in one package) would indeed be prohibitive, but per-service is the right model and PyPI/cibuildwheel make it tractable.

---

Key Links

• obstore — production Rust/PyO3 S3 binding (the template)
• awslabs/aws-sdk-python — AWS's official async Python SDK (pre-alpha)
• smithy-lang/smithy-python — the code generator
• aws/api-models-aws — all AWS Smithy models, Apache-2.0
• smithy-lang/smithy-rs — generates all 300 Rust crates (the codegen template for a Python path)
• pyo3-async-runtimes — Tokio↔asyncio bridge for PyO3
• obstore benchmarks — 2.8× vs aioboto3

Outcome: Research complete — the C++ path has the original concerns and more; the Rust/PyO3 path solves both the "many bindings" and size problems with modern tooling; AWS's own smithy-python is the most likely long-term winner and worth tracking/contributing to.

[dacevedo12]

Hey, a few notes:

• smithy uses the AWS Common Runtime (CRT) libraries, so it's not all pure python as claude said.
• here's a project that can serve as reference for python + aws rust sdk bindings https://github.com/ferrumio/pydynox