@barakalon Hey, I've used v2.5.5 in an internal system, since then you've improved performance with PR 51 (more efficient buffer drain) and PR 71 - fix O(n²) encoding.
I found a few more small bottlenecks below for prepared statements. I have a local branch with fixes but no write access, if you're interested please grant me write access so I can submit a PR for your review
PERFORMANCE ANALYSIS
- Binary protocol still flushes after every row
- Text protocol buffers the whole resultset in memory before sending anything
- Inbound packets hog memory - Inbound packets over 16 MB arrive as chunks and are combined with
bytes +=
- **
read_str_null - same issue as 3
LOCAL BENCHMARK
Binary protocol:
| Build |
Before |
After |
| pure Python |
88k rows/s |
250k rows/s (×2.8) |
| mypyc |
112k rows/s |
321k rows/s (×2.9) |
Flushes for a 10k-row query: 10,010 → 20.
Text protocol:
×1.5 on pure Python (214k → 315k rows/s), no change on mypyc
Quadratic loops:
- 1 GB inbound packet: 22.3 s → 0.54 s
- 1 MB null-terminated string: 15.3 s → 0.08 s
@barakalon Hey, I've used v2.5.5 in an internal system, since then you've improved performance with PR 51 (more efficient buffer drain) and PR 71 - fix O(n²) encoding.
I found a few more small bottlenecks below for prepared statements. I have a local branch with fixes but no write access, if you're interested please grant me write access so I can submit a PR for your review
PERFORMANCE ANALYSIS
bytes +=read_str_null- same issue as 3LOCAL BENCHMARK
Binary protocol:
Flushes for a 10k-row query: 10,010 → 20.
Text protocol:
×1.5 on pure Python (214k → 315k rows/s), no change on mypyc
Quadratic loops: