Skip to content

Per-row drain in binary protocol + memory issue in text path #82

Description

@Selerski

@barakalon Hey, I've used v2.5.5 in an internal system, since then you've improved performance with PR 51 (more efficient buffer drain) and PR 71 - fix O(n²) encoding.

I found a few more small bottlenecks below for prepared statements. I have a local branch with fixes but no write access, if you're interested please grant me write access so I can submit a PR for your review

PERFORMANCE ANALYSIS

  1. Binary protocol still flushes after every row
  2. Text protocol buffers the whole resultset in memory before sending anything
  3. Inbound packets hog memory - Inbound packets over 16 MB arrive as chunks and are combined with bytes +=
  4. **read_str_null - same issue as 3

LOCAL BENCHMARK

Binary protocol:

Build Before After
pure Python 88k rows/s 250k rows/s (×2.8)
mypyc 112k rows/s 321k rows/s (×2.9)

Flushes for a 10k-row query: 10,010 → 20.

Text protocol:

×1.5 on pure Python (214k → 315k rows/s), no change on mypyc

Quadratic loops:

  • 1 GB inbound packet: 22.3 s → 0.54 s
  • 1 MB null-terminated string: 15.3 s → 0.08 s

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions