Skip to content

Project: Explode locations figure export - #762

Draft
tnagorra wants to merge 4 commits into
developfrom
feat/explode-locations-figure-export
Draft

Project: Explode locations figure export#762
tnagorra wants to merge 4 commits into
developfrom
feat/explode-locations-figure-export

Conversation

@tnagorra

Copy link
Copy Markdown
Contributor

Changes

Benchmarked on the dataset with 186,321 figures, comparing the new
explode-by-locations export against the existing regular figure export.

figure export wall-clock peak RSS queries file size
regular (existing) 7.6 min ~334 MB 2 ~168 MB
explode (new) 8.3 min ~334 MB 89 ~194 MB

RSS = Resident Set Size (the amount of a process's memory that is in RAM)

Takeaways

  • Explode runs ~9% slower than the existing export
  • Memory is flat and bounded (~334 MB RSS, ~148 MB Python), independent of row
    count. Both the read and write paths stream to disk
  • Query count is bounded 2 (regular) vs 89 (explode = 1 base query +
    87 batched per-chunk side queries). Query count scales with chunks, not figures.

Optimizations

  • Dropped the unused locations ArrayAgg on explode
  • Replaced the .filter(geo_locations__isnull=False).distinct() filter with an
    Exists() semi-join

Together these cut the explode-specific overhead by ~44% (from +72 s / 15.6%
to +43 s / 9.6% over the regular export)

Known limitation

A full unfiltered export (~8.3 min) runs close to the 10-min Celery task time_limit. Follow-ups:

  • raise the export time_limit
  • adopt xlsxwriter for faster writes
  • validate if we reach 1,048,576-row Excel limitation

This PR doesn't introduce any:

  • temporary files, auto-generated files or secret keys
  • n+1 queries
  • flake8 issues
  • print
  • typos
  • unwanted comments

This PR contains valid:

  • tests
  • permission checks (tests here too)
  • translations

@tnagorra
tnagorra force-pushed the feat/explode-locations-figure-export branch from 4ad222f to c50d909 Compare June 17, 2026 08:56
tnagorra added 3 commits June 18, 2026 07:15
- Add metadata in excel download mutation
- Add metadata in excel download query
@tnagorra
tnagorra force-pushed the feat/explode-locations-figure-export branch from c50d909 to d096d35 Compare June 18, 2026 01:40
@tnagorra
tnagorra force-pushed the feat/explode-locations-figure-export branch from d096d35 to a148d03 Compare June 19, 2026 06:21
@tnagorra
tnagorra marked this pull request as ready for review June 19, 2026 06:22
@tnagorra tnagorra changed the title Explode locations figure export Project: Explode locations figure export Jun 30, 2026
@tnagorra
tnagorra marked this pull request as draft July 16, 2026 01:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants