fix bulk progress bars

[MattShirley]

Currently if a large amount of files are part of the result set, there will be a long delay before downloads start as documented in #703. This is resolved by disabling refresh behavior until all downloads are started.

To test, use the following script.py and metadata file (adapted from Artur's Jupyter notebook). Current master branch will hang for a long time before downloads start (Artur has estimated ~15 minutes). The new branch should start downloads within 15-20 seconds.

file_metadata.json.gz
script.py

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.57%. Comparing base (c57b431) to head (bc3c5e4).

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #718      +/-   ##
==========================================
+ Coverage   98.43%   98.57%   +0.14%     
==========================================
  Files          31       31              
  Lines        2233     2251      +18     
==========================================
+ Hits         2198     2219      +21     
+ Misses         35       32       -3     

Flag	Coverage Δ
unittests	98.57% <100.00%> (+0.14%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ArturU043]

I tested this with the same query showed in #703
I updated it to use this PR branch.

The Jupyter cell output does show all transform requests immediately

The backend dashboard is completely populated a few seconds after calling deliver().
This fixes the delay problem.

However, on the dashboard, I see 362 requests starting while the client should have sent 364 (this may be unrelated)

[ponyisi]

@MattShirley can you lead me through the parts having to do with the cache? Naively I would have assumed this was only going to touch progressbar code?

[MattShirley]

@MattShirley can you lead me through the parts having to do with the cache? Naively I would have assumed this was only going to touch progressbar code?

It's a deficiency of TinyDB. It doesn't store anything in memory by default and therefore must read the entire flat file every time it queries something. A big part of this PR is identifying cache invalidations (which will now be the only time the JSON file is reloaded for reads).

There might be a TinyDB solution to fix this (see TinyDB's extensions, particularly BetterJSONStorage), but I just wrote my own.