feat: add Arraylake source control

[ghostiee-11]

Adds ArraylakeSourceControls, a parametric input control that opens an Arraylake repository (repo / branch / group) and registers it as an XArraySQLSource, so Lumen AI can query Icechunk/Zarr stores the same way uploaded CSVs become a DuckDBSource. It drops 0-dim scalar metadata vars, unifies inconsistent chunks, and sizes the Icechunk snapshot cache so real-world stores like GOES register cleanly and fast (~200s to ~16s open).

from lumen.ai.controls import ArraylakeSourceControls

ui = ExplorerUI(source_controls=[ArraylakeSourceControls])
# User enters repo="earthmover-public/goes-16",
# group="ABI-L2-MCMIPF/post-2023-04-19"  ->  registered as a queryable source

Result (live earthmover-public/goes-16):

[codecov]

Codecov Report

❌ Patch coverage is 91.48936% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.69%. Comparing base (7cc94fb) to head (6c6c052).
⚠️ Report is 70 commits behind head on main.

Files with missing lines	Patch %	Lines
lumen/ai/controls/ingest/arraylake.py	89.28%	6 Missing ⚠️
lumen/ai/controls/ingest/__init__.py	50.00%	2 Missing ⚠️
lumen/tests/ai/test_controls/test_arraylake.py	97.01%	2 Missing ⚠️
lumen/ai/controls/ingest/utils.py	90.90%	1 Missing ⚠️
lumen/ai/ui.py	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1889      +/-   ##
==========================================
+ Coverage   70.60%   70.69%   +0.08%     
==========================================
  Files         195      197       +2     
  Lines       33630    33770     +140     
==========================================
+ Hits        23746    23875     +129     
- Misses       9884     9895      +11     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ahuang11]

Should this be guarded if imports are available? Try/except?

[ghostiee-11]

Yess!! we can guard it and, add try/except.

[ahuang11]

Maybe use as util?

[ghostiee-11]

Yeah, will add this in util...

[ahuang11]

Can you try it out on the UI? Ask the agent to query for something.

[ghostiee-11]

Dug into the slow load and validated it. The open is fine now (sizing the Icechunk snapshot cache took it from ~205s to ~16s, that config is in the control). The rest is registration, and it is an xarray-sql scaling issue rather than a control problem.

xarray-sql creates one scan partition per native chunk and enumerates them all at registration. I measured it on tiny synthetic data (same data, only the chunking changes):

partitions	register time
1,000	0.06s
10,000	0.24s
100,000	2.45s

So ~25us per partition, linear in the number of chunks.

GOES is chunked one timestep per chunk: CMI_C13 is t=102,988 chunks of size 1, y/x are 24 chunks of 226. That is 102988 x 24 x 24 = ~59M partitions for a single variable, which is ~25-30 min just to register one variable. XArraySQLSource registers all 188 variables separately on top of that, so it never finishes. Coord pre-loading and coarse time-chunking on the full store did not help (still minutes).

The real fix is upstream in xarray-sql: coalesce native chunks into a bounded number of scan partitions (or enumerate lazily) so registration cost is independent of how finely the store is chunked. That fixes it for the current XArraySQLSource without changes here. A secondary Lumen improvement would be registering per dimension-group instead of per-variable, which also gives cleaner tables. Filed the xarray-sql issue with the repro: alxmrs/xarray-sql.

The control itself works well on a subset (registers in ~16s, queries in ~1s), and it shows up and loads in the UI under its own Arraylake tab.

[ahuang11]

Nice find! Can you show a screen recording of it please?

[ghostiee-11]

Suree!!

[ghostiee-11]

Screen.Recording.2026-06-13.at.12.54.13.AM.mov

[ahuang11]

Thanks for the screenshot; is there a way for users to discover these values? i.e. can we somehow have a dropdown instead? Perhaps there are central repos like earthmover? I'm thinking a new Select, EarthMover, NWS, AWS, Custom, and for Custom, users input as you did, but for the others, ideally there would be a dropdown.

[ghostiee-11]

Hey @ahuang11, On making the dropdowns dynamic: the group can be auto-discovered, but the repo list can't.

list_repos returns 403 for any org you don't own, so earthmover-public can't be enumerated, and S3 buckets aren't listable. So the provider/repo list stays curated.

Groups are dynamic. Once a repo is open I walk its zarr tree and populate the Group dropdown from the groups that actually hold data, so no group paths are hardcoded (they were fragile anyway, ERA5's real group is single/spatial, not single).

So, How should we move forward ?

[ahuang11]

Can you share how you manually discover the repos? Is there a link?

[ghostiee-11]

No link to list them. list_repos 403s for earthmover-public even when logged in (it needs org membership, not just auth). Public repos are readable one at a time via get_repo, but the org's repo list stays private, so it can't be enumerated.

So I found them by probing names with get_repo. goes-16 is in their virtual-zarr blog; era5/gfs/hrrr I confirmed the same way. Net: repos stay curated, groups stay dynamic.

[ahuang11]

Do you mind asking around in https://discourse.pangeo.io/?

[ghostiee-11]

Do you mind asking around in https://discourse.pangeo.io/?

Yup, Thanks for sharing this, will create a Topic here

[ahuang11]

I created one https://discourse.pangeo.io/t/how-to-see-all-the-repos-in-arraylake/5764

[ghostiee-11]

Thanks a lot, Andrew!!

[ahuang11]

Can the tooltip give an example like the repo?

Also, in my testing, it doesn't look like it's responsive after clicking Fetch Data, which is poor UX. We need to allow the main thread to trigger loading=True by asyncio.sleep(0.01).

#1889 (comment)

Also, would it be possible to subset tables so we don't have to wait for all the tables to load?