unacceptable memory increase with closed datasets

[itcarroll]

Description

I've verified a user experience reported on the CryoCloud slack: memory is held after datasets are closed when they've been opened from an earthaccess.store.EarthAccessFile. Running the following, I watched JupyterLab reported memory use climb from 600MB before the loop to 3.58 GB before crashing the kernel.

import earthaccess
import xarray

earthaccess.login()
results = earthaccess.search_data(short_name="PACE_OCI_L3M_BGC")
paths = earthaccess.open(results)

for item in paths:
    with xarray.open_dataset(item) as ds:
        pass
    # item.close()  <--- does not help?!

I'm sad that item.close() does not seem to help. Need to dig deeper into whether the item.cache is the culprit or something else.

I'll note right away that making earthaccess.open return an object derived from an fsspec.OpenFile with a context manager will probably not help a lot of users, since context managers don't work across Jupyter notebook cells. But that may be irrelevant, if the memory leak has some other source.

One check would be to give xarray.open_* the s3 url and necessary credentials as kwargs; this will help us know if xarray's internal use of fsspec for remote files does anything different.

earthaccess: 0.18.0
fsspec: 2026.2.0

[betolink]

This can also be the I/O cache that fsspec has, I think we can also test something like item.fs.clear_instance_cache()? I'll experiment with it.