Loading data

Use get_lf to read a consolidated table as a polars.LazyFrame; apply .select(...) to choose specific columns and .collect() to materialize as a DataFrame.

Pass session_id to restrict the result to one session, or nwb=True to read directly from NWB files:

import polars as pl
from dr_datacube import get_lf

performance = get_lf("performance").collect()
session_units = (
    get_lf("units", session_id="123456_2024-01-01", nwb=True)
    # filter to reduce the number of rows fetched:
    .filter(
        'is_qc_pass', 
        pl.col('structure') == 'MOs',
    )
    # drop large columns that aren't needed to save time/memory:
    .drop('spike_amplitudes', 'waveform_mean', 'waveform_std')
    # alternatively, select the columns you need directly:
    .select('spike_times', 'unit_id') 
    .collect()
)

Use get_session_table for the standard session sets. By default it applies the behavior filter and keeps only sessions in the configured data asset:

from dr_datacube import get_session_table

brainwide = get_session_table(["brainwide", "templeton"])
all_sessions = get_session_table(None, with_behavior_filter=False)

Datacube session IDs

A list of standard ephys sets is published in assets/datacube_sessions.csv. The table contains the session and subject IDs, the standard session type (brainwide, naive, and templeton), and whether the session passed the relevant behavior filter for its type. The table is generated by scripts/dump_datacube_sessions.py and updated automatically in GitHub Actions when changes are made to the filtering logic or the datacube default version (as it requires CO and AWS credentials).

Read it directly from GitHub with pandas:

import pandas as pd

url = "https://raw.githubusercontent.com/allenneuraldynamics/dr-datacube/main/assets/datacube_sessions.csv"
session_ids = pd.read_csv(url).query("is_behavior_pass and session_type == 'brainwide'")["session_id"].tolist()

Or with Polars:

import polars as pl

url = "https://raw.githubusercontent.com/allenneuraldynamics/dr-datacube/main/assets/datacube_sessions.csv"
session_ids = (
    pl.read_csv(url)
    .filter(pl.col("is_behavior_pass") & (pl.col("session_type") == "brainwide"))["session_id"]
    .to_list()
)

This package also provides a convenience function to get the list of session IDs directly:

from dr_datacube import get_session_ids_from_github

good_session_ids = get_session_ids_from_github("brainwide")
all_session_ids = get_session_ids_from_github(["brainwide", "templeton"], with_behavior_filter=False)

Data source and configuration

The datacube version and data source are controlled by dr_datacube.config. Set configuration near the top of your script or notebook before loading data:

import dr_datacube

dr_datacube.config.version = "v0.0.289"

By default, the package uses a matching local dynamicrouting_datacube asset when running on Code Ocean. If no matching local asset is available, it streams from the datacube asset directory on S3. The S3 asset directory is currently private, so credentials are required for this fallback.

To stream from the public scratch-bucket cache instead, enable the cache:

dr_datacube.config.use_cache = True

If credentials are not available, enable anonymous access as well:

dr_datacube.config.anon = True

Temporary configuration

Use config.override(...) when an operation needs different settings without changing the module-level config. The temporary configuration is restored when the context exits, even if an exception is raised:

from dr_datacube import config, get_lf

with config.override(use_cache=True):
    units = get_lf("units")