feat(encode): add gget encode module to query and download ENCODE data (#151)

[Elarwei001]

Resolves #151

Summary

gget encode: new module to query and download data from the ENCODE project via its official REST API. Two modes, chosen automatically from the input:

• Accession (e.g. ENCSR000AKS experiment or ENCFF000BXK file) → lists the object's file(s) with download URLs, optionally filtered by assembly / file_format / output_type and fetched with download=True.
• Free text → searches ENCODE objects of a given type (default Experiment).

Available in the Python API and on the command line.

Example return values

The file listings also include a url column (the direct @@download link) — it is omitted from the tables below for width, but it is part of the returned DataFrame, as the JSON snippet further down shows.

Mode A — a single file — gget.encode("ENCFF000BXK"):

file_accession file_format output_type assembly  file_size   status
   ENCFF000BXK         bam  alignments     hg19  964104660 archived

Mode A — an experiment's files, filtered to BAMs — gget.encode("ENCSR000AKS", file_format="bam") (20 of the experiment's 82 files; note it spans assemblies):

file_accession file_format           output_type assembly  file_size   status
   ENCFF000BXK         bam            alignments     hg19  964104660 archived
   ENCFF130RCQ         bam unfiltered alignments   GRCh38  806537790 released
   ENCFF778EZR         bam            alignments   GRCh38  419581950 released
   ...

Mode B — free-text search — gget.encode("CTCF K562", type="Experiment", limit=5):

  accession    assay_title            biosample_summary     target   status
ENCSR092LZV  siRNA RNA-seq  Homo sapiens K562 ... CTCF/RAD21 CTCF+RAD21 released
ENCSR597AKG       ChIA-PET            Homo sapiens K562       CTCF released
ENCSR000AKO    TF ChIP-seq            Homo sapiens K562       CTCF released
   ...

With json=True (the command-line default) the same rows come back as records — e.g. the file above:

[
  {
    "file_accession": "ENCFF000BXK",
    "file_format": "bam",
    "output_type": "alignments",
    "assembly": "hg19",
    "file_size": 964104660,
    "status": "archived",
    "url": "https://www.encodeproject.org/files/ENCFF000BXK/@@download/ENCFF000BXK.bam"
  }
]

Add download=True (CLI --download, with --out_dir) to fetch the files in the url column. ⚠️ Download pulls every file left after filtering — an experiment has ~82 files (tens of GB), so narrow with assembly / file_format / output_type first.

Full example output — the file (ENCFF000BXK), the experiment's complete 82-file listing (ENCSR000AKS), and a CTCF K562 experiment search (CSV + JSON) are attached here: https://gist.github.com/Elarwei001/7cee4ce617ee7a239b3a6ac104790e05

Testing

Unit tests — tests/test_encode.py (offline / mocked, fixture in tests/fixtures/):

• the accession router and the three object paths (file / experiment-with-files / generic metadata);
• the assembly / file_format / output_type filters, and free-text search;
• the download path — _download_files streaming, the download=True routing, and the search-mode "download not supported" warning;
• the error paths — 404 → not found, 5xx, request failure, empty term.

Live tests — hit the real ENCODE API (skipped when offline, and they never trigger a download):

• a ENCFF000BXK file (asserts file_format == "bam");
• the ENCSR000AKS experiment listing (many files, incl. BAMs);
• a CTCF experiment search.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.66%. Comparing base (e43a804) to head (74a9cb0).
⚠️ Report is 2 commits behind head on dev.

Additional details and impacted files

@@            Coverage Diff             @@
##              dev     #231      +/-   ##
==========================================
- Coverage   56.70%   56.66%   -0.05%     
==========================================
  Files          29       30       +1     
  Lines        9392     9497     +105     
==========================================
+ Hits         5326     5381      +55     
- Misses       4066     4116      +50     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Elarwei001]

Hi @lauraluebbert — this PR is ready for review whenever you have time. Quick overview:

1. What it adds — gget encode, a module to query and download data from the ENCODE project. Resolves ENCODE downloader #151.

2. How it works — two modes, chosen automatically from the input:

• an ENCODE accession (e.g. ENCSR000AKS experiment or ENCFF000BXK file) → lists the object's file(s) with direct download URLs, optionally filtered by assembly / file_format / output_type and fetched with --download;
• free text → a /search over ENCODE objects of a given --type.

It's built on ENCODE's current REST API (the /{accession}/ and /search/ endpoints, plus the files' direct-download links), not the older kundajelab/ENCODE_downloader script the issue referenced (which you and @akundaje noted was pretty old). The PR description has real return-value examples (tables + JSON) and a full-output attachment.

3. A download caveat — --download fetches every file left after filtering, and an experiment can be ~82 files / tens of GB, so the docs steer users to filter first. Nothing is downloaded unless --download is passed.

4. Testing

• Unit tests (offline/mocked): the accession router, the three object paths, the filters, search, the download path, and the error branches;
• Live tests (hit the real ENCODE API, skipped when offline, never download): a file, an experiment listing, and a search.

5. CI note — the only failing check is test_cellxgene_adata, unrelated to this PR (a cellxgene live-data drift, CI: test_cellxgene_adata fails — cellxgene-census AnnData now exposes a layers entry (live-data drift) #265). It fails on py3.12/3.13 and is skipped on py3.14 (where cellxgene-census has no wheel). All gget encode tests pass on every version.

Happy to adjust anything — thanks!