Arborist

Disclaimer: This repository is generated and managed by GitHub Copilot's multi-agent functionality.

Arborist is a Python toolkit for scanning large, unorganized scientific datasets stored on Globus endpoints. It normalizes inconsistent naming—resolving researcher aliases and scientific synonyms—and prepares an indexed, queryable catalog so that lab personnel can find relevant data even when folder and file names are vague, abbreviated, or inconsistent.

Why Arborist?

Labs accumulate years of data across many researchers and projects. A folder named sxs321/sc_exp_v3 may be impossible to find unless you already know that sxs321 is Dr. Sarah Smith and sc stands for single-cell RNA sequencing. Arborist solves this by:

• Crawling a Globus endpoint recursively to build a file inventory.
• Resolving aliases — mapping researcher usernames, initials, and informal names to a canonical person ID.
• Resolving science synonyms — mapping shorthand (scrna, sc data, single cell) to a canonical assay term (scRNA-seq).
• Indexing the normalized metadata into Globus Search so the full catalog is queryable.
• Proposing reorganization — generating human-approved rename manifests without moving anything automatically.

---

Architecture

Globus Endpoint
      │
      ▼
  crawler.py          ← recursively lists files via operation_ls
      │ FileMetadata[]
      ▼
  reference_db.py     ← resolves aliases and synonyms against reference_seed.json
      │ resolved IDs / terms
      ▼
  search_indexer.py   ← builds normalized documents, ingests into Globus Search
      │
      ▼
  Globus Search Index ← queryable by lab personnel

  reorganization.py   ← optional: generates rename manifests for human approval

---

Installation

Requires Python ≥ 3.10.

pip install -e .

---

Configuration

1. Globus credentials

Copy the example config and fill in your real IDs:

cp config/globus.example.json config/globus.json

Edit config/globus.json:

{
  "globus_client_id": "<your-native-app-client-id>",
  "source_endpoint_id": "<your-data-endpoint-uuid>",
  "search_index_id": "<your-globus-search-index-uuid>"
}

• globus_client_id — Register a Native App at developers.globus.org to obtain this.
• source_endpoint_id — The UUID of the Globus endpoint that holds your datasets.
• search_index_id — The UUID of your Globus Search index (create one via the Globus Search portal).

config/tokens.json is written automatically after first authentication and should be added to .gitignore — it contains refresh tokens.

2. Reference database

config/reference_seed.json defines the alias/synonym mappings used during indexing. Edit it to reflect your lab's personnel and scientific terminology before running Arborist:

{
  "people": [
    {
      "canonical_id": "UID-SARAH-SMITH",
      "aliases": ["sxs321", "Dr. Smith", "Sarah Smith", "s.smith"]
    }
  ],
  "science_terms": [
    {
      "canonical_term": "scRNA-seq",
      "synonyms": ["single cell", "sc data", "scrna", "10x", "dropseq"]
    }
  ]
}

See Extending the reference database for guidance on building out this file for your lab.

---

User guide

The following walkthrough shows a complete end-to-end workflow. Each step maps to one Arborist module.

Step 1 — Authenticate with Globus

import json
from arborist.auth import GlobusAuthManager

config = json.load(open("config/globus.json"))
manager = GlobusAuthManager(client_id=config["globus_client_id"])

# Open the URL in a browser, log in, and paste the authorization code back.
print("Open this URL in your browser:")
print(manager.get_authorize_url())

auth_code = input("Paste the authorization code: ").strip()
manager.exchange_code_for_tokens(auth_code)  # saves tokens to config/tokens.json

# Build an authenticated transfer client for subsequent operations.
transfer_client = manager.build_transfer_client()

Tokens are cached in config/tokens.json and refreshed automatically on subsequent runs. After first authentication you only need:

transfer_client = manager.build_transfer_client()

Step 2 — Crawl your data endpoint

from arborist.crawler import scan_directory_tree

files = scan_directory_tree(
    transfer_client=transfer_client,
    endpoint_id=config["source_endpoint_id"],
    root_path="/datasets",          # root directory on the endpoint
)

print(f"Found {len(files)} files")
for f in files[:5]:
    print(f.path, f.size, f.modified)

Each item in files is a FileMetadata object:

Attribute	Type	Description
path	str	Full remote path on the endpoint
size	int	File size in bytes
modified	str	Last-modified timestamp string
header	dict or None	Optional top-of-file metadata (see below)

Optional: extract file headers

If your files carry embedded metadata (e.g., HDF5 attributes, NetCDF global attributes, CSV header rows), implement the HeaderExtractor protocol and pass it to the crawler:

from arborist.crawler import HeaderExtractor

class MyHDF5Extractor:
    def extract(self, endpoint_id: str, path: str) -> dict[str, str] | None:
        # Download just the header portion and return key/value pairs.
        # Return None if extraction fails or is not applicable.
        ...

files = scan_directory_tree(
    transfer_client, config["source_endpoint_id"], "/datasets",
    header_extractor=MyHDF5Extractor(),
)

Step 3 — Load the reference database

from arborist.reference_db import ReferenceDatabase

db = ReferenceDatabase.from_json("config/reference_seed.json")

# Resolve a researcher alias to a canonical ID:
print(db.resolve_person("sxs321"))        # → "UID-SARAH-SMITH"
print(db.resolve_person("Dr. Smith"))     # → "UID-SARAH-SMITH"

# Resolve a science synonym to a canonical term:
print(db.resolve_science_term("single cell"))  # → "scRNA-seq"
print(db.resolve_science_term("scrna"))         # → "scRNA-seq"

# Returns None when no match is found:
print(db.resolve_person("unknown_user"))  # → None

Step 4 — Build and ingest search documents

The indexer combines the crawler output with the reference database, normalizing paths into searchable documents, then ingests them into Globus Search.

from globus_sdk import SearchClient
from arborist.search_indexer import build_search_documents, ingest_documents

# Build normalized documents (no network calls needed).
documents = build_search_documents(dataset_records=files, references=db)

# Each document includes resolved canonical IDs and terms, e.g.:
# {
#   "path": "/datasets/sxs321/sc_exp_v3/cells.h5ad",
#   "size": 4096000,
#   "modified": "2025-03-15T10:00:00",
#   "header": {},
#   "indexed_at": "2026-07-13T12:00:00+00:00",
#   "resolved_people_ids": ["UID-SARAH-SMITH"],
#   "resolved_science_terms": ["scRNA-seq"],
# }

# Ingest into Globus Search.
search_client = SearchClient()  # authenticated via environment or token
ingest_documents(search_client, config["search_index_id"], documents)
print(f"Ingested {len(documents)} documents")

Step 5 — Query for data

Build a structured query and run it against your Globus Search index:

from arborist.search_indexer import build_structured_query

# Find all scRNA-seq data by Dr. Smith:
query = build_structured_query({
    "person": "UID-SARAH-SMITH",
    "assay": "scRNA-seq",
})

results = search_client.post_search(config["search_index_id"], query)
for hit in results["gmeta"]:
    print(hit["content"][0]["path"])

Lab members can query by canonical IDs and terms regardless of how the original files were named—Arborist's indexing step ensures the normalization is already done.

Step 6 — Propose reorganization (optional)

When you want to standardize folder names, Arborist generates a manifest for human review instead of moving anything automatically:

from arborist.reorganization import build_manifest_of_change

# Propose renaming researcher-initials folders to canonical IDs:
path_mapping = {
    "/datasets/sxs321": "/datasets/UID-SARAH-SMITH",
    "/datasets/jdoe99": "/datasets/UID-JOHN-DOE",
}

manifest = build_manifest_of_change(path_mapping, reason="Standardize folder names to canonical IDs")

print(manifest["requires_user_approval"])  # True — always requires sign-off
for change in manifest["changes"]:
    print(f"  {change['old_path']}  →  {change['new_path']}")

The manifest is a plain Python dict (JSON-serializable). Store it, review it, and only execute the moves once approved by a data manager. Arborist never moves data automatically.

---

Extending the reference database

Adding researchers

Add an entry to people for each lab member. Include every form of their name or username that might appear in a folder or file name:

{
  "canonical_id": "UID-JOHN-DOE",
  "aliases": ["jdoe99", "j.doe", "John Doe", "Dr. Doe", "jdoe"]
}

canonical_id can be any unique string — using a format like UID-FIRSTNAME-LASTNAME makes manifests and search results human-readable.

Adding science synonyms

Add an entry to science_terms for each assay, technique, or experimental type used in your lab. Cast a wide net — include every abbreviation or informal name that staff actually use in folder names:

{
  "canonical_term": "bulk RNA-seq",
  "synonyms": ["rnaseq", "rna-seq", "bulkrna", "total rna", "polyA"]
}

After editing the seed file, re-run the crawler and indexer to update the search index with the new mappings.

---

Module reference

Module	Purpose
arborist/auth.py	OAuth2 bootstrap and token-backed TransferClient creation. Handles the browser-based authorization flow and caches refresh tokens.
arborist/crawler.py	Recursively walks endpoint directories using operation_ls and returns a list of FileMetadata objects. Supports an optional HeaderExtractor for reading embedded file metadata.
arborist/reference_db.py	Pydantic models for the people-alias and science-synonym lookup tables. Loads from reference_seed.json and exposes resolve_person / resolve_science_term methods.
arborist/search_indexer.py	Combines FileMetadata with resolved aliases to build normalized search documents, ingests them into Globus Search, and provides a helper to build structured queries.
arborist/reorganization.py	Generates a human-in-the-loop rename/move manifest. Always sets requires_user_approval: true. Does not execute any file operations.

---

Running tests

python -m unittest discover -s tests -v

Expected output:

test_scan_directory_tree_collects_recursive_files ... ok
test_manifest_always_requires_approval ... ok
test_round_trip_json ... ok
test_seed_file_resolves_aliases_and_terms ... ok

Ran 4 tests in 0.00s

OK