Add GitHub-issue-driven test framework (+ src/ library carve-out)

.github/workflows/tests.yaml

@@ -0,0 +1,17 @@
+name: Tests
+
+on:
+  pull_request:
+
+jobs:
+  unit-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: astral-sh/setup-uv@v6
+
+      - name: Run unit tests
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
+        run: uv run pytest -m unit -v

.gitignore

@@ -1,3 +1,6 @@
+# Ignore the root .env file.
+/.env
+
 # Ignore all data files.
 data/
 

CLAUDE.md

@@ -20,7 +20,14 @@ pytest --target all                    # Run against all targets in targets.ini
 pytest --category "Unit Tests"         # Filter by Google Sheet category
 pytest --category-exclude "Slow"       # Exclude a category
 pytest tests/nodenorm/test_nodenorm_from_gsheet.py  # Run a specific test file
-pytest tests/nodenorm/test_nodenorm_from_gsheet.py -k "row=42"  # Run a specific test row
+pytest -m unit                         # Offline unit suite (no network/token) — what CI runs on PRs
+```
+
+To run a single Google Sheet row, pass its full node ID (NOT `-k "row=42"` — pytest's
+`-k` expression parser rejects the `=`):
+
+```bash
+pytest "tests/nodenorm/test_nodenorm_from_gsheet.py::test_normalization[dev-gsheet:row=42]"
 ```
 
 ### Code Formatting
@@ -46,21 +53,31 @@ cd website && npm install && npm run dev   # Dev server at localhost:4321
 
 ## Architecture
 
+### Library (`src/babel_validation/`)
+
+Shared library code used by the tests and potentially other consumers.
+
+- `core/testrow.py` — `TestRow` dataclass (models a single Google Sheet test row), `TestStatus` enum, `TestResult` dataclass
+- `services/nodenorm.py` — `CachedNodeNorm`: wraps the NodeNorm `get_normalized_nodes` API with per-instance caching
+- `services/nameres.py` — `CachedNameRes`: wraps the NameRes `lookup`/`bulk-lookup` APIs with per-instance caching
+- `sources/google_sheets/google_sheet_test_cases.py` — `GoogleSheetTestCases`: downloads and parses the shared Google Sheet into `TestRow` instances and pytest `ParameterSet` lists
+- `sources/github/github_issues_test_cases.py` — `GitHubIssueTest` and `GitHubIssuesTestCases`: pull test cases embedded in GitHub issues (wiki or YAML syntax) and evaluate them against NodeNorm/NameRes
+
 ### Test Framework (`tests/`)
 
 The core of this project. Tests validate NodeNorm and NameRes services across multiple deployment environments.
 
 **Target system:** `tests/targets.ini` defines endpoints for each environment (dev, prod, test, ci, exp, localhost). Tests use `target_info` fixture to get URLs. The `conftest.py` parametrizes tests across targets via `--target` CLI option; default is `dev`.
 
-**Google Sheet integration:** ~2000+ test cases are pulled from a [shared Google Sheet](https://docs.google.com/spreadsheets/d/11zebx8Qs1Tc3ShQR9nh4HRW8QSoo8k65w_xIaftN0no/). `tests/common/google_sheet_test_cases.py` fetches and parses these into `TestRow` dataclasses. Rows marked as not expected to pass are wrapped with `pytest.mark.xfail(strict=True)`. Tests are parametrized by row, with IDs like `gsheet:row=42`.
+**Google Sheet integration:** ~2000+ test cases are pulled from a [shared Google Sheet](https://docs.google.com/spreadsheets/d/11zebx8Qs1Tc3ShQR9nh4HRW8QSoo8k65w_xIaftN0no/). `src/babel_validation/sources/google_sheets/google_sheet_test_cases.py` fetches and parses these into `TestRow` dataclasses. Rows marked as not expected to pass are wrapped with `pytest.mark.xfail(strict=True)`. Tests are parametrized by row, with IDs like `gsheet:row=42`.
 
 **Category filtering:** Google Sheet rows have a Category column. The `test_category` fixture (from conftest.py) returns a callable that tests use to `pytest.skip()` rows not matching `--category`/`--category-exclude` filters.
 
 **Test modules:**
 - `tests/nodenorm/` — NodeNorm tests (normalization accuracy, preferred IDs/labels, Biolink types, conflation, descriptions, OpenAPI spec, setid endpoint)
 - `tests/nameres/` — NameRes tests (label lookup, autocomplete, Biolink type filtering, blocklist, taxon_specific flag)
-- `tests/nodenorm/by_issue/` — Tests tied to specific GitHub issues
-- `tests/common/` — Shared utilities (`GoogleSheetTestCases`, `TestRow`)
+- `tests/nodenorm/by_issue/` — Per-issue regression tests for NodeNorm (hand-written)
+- `tests/github_issues/` — Dynamically-generated tests pulled from GitHub issue bodies via `GitHubIssuesTestCases`
 
 ### Web Applications
 
@@ -79,4 +96,22 @@ When writing new tests:
 - Use the `target_info` fixture to get NodeNorm/NameRes URLs from targets.ini
 - For Google Sheet-based tests, parametrize with `gsheet.test_rows()` and use the `test_category` fixture for category filtering
 - Use `pytest.mark.xfail(strict=True)` for known failures (strict=True means unexpected passes also fail)
-- Issue-specific tests go in `tests/nodenorm/by_issue/` or `tests/github_issues/`
+- Hand-written per-issue regression tests go in `tests/nodenorm/by_issue/`
+- GitHub-issue-driven tests are picked up automatically by `tests/github_issues/test_github_issues.py` via `GitHubIssuesTestCases`
+- Import shared classes from `src.babel_validation.*` (e.g. `from src.babel_validation.services.nodenorm import CachedNodeNorm`)
+
+## Gotchas (learned the hard way)
+
+- **Offline vs. networked tests.** `pytest -m unit` is the offline suite CI runs on PRs (no
+  network, no `GITHUB_TOKEN`). Network-backed modules (Google Sheet, GitHub issues) defer all
+  fetching to `pytest_generate_tests` and call `tests/_pytest_helpers.deselected_by_markexpr`
+  so a marker-deselected run never hits the network. If you add a module that fetches at
+  import/collection time, replicate that pattern or you'll break `-m unit`.
+- **`src/babel_validation` is importable only because** `[tool.hatch.build.targets.wheel]` in
+  `pyproject.toml` packages `src/`. Imports use the full `src.babel_validation.*` path.
+- **Black is not strictly enforced** — `black --check` flags much of the existing tree. Match
+  the surrounding style of the file you're editing; don't mass-reformat (it creates huge,
+  review-hostile diffs).
+- **Commits are signed via 1Password's SSH agent** (`op-ssh-sign`). It can lock mid-session;
+  a `failed to fill whole buffer` / `failed to write commit object` error means 1Password
+  needs unlocking, not a git problem.

pyproject.toml

@@ -7,12 +7,41 @@ readme = "README.md"
 requires-python = ">=3.11"
 dependencies = [
     "black>=25.9.0",
+    "click>=8.1",
+    "pyyaml>=6.0",
     "requests>=2.32.5",
+    "tqdm>=4.67.1",
+    "filelock",
     "deepdiff>=8.6.1",
+    "python-dotenv>=0.9.9",
     "openapi-spec-validator>=0.7.2",
-    "pytest>=8.4.2",
+    "pygithub>=2.8.1",
+    "pytest>=9.0.2",
     "pytest-timeout>=2.4.0",
+    "pytest-xdist[psutil]",
+    "pytest-subtests",
 ]
 
 [project.urls]
 Repository = "https://github.com/TranslatorSRI/babel-validation"
+
+[project.scripts]
+csv-to-babeltests = "src.babel_validation.tools.csv_to_babeltests:main"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+# Package the entire `src/` directory so existing imports
+# (`from src.babel_validation.X import Y`) continue to work after install.
+packages = ["src"]
+
+[tool.pytest.ini_options]
+# Without testpaths, a bare `pytest` would also scan the website directories
+# (including node_modules) during collection.
+testpaths = ["tests"]
+timeout = 300
+markers = [
+    "unit: unit tests that do not require network access",
+]

src/babel_validation/assertions/README.md

@@ -0,0 +1,243 @@
+<!-- AUTO-GENERATED — do not edit by hand.
+     Regenerate with: uv run python -m src.babel_validation.assertions.gen_docs -->
+
+# BabelTest Assertion Types
+
+This package defines the assertion types that can be embedded in GitHub issue bodies and evaluated against the NodeNorm and NameRes services.
+
+## Embedding Tests in Issues
+
+Two syntaxes are supported:
+
+**Wiki syntax** (one assertion per line):
+```
+{{BabelTest|AssertionType|param1|param2|...}}
+```
+
+**YAML syntax** (multiple assertions, multiple param sets):
+````
+```yaml
+babel_tests:
+  AssertionType:
+    - param1
+    - [param1, param2]
+```
+````
+
+Assertion names are case-insensitive, as is the `{{BabelTest|...}}` marker itself.
+
+## Param Sets
+
+Each assertion can be invoked with one or more **param sets** — independent groups of
+parameters that are each evaluated separately.
+
+- **Wiki syntax** — each `{{BabelTest|...}}` line is one param set.
+- **YAML syntax** — each list entry under an assertion key is one param set; a bare string
+  is a single-element param set, a YAML list is a multi-element param set.
+
+The meaning of each element in a param set depends on the assertion type (see below).
+For most assertions the elements are CURIEs; for `HasLabel` the second element is a
+label string; for `ResolvesWithType` the first element is a Biolink type.
+
+---
+
+## NodeNorm Assertions
+
+These assertions test the [NodeNorm](https://nodenorm.transltr.io/docs) service.
+
+### Resolves
+
+**Applies to:** NodeNorm
+
+Each CURIE in each param_set must resolve to a non-null result in NodeNorm.
+
+**Parameters:** One or more CURIEs per param_set.
+
+**Wiki syntax:**
+```
+{{BabelTest|Resolves|CHEBI:15365}}
+{{BabelTest|Resolves|MONDO:0005015|DOID:9351}}
+```
+
+**YAML syntax:**
+```yaml
+babel_tests:
+  Resolves:
+    - CHEBI:15365
+    - [MONDO:0005015, DOID:9351]
+```
+
+---
+
+### DoesNotResolve
+
+**Applies to:** NodeNorm
+
+Each CURIE in each param_set must fail to resolve (return null) in NodeNorm. Use this to confirm that an identifier is intentionally not normalizable.
+
+**Parameters:** One or more CURIEs per param_set.
+
+**Wiki syntax:**
+```
+{{BabelTest|DoesNotResolve|FAKENS:99999}}
+```
+
+**YAML syntax:**
+```yaml
+babel_tests:
+  DoesNotResolve:
+    - FAKENS:99999
+```
+
+---
+
+### ResolvesWith
+
+**Applies to:** NodeNorm
+
+All CURIEs within each param_set must resolve to the identical normalized result. Use this to assert that two identifiers are equivalent.
+
+**Parameters:** Two or more CURIEs per param_set. All must resolve to the same result.
+
+**Wiki syntax:**
+```
+{{BabelTest|ResolvesWith|CHEBI:15365|PUBCHEM.COMPOUND:1}}
+```
+
+**YAML syntax:**
+```yaml
+babel_tests:
+  ResolvesWith:
+    - [CHEBI:15365, PUBCHEM.COMPOUND:1]
+    - [MONDO:0005015, DOID:9351]
+```
+
+---
+
+### DoesNotResolveWith
+
+**Applies to:** NodeNorm
+
+The CURIEs within each param_set must NOT all resolve to the same normalized result. Use this to assert that two identifiers are intentionally distinct entities.
+
+**Parameters:** Two or more CURIEs per param_set. They must not all resolve to the same result.
+
+**Wiki syntax:**
+```
+{{BabelTest|DoesNotResolveWith|CHEBI:15365|CHEBI:16856}}
+```
+
+**YAML syntax:**
+```yaml
+babel_tests:
+  DoesNotResolveWith:
+    - [CHEBI:15365, CHEBI:16856]
+```
+
+---
+
+### HasLabel
+
+**Applies to:** NodeNorm
+
+The CURIE must resolve in NodeNorm and its primary label (id.label) must match the expected label exactly (case-sensitive).
+
+**Parameters:** Exactly two elements per param_set: a CURIE, then the expected label string.
+
+**Wiki syntax:**
+```
+{{BabelTest|HasLabel|CHEBI:15365|aspirin}}
+```
+
+**YAML syntax:**
+```yaml
+babel_tests:
+  HasLabel:
+    - [CHEBI:15365, aspirin]
+```
+
+---
+
+### ResolvesWithType
+
+**Applies to:** NodeNorm
+
+Each param_set must have at least two elements: the first is the expected Biolink type (e.g. 'biolink:Gene'), and the remainder are CURIEs that must resolve with that type.
+
+**Parameters:** Each param_set: first element is the expected Biolink type (e.g. `biolink:Gene`), remaining elements are CURIEs.
+
+**Wiki syntax:**
+```
+{{BabelTest|ResolvesWithType|biolink:Gene|NCBIGene:1}}
+```
+
+**YAML syntax:**
+```yaml
+babel_tests:
+  ResolvesWithType:
+    - [biolink:Gene, NCBIGene:1, HGNC:5]
+```
+
+---
+
+## NameRes Assertions
+
+These assertions test the [NameRes](https://name-lookup.transltr.io/docs) service.
+
+### SearchByName
+
+**Applies to:** NameRes
+
+Each param_set must have at least two elements: a search query string and an expected CURIE. The test passes if the CURIE's normalized identifier appears within the top N results (default N=5) when NameRes looks up the search query.
+
+**Parameters:** Each param_set: the **search query string** and the **expected CURIE**. The CURIE is normalized via NodeNorm (drug/chemical conflation enabled) before matching.
+
+**Wiki syntax:**
+```
+{{BabelTest|SearchByName|water|CHEBI:15377}}
+```
+
+**YAML syntax:**
+```yaml
+babel_tests:
+  SearchByName:
+    - [water, CHEBI:15377]
+    - [diabetes, MONDO:0005015]
+```
+
+---
+
+## Special Assertions
+
+### Needed
+
+**Applies to:** NodeNorm and NameRes
+
+Marks an issue as needing a test — always fails as a reminder to add real assertions.
+
+**Wiki syntax:**
+```
+{{BabelTest|Needed}}
+```
+
+**YAML syntax:**
+```yaml
+babel_tests:
+  Needed:
+    - placeholder
+```
+
+---
+
+## Adding a New Assertion Type
+
+1. Choose the right module:
+   - `nodenorm.py` — for NodeNorm-only assertions (subclass `NodeNormTest`, override `test_param_set`)
+   - `nameres.py` — for NameRes-only assertions (subclass `NameResTest`, override `test_param_set`)
+   - `common.py` — for assertions that apply to both services (subclass `AssertionHandler`, override `test_with_nodenorm` and/or `test_with_nameres`)
+
+2. Define the class with `NAME`, `DESCRIPTION`, `PARAMETERS`, `WIKI_EXAMPLES`, `YAML_PARAMS`, and `test_param_set()` (or both `test_with_*` methods for `AssertionHandler` subclasses).
+
+3. Import it in `__init__.py` and add an instance to `ASSERTION_HANDLERS`.
+
+4. Run `uv run python -m src.babel_validation.assertions.gen_docs` to regenerate `README.md`.