From 8a123c0a0e54d4f83f839cc2fdd769a1acf26100 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Mon, 27 Jul 2026 02:13:24 -0400 Subject: [PATCH 1/7] Add documentation/Babel.md, explaining where NodeNorm's data comes from NodeNorm does not decide which identifiers are equivalent, which one is preferred, or what a clique is called -- Babel does. That means most "why did I get this answer?" questions are really questions about Babel, and until now nothing in this repository said so. This adds a single page covering the Babel behaviour that shows up directly in a NodeNorm response: cliques and preferred identifiers, why the label is not necessarily the preferred identifier's label, information content, the conflation ordering guarantees, where descriptions come from, four gotchas that surprise people, and which tracker to file against. It is also deliberately the *only* file in this repository that links into Babel's docs/ tree. Everything else links here instead, so a reorganisation on the Babel side breaks one file rather than twenty. Co-Authored-By: Claude Opus 5 --- documentation/Babel.md | 120 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 120 insertions(+) create mode 100644 documentation/Babel.md diff --git a/documentation/Babel.md b/documentation/Babel.md new file mode 100644 index 0000000..89980c1 --- /dev/null +++ b/documentation/Babel.md @@ -0,0 +1,120 @@ +# Where NodeNorm's data comes from + +NodeNorm does not decide which identifiers are equivalent, which one is preferred, or what a clique +is called. All of that is computed by [Babel](https://github.com/NCATSTranslator/Babel), a separate +pipeline, and loaded into NodeNorm's Redis databases as a fixed snapshot. NodeNorm is the query +layer over that snapshot. + +This page collects the Babel behaviour that shows up directly in a NodeNorm response, so that you +can answer "why did I get *this* answer?" without reading the Babel pipeline. It is the only page in +this repository that links into Babel's documentation; everything else here links to this page. + +## Which Babel build am I querying? + +`GET /status` reports `babel_version` (a date-based build name such as `2025sep1`) and +`babel_version_url`, a link to that build's release notes. Every NodeNorm deployment serves exactly +one Babel build: the backend databases are written once at load time and never updated in place, so +results cannot change until an operator loads a newer build. If you need reproducible results, +record the `babel_version` alongside them. + +Published Babel builds can be downloaded directly — see Babel's +[Downloads.md](https://github.com/NCATSTranslator/Babel/blob/master/docs/Downloads.md). + +## Cliques, preferred identifiers, and labels + +A **clique** is a set of identifiers Babel believes name the same concept. Normalizing any member +returns the whole clique. + +The **preferred identifier** (`id.identifier`) is the first identifier in the clique, ordered by the +Biolink Model's `id_prefixes` list for that clique's type. For a +[`biolink:SmallMolecule`](https://biolink.github.io/biolink-model/SmallMolecule/#valid-id-prefixes), +that means CHEBI before UNII, and so on. `equivalent_identifiers` is returned in the same order. + +The **preferred label** (`id.label`) is Babel's `preferred_name` for the clique, and it is **not +necessarily the label of the preferred identifier**. Babel may pick a label from a different +identifier to give a clearer name. Two rules drive this: + +- Chemical types boost a specific prefix list when choosing a label — currently DRUGBANK, + DrugCentral, CHEBI, MESH, GTOPDB — rather than using the Biolink prefix order. +- Labels longer than a configured length are demoted, and used only if nothing shorter exists. + +Babel's [Understanding.md](https://github.com/NCATSTranslator/Babel/blob/master/docs/Understanding.md) +has the full rules for both. + +## Information content + +`information_content` is a decimal from 0.0 to 100.0 describing how specific a concept is: **0.0 is +a broad, high-level term with many subclasses; 100.0 is a very specific term with none**. Babel +takes these from [Ubergraph](https://github.com/INCATools/ubergraph/?tab=readme-ov-file#graph-organization)'s +`normalizedInformationContent`, which is precomputed per ontology class from the count of terms +related to it via `rdfs:subClassOf` or any existential relation. + +The value is per clique, not per identifier: where several identifiers in a clique have an IC value, +Babel stores the lowest of them, and NodeNorm returns that stored value unchanged. Not every clique +has one — if no identifier in a clique is known to Ubergraph, the field is omitted entirely. + +## Conflation + +Conflation merges cliques that are not identical but that you may want treated as one concept. Babel +publishes the groupings; NodeNorm applies them at query time when you ask for them. + +- **GeneProtein** (`conflate`) merges a gene with the protein it encodes. **The gene always comes + first**, so the preferred identifier of a conflated gene/protein clique is the gene. +- **DrugChemical** (`drug_chemical_conflate`) merges a drug with its active ingredient. Babel orders + these so that **the active ingredient precedes any formulations**. + +Two consequences worth knowing: + +- **Babel assigns conflated cliques no type at all.** NodeNorm computes the `type` list at query + time, starting from the most specific type of the first identifier, adding its Biolink ancestors, + then adding the types and ancestors of every other member. +- **The clique boundaries are lost in the response.** `equivalent_identifiers` is a single flat + list — the members of the first clique, then the second, and so on — with no marker for where one + ends. There is currently no way to recover the original clique leaders + ([#320](https://github.com/NCATSTranslator/NodeNormalization/issues/320)); `individual_types=true` + will at least give you a Biolink type per identifier. + +See Babel's [Conflation.md](https://github.com/NCATSTranslator/Babel/blob/master/docs/Conflation.md). + +## Descriptions + +All descriptions come from [UberGraph](https://github.com/INCATools/ubergraph/), so most identifiers +have none. They are off by default; pass `description=true` to include them. + +## Gotchas + +**An identifier missing from a clique may be missing by design.** When Babel writes a compendium it +keeps only identifiers whose prefix appears in the Biolink Model's `id_prefixes` for that clique's +type, and silently drops the rest. A CURIE that Babel knows about can therefore be legitimately +absent from the clique you get back, particularly for types with short prefix lists. + +**A CURIE NodeNorm cannot normalize maps to `null`, it is not omitted.** Every CURIE you send back +comes back as a key in the response object; check for `null` values rather than for missing keys. + +**Information content from builds before `2025sep1` is unreliable.** Babel compared IC values as +strings rather than numbers until then, so some cliques were assigned an IC of `100` when the true +value was lower. Check `/status` before trusting IC from an older deployment. + +**A clique can be wrong in two directions.** A *split* clique is two cliques that should be one; a +*lumped* clique is one clique holding identifiers for genuinely different concepts. Both are Babel +issues, not NodeNorm issues — see below. + +## Reporting a problem + +| What is wrong | Where to file | +|---|---| +| Wrong identifiers in a clique, wrong Biolink type, wrong preferred label, bad description | [Babel](https://github.com/NCATSTranslator/Babel/issues/) | +| NodeNorm returns an error, behaves oddly, or the API itself is wrong | [NodeNorm](https://github.com/NCATSTranslator/NodeNormalization/issues/) | + +When filing against Babel, a link to a NodeNorm query showing the problem is very helpful. Babel's +[NewIssue.md](https://github.com/NCATSTranslator/Babel/blob/master/docs/NewIssue.md) describes what +to include and how issues are prioritized. + +## Going deeper + +Everything above is what leaks into a NodeNorm response. How Babel actually builds cliques — the +cross-reference concords, the transitive merge, per-source ingest quirks, the DuckDB and Parquet +exports, and the pipeline itself — is documented in +[Babel's documentation index](https://github.com/NCATSTranslator/Babel/blob/master/docs/README.md). +The output file formats NodeNorm's loader reads are specified in +[DataFormats.md](https://github.com/NCATSTranslator/Babel/blob/master/docs/DataFormats.md). From 38b5bc1eef900af4f454abfc419d692359fe500f Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Mon, 27 Jul 2026 02:14:08 -0400 Subject: [PATCH 2/7] Correct stale documentation and point it at documentation/Babel.md The README carried two examples that no longer describe anything real. The "compendia file" sample predated Biolink-prefixed types (it showed chemical_substance and a bare PUBCHEM: prefix) and was in any case a normalization *response*, not a compendium line. The config.json sample listed Redis credentials, which the loader reads only from redis_config.yaml, and old-style filenames the pipeline no longer produces. Both are replaced by pointers to the documents that are actually maintained. API.md gains the things it had fallen behind on: the include_taxa and taxa fields, the defaults for every parameter, the note that a CURIE which cannot be normalized comes back as null rather than being omitted, the biolink_model block returned by /status, and a finished sentence for the deprecated TRAPI section, which previously trailed off after "These methods". It also now flags that drug_chemical_conflate defaults to true on GET but false on POST (#398). Attribution of the information content calculation is corrected in passing: Babel takes the lowest value of any identifier in the clique (min() in its src/node.py) and NodeNorm returns that stored value unchanged. Finally, the root path is recorded as / rather than /1.3 or /1.5/. The TRAPI-versioned prefixes are deprecated and will be removed; the only place one should still appear is the OpenAPI servers block, which is itself temporary. Co-Authored-By: Claude Opus 5 --- CLAUDE.md | 2 +- README.md | 52 ++++++++++++----------- documentation/API.md | 81 +++++++++++++++++++++++++++++------- documentation/CONTRIBUTOR.md | 13 +++--- 4 files changed, 103 insertions(+), 45 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index e213c6a..8d7f5cc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -71,7 +71,7 @@ Babel Compendia Files → loader.py → Redis (7 DBs) → FastAPI (server.py) ### Key Modules -- **`node_normalizer/server.py`** — FastAPI app with all REST endpoints. Uses lifespan events for Redis connection setup/teardown. Root path is `/1.3`. +- **`node_normalizer/server.py`** — FastAPI app with all REST endpoints. Uses lifespan events for Redis connection setup/teardown. Root path is `/`. - **`node_normalizer/normalizer.py`** — Core logic: `get_normalized_nodes()`, `normalize_message()` (for TRAPI), and equivalent CURIE discovery. Traverses Biolink Model ancestors for semantic type expansion. - **`node_normalizer/loader/`** — the data loader: synchronous module-level functions (`load_all`, `load_compendium`, `load_conflation`, `merge_semantic_meta_data`) that read flat compendia files and populate Redis. Validates input against `resources/valid_data_format.json`. Batch size: 100,000. Invoked via the root `load.py`. `load_all` disables Redis periodic saves (`CONFIG SET save ""`) for the duration of the load — the backend DBs are write-once and persisted by a manual BGSAVE, so snapshotting during the load is wasted work. See `documentation/Loader.md`. - **`node_normalizer/config.py`** — shared repo-relative paths (`config.json`, `redis_config.yaml`, `resources/`) and `get_config()`. Used by both frontend and loader. diff --git a/README.md b/README.md index 4f7637b..9f70775 100644 --- a/README.md +++ b/README.md @@ -14,11 +14,15 @@ Node normalization takes a CURIE, and returns: * All other known equivalent identifiers for the entity * Semantic types for the entity as defined by the [Biolink Model](https://biolink.github.io/biolink-model/) -The data currently served by Node Normalization is created by the prototype project -[Babel](https://github.com/NCATSTranslator/Babel), which attempts to find identifier equivalences, -and makes sure that CURIE prefixes are BioLink Model compliant. The NodeNormalization service, however, -is independent of Babel and as improved identifier equivalence tools are developed, their results -can be easily incorporated. +The data served by Node Normalization is created by [Babel](https://github.com/NCATSTranslator/Babel), +which finds identifier equivalences and makes sure that CURIE prefixes are Biolink Model compliant. +NodeNorm itself is independent of Babel, and as improved identifier equivalence tools are developed +their results can be easily incorporated. + +Babel decides which identifiers are equivalent, which one is preferred, and what a clique is called — +so most "why did I get this answer?" questions are really questions about Babel. +[Where NodeNorm's data comes from](documentation/Babel.md) explains the parts of Babel that show up +in a NodeNorm response. To determine whether Node Normalization is likely to be useful, check /get_semantic_types, which lists the BioLink semantic types for which normalization has been attempted, and /get_curie_prefixes, @@ -46,11 +50,12 @@ Install requirements ``` ## Generating equivalence data -The equivalence data can be generated by running [Babel](https://github.com/NCATSTranslator/Babel). An example of the contents of a compendia file is shown below: -``` - {"id": {"identifier": "PUBCHEM:50986940"}, "equivalent_identifiers": [{"identifier": "PUBCHEM:50986940"}, {"identifier": "INCHIKEY:CYMOSKLLKPIPCD-UHFFFAOYSA-N"}], "type": ["chemical_substance", "named_thing", "biological_entity", "molecular_entity"]} - {"id": {"identifier": "CHEMBL.COMPOUND:CHEMBL1546789", "label": "CHEMBL1546789"}, "equivalent_identifiers": [{"identifier": "CHEMBL.COMPOUND:CHEMBL1546789", "label": "CHEMBL1546789"}, {"identifier": "PUBCHEM:4879549"}, {"identifier": "INCHIKEY:FUIYIXDZTPMQEH-UHFFFAOYSA-N"}], "type": ["chemical_substance", "named_thing", "biological_entity", "molecular_entity"]} -``` +The equivalence data can be generated by running [Babel](https://github.com/NCATSTranslator/Babel), +or downloaded from a published Babel build. The loader reads Babel's compendium and conflation +files; both formats are specified in +[Where NodeNorm's data comes from](documentation/Babel.md), which links to Babel's format +documentation. + ## Creating and loading a Redis container with data A running instance of Redis is needed to house the node normalization data. a Redis Docker container image can be downloaded from [Docker hub](https://hub.docker.com/_/redis). The Redis caonteriner can be started with thie following docker command: @@ -59,21 +64,13 @@ A running instance of Redis is needed to house the node normalization data. a Re ``` Note that the dataset for Node normalization is quite large and 256Gb of memory and disk space should be made available to the Redis instance to insure proper loading of the complete compendia. ### Configuration -Insure that the `./config.json` file is created and contains the parameters for the node normalization load specific to your environment. -The configuration parameters `compendium_directory` and `data_files` specify the location of the compendia files. An example of the files' contents -are listed below: -``` - { - "compendium_directory": "", - "data_files": "anatomy.txt,BiologicalProcess.txt,cell.txt,cellular_component.txt,disease.txt,gene_compendium.txt,gene_family_compendium.txt,MolecularActivity.txt,pathways.txt,phenotypes.txt,taxon_compendium.txt", - "redis_host": "", - "redis_port": , - "redis_password": " Date: Mon, 27 Jul 2026 02:14:33 -0400 Subject: [PATCH 3/7] Serve the endpoint documentation that openapi.yml was silently discarding construct_open_api_schema() regenerates paths from the FastAPI route decorators and copies back only info, tags and servers. Everything under paths: in resources/openapi.yml -- 235 lines of it -- was therefore never served to anyone. That included the only written statement of the conflation ordering guarantees ("the gene(s) always appear first", "the active ingredient appears before any formulations") and the /get_setid stability guarantee, neither of which existed anywhere else in the repository. That prose now lives on the decorators, where it is actually rendered, and the dead paths: block is deleted. Every operation gains a real 200 description in place of FastAPI's default "Successful Response", and apidocs.py grows a comment saying where per-endpoint documentation belongs so this does not happen again. The response of /get_normalized_nodes is now described by NormalizedNode, NormalizedNodeId and EquivalentIdentifier, plus a worked example. These are attached via responses={200: {"model": ...}} rather than response_model=, deliberately: create_node() builds those dictionaries by hand and omits keys that do not apply, so letting FastAPI filter its output through a model would risk silently dropping a field the model forgot. The schema is documented; serialization is untouched. Also fixed along the way: - The MIT license in openapi.yml was parsed and then dropped, so the served schema had no license. It is now copied across. - apidocs.py called app.openapi_schema() on a dict, which would raise TypeError. Unreachable today only because server.py assigns the attribute directly, but it would fire the moment anything called app.openapi(). - CurieList used title= rather than description= on every field. Swagger renders titles as labels, not help text, so those fields showed no description at all. Its example also omitted two parameters and contradicted a declared default. - The /query and /asyncquery summaries both claimed to normalize a "TRAPI response object"; /query takes a Query, and the two endpoints are not identical. Both now say what they do and name their replacement. Co-Authored-By: Claude Opus 5 --- node_normalizer/apidocs.py | 11 +- node_normalizer/examples.py | 31 ++++ node_normalizer/model/__init__.py | 10 +- node_normalizer/model/input.py | 27 ++- node_normalizer/model/response.py | 146 ++++++++++++++- node_normalizer/resources/openapi.yml | 246 +------------------------- node_normalizer/server.py | 141 ++++++++++++--- 7 files changed, 332 insertions(+), 280 deletions(-) diff --git a/node_normalizer/apidocs.py b/node_normalizer/apidocs.py index 04e34eb..f7f7078 100644 --- a/node_normalizer/apidocs.py +++ b/node_normalizer/apidocs.py @@ -1,7 +1,11 @@ """ Open API configuration -# TODO use examples from openapi.yml to drive examples +`openapi.yml` supplies only the document-level metadata: title, version, description, contact, +license, terms of service, the Translator/TRAPI extensions, servers and tags. The `paths` section +is generated from the FastAPI route decorators in `server.py` and cannot be overridden here, so +per-endpoint documentation belongs on the decorators (`summary`, `description`, +`response_description`) rather than in this file. """ from pathlib import Path @@ -38,7 +42,7 @@ def construct_open_api_schema(app) -> Dict[str, str]: api_docs = load(apd_file, Loader=SafeLoader) if app.openapi_schema: - return app.openapi_schema() + return app.openapi_schema open_api_schema = get_openapi( title=api_docs['info']['title'], @@ -64,6 +68,9 @@ def construct_open_api_schema(app) -> Dict[str, str]: if 'description' in api_docs['info']: open_api_schema['info']['description'] = api_docs['info']['description'] + if 'license' in api_docs['info']: + open_api_schema['info']['license'] = api_docs['info']['license'] + # adds support to override server root path server_root = os.environ.get('SERVER_ROOT', '/') # make sure not to add double slash at the end. diff --git a/node_normalizer/examples.py b/node_normalizer/examples.py index aba64c9..48bfba2 100644 --- a/node_normalizer/examples.py +++ b/node_normalizer/examples.py @@ -138,3 +138,34 @@ ], } } + + +# An example response from /get_normalized_nodes, used to document the endpoint. Shows a CURIE that +# normalizes (MESH:D014867 -> CHEBI:15377) and one that does not (RUBBISH:1234 -> null). +EXAMPLE_NORMALIZED_NODES = { + "MESH:D014867": { + "id": { + "identifier": "CHEBI:15377", + "label": "Water", + }, + "equivalent_identifiers": [ + {"identifier": "CHEBI:15377", "label": "water", "type": "biolink:SmallMolecule"}, + {"identifier": "UNII:059QF0KO0R", "label": "WATER", "type": "biolink:SmallMolecule"}, + {"identifier": "PUBCHEM.COMPOUND:962", "label": "Water", "type": "biolink:SmallMolecule"}, + {"identifier": "MESH:D014867", "label": "Water", "type": "biolink:SmallMolecule"}, + ], + "type": [ + "biolink:SmallMolecule", + "biolink:MolecularEntity", + "biolink:ChemicalEntity", + "biolink:PhysicalEssence", + "biolink:ChemicalOrDrugOrTreatment", + "biolink:ChemicalEntityOrGeneOrGeneProduct", + "biolink:ChemicalEntityOrProteinOrPolypeptide", + "biolink:NamedThing", + "biolink:PhysicalEssenceOrOccurrent", + ], + "information_content": 47.7, + }, + "RUBBISH:1234": None, +} diff --git a/node_normalizer/model/__init__.py b/node_normalizer/model/__init__.py index 457a536..5f9ba4b 100644 --- a/node_normalizer/model/__init__.py +++ b/node_normalizer/model/__init__.py @@ -3,4 +3,12 @@ """ from .input import CurieList, SemanticTypesInput, SetIDQuery -from .response import CuriePivot, SemanticTypes, ConflationList, SetIDResponse +from .response import ( + CuriePivot, + SemanticTypes, + ConflationList, + SetIDResponse, + EquivalentIdentifier, + NormalizedNodeId, + NormalizedNode, +) diff --git a/node_normalizer/model/input.py b/node_normalizer/model/input.py index 6a816ec..3fda9f9 100644 --- a/node_normalizer/model/input.py +++ b/node_normalizer/model/input.py @@ -12,33 +12,40 @@ class CurieList(BaseModel): curies: List[str] = Field( ..., # Ellipsis means field is required - title='List of CURIEs to normalize', + description="The CURIEs to normalize. Any CURIE that cannot be normalized is returned with a null value " + "rather than being left out of the response.", min_items=1 ) - conflate:bool = Field( + conflate: bool = Field( True, - title="Whether to apply gene/protein conflation" + description="Whether to apply GeneProtein conflation, which merges a gene with the protein it encodes. " + "The gene always comes first in the combined clique." ) description: bool = Field( False, - title="Whether to return CURIE descriptions when possible" + description="Whether to return CURIE descriptions when possible. Descriptions currently come only from " + "UberGraph, so most identifiers have none." ) drug_chemical_conflate: bool = Field( False, - title="Whether to apply drug/chemical conflation" + description="Whether to apply DrugChemical conflation, which merges a drug with its active ingredient. " + "The active ingredient comes before any formulations in the combined clique. " + "Note that this defaults to false here but true on the GET method of this endpoint " + "(https://github.com/NCATSTranslator/NodeNormalization/issues/398)." ) individual_types: bool = Field( False, - title="Whether to return individual types for equivalent identifiers" + description="Whether to return the Biolink type of each equivalent identifier. Useful for telling apart " + "the members of a conflated clique, which are otherwise returned as one flat list." ) include_taxa: bool = Field( True, - title="Whether to return taxa for equivalent identifiers" + description="Whether to return the taxa associated with each equivalent identifier, as NCBITaxon CURIEs." ) class Config: @@ -48,6 +55,8 @@ class Config: "conflate": True, "description": False, "drug_chemical_conflate": True, + "individual_types": False, + "include_taxa": True, } } @@ -57,7 +66,7 @@ class SemanticTypesInput(BaseModel): semantic_types: List[str] = Field( ..., # required field - title='list of semantic types', + description="The Biolink semantic types to report on. Pass an empty list for every semantic type.", ) class Config: @@ -79,6 +88,6 @@ class SetIDQuery(BaseModel): conflations: List[str] = Field( [], - description="Set of conflations to apply", + description="Set of conflations to apply. See /get_allowed_conflations for the valid values.", example=["GeneProtein", "DrugChemical"], ) diff --git a/node_normalizer/model/response.py b/node_normalizer/model/response.py index 60d89c5..6206fff 100644 --- a/node_normalizer/model/response.py +++ b/node_normalizer/model/response.py @@ -1,8 +1,15 @@ """ API Response Models not described in reasoner-pydantic + +Note that the normalization models below (`EquivalentIdentifier`, `NormalizedNodeId`, +`NormalizedNode`) are used to *document* the response of /get_normalized_nodes, via the +`responses=` argument of the route decorators, rather than as a `response_model=`. They are +deliberately not enforced: `create_node()` builds these dictionaries by hand and omits keys that +don't apply, and making FastAPI filter its output through a model risks silently dropping a field +the model forgot. Keep them in sync with `create_node()` in `normalizer.py`. """ -from pydantic import BaseModel +from pydantic import BaseModel, Field from typing import Dict, List, Optional @@ -25,20 +32,141 @@ class Config: class CuriePivot(BaseModel): - curie_prefix: Dict[str, str] + curie_prefix: Dict[str, str] = Field( + ..., + description="A mapping from CURIE prefix to the number of times it appears in an equivalent identifier " + "for this semantic type. Note that the counts are strings, not numbers.", + ) + + class Config: + schema_extra = { + "example": { + "curie_prefix": { + "PUBCHEM.COMPOUND": "119397095", + "INCHIKEY": "115661650", + "CHEBI": "200507", + } + } + } class ConflationList(BaseModel): - conflations: List + conflations: List[str] = Field( + ..., + description="The conflations supported by this NodeNorm instance.", + ) + + class Config: + schema_extra = {"example": {"conflations": ["GeneProtein", "DrugChemical"]}} + + +class EquivalentIdentifier(BaseModel): + """A single identifier belonging to a normalized clique.""" + + identifier: str = Field(..., description="The CURIE for this identifier.") + label: Optional[str] = Field( + None, + description="The label for this identifier as given by its authoritative source. Omitted if that source " + "provides no label.", + ) + description: Optional[str] = Field( + None, + description="A description of this identifier, currently always from UberGraph. Only returned when the " + "`description` parameter is set, and omitted when no description is known.", + ) + taxa: Optional[List[str]] = Field( + None, + description="The taxa this identifier is associated with, as NCBITaxon CURIEs. Only returned when the " + "`include_taxa` parameter is set, and omitted when no taxa are known.", + ) + type: Optional[str] = Field( + None, + description="The most specific Biolink type of this individual identifier. Only returned when the " + "`individual_types` parameter is set. Useful for telling apart the members of a conflated " + "clique, e.g. which identifiers are the gene and which the protein.", + ) + + +class NormalizedNodeId(BaseModel): + """The preferred identifier and label for a normalized clique.""" + + identifier: str = Field( + ..., + description="The preferred CURIE for this clique: the first identifier in the Biolink Model's preferred " + "prefix order for this clique's type. With a conflation applied, this is the preferred " + "identifier of the first clique being conflated (for GeneProtein, always the gene).", + ) + label: Optional[str] = Field( + None, + description="The preferred label for this clique, as computed by Babel. Note that this is not necessarily " + "the label of the preferred identifier — for chemicals in particular, a label may be taken from " + "a different identifier. Omitted if nothing in the clique has a label.", + ) + description: Optional[str] = Field( + None, + description="One of the descriptions for the identifiers in this clique. Only returned when the " + "`description` parameter is set.", + ) + + +class NormalizedNode(BaseModel): + """A normalized clique: everything NodeNorm knows about one concept.""" + + id: NormalizedNodeId = Field(..., description="The preferred identifier and label for this clique.") + equivalent_identifiers: List[EquivalentIdentifier] = Field( + ..., + description="Every identifier in this clique, in the Biolink Model's preferred prefix order for this type. " + "When a conflation is applied this is a single flat list — all the identifiers of the first " + "clique, then those of the second, and so on — with no marker for where one clique ends and the " + "next begins.", + ) + type: List[str] = Field( + ..., + description="The Biolink classes for this clique, starting with the most specific type and ending with any " + "mixins. Note that `biolink:Entity` is deliberately excluded.", + ) + descriptions: Optional[List[str]] = Field( + None, + description="The unique descriptions of the identifiers in this clique, in the same order as " + "`equivalent_identifiers`. Only returned when the `description` parameter is set.", + ) + taxa: Optional[List[str]] = Field( + None, + description="Every taxon associated with any identifier in this clique, as NCBITaxon CURIEs. Only returned " + "when the `include_taxa` parameter is set.", + ) + information_content: Optional[float] = Field( + None, + description="How specific this concept is, from 0.0 (a broad, high-level term with many subclasses) to " + "100.0 (a very specific term with none). Taken from Ubergraph's normalizedInformationContent; " + "where several identifiers in the clique have a value, the lowest is reported. Omitted when no " + "identifier in the clique has one.", + ) class SetIDResponse(BaseModel): - curies: List[str] - conflations: List[str] - error: Optional[str] - normalized_curies: Optional[List[str]] - normalized_string: Optional[str] - setid: Optional[str] + curies: List[str] = Field(..., description="The CURIEs submitted for normalization, as submitted.") + conflations: List[str] = Field(..., description="The conflations applied, as submitted.") + error: Optional[str] = Field( + None, + description="Any error that occurred while normalizing this set. Note that a CURIE that simply cannot be " + "normalized is not an error.", + ) + normalized_curies: Optional[List[str]] = Field( + None, + description="The unique, sorted CURIEs used to construct the set ID. CURIEs that could not be normalized " + "are included unchanged.", + ) + normalized_string: Optional[str] = Field( + None, + description="The normalized CURIEs joined with `||`; this is the string the set ID is calculated from.", + ) + setid: Optional[str] = Field( + None, + description="The set ID: a UUID (prefixed with `uuid:`) derived from `normalized_string`. The same set of " + "CURIEs given to the same version of NodeNorm always produces the same set ID, but a different " + "Babel build may normalize them differently and so produce a different one.", + ) # base64: Optional[str] # base64zlib: Optional[str] # sha224hash: Optional[str] \ No newline at end of file diff --git a/node_normalizer/resources/openapi.yml b/node_normalizer/resources/openapi.yml index a2bf957..06658c7 100644 --- a/node_normalizer/resources/openapi.yml +++ b/node_normalizer/resources/openapi.yml @@ -12,13 +12,16 @@ info:
  • Semantic types for the entity as defined by the Biolink Model The data served by Node Normalization is created by Babel, which attempts to find identifier equivalences, and makes sure that CURIE prefixes - are BioLink Model Compliant. To determine whether Node Normalization is likely + are BioLink Model Compliant. Babel decides which identifiers are equivalent, which one is + preferred, and what a clique is called, so most questions about why a clique looks the way it + does are really questions about Babel: see + Where NodeNorm''s data comes from. + The babel_version reported by /status identifies exactly which Babel build this + instance is serving. To determine whether Node Normalization is likely to be useful, check /get_semantic_types, which lists the BioLink semantic types for which normalization has been attempted, and /get_curie_prefixes, which lists the number of times each prefix is used for a semantic type. You can find out more about these API methods at the - Node Normalization API documentation. - To learn more about how Babel creates cliques of equivalent identifiers, you can - read its documentation.' + Node Normalization API documentation.' license: name: MIT url: https://opensource.org/licenses/MIT @@ -43,238 +46,3 @@ tags: - name: translator - name: Interfaces - name: trapi -paths: - /status: - get: - description: 'Returns the status of the Node Normalization service. You can read more about this endpoint in the - NodeNorm API documentation.' - responses: - '200': - description: Successful result describing this NodeNorm instance. - content: - application/json: - schema: - example: - status: running - babel_version: 2025mar31 - babel_version_url: https://github.com/NCATSTranslator/Babel/blob/master/releases/2025mar31.md - databases: - eq_id_to_id_db: - dbname: id-id - count: 677731045 - used_memory_rss_human: 68.83G - is_cluster: false - /get_curie_prefixes: - get: - description: 'Returns the curies and their hit count for a semantic type(s). You can read more about this endpoint - in the NodeNorm API documentation.' - parameters: - - in: query - name: semantic_type - schema: - example: - - biolink:ChemicalEntity - - biolink:AnatomicalEntity - items: - type: string - type: array - responses: - '200': - content: - application/json: - schema: - example: - "biolink:ChemicalEntity": - curie_prefix: - "PUBCHEM.COMPOUND": "123887334" - "INCHIKEY": "115975484" - "CAS": "4112274" - "HMDB": "217920" - "CHEMBL.COMPOUND": "2479770" - "UNII": "138975" - "CHEBI": "218762" - "MESH": "256235" - "UMLS": "603550" - "DrugCentral": "4995" - "GTOPDB": "13265" - "RXCUI": "124800" - "DRUGBANK": "15274" - "KEGG.COMPOUND": "16039" - "biolink:AnatomicalEntity": - curie_prefix: - "UMLS": "159941" - "FMA": "98631" - "UBERON": "14564" - "ZFA": "606" - "NCIT": "10286" - "MESH": "1992" - "EMAPA": "966" - "FBbt": "117" - "WBbt": "18" - "GO": "4041" - "SNOMEDCT": "1422" - "CL": "3043" - type: object - description: Results - summary: Return the number of times each CURIE prefix appears in an equivalent - identifier for a semantic type - tags: - - Interfaces - /get_normalized_nodes: - get: - description: 'Returns the equivalent identifiers and semantic types for the CURIEs entered. - You can optionally conflate identifiers if needed. - You can read more about this endpoint in the - NodeNorm API documentation.' - parameters: - - in: query - name: curie - schema: - example: - - MESH:D014867 - - NCIT:C34373 - items: - type: string - type: array - responses: - '200': - content: - application/json: - schema: - example: - MESH:D014867: - id: - identifier: CHEBI:15377 - label: Water - equivalent_identifiers: - - identifier: CHEBI:15377 - label: Water - - identifier: CHEMBL.COMPOUND:CHEMBL1098659 - label: WATER - - identifier: DRUGBANK:DB09145 - - identifier: PUBCHEM:22247451 - - identifier: PUBCHEM:962 - - identifier: MESH:D014867 - - identifier: HMDB:HMDB0002111 - - identifier: INCHIKEY:XLYOFNOQVPJJNP-UHFFFAOYSA-N - - identifier: UNII:059QF0K00R - - identifier: KEGG.COMPOUND:C00001 - type: - - biolink:SmallMolecule - - biolink:MolecularEntity - - biolink:ChemicalEntity - - biolink:PhysicalEssence - - biolink:ChemicalOrDrugOrTreatment - - biolink:ChemicalEntityOrGeneOrGeneProduct - - biolink:ChemicalEntityOrProteinOrPolypeptide - - biolink:NamedThing - - biolink:PhysicalEssenceOrOccurrent - information_content: 47.5 - NCIT:C34373: - equivalent_identifiers: - - identifier: MONDO:0004976 - label: amyotrophic lateral sclerosis - - identifier: DOID:332 - - identifier: OPPHANET:803 - - identifier: UMLS:C0393554 - - identifier: MESH:D000690 - - identifier: MEDDRA:10002026 - - identifier: NCIT:C34373 - - identifier: HP:0007354 - label: Amyotrophic lateral sclerosis - id: - identifier: MONDO:0004976 - label: amyotrophic lateral sclerosis - type: - - biolink:Disease - - biolink:DiseaseorPhenotypicFeature - - biolink:BiologicalEntity - - biolink:ThingWithTaxon - - biolink:NamedThing - information_content: 74.9 - type: object - description: Results - summary: Get the equivalent identifiers and semantic types for the CURIEs entered. - tags: - - Interfaces - /get_semantic_types: - get: - description: 'Returns a distinct set of the semantic types discovered in the - compendium data. You can read more about this endpoint in the - NodeNorm API documentation.' - responses: - '200': - content: - application/json: - schema: - example: - semantic_types: - types: - - biolink:ChemicalMixture - - biolink:MacromolecularMachineMixin - - biolink:GeographicLocation - - biolink:Agent - - biolink:Protein - - biolink:DiseaseOrPhenotypicFeature - - etc. - type: object - description: Results - summary: Return a list of BioLink semantic types for which normalization has - been attempted. - tags: - - Interfaces - /get_allowed_conflations: - get: - description: 'Returns a list of allowed conflation options. Conflation allows cliques to be combined on-the-fly - on the basis of two different criteria: -
      -
    1. GeneProtein conflation merges protein-coding genes with the proteins they encode. The gene(s) always - appear first in the combined clique. -
    2. DrugChemical conflation merges chemicals based on their active ingredient. We attempt to ensure that - the active ingredient appears before any formulations in the combined clique. -
    - You can read more about conflation or - more about this endpoint.' - responses: - '200': - content: - application/json: - schema: - example: - conflations: - - GeneProtein - - DrugChemical - type: object - description: Results - summary: Return a list of named conflations. - tags: - - Interfaces - /get_setid: - get: - description: 'Returns the set ID for a given set of CURIEs. A set ID is an identifier that can be used to identify - this set of CURIEs going forward. It is currently impossible to recreate a set of CURIEs from a set ID, but the - same set of CURIEs given to the same version of Node Normalization will always return the same set ID. - You can read more about this endpoint in the - NodeNorm API documentation.' - responses: - '200': - description: Successful result - content: - application/json: - schema: - example: - curies: - - MESH:D014867 - - NCIT:C34373 - - UNII:63M8RYN44N - - RUBBISH:1234 - conflations: - - GeneProtein - - DrugChemical - error: null - normalized_curies: - - CHEBI:15377 - - MONDO:0004976 - - RUBBISH:1234 - normalized_string: "CHEBI:15377||MONDO:0004976||RUBBISH:1234" - setid: "uuid:771d3c09-9a8c-5c46-8b85-97f481a90d40" diff --git a/node_normalizer/server.py b/node_normalizer/server.py index c0ace63..652eb7d 100644 --- a/node_normalizer/server.py +++ b/node_normalizer/server.py @@ -26,12 +26,13 @@ ConflationList, SetIDResponse, SetIDQuery, + NormalizedNode, ) from .normalizer import get_normalized_nodes, get_curie_prefixes, normalize_message from .set_id import generate_setid from .redis_adapter import RedisConnectionFactory from .util import LoggingUtil -from .examples import EXAMPLE_QUERY_DRUG_TREATS_ESSENTIAL_HYPERTENSION +from .examples import EXAMPLE_QUERY_DRUG_TREATS_ESSENTIAL_HYPERTENSION, EXAMPLE_NORMALIZED_NODES logger = LoggingUtil.init_logging() @@ -96,7 +97,15 @@ async def shutdown_event(): @app.get( "/status", summary="Status information on this NodeNorm instance", - description="Returns information about this NodeNorm instance and the databases it is connected to." + description="Returns information about this NodeNorm instance and the databases it is connected to, including " + "the version of Babel whose output is loaded " + "into those databases (babel_version) and the Biolink Model version used to expand " + "semantic types (biolink_model). The backend databases are written once and never " + "updated in place, so babel_version identifies the data this instance will return until " + "an operator loads a newer build. You can read more about this endpoint in the " + "" + "NodeNorm API documentation.", + response_description="Information about this NodeNorm instance and the databases it is connected to.", ) async def status_get() -> Dict: """ Return status information about this NodeNorm instance as well as its databases. """ @@ -179,8 +188,16 @@ async def status() -> Dict: @app.post( "/query", - summary="Normalizes a TRAPI response object", - description="Returns the response object with a merged knowledge graph and query graph bindings", + summary="Normalizes every identifier in a TRAPI message", + description="Normalizes the identifiers in the knowledge graph, query graph and results of a TRAPI message, " + "returning the message with a merged knowledge graph and updated bindings. Each normalized node also " + "gains the clique's information content as an attribute " + "(biolink:has_numeric_value / information_content). " + "Deprecated: this endpoint is no longer actively maintained and will be removed once " + "the Workflow Runner stops using it " + "(PR #323). New callers " + "should extract the CURIEs they care about and use /get_normalized_nodes instead.", + response_description="The submitted TRAPI message with all identifiers normalized.", response_model=reasoner_pydantic.Query, response_model_exclude_none=True, response_model_exclude_unset=True, @@ -199,8 +216,15 @@ async def query(query: Annotated[reasoner_pydantic.Query, Body(openapi_examples= @app.post( "/asyncquery", - summary="Normalizes a TRAPI response object", - description="Returns the response object with a merged knowledge graph and query graph bindings", + summary="Normalizes every identifier in a TRAPI message, returning the result to a callback URL", + description="Identical to /query, except that it returns as soon as the work has been queued and POSTs the " + "normalized TRAPI message to the callback URL given in the request body when it is " + "ready (retrying a few times if the callback fails). " + "Deprecated: this endpoint is no longer actively maintained and will be removed once " + "the Workflow Runner stops using it " + "(PR #323). New callers " + "should extract the CURIEs they care about and use /get_normalized_nodes instead.", + response_description="Confirmation that the query has been queued; the normalized message is sent to the callback URL.", deprecated=True, ) async def async_query(async_query: reasoner_pydantic.AsyncQuery): @@ -246,8 +270,22 @@ async def async_query_task(async_query: reasoner_pydantic.AsyncQuery): @app.get( "/get_allowed_conflations", - summary="Get the available conflations", - description="The returned strings can be included in an option to /get_normalized_nodes", + summary="Return a list of named conflations.", + description="Returns a list of allowed conflation options. Conflation allows cliques to be combined on-the-fly " + "on the basis of two different criteria:" + "
      " + "
    1. GeneProtein conflation merges protein-coding genes with the proteins they encode. " + "The gene(s) always appear first in the combined clique." + "
    2. DrugChemical conflation merges chemicals based on their active ingredient. We " + "attempt to ensure that the active ingredient appears before any formulations in the combined clique." + "
    " + "The returned strings can be used with the conflation parameter of /get_setid; " + "/get_normalized_nodes has a separate boolean flag for each. You can read " + "" + "more about conflation or " + "" + "more about this endpoint.", + response_description="The list of conflations supported by this NodeNorm instance.", ) async def get_conflations() -> ConflationList: """ @@ -261,10 +299,26 @@ async def get_conflations() -> ConflationList: @app.get( "/get_normalized_nodes", summary="Get the equivalent identifiers and semantic types for the CURIEs entered.", - description="Returns the equivalent identifiers and semantic types for the CURIEs entered." - "You can optionally conflate identifiers if needed." + description="Returns the equivalent identifiers and semantic types for the CURIEs entered. " + "A CURIE that cannot be normalized is returned as a key with a null value rather than " + "being left out of the response. " + "You can optionally " + "" + "conflate identifiers if needed: conflate merges genes with the proteins they encode " + "(the gene comes first), and drug_chemical_conflate merges drugs with their active " + "ingredient (the ingredient comes before any formulations). Conflated cliques are returned as a single " + "flat list of equivalent identifiers, so use individual_types if you need to tell the " + "members apart. " "You can read more about this endpoint in the " - "NodeNorm API documentation.", + "NodeNorm API documentation, " + "and about where the identifiers, labels and information content values come from in " + "" + "Where NodeNorm's data comes from.", + responses={200: { + "model": Dict[str, Optional[NormalizedNode]], + "description": "A mapping from each CURIE queried to its normalized clique, or to null if it could not be normalized.", + "content": {"application/json": {"example": EXAMPLE_NORMALIZED_NODES}}, + }}, ) async def get_normalized_node_handler( curie: List[str] = fastapi.Query( @@ -299,8 +353,21 @@ async def get_normalized_node_handler( @app.post( "/get_normalized_nodes", - summary="Get the equivalent identifiers and semantic types for the curie(s) entered.", - description="Returns the equivalent identifiers and semantic types for the curie(s). Use the `conflate` flag to choose whether to apply conflation.", + summary="Get the equivalent identifiers and semantic types for the CURIEs entered.", + description="Returns the equivalent identifiers and semantic types for the CURIEs entered. Identical to the GET " + "method of this endpoint, but takes a curies list in a JSON body instead of repeated " + "curie query parameters. " + "Note that drug_chemical_conflate currently defaults to false here " + "but true on the GET method " + "(#398); set it " + "explicitly if you care which conflations are applied. " + "You can read more about this endpoint in the " + "NodeNorm API documentation.", + responses={200: { + "model": Dict[str, Optional[NormalizedNode]], + "description": "A mapping from each CURIE queried to its normalized clique, or to null if it could not be normalized.", + "content": {"application/json": {"example": EXAMPLE_NORMALIZED_NODES}}, + }}, ) async def get_normalized_node_handler_post(curies: CurieList): """ @@ -322,7 +389,18 @@ async def get_normalized_node_handler_post(curies: CurieList): @app.get( "/get_setid", response_model=SetIDResponse, - summary="Normalize and deduplicate a set of identifiers and return a single hash that represents this set." + summary="Normalize and deduplicate a set of identifiers and return a single hash that represents this set.", + description="Returns the set ID for a given set of CURIEs. CURIEs that can be normalized are normalized (using " + "the conflations provided); those that cannot are kept as-is. Duplicates are then removed, the " + "remaining CURIEs are sorted, and a hash is generated from them. " + "A set ID is an identifier that can be used to identify this set of CURIEs going forward. It is " + "currently impossible to recreate a set of CURIEs from a set ID, but the same set of CURIEs given to " + "the same version of Node Normalization will always return the same set ID. Note that a different " + "Babel build may normalize the same CURIEs differently and so produce a different set ID — see " + "babel_version in /status. " + "You can read more about this endpoint in the " + "NodeNorm API documentation.", + response_description="The normalized CURIEs and the set ID calculated from them.", ) async def get_setid( curie: List[str] = fastapi.Query( @@ -343,7 +421,13 @@ async def get_setid( @app.post( "/get_setid", response_model=List[SetIDResponse], - summary="Normalize and deduplicate a set of identifiers and return a single hash that represents this set." + summary="Normalize and deduplicate a set of identifiers and return a single hash that represents this set.", + description="Identical to the GET method of this endpoint, but calculates a set ID for several sets at once. " + "Takes a list of sets, each with its own curies and optional conflations, " + "and returns a list of results in the same order. " + "You can read more about this endpoint in the " + "NodeNorm API documentation.", + response_description="One result per set submitted, in the order submitted.", ) async def get_setid( sets: List[SetIDQuery] = fastapi.Body([], @@ -367,7 +451,12 @@ async def get_setid( "/get_semantic_types", response_model=SemanticTypes, summary="Return a list of BioLink semantic types for which normalization has been attempted.", - description="Returns a distinct set of the semantic types discovered in the compendium data.", + description="Returns a distinct, unordered set of the Biolink semantic types found in the " + "Babel compendia loaded into this instance. " + "Returns 404 if no semantic types could be found, which usually means the databases have not been " + "loaded. You can read more about this endpoint in the " + "NodeNorm API documentation.", + response_description="The distinct Biolink semantic types present in this instance.", ) async def get_semantic_types_handler() -> SemanticTypes: # look for all biolink semantic types @@ -389,11 +478,20 @@ async def get_semantic_types_handler() -> SemanticTypes: "/get_curie_prefixes", response_model=Dict[str, CuriePivot], summary="Return the number of times each CURIE prefix appears in an equivalent identifier for a semantic type", - description="Returns the curies and their hit count for a semantic type(s).", + description="Returns the CURIE prefixes and their hit counts for one or more semantic types. Omit " + "semantic_type to get every semantic type. Counts are returned as strings. " + "These counts are tallied when the " + "Babel compendia are loaded into this " + "instance and are approximate — the load aggregates them concurrently without locking " + "(#380), so prefer " + "Babel's own reports if you need exact figures. You can read more about this endpoint in the " + "NodeNorm API documentation.", + response_description="A mapping from each semantic type requested to its CURIE prefix counts.", ) async def get_curie_prefixes_handler( - semantic_type: Optional[List[str]] = fastapi.Query([], description="e.g. biolink:ChemicalEntity, " - "biolink:AnatomicalEntity") + semantic_type: Optional[List[str]] = fastapi.Query([], description="The semantic types to report on, e.g. " + "biolink:ChemicalEntity, " + "biolink:AnatomicalEntity. Omit for all types.") ) -> Dict[str, CuriePivot]: return await get_curie_prefixes(app, semantic_type) @@ -402,7 +500,10 @@ async def get_curie_prefixes_handler( "/get_curie_prefixes", response_model=Dict[str, CuriePivot], summary="Return the number of times each CURIE prefix appears in an equivalent identifier for a semantic type", - description="Returns the curies and their hit count for a semantic type(s).", + description="Identical to the GET method of this endpoint, but takes the list of semantic types in a JSON body. " + "You can read more about this endpoint in the " + "NodeNorm API documentation.", + response_description="A mapping from each semantic type requested to its CURIE prefix counts.", ) async def get_curie_prefixes_handler( semantic_types: SemanticTypesInput, From 69746f07dab507eb1a3db599bbee1a2fa4555c49 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Mon, 27 Jul 2026 05:23:33 -0400 Subject: [PATCH 4/7] Point every cross-repo link at main, not master NodeNormalization, Babel and NameResolution all use main as their default branch. A /blob/master/ URL resolves today only through GitHub's post-rename redirect, which is exactly the kind of thing that disappears quietly -- so these are link rot rather than an acceptable alias. 25 links across five files: the endpoint descriptions in server.py (12), this branch's new Babel.md (6), the /status examples in API.md (2), openapi.yml (2), and the notebook including its Colab badge (3). Most predate this branch; they are fixed here because this is the documentation PR and because the skill branch stacked on it inherits them. raw.githubusercontent.com/biolink/biolink-model/master/... is deliberately not touched: biolink-model really does default to master. Verified: every rewritten github.com URL returns 200, the notebook is still valid JSON, and tests/frontend passes (22 passed, 4 skipped). Co-Authored-By: Claude Opus 5 --- documentation/API.md | 4 ++-- documentation/Babel.md | 12 ++++++------ documentation/NodeNormalization.ipynb | 6 +++--- node_normalizer/resources/openapi.yml | 4 ++-- node_normalizer/server.py | 24 ++++++++++++------------ uv.lock | 3 +++ 6 files changed, 28 insertions(+), 25 deletions(-) create mode 100644 uv.lock diff --git a/documentation/API.md b/documentation/API.md index 62bc5de..bebe5cc 100644 --- a/documentation/API.md +++ b/documentation/API.md @@ -202,7 +202,7 @@ Example output: "version": "2.5.1", "backend": "redis", "babel_version": "2025mar31", - "babel_version_url": "https://github.com/NCATSTranslator/Babel/blob/master/releases/2025mar31.md", + "babel_version_url": "https://github.com/NCATSTranslator/Babel/blob/main/releases/2025mar31.md", "biolink_model": { "tag": "v4.2.6-rc2", "url": "https://github.com/biolink/biolink-model/tree/v4.2.6-rc2", @@ -229,7 +229,7 @@ Output values: the cliques being presented. These are usually date-based versions indicating approximately when the Babel build was completed. Since the backend databases are written once and never updated in place, this identifies the data this instance will return until an operator loads a newer build — record it if you need reproducible results. -* `babel_version_url` (example: https://github.com/NCATSTranslator/Babel/blob/master/releases/2025mar31.md): A URL you +* `babel_version_url` (example: https://github.com/NCATSTranslator/Babel/blob/main/releases/2025mar31.md): A URL you can use to learn more about this version of Babel, and how it differs from previous and future versions. * `biolink_model`: The version of the [Biolink Model](https://github.com/biolink/biolink-model) this instance uses to expand a clique's most specific type into the full list of ancestors returned in `type`. Set with the diff --git a/documentation/Babel.md b/documentation/Babel.md index 89980c1..655885f 100644 --- a/documentation/Babel.md +++ b/documentation/Babel.md @@ -18,7 +18,7 @@ results cannot change until an operator loads a newer build. If you need reprodu record the `babel_version` alongside them. Published Babel builds can be downloaded directly — see Babel's -[Downloads.md](https://github.com/NCATSTranslator/Babel/blob/master/docs/Downloads.md). +[Downloads.md](https://github.com/NCATSTranslator/Babel/blob/main/docs/Downloads.md). ## Cliques, preferred identifiers, and labels @@ -38,7 +38,7 @@ identifier to give a clearer name. Two rules drive this: DrugCentral, CHEBI, MESH, GTOPDB — rather than using the Biolink prefix order. - Labels longer than a configured length are demoted, and used only if nothing shorter exists. -Babel's [Understanding.md](https://github.com/NCATSTranslator/Babel/blob/master/docs/Understanding.md) +Babel's [Understanding.md](https://github.com/NCATSTranslator/Babel/blob/main/docs/Understanding.md) has the full rules for both. ## Information content @@ -74,7 +74,7 @@ Two consequences worth knowing: ([#320](https://github.com/NCATSTranslator/NodeNormalization/issues/320)); `individual_types=true` will at least give you a Biolink type per identifier. -See Babel's [Conflation.md](https://github.com/NCATSTranslator/Babel/blob/master/docs/Conflation.md). +See Babel's [Conflation.md](https://github.com/NCATSTranslator/Babel/blob/main/docs/Conflation.md). ## Descriptions @@ -107,7 +107,7 @@ issues, not NodeNorm issues — see below. | NodeNorm returns an error, behaves oddly, or the API itself is wrong | [NodeNorm](https://github.com/NCATSTranslator/NodeNormalization/issues/) | When filing against Babel, a link to a NodeNorm query showing the problem is very helpful. Babel's -[NewIssue.md](https://github.com/NCATSTranslator/Babel/blob/master/docs/NewIssue.md) describes what +[NewIssue.md](https://github.com/NCATSTranslator/Babel/blob/main/docs/NewIssue.md) describes what to include and how issues are prioritized. ## Going deeper @@ -115,6 +115,6 @@ to include and how issues are prioritized. Everything above is what leaks into a NodeNorm response. How Babel actually builds cliques — the cross-reference concords, the transitive merge, per-source ingest quirks, the DuckDB and Parquet exports, and the pipeline itself — is documented in -[Babel's documentation index](https://github.com/NCATSTranslator/Babel/blob/master/docs/README.md). +[Babel's documentation index](https://github.com/NCATSTranslator/Babel/blob/main/docs/README.md). The output file formats NodeNorm's loader reads are specified in -[DataFormats.md](https://github.com/NCATSTranslator/Babel/blob/master/docs/DataFormats.md). +[DataFormats.md](https://github.com/NCATSTranslator/Babel/blob/main/docs/DataFormats.md). diff --git a/documentation/NodeNormalization.ipynb b/documentation/NodeNormalization.ipynb index d4efbee..872a1e8 100644 --- a/documentation/NodeNormalization.ipynb +++ b/documentation/NodeNormalization.ipynb @@ -14,7 +14,7 @@ { "metadata": {}, "cell_type": "markdown", - "source": "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NCATSTranslator/NodeNormalization/blob/master/documentation/NodeNormalization.ipynb)" + "source": "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NCATSTranslator/NodeNormalization/blob/main/documentation/NodeNormalization.ipynb)" }, { "cell_type": "markdown", @@ -178,7 +178,7 @@ } }, "source": [ - "In this example, `get_normalized_node` is called with a MeSH identifier for water. This is normalized to [CHEBI:15377](https://www.ebi.ac.uk/chebi/CHEBI:15377), the ChEBI identifier for water. Other equivalent identifiers are returned as well, including UNII, PubChem Compound, ChEMBL Compound and DrugBank identifiers. Note that several MeSH identifiers are combined, including MESH:D060766 \"Drinking Water\". This class is typed as a [biolink:SmallMolecule](https://biolink.github.io/biolink-model/SmallMolecule/), and has an [information content](https://github.com/NCATSTranslator/Babel/blob/master/docs/README.md#what-are-information-content-values) of 47.5." + "In this example, `get_normalized_node` is called with a MeSH identifier for water. This is normalized to [CHEBI:15377](https://www.ebi.ac.uk/chebi/CHEBI:15377), the ChEBI identifier for water. Other equivalent identifiers are returned as well, including UNII, PubChem Compound, ChEMBL Compound and DrugBank identifiers. Note that several MeSH identifiers are combined, including MESH:D060766 \"Drinking Water\". This class is typed as a [biolink:SmallMolecule](https://biolink.github.io/biolink-model/SmallMolecule/), and has an [information content](https://github.com/NCATSTranslator/Babel/blob/main/docs/README.md#what-are-information-content-values) of 47.5." ] }, { @@ -912,7 +912,7 @@ "* GeneProtein conflation combines protein-encoding genes with their gene products, which can combine\n", "* DrugChemical conflation combines drug formulations with their active ingredients, allowing you to normalize a specific formulation of a drug (such as [UMLS:C0704942 \"acetaminophen 16 MG/ML Oral Solution\"](https://uts.nlm.nih.gov/uts/umls/concept/C0704942)) to the active ingredient (in this case, [CHEBI:46195 \"Acetaminophen\"](https://www.ebi.ac.uk/chebi/CHEBI:46195)).\n", "\n", - "More details about conflation is available [in the Babel repository](https://github.com/NCATSTranslator/Babel/blob/master/docs/Conflation.md)." + "More details about conflation is available [in the Babel repository](https://github.com/NCATSTranslator/Babel/blob/main/docs/Conflation.md)." ] }, { diff --git a/node_normalizer/resources/openapi.yml b/node_normalizer/resources/openapi.yml index 06658c7..de72d28 100644 --- a/node_normalizer/resources/openapi.yml +++ b/node_normalizer/resources/openapi.yml @@ -15,13 +15,13 @@ info: are BioLink Model Compliant. Babel decides which identifiers are equivalent, which one is preferred, and what a clique is called, so most questions about why a clique looks the way it does are really questions about Babel: see - Where NodeNorm''s data comes from. + Where NodeNorm''s data comes from. The babel_version reported by /status identifies exactly which Babel build this instance is serving. To determine whether Node Normalization is likely to be useful, check /get_semantic_types, which lists the BioLink semantic types for which normalization has been attempted, and /get_curie_prefixes, which lists the number of times each prefix is used for a semantic type. You can find out more about these API methods at the - Node Normalization API documentation.' + Node Normalization API documentation.' license: name: MIT url: https://opensource.org/licenses/MIT diff --git a/node_normalizer/server.py b/node_normalizer/server.py index 652eb7d..05abd10 100644 --- a/node_normalizer/server.py +++ b/node_normalizer/server.py @@ -103,7 +103,7 @@ async def shutdown_event(): "semantic types (biolink_model). The backend databases are written once and never " "updated in place, so babel_version identifies the data this instance will return until " "an operator loads a newer build. You can read more about this endpoint in the " - "" + "" "NodeNorm API documentation.", response_description="Information about this NodeNorm instance and the databases it is connected to.", ) @@ -281,9 +281,9 @@ async def async_query_task(async_query: reasoner_pydantic.AsyncQuery): "" "The returned strings can be used with the conflation parameter of /get_setid; " "/get_normalized_nodes has a separate boolean flag for each. You can read " - "" + "" "more about conflation or " - "" + "" "more about this endpoint.", response_description="The list of conflations supported by this NodeNorm instance.", ) @@ -303,16 +303,16 @@ async def get_conflations() -> ConflationList: "A CURIE that cannot be normalized is returned as a key with a null value rather than " "being left out of the response. " "You can optionally " - "" + "" "conflate identifiers if needed: conflate merges genes with the proteins they encode " "(the gene comes first), and drug_chemical_conflate merges drugs with their active " "ingredient (the ingredient comes before any formulations). Conflated cliques are returned as a single " "flat list of equivalent identifiers, so use individual_types if you need to tell the " "members apart. " "You can read more about this endpoint in the " - "NodeNorm API documentation, " + "NodeNorm API documentation, " "and about where the identifiers, labels and information content values come from in " - "" + "" "Where NodeNorm's data comes from.", responses={200: { "model": Dict[str, Optional[NormalizedNode]], @@ -362,7 +362,7 @@ async def get_normalized_node_handler( "(#398); set it " "explicitly if you care which conflations are applied. " "You can read more about this endpoint in the " - "NodeNorm API documentation.", + "NodeNorm API documentation.", responses={200: { "model": Dict[str, Optional[NormalizedNode]], "description": "A mapping from each CURIE queried to its normalized clique, or to null if it could not be normalized.", @@ -399,7 +399,7 @@ async def get_normalized_node_handler_post(curies: CurieList): "Babel build may normalize the same CURIEs differently and so produce a different set ID — see " "babel_version in /status. " "You can read more about this endpoint in the " - "NodeNorm API documentation.", + "NodeNorm API documentation.", response_description="The normalized CURIEs and the set ID calculated from them.", ) async def get_setid( @@ -426,7 +426,7 @@ async def get_setid( "Takes a list of sets, each with its own curies and optional conflations, " "and returns a list of results in the same order. " "You can read more about this endpoint in the " - "NodeNorm API documentation.", + "NodeNorm API documentation.", response_description="One result per set submitted, in the order submitted.", ) async def get_setid( @@ -455,7 +455,7 @@ async def get_setid( "Babel compendia loaded into this instance. " "Returns 404 if no semantic types could be found, which usually means the databases have not been " "loaded. You can read more about this endpoint in the " - "NodeNorm API documentation.", + "NodeNorm API documentation.", response_description="The distinct Biolink semantic types present in this instance.", ) async def get_semantic_types_handler() -> SemanticTypes: @@ -485,7 +485,7 @@ async def get_semantic_types_handler() -> SemanticTypes: "instance and are approximate — the load aggregates them concurrently without locking " "(#380), so prefer " "Babel's own reports if you need exact figures. You can read more about this endpoint in the " - "NodeNorm API documentation.", + "NodeNorm API documentation.", response_description="A mapping from each semantic type requested to its CURIE prefix counts.", ) async def get_curie_prefixes_handler( @@ -502,7 +502,7 @@ async def get_curie_prefixes_handler( summary="Return the number of times each CURIE prefix appears in an equivalent identifier for a semantic type", description="Identical to the GET method of this endpoint, but takes the list of semantic types in a JSON body. " "You can read more about this endpoint in the " - "NodeNorm API documentation.", + "NodeNorm API documentation.", response_description="A mapping from each semantic type requested to its CURIE prefix counts.", ) async def get_curie_prefixes_handler( diff --git a/uv.lock b/uv.lock new file mode 100644 index 0000000..9431a63 --- /dev/null +++ b/uv.lock @@ -0,0 +1,3 @@ +version = 1 +revision = 3 +requires-python = ">=3.11" From 63844353e31d026962b3a02ee0b3f1db12fc31b0 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Mon, 27 Jul 2026 05:30:33 -0400 Subject: [PATCH 5/7] Enforce the link rules with a test, ported from NameRes Without something failing CI, the master links just fixed drift back. This is NameResolution's tests/test_docs_links.py adapted to this repo's paths, so the two services enforce the same rules: relative links resolve, heading anchors exist, the API.md anchors used in the endpoint descriptions exist, and no link is pinned to `master`. All offline -- a link check that fails when GitHub is slow is one people learn to ignore. It earned its place immediately: it found 12 more links, in releases/, that a grep for NCATSTranslator//blob/master/ had missed because they use the pre-rename TranslatorSRI org as well as the pre-rename branch. Those are now NCATSTranslator/Babel/blob/main/ and all five distinct targets return 200. The other three checks passed on this repo unchanged, so the whole file came across rather than just the master ban. Co-Authored-By: Claude Opus 5 --- releases/TranslatorFuguJuly2024.md | 4 +- releases/TranslatorGuppyAugust2024.md | 4 +- releases/TranslatorHammerheadNovember2024.md | 4 +- releases/v2.3.19.md | 4 +- releases/v2.3.26.md | 4 +- releases/v2.4.1.md | 4 +- tests/test_docs_links.py | 105 +++++++++++++++++++ 7 files changed, 117 insertions(+), 12 deletions(-) create mode 100644 tests/test_docs_links.py diff --git a/releases/TranslatorFuguJuly2024.md b/releases/TranslatorFuguJuly2024.md index 5d84318..a970fae 100644 --- a/releases/TranslatorFuguJuly2024.md +++ b/releases/TranslatorFuguJuly2024.md @@ -1,6 +1,6 @@ # NodeNorm Translator "Fugu" July 2024 Release - Babel: [2024jul13](https://stars.renci.org/var/babel_outputs/2024jul13/) - ([Babel Translator July 2024 Release](https://github.com/TranslatorSRI/Babel/blob/master/releases/TranslatorFuguJuly2024.md)) + ([Babel Translator July 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorFuguJuly2024.md)) - NodeNorm: [v2.3.15](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.15) Previous release: [Translator May 2024](./TranslatorMay2024.md) @@ -14,7 +14,7 @@ Next release: [Translator "Guppy" August 2024](./TranslatorGuppyAugust2024.md) ## Technical debt * Improved database names in source code. -## Babel updates (from [Babel Translator July 2024 Release](https://github.com/TranslatorSRI/Babel/blob/master/releases/TranslatorFuguJuly2024.md)) +## Babel updates (from [Babel Translator July 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorFuguJuly2024.md)) * [Feature] Added manual disease concords, and used that to do a better job of combining opioid use disorder and alcohol use disorder. * [Feature] Moved ensembl_datasets_to_skip into the config file. * [Bugfix] Eliminated preferred prefix overrides in Babel; we now trust the preferred prefixes from the Biolink Model. diff --git a/releases/TranslatorGuppyAugust2024.md b/releases/TranslatorGuppyAugust2024.md index 6be4653..5c35897 100644 --- a/releases/TranslatorGuppyAugust2024.md +++ b/releases/TranslatorGuppyAugust2024.md @@ -1,6 +1,6 @@ # NodeNorm Translator "Guppy" August 2024 Release - Babel: [2024aug18](https://stars.renci.org/var/babel_outputs/2024aug18/) - ([Babel Translator August 2024 Release](https://github.com/TranslatorSRI/Babel/blob/master/releases/TranslatorGuppyAugust2024.md)) + ([Babel Translator August 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorGuppyAugust2024.md)) - NodeNorm: [v2.3.16](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.16) Next release: [Translator "Hammerhead" November 2024](./TranslatorHammerheadNovember2024.md) @@ -13,7 +13,7 @@ Previous release: [Translator "Fugu" July 2024](./TranslatorFuguJuly2024.md) ## Technical debt * Several improvements to the compendium loader by @gaurav in [#291](https://github.com/TranslatorSRI/NodeNormalization/pull/291) -## Babel updates (from [Babel Translator "Guppy" August 2024 Release](https://github.com/TranslatorSRI/Babel/blob/master/releases/TranslatorGuppyAugust2024.md)) +## Babel updates (from [Babel Translator "Guppy" August 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorGuppyAugust2024.md)) * [Feature] Added support for generating DuckDB and Parquet files from the compendium and synonym files, allowing us to run queries such as looking for all the identically labeled cliques across all the compendia. Increased Babel Outputs file size to support DuckDB. diff --git a/releases/TranslatorHammerheadNovember2024.md b/releases/TranslatorHammerheadNovember2024.md index b21f1c6..a7aafd9 100644 --- a/releases/TranslatorHammerheadNovember2024.md +++ b/releases/TranslatorHammerheadNovember2024.md @@ -1,6 +1,6 @@ # NodeNorm Translator "Hammerhead" November 2024 Release - Babel: [2024oct24](https://stars.renci.org/var/babel_outputs/2024oct24/) - ([Babel Translator November 2024 Release](https://github.com/TranslatorSRI/Babel/blob/master/releases/TranslatorHammerheadNovember2024.md)) + ([Babel Translator November 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorHammerheadNovember2024.md)) - NodeNorm: [v2.3.18](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.18) Next release: _None as yet_ @@ -16,7 +16,7 @@ Previous release: [Translator "Guppy" August 2024](./TranslatorGuppyAugust2024.m ## Documentation * Added Guppy release notes ([#294](https://github.com/TranslatorSRI/NodeNormalization/pull/294)). -## Babel updates (from [Babel Translator "Hammerhead" November 2024 Release](https://github.com/TranslatorSRI/Babel/blob/master/releases/TranslatorHammerheadNovember2024.md)) +## Babel updates (from [Babel Translator "Hammerhead" November 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorHammerheadNovember2024.md)) - [New features] Added taxon information to proteins ([#349](https://github.com/TranslatorSRI/Babel/pull/349)) - [Updates] Upgraded RxNorm to 09032024. - [Updates] Changed NCBIGene download from FTP to HTTP. diff --git a/releases/v2.3.19.md b/releases/v2.3.19.md index fb4b6f5..c9b031c 100644 --- a/releases/v2.3.19.md +++ b/releases/v2.3.19.md @@ -1,6 +1,6 @@ # NodeNorm v2.3.19 - Babel: [2025jan23](https://stars.renci.org/var/babel_outputs/2025jan23/) - ([Babel 2025jan23](https://github.com/TranslatorSRI/Babel/blob/master/releases/2025jan23.md)) + ([Babel 2025jan23](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025jan23.md)) - NodeNorm: [v2.3.19](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.19) Next release: [v2.3.26](./v2.3.26.md) @@ -12,7 +12,7 @@ Previous release: [Translator "Hammerhead" November 2024](./TranslatorHammerhead ## Updates * _None_ -## Babel updates (from [Babel 2025jan23](https://github.com/TranslatorSRI/Babel/blob/master/releases/2025jan23.md)) +## Babel updates (from [Babel 2025jan23](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025jan23.md)) - [New feature] Added a check for duplicate CURIEs [#342](https://github.com/TranslatorSRI/Babel/pull/342). - [New feature] Added some additional manual concords for Disease/Phenotype cliques and DrugChemical conflation [#360](https://github.com/TranslatorSRI/Babel/pull/360). diff --git a/releases/v2.3.26.md b/releases/v2.3.26.md index a68d577..750869a 100644 --- a/releases/v2.3.26.md +++ b/releases/v2.3.26.md @@ -1,6 +1,6 @@ # NodeNorm v2.3.26 - Babel: [2025sep1](https://stars.renci.org/var/babel_outputs/2025sep1/) - ([Babel 2025sep1](https://github.com/TranslatorSRI/Babel/blob/master/releases/2025sep1.md)) + ([Babel 2025sep1](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025sep1.md)) - NodeNorm: [v2.3.26](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.26) Next release: [v2.4.1](./v2.4.1.md) @@ -16,7 +16,7 @@ Previous release: [v2.3.19](./v2.3.19.md) * Added release notes for November 2024 and January 2025. * Updated renci-python-image to GitHub Packages. -## Babel updates (from [Babel 2025sep1](https://github.com/TranslatorSRI/Babel/blob/master/releases/2025sep1.md)) +## Babel updates (from [Babel 2025sep1](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025sep1.md)) - [MAJOR] Information content values were incorrectly calculated as `100` for some cliques when they were in fact lower. - [MAJOR] UMLS proteins have now been removed from chemicals so that they only appear in proteins. - [MAJOR] Improved DrugChemical conflation order. diff --git a/releases/v2.4.1.md b/releases/v2.4.1.md index 6a2ae19..69d36a9 100644 --- a/releases/v2.4.1.md +++ b/releases/v2.4.1.md @@ -1,6 +1,6 @@ # NodeNorm v2.4.1 - Babel: [2025sep1](https://stars.renci.org/var/babel_outputs/2025sep1/) - ([Babel 2025sep1](https://github.com/TranslatorSRI/Babel/blob/master/releases/2025sep1.md)) + ([Babel 2025sep1](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025sep1.md)) - NodeNorm: [v2.4.1](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.4.1) Next release: _None as yet_ @@ -23,7 +23,7 @@ Previous release: [v2.3.26](./v2.3.26.md) * (v2.3.27) NEW BUG: `description` accidentally moved from `result['id']['description']` to `result['description']` ([#345](https://github.com/NCATSTranslator/NodeNormalization/pull/345)). * (v2.3.28) Fixed description bug introduced in NodeNorm v2.3.27 by @gaurav in [#346](https://github.com/NCATSTranslator/NodeNormalization/pull/346) -## Babel updates (from [Babel 2025sep1](https://github.com/TranslatorSRI/Babel/blob/master/releases/2025sep1.md)) +## Babel updates (from [Babel 2025sep1](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025sep1.md)) Babel was not updated in this release. ## Releases since [NodeNorm v2.3.26](./v2.3.26.md) diff --git a/tests/test_docs_links.py b/tests/test_docs_links.py new file mode 100644 index 0000000..7de6309 --- /dev/null +++ b/tests/test_docs_links.py @@ -0,0 +1,105 @@ +"""Checks on the links in our documentation and in the endpoint descriptions. + +Ported from NameResolution's tests/test_docs_links.py, so the two services enforce the same +rules. Keep them in sync when either changes. + +Everything here is offline: it resolves relative paths and heading anchors on disk and +greps for a banned URL form. Nothing fetches a URL, because a test that fails when +GitHub is slow is a test people learn to ignore. +""" + +import re +from pathlib import Path + +REPO_ROOT = Path(__file__).parent.parent + +# Markdown we own. The notebook and anything vendored is deliberately excluded. +MARKDOWN_FILES = sorted( + p for p in REPO_ROOT.rglob("*.md") + if not any(part in {"venv", ".venv", "node_modules", ".git", ".pytest_cache"} for part in p.parts) +) + +# Files that carry links out to GitHub in code rather than in prose. +SOURCE_WITH_LINKS = [REPO_ROOT / "node_normalizer" / "server.py", REPO_ROOT / "node_normalizer" / "resources" / "openapi.yml"] + +INLINE_LINK = re.compile(r"\[[^\]]*\]\(([^)\s]+)\)") +HEADING = re.compile(r"^#+\s+(.*)$", re.MULTILINE) + + +def anchors_for(markdown_text): + """The anchor slugs GitHub generates for a document's headings.""" + slugs = set() + for heading in HEADING.findall(markdown_text): + slug = re.sub(r"[`*]", "", heading.strip().lower()) + slug = re.sub(r"[^a-z0-9 _-]", "", slug) + slugs.add(slug.replace(" ", "-")) + return slugs + + +def test_relative_links_resolve(): + """A relative link is written relative to the file it sits in, which is easy to get + wrong when a document moves into a subdirectory.""" + broken = [] + for path in MARKDOWN_FILES: + for target in INLINE_LINK.findall(path.read_text()): + if target.startswith(("http://", "https://", "#", "mailto:")): + continue + resolved = (path.parent / target.split("#")[0]).resolve() + if not resolved.exists(): + broken.append(f"{path.relative_to(REPO_ROOT)} -> {target}") + assert not broken, "Relative links that do not resolve:\n " + "\n ".join(broken) + + +def test_anchors_resolve(): + """Anchors are case-sensitive and GitHub lower-cases them, so `#Conflation` silently + lands at the top of the page rather than at the heading.""" + broken = [] + for path in MARKDOWN_FILES: + for target in INLINE_LINK.findall(path.read_text()): + if "#" not in target or target.startswith(("http://", "https://", "mailto:")): + continue + rel, _, anchor = target.partition("#") + if not anchor: + continue + target_path = path if rel == "" else (path.parent / rel) + if target_path.suffix != ".md" or not target_path.exists(): + continue + if anchor not in anchors_for(target_path.read_text()): + broken.append(f"{path.relative_to(REPO_ROOT)} -> {target}") + assert not broken, "Anchors with no matching heading:\n " + "\n ".join(broken) + + +def test_in_repo_anchors_in_source_resolve(): + """The endpoint descriptions link into documentation/API.md by absolute URL, so the + check above cannot see them.""" + api_md = REPO_ROOT / "documentation" / "API.md" + valid = anchors_for(api_md.read_text()) + broken = [] + for path in SOURCE_WITH_LINKS: + for anchor in re.findall(r"documentation/API\.md#([A-Za-z0-9_-]+)", path.read_text()): + if anchor not in valid: + broken.append(f"{path.relative_to(REPO_ROOT)} -> API.md#{anchor}") + assert not broken, "Anchors with no matching heading in API.md:\n " + "\n ".join(broken) + + +#: A GitHub link pinned to `master`, in either the blob or the raw form. The raw half is scoped to +#: NCATSTranslator on purpose: api/server.py legitimately builds +#: raw.githubusercontent.com/biolink/biolink-model/master/... and biolink-model really does default +#: to master. +MASTER_LINK = re.compile( + r"github\.com/[^/\s]+/[^/\s]+/blob/master/" + r"|raw\.githubusercontent\.com/NCATSTranslator/[^/\s]+/master/", + re.IGNORECASE, +) + + +def test_no_master_branch_links(): + """NodeNormalization, Babel and NodeNormalization all default to `main`. A master URL resolves + only through GitHub's post-rename redirect, so it looks fine right up until that redirect goes + away. See CLAUDE.md.""" + offenders = [] + for path in MARKDOWN_FILES + SOURCE_WITH_LINKS: + for line_no, line in enumerate(path.read_text().splitlines(), start=1): + if MASTER_LINK.search(line): + offenders.append(f"{path.relative_to(REPO_ROOT)}:{line_no}") + assert not offenders, "Links pinned to `master` instead of `main`:\n " + "\n ".join(offenders) From 89216dba98bab6cc5e332557ebefca44f5cfb81f Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Mon, 27 Jul 2026 05:46:07 -0400 Subject: [PATCH 6/7] Use the current org in the four user-facing TranslatorSRI links NodeNormalization moved to NCATSTranslator; these four still named the old org in places a user actually lands on: - The release badge in README.md linked to TranslatorSRI while its image already pointed at NCATSTranslator, so clicking the badge left the org the badge itself reported. - Two PR #323 links in server.py are served in the OpenAPI descriptions, so they show up in Swagger. - One more in documentation/API.md. An org-rename redirect is more durable than the branch-rename one cleaned up earlier, so this is consistency rather than an outage waiting to happen. The ~77 remaining references are all in releases/, i.e. historical notes recording what was true at the time; whether to sweep those is tracked separately in NameResolution#292, which covers the same question for that repo. Left alone deliberately: TranslatorSRI/r3 in README.md. That repository really does still live under TranslatorSRI -- NCATSTranslator/r3 is a 404. Co-Authored-By: Claude Opus 5 --- README.md | 2 +- documentation/API.md | 2 +- node_normalizer/server.py | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 9f70775..f2e3d4b 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.18489030.svg)](https://doi.org/10.5281/zenodo.18489030) [![arXiv](https://img.shields.io/badge/arXiv-2601.10008-b31b1b.svg)](https://arxiv.org/abs/2601.10008) ## Introduction diff --git a/documentation/API.md b/documentation/API.md index bebe5cc..c1d1756 100644 --- a/documentation/API.md +++ b/documentation/API.md @@ -352,7 +352,7 @@ Example output: ## TRAPI Normalization (deprecated) These two endpoints normalize an entire TRAPI message rather than a list of CURIEs. Both are -deprecated ([PR #323](https://github.com/TranslatorSRI/NodeNormalization/pull/323)): they are no +deprecated ([PR #323](https://github.com/NCATSTranslator/NodeNormalization/pull/323)): they are no longer actively maintained, and will be removed once the Workflow Runner stops using them. New callers should extract the CURIEs they care about and use `/get_normalized_nodes` instead. diff --git a/node_normalizer/server.py b/node_normalizer/server.py index 05abd10..6875d66 100644 --- a/node_normalizer/server.py +++ b/node_normalizer/server.py @@ -195,7 +195,7 @@ async def status() -> Dict: "(biolink:has_numeric_value / information_content). " "Deprecated: this endpoint is no longer actively maintained and will be removed once " "the Workflow Runner stops using it " - "(PR #323). New callers " + "(PR #323). New callers " "should extract the CURIEs they care about and use /get_normalized_nodes instead.", response_description="The submitted TRAPI message with all identifiers normalized.", response_model=reasoner_pydantic.Query, @@ -222,7 +222,7 @@ async def query(query: Annotated[reasoner_pydantic.Query, Body(openapi_examples= "ready (retrying a few times if the callback fails). " "Deprecated: this endpoint is no longer actively maintained and will be removed once " "the Workflow Runner stops using it " - "(PR #323). New callers " + "(PR #323). New callers " "should extract the CURIEs they care about and use /get_normalized_nodes instead.", response_description="Confirmation that the query has been queued; the normalized message is sent to the callback URL.", deprecated=True, From 9d2fa3791bf046d7caa68f6eac5559e4a74855cd Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Mon, 27 Jul 2026 05:54:42 -0400 Subject: [PATCH 7/7] Use the current org in release-note links, and ban the old one NodeNormalization, Babel and NameResolution all moved from TranslatorSRI to NCATSTranslator. 79 links across the nine release-note files still named the old org -- they resolve through GitHub's org-rename redirect, but point at an org that no longer owns the code. The new test is scoped to those three repositories by name rather than banning the TranslatorSRI string, because TranslatorSRI/r3 (referenced from README.md) really does still live under that org and NCATSTranslator/r3 is a 404. A blanket ban would push someone to "fix" a correct link into a broken one. Also excludes data/ from the markdown scan: it holds untracked working copies of helm charts, so a scratch file there could fail the suite for reasons that have nothing to do with the repository. Matches NameResolution#262, which does the same sweep and adds the same test. Co-Authored-By: Claude Opus 5 --- releases/TranslatorDecember2023.md | 16 +++++------ releases/TranslatorFuguJuly2024.md | 16 +++++------ releases/TranslatorGuppyAugust2024.md | 30 ++++++++++---------- releases/TranslatorHammerheadNovember2024.md | 26 ++++++++--------- releases/TranslatorMay2024.md | 18 ++++++------ releases/TranslatorOctober2023.md | 4 +-- releases/v2.3.19.md | 14 ++++----- releases/v2.3.26.md | 22 +++++++------- releases/v2.4.1.md | 8 +++--- tests/test_docs_links.py | 24 +++++++++++++++- 10 files changed, 100 insertions(+), 78 deletions(-) diff --git a/releases/TranslatorDecember2023.md b/releases/TranslatorDecember2023.md index 8da24fb..1043766 100644 --- a/releases/TranslatorDecember2023.md +++ b/releases/TranslatorDecember2023.md @@ -1,7 +1,7 @@ # NodeNorm Translator December 2023 Release - Babel: [2023nov5](https://stars.renci.org/var/babel_outputs/2023nov5/) -- NodeNorm: [v2.3.5](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.5) +- NodeNorm: [v2.3.5](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.5) Next release: [May 2024](TranslatorMay2024.md) @@ -24,26 +24,26 @@ Next release: [May 2024](TranslatorMay2024.md) ## Releases since [Translator October 2023 release](TranslatorOctober2023.md) -* [NodeNorm v2.2.1](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.2.1) +* [NodeNorm v2.2.1](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.2.1) * Node descriptions by @YaphetKG in #212 * Trapi 14 by @cbizon in #208 -* [NodeNorm v2.3.0](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.0) +* [NodeNorm v2.3.0](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.0) * Node descriptions by @YaphetKG in #216 * initial implementation chem conflation by @cbizon in #211 -* [NodeNorm v2.3.1](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.1) +* [NodeNorm v2.3.1](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.1) * Fix Terms of Service and service description by @gaurav in #226 -* [NodeNorm v2.3.2](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.2) +* [NodeNorm v2.3.2](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.2) * Bump httpx from 0.19.0 to 0.23.0 by @dependabot in #195 * Bump requests from 2.28.1 to 2.31.0 by @dependabot in #194 * Upgrade reasoner-pydantic and requirements.lock by @gaurav in #227 -* [NodeNorm v2.3.3](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.3) +* [NodeNorm v2.3.3](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.3) * Add NodeNorm-Loader Docker by @gaurav in #228 * Fix missing node bindings in NodeNorm /query output by @gaurav in #231 * Uniquify semantic types returned from the database by @gaurav in #232 * Remove hardcoded TRAPI version by @gaurav in #233 * Remove biolink:Entity from nodes created by NodeNorm by @gaurav in #234 -* [NodeNorm v2.3.4](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.4) +* [NodeNorm v2.3.4](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.4) * Forgot to update the version number in NodeNorm v2.3.3. -* [NodeNorm v2.3.5](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.5) +* [NodeNorm v2.3.5](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.5) * Workaround for missing type information by @gaurav in #223 * Add Open Telemetry instrumentation by @gaurav in #237 diff --git a/releases/TranslatorFuguJuly2024.md b/releases/TranslatorFuguJuly2024.md index a970fae..6d9900f 100644 --- a/releases/TranslatorFuguJuly2024.md +++ b/releases/TranslatorFuguJuly2024.md @@ -1,15 +1,15 @@ # NodeNorm Translator "Fugu" July 2024 Release - Babel: [2024jul13](https://stars.renci.org/var/babel_outputs/2024jul13/) ([Babel Translator July 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorFuguJuly2024.md)) -- NodeNorm: [v2.3.15](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.15) +- NodeNorm: [v2.3.15](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.15) Previous release: [Translator May 2024](./TranslatorMay2024.md) Next release: [Translator "Guppy" August 2024](./TranslatorGuppyAugust2024.md) ## New features -* Update Biolink model by replacing bmt-lite with bmt ([#282](https://github.com/TranslatorSRI/NodeNormalization/pull/282), - [#284](https://github.com/TranslatorSRI/NodeNormalization/pull/284)) -* Added /get_setid GET endpoint need for Multi-CURIE Query (MCQ) ([#274](https://github.com/TranslatorSRI/NodeNormalization/pull/274)) +* Update Biolink model by replacing bmt-lite with bmt ([#282](https://github.com/NCATSTranslator/NodeNormalization/pull/282), + [#284](https://github.com/NCATSTranslator/NodeNormalization/pull/284)) +* Added /get_setid GET endpoint need for Multi-CURIE Query (MCQ) ([#274](https://github.com/NCATSTranslator/NodeNormalization/pull/274)) ## Technical debt * Improved database names in source code. @@ -28,12 +28,12 @@ Next release: [Translator "Guppy" August 2024](./TranslatorGuppyAugust2024.md) * [Update] Updated PANTHER pathways from SequenceAssociationPathway3.6.7.txt to SequenceAssociationPathway3.6.8.txt. ## Releases since [Translator May 2024 release](TranslatorMay2024.md) -* [NodeNorm v2.3.12](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.12) +* [NodeNorm v2.3.12](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.12) * Rename redis_connection[0-6] to their database names by @gaurav in #271 -* [NodeNorm v2.3.13](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.13): Incremented version, +* [NodeNorm v2.3.13](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.13): Incremented version, which I forgot to do in the previous release. -* [NodeNorm v2.3.14](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.14) +* [NodeNorm v2.3.14](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.14) * Replaced bmt-lite with bmt by @gaurav in #282 * Add a /get_setid GET endpoint by @gaurav in #274 -* [NodeNorm v2.3.15](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.15) +* [NodeNorm v2.3.15](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.15) * Fix BMT import and formatting issues by @gaurav in #284 diff --git a/releases/TranslatorGuppyAugust2024.md b/releases/TranslatorGuppyAugust2024.md index 5c35897..be7b3e2 100644 --- a/releases/TranslatorGuppyAugust2024.md +++ b/releases/TranslatorGuppyAugust2024.md @@ -1,31 +1,31 @@ # NodeNorm Translator "Guppy" August 2024 Release - Babel: [2024aug18](https://stars.renci.org/var/babel_outputs/2024aug18/) ([Babel Translator August 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorGuppyAugust2024.md)) -- NodeNorm: [v2.3.16](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.16) +- NodeNorm: [v2.3.16](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.16) Next release: [Translator "Hammerhead" November 2024](./TranslatorHammerheadNovember2024.md) Previous release: [Translator "Fugu" July 2024](./TranslatorFuguJuly2024.md) ## New features -* Optionally return types for every equivalent identifier by @gaurav in [#289](https://github.com/TranslatorSRI/NodeNormalization/pull/289) -* Add POST SetID by @gaurav in [#290](https://github.com/TranslatorSRI/NodeNormalization/pull/290) +* Optionally return types for every equivalent identifier by @gaurav in [#289](https://github.com/NCATSTranslator/NodeNormalization/pull/289) +* Add POST SetID by @gaurav in [#290](https://github.com/NCATSTranslator/NodeNormalization/pull/290) ## Technical debt -* Several improvements to the compendium loader by @gaurav in [#291](https://github.com/TranslatorSRI/NodeNormalization/pull/291) +* Several improvements to the compendium loader by @gaurav in [#291](https://github.com/NCATSTranslator/NodeNormalization/pull/291) ## Babel updates (from [Babel Translator "Guppy" August 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorGuppyAugust2024.md)) * [Feature] Added support for generating DuckDB and Parquet files from the compendium and synonym files, allowing us to run queries such as looking for all the identically labeled cliques across all the compendia. Increased Babel Outputs file size to support DuckDB. -* [Feature] Added labels from DrugBank (https://github.com/TranslatorSRI/Babel/pull/335). -* [Feature] Improved cell anatomy concords using Wikidata (https://github.com/TranslatorSRI/Babel/pull/329). -* [Feature] Added manual concords for the DrugChemical conflation (https://github.com/TranslatorSRI/Babel/pull/337). -* [Feature] Wrote a script for comparing between two summary files (https://github.com/TranslatorSRI/Babel/pull/320). +* [Feature] Added labels from DrugBank (https://github.com/NCATSTranslator/Babel/pull/335). +* [Feature] Improved cell anatomy concords using Wikidata (https://github.com/NCATSTranslator/Babel/pull/329). +* [Feature] Added manual concords for the DrugChemical conflation (https://github.com/NCATSTranslator/Babel/pull/337). +* [Feature] Wrote a script for comparing between two summary files (https://github.com/NCATSTranslator/Babel/pull/320). * [Feature] Added timestamping as an option to Wget. * [Feature] Reorganized primary label determination so that we can include it in compendia files as well. * This isn't currently used by the loader, but might be in the future. For now, this is only useful in helping track what labels are being chosen as the preferred label. -* [Bugfixes] Added additional ENSEMBL datasets to skip (https://github.com/TranslatorSRI/Babel/pull/297). +* [Bugfixes] Added additional ENSEMBL datasets to skip (https://github.com/NCATSTranslator/Babel/pull/297). * [Bugfixes] Fixed a bug in recognizing the end of file when reading the PubChem ID and SMILES files. * [Bugfixes] Fixed the lack of `clique_identifier_count` in leftover UMLS output. * [Bugfixes] Fixed unraised exception in Ensembl BioMart download. @@ -35,10 +35,10 @@ Previous release: [Translator "Fugu" July 2024](./TranslatorFuguJuly2024.md) * [Updates] Changed DrugBank ID types from 'ChemicalEntity' to 'Drug'. ## Releases since [Translator Fugu July 2024](./TranslatorFuguJuly2024.md) -* [2.3.16](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.16) - * Added TranslatorFuguJuly2024 release information by @gaurav in [#287](https://github.com/TranslatorSRI/NodeNormalization/pull/287) - * Optionally return types for every equivalent identifier by @gaurav in [#289](https://github.com/TranslatorSRI/NodeNormalization/pull/289) - * Bump requests from 2.31.0 to 2.32.0 by @dependabot in [#261](https://github.com/TranslatorSRI/NodeNormalization/pull/261) - * Add POST SetID by @gaurav in [#290](https://github.com/TranslatorSRI/NodeNormalization/pull/290) - * Several improvements to the compendium loader by @gaurav in [#291](https://github.com/TranslatorSRI/NodeNormalization/pull/291) +* [2.3.16](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.16) + * Added TranslatorFuguJuly2024 release information by @gaurav in [#287](https://github.com/NCATSTranslator/NodeNormalization/pull/287) + * Optionally return types for every equivalent identifier by @gaurav in [#289](https://github.com/NCATSTranslator/NodeNormalization/pull/289) + * Bump requests from 2.31.0 to 2.32.0 by @dependabot in [#261](https://github.com/NCATSTranslator/NodeNormalization/pull/261) + * Add POST SetID by @gaurav in [#290](https://github.com/NCATSTranslator/NodeNormalization/pull/290) + * Several improvements to the compendium loader by @gaurav in [#291](https://github.com/NCATSTranslator/NodeNormalization/pull/291) diff --git a/releases/TranslatorHammerheadNovember2024.md b/releases/TranslatorHammerheadNovember2024.md index a7aafd9..ede4016 100644 --- a/releases/TranslatorHammerheadNovember2024.md +++ b/releases/TranslatorHammerheadNovember2024.md @@ -1,31 +1,31 @@ # NodeNorm Translator "Hammerhead" November 2024 Release - Babel: [2024oct24](https://stars.renci.org/var/babel_outputs/2024oct24/) ([Babel Translator November 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorHammerheadNovember2024.md)) -- NodeNorm: [v2.3.18](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.18) +- NodeNorm: [v2.3.18](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.18) Next release: _None as yet_ Previous release: [Translator "Guppy" August 2024](./TranslatorGuppyAugust2024.md) ## New features -* Add a status endpoint with database name, key count and memory usage for each database ([#268](https://github.com/TranslatorSRI/NodeNormalization/pull/268)). -* Upgraded OTEL implementation to use gRPC by @EvanDietzMorris ([#298](https://github.com/TranslatorSRI/NodeNormalization/pull/298)). +* Add a status endpoint with database name, key count and memory usage for each database ([#268](https://github.com/NCATSTranslator/NodeNormalization/pull/268)). +* Upgraded OTEL implementation to use gRPC by @EvanDietzMorris ([#298](https://github.com/NCATSTranslator/NodeNormalization/pull/298)). ## Updates -* Updated NodeNorm preferred label to match Babel's ([#300](https://github.com/TranslatorSRI/NodeNormalization/pull/300)). +* Updated NodeNorm preferred label to match Babel's ([#300](https://github.com/NCATSTranslator/NodeNormalization/pull/300)). ## Documentation -* Added Guppy release notes ([#294](https://github.com/TranslatorSRI/NodeNormalization/pull/294)). +* Added Guppy release notes ([#294](https://github.com/NCATSTranslator/NodeNormalization/pull/294)). ## Babel updates (from [Babel Translator "Hammerhead" November 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorHammerheadNovember2024.md)) -- [New features] Added taxon information to proteins ([#349](https://github.com/TranslatorSRI/Babel/pull/349)) +- [New features] Added taxon information to proteins ([#349](https://github.com/NCATSTranslator/Babel/pull/349)) - [Updates] Upgraded RxNorm to 09032024. - [Updates] Changed NCBIGene download from FTP to HTTP. -- [Updates] Increased DRUG_CHEMICAL_SMALLER_MAX_LABEL_LENGTH (introduced in [#330](https://github.com/TranslatorSRI/Babel/pull/330)) from 30 to 40. +- [Updates] Increased DRUG_CHEMICAL_SMALLER_MAX_LABEL_LENGTH (introduced in [#330](https://github.com/NCATSTranslator/Babel/pull/330)) from 30 to 40. ## Releases since [Translator "Guppy" August 2024](./TranslatorGuppyAugust2024.md) -* [2.3.17](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.17) - * Add a status endpoint with database name, key count and memory usage for each database ([#268](https://github.com/TranslatorSRI/NodeNormalization/pull/268)). - * Added Guppy release notes ([#294](https://github.com/TranslatorSRI/NodeNormalization/pull/294)). -* [2.3.18](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.18) - * Upgraded OTEL implementation to use gRPC by @EvanDietzMorris ([#298](https://github.com/TranslatorSRI/NodeNormalization/pull/298)). - * Updated NodeNorm preferred label to match Babel's ([#300](https://github.com/TranslatorSRI/NodeNormalization/pull/300)). +* [2.3.17](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.17) + * Add a status endpoint with database name, key count and memory usage for each database ([#268](https://github.com/NCATSTranslator/NodeNormalization/pull/268)). + * Added Guppy release notes ([#294](https://github.com/NCATSTranslator/NodeNormalization/pull/294)). +* [2.3.18](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.18) + * Upgraded OTEL implementation to use gRPC by @EvanDietzMorris ([#298](https://github.com/NCATSTranslator/NodeNormalization/pull/298)). + * Updated NodeNorm preferred label to match Babel's ([#300](https://github.com/NCATSTranslator/NodeNormalization/pull/300)). diff --git a/releases/TranslatorMay2024.md b/releases/TranslatorMay2024.md index f989cb6..8cfed7a 100644 --- a/releases/TranslatorMay2024.md +++ b/releases/TranslatorMay2024.md @@ -2,7 +2,7 @@ - Babel: [2024mar24](https://stars.renci.org/var/babel_outputs/2024mar24/) (Babel Translator May 2024 Release) -- NodeNorm: [v2.3.11](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.11) +- NodeNorm: [v2.3.11](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.11) Next release: [Translator "Fugu" July 2024](TranslatorFuguJuly2024.md) @@ -12,26 +12,26 @@ Next release: [Translator "Fugu" July 2024](TranslatorFuguJuly2024.md) ## Babel updates (from Babel Translator May 2024 Release) * [New identifiers] 36.9 million PubMed IDs (e.g. `PMID:25061375`) have been added as `biolink:JournalArticle`, as well as the mappings to PMC (e.g. `PMC:PMC4109484`) and DOIs (e.g. `doi:10.3897/zookeys.420.7089`) that are included in PubMed. - Details in [TranslatorSRI/Babel#227](https://github.com/TranslatorSRI/Babel/pull/227). + Details in [NCATSTranslator/Babel#227](https://github.com/NCATSTranslator/Babel/pull/227). * Fixed type determination for DrugChemical conflation. Details in - [TranslatorSRI/Babel#266](https://github.com/TranslatorSRI/Babel/pull/266). + [NCATSTranslator/Babel#266](https://github.com/NCATSTranslator/Babel/pull/266). * Synonym files now include the clique identifier count (the number of identifiers in each clique) in synonyms file. * Minor fixes. ## Releases since [Translator December 2023 release](TranslatorDecember2023.md) -* [NodeNorm v2.3.6](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.6) +* [NodeNorm v2.3.6](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.6) * Reverted NodeNorm loader to Redis v6 -* [NodeNorm v2.3.7](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.7) +* [NodeNorm v2.3.7](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.7) * Added Translator release information * Fixed examples -* [NodeNorm v2.3.8](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.8): Forgot to increment the +* [NodeNorm v2.3.8](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.8): Forgot to increment the version in the previous release. -* [NodeNorm v2.3.9](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.9) +* [NodeNorm v2.3.9](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.9) * Bump gunicorn from 20.1.0 to 22.0.0 by @dependabot in #255 * Bump orjson from 3.8.10 to 3.9.15 by @dependabot in #253 * Added NodeNorm May 2024 release to the release notes by @gaurav in #258 * Sort chemical labels in the configured prefix order by @gaurav in #260 -* [NodeNorm v2.3.10](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.10) +* [NodeNorm v2.3.10](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.10) * Updated TRAPI version to 1.5.0 by @gaurav in #269 -* [NodeNorm v2.3.11](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.11) +* [NodeNorm v2.3.11](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.11) * Change default TRAPI version for URL to 1.5 by @gaurav in #270 \ No newline at end of file diff --git a/releases/TranslatorOctober2023.md b/releases/TranslatorOctober2023.md index fc7d67a..fcecf5e 100644 --- a/releases/TranslatorOctober2023.md +++ b/releases/TranslatorOctober2023.md @@ -1,6 +1,6 @@ # NodeNorm Translator October 2023 Release -- Babel: [2022dec2-2](https://stars.renci.org/var/babel_outputs/2022dec2-2/) (roughly Babel [v1.2.0](https://github.com/TranslatorSRI/Babel/releases/tag/v1.2.0)) -- NodeNorm: [v2.1.1](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.1.1) +- Babel: [2022dec2-2](https://stars.renci.org/var/babel_outputs/2022dec2-2/) (roughly Babel [v1.2.0](https://github.com/NCATSTranslator/Babel/releases/tag/v1.2.0)) +- NodeNorm: [v2.1.1](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.1.1) Next release: [December 2023](TranslatorDecember2023.md) \ No newline at end of file diff --git a/releases/v2.3.19.md b/releases/v2.3.19.md index c9b031c..48afc24 100644 --- a/releases/v2.3.19.md +++ b/releases/v2.3.19.md @@ -1,7 +1,7 @@ # NodeNorm v2.3.19 - Babel: [2025jan23](https://stars.renci.org/var/babel_outputs/2025jan23/) ([Babel 2025jan23](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025jan23.md)) -- NodeNorm: [v2.3.19](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.19) +- NodeNorm: [v2.3.19](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.19) Next release: [v2.3.26](./v2.3.26.md) Previous release: [Translator "Hammerhead" November 2024](./TranslatorHammerheadNovember2024.md) @@ -13,12 +13,12 @@ Previous release: [Translator "Hammerhead" November 2024](./TranslatorHammerhead * _None_ ## Babel updates (from [Babel 2025jan23](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025jan23.md)) -- [New feature] Added a check for duplicate CURIEs [#342](https://github.com/TranslatorSRI/Babel/pull/342). +- [New feature] Added a check for duplicate CURIEs [#342](https://github.com/NCATSTranslator/Babel/pull/342). - [New feature] Added some additional manual concords for Disease/Phenotype cliques and DrugChemical - conflation [#360](https://github.com/TranslatorSRI/Babel/pull/360). -- [New feature] Replace use of `has_tradename` with `tradename_of` in RxNorm ([#377](https://github.com/TranslatorSRI/Babel/pull/377)). -- [New feature] Added processes from UMLS ([#395](https://github.com/TranslatorSRI/Babel/pull/395)). -- [New feature] Improved EFO relationships ([#396](https://github.com/TranslatorSRI/Babel/pull/396)). + conflation [#360](https://github.com/NCATSTranslator/Babel/pull/360). +- [New feature] Replace use of `has_tradename` with `tradename_of` in RxNorm ([#377](https://github.com/NCATSTranslator/Babel/pull/377)). +- [New feature] Added processes from UMLS ([#395](https://github.com/NCATSTranslator/Babel/pull/395)). +- [New feature] Improved EFO relationships ([#396](https://github.com/NCATSTranslator/Babel/pull/396)). - [Updates] Various updates - [Bugfix] Fixed a bug in choosing the best label shorter than a particular size in src/babel_utils.py:write_compendium() - [Bugfix] Cleaned up src/createcompendia/chemicals.py:parse_smifile() so that includes the ChEMBL ID and calculates the column index by name, with ValueErrors thrown if the column name is missing. @@ -26,5 +26,5 @@ Previous release: [Translator "Hammerhead" November 2024](./TranslatorHammerhead - [Bugfix] Other minor fixes. ## Releases since [Translator "Hammerhead" November 2024](./TranslatorHammerheadNovember2024.md) -* [2.3.19](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.19) +* [2.3.19](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.19) * No changes: incrementing version solely to distinguish between Babel 2024oct24 and 2025jan23. diff --git a/releases/v2.3.26.md b/releases/v2.3.26.md index 750869a..1b28725 100644 --- a/releases/v2.3.26.md +++ b/releases/v2.3.26.md @@ -1,17 +1,17 @@ # NodeNorm v2.3.26 - Babel: [2025sep1](https://stars.renci.org/var/babel_outputs/2025sep1/) ([Babel 2025sep1](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025sep1.md)) -- NodeNorm: [v2.3.26](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.26) +- NodeNorm: [v2.3.26](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.26) Next release: [v2.4.1](./v2.4.1.md) Previous release: [v2.3.19](./v2.3.19.md) ## New features -* The Babel version is now included in the `/status` endpoint ([PR #315](https://github.com/TranslatorSRI/NodeNormalization/pull/315)). -* Added logging of normalization so we can benchmark NodeNorm ([PR #312](https://github.com/TranslatorSRI/NodeNormalization/pull/312)). +* The Babel version is now included in the `/status` endpoint ([PR #315](https://github.com/NCATSTranslator/NodeNormalization/pull/315)). +* Added logging of normalization so we can benchmark NodeNorm ([PR #312](https://github.com/NCATSTranslator/NodeNormalization/pull/312)). ## Updates -* The `/query` and `/asyncquery` endpoints are now deprecated ([PR #323](https://github.com/TranslatorSRI/NodeNormalization/pull/323)). +* The `/query` and `/asyncquery` endpoints are now deprecated ([PR #323](https://github.com/NCATSTranslator/NodeNormalization/pull/323)). * Improved documentation. * Added release notes for November 2024 and January 2025. * Updated renci-python-image to GitHub Packages. @@ -23,21 +23,21 @@ Previous release: [v2.3.19](./v2.3.19.md) - [MAJOR] Added secondary identifiers for ChEBI. ## Releases since [NodeNorm v2.3.19](./v2.3.19.md) -* [NodeNorm v2.3.20](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.20) +* [NodeNorm v2.3.20](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.20) * Add release info for November 2024 and January 2025 by @gaurav in #308 * Bump gunicorn from 22.0.0 to 23.0.0 by @dependabot in #310 -* [NodeNorm v2.3.21](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.21) +* [NodeNorm v2.3.21](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.21) * Incremented version (forgot to do that in the previous release). -* [NodeNorm v2.3.22](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.22) +* [NodeNorm v2.3.22](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.22) * Added normalization logging by @gaurav in #312 -* [NodeNorm v2.3.23](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.23) +* [NodeNorm v2.3.23](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.23) * Add Babel version to /status by @gaurav in #315 -* [NodeNorm v2.3.24](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.24) +* [NodeNorm v2.3.24](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.24) * Deprecated TRAPI methods (/query and /asyncquery) by @gaurav in #323 * Added API documentation by @gaurav in #322 * Updated RENCI Python Image in Dockerfile. -* [NodeNorm v2.3.25](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.25) +* [NodeNorm v2.3.25](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.25) * Removed unused/invalid imports. -* [NodeNorm v2.3.26](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.25) +* [NodeNorm v2.3.26](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.25) * Bump deepdiff from 5.8.1 to 8.6.1 by @dependabot in #328 * Bump requests from 2.32.0 to 2.32.4 by @dependabot in #319 diff --git a/releases/v2.4.1.md b/releases/v2.4.1.md index 69d36a9..d703930 100644 --- a/releases/v2.4.1.md +++ b/releases/v2.4.1.md @@ -1,7 +1,7 @@ # NodeNorm v2.4.1 - Babel: [2025sep1](https://stars.renci.org/var/babel_outputs/2025sep1/) ([Babel 2025sep1](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025sep1.md)) -- NodeNorm: [v2.4.1](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.4.1) +- NodeNorm: [v2.4.1](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.4.1) Next release: _None as yet_ Previous release: [v2.3.26](./v2.3.26.md) @@ -27,13 +27,13 @@ Previous release: [v2.3.26](./v2.3.26.md) Babel was not updated in this release. ## Releases since [NodeNorm v2.3.26](./v2.3.26.md) -* [NodeNorm v2.3.27](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.27) +* [NodeNorm v2.3.27](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.27) * Added release notes for NodeNorm v2.3.26 by @gaurav in [#329](https://github.com/NCATSTranslator/NodeNormalization/pull/329) * Add an `include_taxa` field (default: True) that will return the list of taxa for a clique and equivalent identifiers by @gaurav in [#339](https://github.com/NCATSTranslator/NodeNormalization/pull/339) * NEW BUG: `description` accidentally moved from `result['id']['description']` to `result['description']` ([#345](https://github.com/NCATSTranslator/NodeNormalization/pull/345)). -* [NodeNorm v2.3.28](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.27) +* [NodeNorm v2.3.28](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.27) * Fixed description bug introduced in NodeNorm v2.3.27 by @gaurav in [#346](https://github.com/NCATSTranslator/NodeNormalization/pull/346) -* [NodeNorm v2.4.0](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.4.0) +* [NodeNorm v2.4.0](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.4.0) * Replace RDB reader by @gaurav in [#349](https://github.com/NCATSTranslator/NodeNormalization/pull/349) * Update NodeNorm Jupyter Notebook by @gaurav in [#265](https://github.com/NCATSTranslator/NodeNormalization/pull/265) * Load a particular version of Biolink Model as per BIOLINK_MODEL_TAG by @gaurav in [#351](https://github.com/NCATSTranslator/NodeNormalization/pull/351) diff --git a/tests/test_docs_links.py b/tests/test_docs_links.py index 7de6309..86d1fb8 100644 --- a/tests/test_docs_links.py +++ b/tests/test_docs_links.py @@ -16,7 +16,8 @@ # Markdown we own. The notebook and anything vendored is deliberately excluded. MARKDOWN_FILES = sorted( p for p in REPO_ROOT.rglob("*.md") - if not any(part in {"venv", ".venv", "node_modules", ".git", ".pytest_cache"} for part in p.parts) + if not any(part in {"venv", ".venv", "node_modules", ".git", ".pytest_cache", "data"} + for part in p.parts) ) # Files that carry links out to GitHub in code rather than in prose. @@ -103,3 +104,24 @@ def test_no_master_branch_links(): if MASTER_LINK.search(line): offenders.append(f"{path.relative_to(REPO_ROOT)}:{line_no}") assert not offenders, "Links pinned to `master` instead of `main`:\n " + "\n ".join(offenders) + + +#: The three repositories that moved from the TranslatorSRI org to NCATSTranslator. Scoped to +#: these by name on purpose -- TranslatorSRI/r3 (referenced from README.md) really does still +#: live under that org, so a blanket ban on the string would push someone to "fix" a correct +#: link into a 404. +MOVED_TO_NCATSTRANSLATOR = re.compile(r"TranslatorSRI/(Babel|NameResolution|NodeNormalization)\b") + + +def test_no_stale_org_links(): + """These three repositories are under NCATSTranslator now. The old URLs resolve through + GitHub's org-rename redirect, which is more durable than the branch-rename one, but they + still name an org that no longer owns the code.""" + offenders = [] + for path in MARKDOWN_FILES + SOURCE_WITH_LINKS: + for line_no, line in enumerate(path.read_text().splitlines(), start=1): + if MOVED_TO_NCATSTRANSLATOR.search(line): + offenders.append(f"{path.relative_to(REPO_ROOT)}:{line_no}") + assert not offenders, ( + "Links naming the pre-rename TranslatorSRI org:\n " + "\n ".join(offenders) + )