From 57851472478f7c301dd052fa766e1ed4d26318d1 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Mon, 27 Apr 2026 18:42:09 -0400 Subject: [PATCH 01/46] Add Translator Guide and improve README for issue #133 Creates documentation/TranslatorGuide.md with Translator-specific guidance: what to do when lookup results are unexpected (highlighting, type/prefix/taxa filters, autocomplete), when to use /synonyms vs. NodeNorm, and performance tips (bulk-lookup, filtering, caching). Updates README.md with a Colab badge link to the notebook and a structured Getting Started / Documentation section layout. Co-Authored-By: Claude Sonnet 4.6 --- README.md | 16 ++- documentation/TranslatorGuide.md | 184 +++++++++++++++++++++++++++++++ 2 files changed, 196 insertions(+), 4 deletions(-) create mode 100644 documentation/TranslatorGuide.md diff --git a/README.md b/README.md index 79c60fe0..66dd0810 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,15 @@ Multiple results may be returned representing possible conceptual matches, but a Note that the results returned by this service have been conflated using both GeneProtein and DrugChemical conflation; you can read more about this at the [Conflation documentation](https://github.com/NCATSTranslator/Babel/blob/master/docs/Conflation.md). -* See this [Jupyter Notebook](documentation/NameResolution.ipynb) for examples of use. -* See the [API documentation](documentation/API.md) for information about the NameRes API. -* See [Scoring](documentation/Scoring.md) for information about the scoring algorithm used by NameRes. -* See [Deployment](documentation/Deployment.md) for instructions on deploying NameRes. +## Getting started + +The best place to start is the Jupyter Notebook, which walks through the most common use cases with live examples: + +* [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NCATSTranslator/NameResolution/blob/master/documentation/NameResolution.ipynb) [Jupyter Notebook](documentation/NameResolution.ipynb) — interactive examples covering lookup, filtering, autocomplete, bulk lookup, and synonyms + +## Documentation + +* [Translator Guide](documentation/TranslatorGuide.md) — what to do when results are unexpected, when to use `/synonyms` vs. NodeNorm, and performance tips +* [API documentation](documentation/API.md) — full reference for all NameRes endpoints +* [Scoring](documentation/Scoring.md) — how NameRes scores and ranks results +* [Deployment](documentation/Deployment.md) — instructions for deploying NameRes diff --git a/documentation/TranslatorGuide.md b/documentation/TranslatorGuide.md new file mode 100644 index 00000000..ed5cc81f --- /dev/null +++ b/documentation/TranslatorGuide.md @@ -0,0 +1,184 @@ +# NameRes Translator Guide + +This guide is aimed at Translator developers and users who are integrating NameRes into their workflows. +It covers what to do when results are unexpected, how `/synonyms` (reverse-lookup) relates to NodeNorm, +and tips for improving performance. + +## What to do when a name lookup returns unexpected results + +NameRes ranks results by a [Solr TF*IDF score](./Scoring.md) — the top result is the best *textual* match, +not necessarily the biologically intended concept. If the results don't look right, try these steps. + +### 1. Use `highlighting` to understand what matched + +Set `highlighting=true` on a `/lookup` call to see which label or synonym drove the match: + +``` +GET /lookup?string=cold&highlighting=true&limit=5 +``` + +This tells you which synonym triggered the match, which helps diagnose why an unexpected concept ranked high. + +### 2. Filter by Biolink type + +Use `biolink_type` to restrict results to the category you expect. Multiple types are combined with OR logic: + +``` +GET /lookup?string=cold&biolink_type=Disease&biolink_type=PhenotypicFeature +``` + +Common types: `Disease`, `Gene`, `ChemicalEntity`, `PhenotypicFeature`, `BiologicalProcess`, `AnatomicalEntity`. +Types can be specified with or without the `biolink:` prefix. + +### 3. Restrict to trusted prefixes + +Use `only_prefixes` to limit results to a specific ontology, or `exclude_prefixes` to drop a noisy one. +Prefixes are pipe-separated and case-sensitive: + +``` +# Only MONDO disease identifiers +GET /lookup?string=diabetes&biolink_type=Disease&only_prefixes=MONDO + +# Exclude UMLS (often produces many ambiguous matches) +GET /lookup?string=NIH&exclude_prefixes=UMLS +``` + +Common trusted prefixes by category: + +| Category | Recommended prefixes | +|---|---| +| Disease | `MONDO`, `OMIM`, `ORPHANET` | +| Gene | `NCBIGene`, `HGNC` | +| Chemical/Drug | `CHEBI`, `DRUGBANK` | +| Phenotype | `HP`, `MP` | +| Anatomy | `UBERON`, `CL` | + +### 4. Filter by taxon for gene/protein queries + +When searching for a gene or protein, results may include entries from multiple species. Use `only_taxa` +to restrict to a specific organism. The value is a pipe-separated list of NCBI Taxon CURIEs: + +``` +# Human genes only +GET /lookup?string=APOE&biolink_type=Gene&only_taxa=NCBITaxon:9606 + +# Human and mouse +GET /lookup?string=APOE&only_taxa=NCBITaxon:9606|NCBITaxon:10090 +``` + +Common taxa: human `NCBITaxon:9606`, mouse `NCBITaxon:10090`, rat `NCBITaxon:10116`, zebrafish `NCBITaxon:7955`. + +### 5. Try autocomplete mode for partial strings + +If your search string is a fragment of a name (e.g., typed by a user mid-word), set `autocomplete=true`. +This expands the final word with a wildcard so that `"diab"` matches `"diabetes"`, `"diabetic"`, etc.: + +``` +GET /lookup?string=diab&autocomplete=true&limit=5 +``` + +Without `autocomplete`, `"diab"` will only match documents that literally contain the token `"diab"`. + +### 6. If the correct concept is consistently missing + +If your filtering is correct but the expected result never appears, the concept may be missing from the +Babel data that NameRes is built from. Consider filing an issue on: +- [NameRes GitHub](https://github.com/NCATSTranslator/NameResolution/issues) — for search/ranking problems +- [Babel GitHub](https://github.com/NCATSTranslator/Babel/issues) — for missing synonyms or identifiers + +--- + +## Using `/synonyms` (reverse-lookup) vs. NodeNorm + +These two services answer different questions. + +### Use `/synonyms` when you want to inspect synonyms for a known CURIE + +The `/synonyms` endpoint returns all names and synonyms that NameRes knows for a given concept, along with +its Biolink types, taxa, and clique identifier count. This is useful for verifying synonym coverage or +debugging why a particular name did or did not match. + +``` +GET /synonyms?preferred_curies=NCBIGene:1756 +``` + +**Important:** `/synonyms` requires the *preferred* (normalized) CURIE. If you pass a non-preferred +identifier (e.g. a UniProtKB accession for a gene), you will get an empty result. Before calling +`/synonyms`, normalize your CURIE with NodeNorm (see below). + +You can look up multiple CURIEs in one request: + +``` +GET /synonyms?preferred_curies=MONDO:0005148&preferred_curies=NCBIGene:1756 +``` + +### Use NodeNorm when you need identifier normalization or equivalent identifiers + +The [Node Normalization service](https://nodenormalization-sri.renci.org/) is the right tool when you need to: + +- Convert a non-preferred identifier to its preferred CURIE +- Find all equivalent identifiers for a concept across ontologies +- Check which Biolink types a CURIE maps to +- Determine whether two CURIEs refer to the same concept + +To normalize a CURIE before passing it to `/synonyms`, call NodeNorm with GeneProtein and DrugChemical +conflation enabled (to match the conflation used by NameRes): + +``` +GET https://nodenormalization-sri.renci.org/get_normalized_nodes?curie=UniProtKB:A0A0S2Z3B5&conflate=true&drug_chemical=true +``` + +The `id.identifier` field in the response is the preferred CURIE you can then pass to `/synonyms`. + +### Quick decision guide + +| Question | Tool | +|---|---| +| What synonyms does NameRes know for this CURIE? | `/synonyms` | +| What is the preferred identifier for this concept? | NodeNorm | +| Are these two CURIEs equivalent? | NodeNorm | +| What Biolink types does this CURIE have? | NodeNorm | +| Why didn't a particular name match in `/lookup`? | `/synonyms` + `highlighting` | + +--- + +## Performance tips + +### Batch multiple queries with `/bulk-lookup` + +Instead of making N separate `/lookup` calls, send them all in one POST request to `/bulk-lookup`. +It returns a dictionary keyed by input string: + +```json +POST /bulk-lookup +{ + "strings": ["diabetes", "hypertension", "asthma"], + "limit": 5, + "biolink_types": ["Disease"] +} +``` + +This is significantly more efficient than sequential individual requests. + +### Add filters before processing results + +Apply `biolink_type`, `only_prefixes`, and `only_taxa` at query time rather than filtering the response +yourself. Server-side filtering reduces the result set before it is serialized and transmitted. + +### Set `limit` to what you actually need + +The default `limit` is 10 and the maximum is 1000. If you only need the top result, set `limit=1`. +If you need to page through a large result set, use `offset` for server-side pagination rather than +requesting a large `limit` and slicing client-side. + +### Cache results between Babel data releases + +NameRes results are stable between Babel data releases (which happen a few times per year). If your +application calls NameRes repeatedly for the same input strings, cache the results locally. Check the +`/status` endpoint to detect when the Babel version changes and invalidate your cache accordingly: + +``` +GET /status +``` + +The `babel_version` field in the response changes with each data release. From 17fb4a196bdab0f2b1292cf42d352b13219bcce1 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Mon, 27 Apr 2026 18:59:57 -0400 Subject: [PATCH 02/46] Document conflations in OpenAPI spec and /status endpoint - Fix typo and improve conflation paragraph in openapi.yml; update Babel Conflation.md link from master to main - Add conflations list and conflation_url to /status response, driven by a CONFLATIONS env var (default: GeneProtein,DrugChemical) so each deployment can self-describe - Fix incomplete sentence and improve conflation section in API.md; add new /status fields to example response - Add /status to the quick decision table in TranslatorGuide.md Closes #202 Co-Authored-By: Claude Sonnet 4.6 --- api/resources/openapi.yml | 6 ++++-- api/server.py | 9 +++++++++ documentation/API.md | 20 +++++++++++++------- documentation/TranslatorGuide.md | 1 + 4 files changed, 27 insertions(+), 9 deletions(-) diff --git a/api/resources/openapi.yml b/api/resources/openapi.yml index 07313170..6a9ac0dc 100644 --- a/api/resources/openapi.yml +++ b/api/resources/openapi.yml @@ -13,8 +13,10 @@ info: have been correctly normalized using the Node Normalization service.

You can read more about this API on the NameResolution GitHub repository.

- Note that the returned by this service have been conflated using both GeneProtein and DrugChemical conflation; - you can read more about this at the Conflation documentation.' +

Note that the results returned by this service have been conflated using both GeneProtein and DrugChemical + conflation; you can read more about this at the + Conflation documentation. + The active conflations for any deployment can be discovered via the /status endpoint.

' license: name: MIT url: https://opensource.org/licenses/MIT diff --git a/api/server.py b/api/server.py index 2a2277b6..e6e3cb5e 100755 --- a/api/server.py +++ b/api/server.py @@ -71,6 +71,11 @@ async def status() -> Dict: babel_version = os.environ.get("BABEL_VERSION", "unknown") babel_version_url = os.environ.get("BABEL_VERSION_URL", "") + # Which conflations are active in this deployment? Baked in at data-loading time. + conflations_raw = os.environ.get("CONFLATIONS", "GeneProtein,DrugChemical") + conflations = [c.strip() for c in conflations_raw.split(",") if c.strip()] + conflation_url = "https://github.com/NCATSTranslator/Babel/blob/main/docs/Conflation.md" + # Look up the BIOLINK_MODEL_TAG. # Note: this should be a tag from the Biolink Model repo, e.g. "master" or "v4.3.6". biolink_model_tag = os.environ.get("BIOLINK_MODEL_TAG", "master") @@ -101,6 +106,8 @@ async def status() -> Dict: 'url': biolink_model_url, 'download_url': biolink_model_download_url, }, + 'conflations': conflations, + 'conflation_url': conflation_url, 'nameres_version': nameres_version, 'startTime': core['startTime'], 'numDocs': index.get('numDocs', ''), @@ -122,6 +129,8 @@ async def status() -> Dict: 'url': biolink_model_url, 'download_url': biolink_model_download_url, }, + 'conflations': conflations, + 'conflation_url': conflation_url, 'nameres_version': nameres_version, } diff --git a/documentation/API.md b/documentation/API.md index 57bcbdea..8db89f68 100644 --- a/documentation/API.md +++ b/documentation/API.md @@ -91,13 +91,17 @@ The Name Resolver largely consists of two [search endpoints](#search-endpoints): ## Conflation Unlike the Node Normalizer, the Name Resolution Service does not currently support on-the-fly conflation. Instead, -all the [Babel conflations](https://github.com/NCATSTranslator/Babel/blob/master/docs/Conflation.md) are turned on when Solr database is built. At the moment, this includes: -* GeneProtein conflation: protein-encoding genes are conflated with the protein(s) they encode, and the gene identifier - is used to identify this concept. Therefore, if you search for "" -* DrugChemical conflation: drugs are conflated with their active ingredient, and the identifier for the active ingredient - is used to identify this concept. -This means that -- for example -- protein-encoding genes will include the synonyms found -for the protein they encode, and that no separate entry will be available for those proteins. +all the [Babel conflations](https://github.com/NCATSTranslator/Babel/blob/main/docs/Conflation.md) are baked in when the Solr database is built. At the moment, this includes: +* **GeneProtein conflation:** protein-encoding genes are conflated with the protein(s) they encode, and the gene identifier + is used to identify this concept. Therefore, if you search for a protein name, you will typically receive the gene + identifier (e.g., searching for "dystrophin" returns `NCBIGene:1756` rather than a UniProtKB identifier). +* **DrugChemical conflation:** drugs are conflated with their active ingredient, and the identifier for the active + ingredient is used to identify this concept. + +This means that protein-encoding genes include the synonyms found for the protein they encode, and no separate +entry is available for those proteins in NameRes. + +The active conflations for any NameRes deployment can be queried programmatically via the [`/status` endpoint](#status). Once you have an identifier from Name Resolver, you can use the [Node Normalizer](https://nodenormalization-sri.renci.org/) to look up the equivalent identifiers for that CURIE with and without conflation. Please use the Node Normalizer @@ -325,6 +329,8 @@ Solr database. "url": "https://github.com/biolink/biolink-model/tree/v4.2.6-rc5", "download_url": "https://raw.githubusercontent.com/biolink/biolink-model/v4.2.6-rc5/biolink-model.yaml" }, + "conflations": ["GeneProtein", "DrugChemical"], + "conflation_url": "https://github.com/NCATSTranslator/Babel/blob/main/docs/Conflation.md", "nameres_version": "v1.5.1", "startTime": "2025-12-19T11:53:09.638Z", "numDocs": 425583391, diff --git a/documentation/TranslatorGuide.md b/documentation/TranslatorGuide.md index ed5cc81f..763412cf 100644 --- a/documentation/TranslatorGuide.md +++ b/documentation/TranslatorGuide.md @@ -139,6 +139,7 @@ The `id.identifier` field in the response is the preferred CURIE you can then pa | Are these two CURIEs equivalent? | NodeNorm | | What Biolink types does this CURIE have? | NodeNorm | | Why didn't a particular name match in `/lookup`? | `/synonyms` + `highlighting` | +| Which conflations are active in this NameRes deployment? | `/status` (`conflations` field) | --- From 27c74ada422cfbc7853347931348c3aaa4465eee Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Sat, 16 May 2026 01:11:46 -0400 Subject: [PATCH 03/46] Fix descriptions of the `limit` GET/POST parameter. Closes #269. --- api/server.py | 6 +++--- documentation/API.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/api/server.py b/api/server.py index 011c5d2b..9f1fcc08 100755 --- a/api/server.py +++ b/api/server.py @@ -281,7 +281,7 @@ async def lookup_curies_get( ge=0 )] = 0, limit: Annotated[int, Query( - description="The number of results to skip. Can be used to page through the results of a query.", + description="The number of results to return. Can be used to page through the results of a query.", # Limit should be greater than or equal to zero and less than or equal to 1000. ge=0, le=1000 @@ -345,7 +345,7 @@ async def lookup_curies_post( ge=0 )] = 0, limit: Annotated[int, Query( - description="The number of results to skip. Can be used to page through the results of a query.", + description="The number of results to return. Can be used to page through the results of a query.", # Limit should be greater than or equal to zero and less than or equal to 1000. ge=0, le=1000 @@ -638,7 +638,7 @@ class NameResQuery(BaseModel): ) limit: Optional[int] = Field( 10, - description="The number of results to skip. Can be used to page through the results of a query.", + description="The number of results to return. Can be used to page through the results of a query.", # Limit should be greater than or equal to zero and less than or equal to 1000. ge=0, le=1000 diff --git a/documentation/API.md b/documentation/API.md index 57bcbdea..5311a2d5 100644 --- a/documentation/API.md +++ b/documentation/API.md @@ -122,7 +122,7 @@ Search for cliques by a fragment of a name or synonym. - `autocomplete` (optional, boolean, default: false): If `true`, treats the input string as incomplete and looks for terms that start with the final word. If `false`, treats the entire phrase as complete (entity linker mode). - `highlighting` (optional, boolean, default: false): If `true`, returns information on which labels and synonyms matched the search query. - `offset` (optional, integer, default: 0, minimum: 0): The number of results to skip. Used for pagination. -- `limit` (optional, integer, default: 10, minimum: 0, maximum: 1000): The maximum number of results to return. Used for pagination. +- `limit` (optional, integer, default: 10, minimum: 0, maximum: 1000): The number of results to return. Used for pagination. - `biolink_type` (optional, list of strings): Filter results to specific Biolink types. Types can be specified with or without the `biolink:` prefix (e.g., `biolink:Disease` or `Disease`). Multiple types are combined with OR logic, so filtering for `PhenotypicFeature` and `Disease` will return concepts that are either phenotypic features OR diseases. - `only_prefixes` (optional, string): Pipe-separated, case-sensitive list of CURIE prefixes to include (e.g., `MONDO|EFO`). Only results with matching prefixes will be returned. - `exclude_prefixes` (optional, string): Pipe-separated, case-sensitive list of CURIE prefixes to exclude (e.g., `UMLS|EFO`). Results with matching prefixes will be filtered out. From 5c7a1a425ec1207c2c41084334c2bb0b183410e9 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 16 Jul 2026 10:24:04 -0400 Subject: [PATCH 04/46] Add checked-in Solr configset and parallel data loader with doc-count guard The Solr schema was defined twice -- in setup_solr.sh at load time and again in the Helm restore script at serve time -- and had already drifted. Make the checked-in configset (configsets/name_lookup/conf) the single source of truth, generated from a real Solr so behaviour is unchanged, with the malformed `types` field fixed, a larger index buffer, autoSoftCommit off, and a grown queryResultCache (closes #266). Rewrite setup-and-load-solr.sh to stream files to Solr in parallel with a single deferred commit (no per-file commit, no sleeps), and guard against dropped data by counting input documents before the load and comparing against Solr's count afterward, with curl --fail on every upload. The core is now created from the configset (see the Makefile / CI), so this script no longer sets up the schema and setup_solr.sh is deleted. Convert the test fixture to JSON-lines to match the Babel production format so CI exercises the line-counting guard. Co-Authored-By: Claude Opus 4.8 --- .../name_lookup/conf/configoverlay.json | 1 + .../name_lookup/conf/lang/contractions_ca.txt | 8 + .../name_lookup/conf/lang/contractions_fr.txt | 15 + .../name_lookup/conf/lang/contractions_ga.txt | 5 + .../name_lookup/conf/lang/contractions_it.txt | 23 + .../name_lookup/conf/lang/hyphenations_ga.txt | 5 + .../name_lookup/conf/lang/stemdict_nl.txt | 6 + .../name_lookup/conf/lang/stoptags_ja.txt | 420 ++++ .../name_lookup/conf/lang/stopwords_ar.txt | 125 + .../name_lookup/conf/lang/stopwords_bg.txt | 193 ++ .../name_lookup/conf/lang/stopwords_ca.txt | 220 ++ .../name_lookup/conf/lang/stopwords_cz.txt | 172 ++ .../name_lookup/conf/lang/stopwords_da.txt | 110 + .../name_lookup/conf/lang/stopwords_de.txt | 294 +++ .../name_lookup/conf/lang/stopwords_el.txt | 78 + .../name_lookup/conf/lang/stopwords_en.txt | 54 + .../name_lookup/conf/lang/stopwords_es.txt | 356 +++ .../name_lookup/conf/lang/stopwords_et.txt | 1603 ++++++++++++ .../name_lookup/conf/lang/stopwords_eu.txt | 99 + .../name_lookup/conf/lang/stopwords_fa.txt | 313 +++ .../name_lookup/conf/lang/stopwords_fi.txt | 96 + .../name_lookup/conf/lang/stopwords_fr.txt | 186 ++ .../name_lookup/conf/lang/stopwords_ga.txt | 110 + .../name_lookup/conf/lang/stopwords_gl.txt | 161 ++ .../name_lookup/conf/lang/stopwords_hi.txt | 235 ++ .../name_lookup/conf/lang/stopwords_hu.txt | 211 ++ .../name_lookup/conf/lang/stopwords_hy.txt | 46 + .../name_lookup/conf/lang/stopwords_id.txt | 359 +++ .../name_lookup/conf/lang/stopwords_it.txt | 303 +++ .../name_lookup/conf/lang/stopwords_ja.txt | 127 + .../name_lookup/conf/lang/stopwords_lv.txt | 172 ++ .../name_lookup/conf/lang/stopwords_nl.txt | 121 + .../name_lookup/conf/lang/stopwords_no.txt | 190 ++ .../name_lookup/conf/lang/stopwords_pt.txt | 253 ++ .../name_lookup/conf/lang/stopwords_ro.txt | 233 ++ .../name_lookup/conf/lang/stopwords_ru.txt | 244 ++ .../name_lookup/conf/lang/stopwords_sv.txt | 133 + .../name_lookup/conf/lang/stopwords_th.txt | 119 + .../name_lookup/conf/lang/stopwords_tr.txt | 212 ++ .../name_lookup/conf/lang/userdict_ja.txt | 29 + .../name_lookup/conf/managed-schema.xml | 549 +++++ .../configsets/name_lookup/conf/protwords.txt | 21 + .../name_lookup/conf/solrconfig.xml | 1032 ++++++++ .../configsets/name_lookup/conf/stopwords.txt | 14 + .../configsets/name_lookup/conf/synonyms.txt | 29 + data-loading/setup-and-load-solr.sh | 102 +- data-loading/setup_solr.sh | 147 -- tests/create_test_synonyms.py | 7 +- tests/data/test-synonyms.json | 2172 +---------------- 49 files changed, 9456 insertions(+), 2257 deletions(-) create mode 100644 data-loading/configsets/name_lookup/conf/configoverlay.json create mode 100644 data-loading/configsets/name_lookup/conf/lang/contractions_ca.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/contractions_fr.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/contractions_ga.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/contractions_it.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/hyphenations_ga.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stemdict_nl.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stoptags_ja.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_ar.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_bg.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_ca.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_cz.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_da.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_de.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_el.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_en.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_es.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_et.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_eu.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_fa.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_fi.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_fr.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_ga.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_gl.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_hi.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_hu.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_hy.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_id.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_it.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_ja.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_lv.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_nl.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_no.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_pt.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_ro.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_ru.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_sv.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_th.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/stopwords_tr.txt create mode 100644 data-loading/configsets/name_lookup/conf/lang/userdict_ja.txt create mode 100644 data-loading/configsets/name_lookup/conf/managed-schema.xml create mode 100644 data-loading/configsets/name_lookup/conf/protwords.txt create mode 100644 data-loading/configsets/name_lookup/conf/solrconfig.xml create mode 100644 data-loading/configsets/name_lookup/conf/stopwords.txt create mode 100644 data-loading/configsets/name_lookup/conf/synonyms.txt delete mode 100644 data-loading/setup_solr.sh diff --git a/data-loading/configsets/name_lookup/conf/configoverlay.json b/data-loading/configsets/name_lookup/conf/configoverlay.json new file mode 100644 index 00000000..271cd0a4 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/configoverlay.json @@ -0,0 +1 @@ +{"userProps":{"update.autoCreateFields":"false"}} \ No newline at end of file diff --git a/data-loading/configsets/name_lookup/conf/lang/contractions_ca.txt b/data-loading/configsets/name_lookup/conf/lang/contractions_ca.txt new file mode 100644 index 00000000..307a85f9 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/contractions_ca.txt @@ -0,0 +1,8 @@ +# Set of Catalan contractions for ElisionFilter +# TODO: load this as a resource from the analyzer and sync it in build.xml +d +l +m +n +s +t diff --git a/data-loading/configsets/name_lookup/conf/lang/contractions_fr.txt b/data-loading/configsets/name_lookup/conf/lang/contractions_fr.txt new file mode 100644 index 00000000..f1bba51b --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/contractions_fr.txt @@ -0,0 +1,15 @@ +# Set of French contractions for ElisionFilter +# TODO: load this as a resource from the analyzer and sync it in build.xml +l +m +t +qu +n +s +j +d +c +jusqu +quoiqu +lorsqu +puisqu diff --git a/data-loading/configsets/name_lookup/conf/lang/contractions_ga.txt b/data-loading/configsets/name_lookup/conf/lang/contractions_ga.txt new file mode 100644 index 00000000..9ebe7fa3 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/contractions_ga.txt @@ -0,0 +1,5 @@ +# Set of Irish contractions for ElisionFilter +# TODO: load this as a resource from the analyzer and sync it in build.xml +d +m +b diff --git a/data-loading/configsets/name_lookup/conf/lang/contractions_it.txt b/data-loading/configsets/name_lookup/conf/lang/contractions_it.txt new file mode 100644 index 00000000..cac04095 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/contractions_it.txt @@ -0,0 +1,23 @@ +# Set of Italian contractions for ElisionFilter +# TODO: load this as a resource from the analyzer and sync it in build.xml +c +l +all +dall +dell +nell +sull +coll +pell +gl +agl +dagl +degl +negl +sugl +un +m +t +s +v +d diff --git a/data-loading/configsets/name_lookup/conf/lang/hyphenations_ga.txt b/data-loading/configsets/name_lookup/conf/lang/hyphenations_ga.txt new file mode 100644 index 00000000..4d2642cc --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/hyphenations_ga.txt @@ -0,0 +1,5 @@ +# Set of Irish hyphenations for StopFilter +# TODO: load this as a resource from the analyzer and sync it in build.xml +h +n +t diff --git a/data-loading/configsets/name_lookup/conf/lang/stemdict_nl.txt b/data-loading/configsets/name_lookup/conf/lang/stemdict_nl.txt new file mode 100644 index 00000000..44107297 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stemdict_nl.txt @@ -0,0 +1,6 @@ +# Set of overrides for the dutch stemmer +# TODO: load this as a resource from the analyzer and sync it in build.xml +fiets fiets +bromfiets bromfiets +ei eier +kind kinder diff --git a/data-loading/configsets/name_lookup/conf/lang/stoptags_ja.txt b/data-loading/configsets/name_lookup/conf/lang/stoptags_ja.txt new file mode 100644 index 00000000..71b75084 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stoptags_ja.txt @@ -0,0 +1,420 @@ +# +# This file defines a Japanese stoptag set for JapanesePartOfSpeechStopFilter. +# +# Any token with a part-of-speech tag that exactly matches those defined in this +# file are removed from the token stream. +# +# Set your own stoptags by uncommenting the lines below. Note that comments are +# not allowed on the same line as a stoptag. See LUCENE-3745 for frequency lists, +# etc. that can be useful for building you own stoptag set. +# +# The entire possible tagset is provided below for convenience. +# +##### +# noun: unclassified nouns +#名詞 +# +# noun-common: Common nouns or nouns where the sub-classification is undefined +#名詞-一般 +# +# noun-proper: Proper nouns where the sub-classification is undefined +#名詞-固有名詞 +# +# noun-proper-misc: miscellaneous proper nouns +#名詞-固有名詞-一般 +# +# noun-proper-person: Personal names where the sub-classification is undefined +#名詞-固有名詞-人名 +# +# noun-proper-person-misc: names that cannot be divided into surname and +# given name; foreign names; names where the surname or given name is unknown. +# e.g. お市の方 +#名詞-固有名詞-人名-一般 +# +# noun-proper-person-surname: Mainly Japanese surnames. +# e.g. 山田 +#名詞-固有名詞-人名-姓 +# +# noun-proper-person-given_name: Mainly Japanese given names. +# e.g. 太郎 +#名詞-固有名詞-人名-名 +# +# noun-proper-organization: Names representing organizations. +# e.g. 通産省, NHK +#名詞-固有名詞-組織 +# +# noun-proper-place: Place names where the sub-classification is undefined +#名詞-固有名詞-地域 +# +# noun-proper-place-misc: Place names excluding countries. +# e.g. アジア, バルセロナ, 京都 +#名詞-固有名詞-地域-一般 +# +# noun-proper-place-country: Country names. +# e.g. 日本, オーストラリア +#名詞-固有名詞-地域-国 +# +# noun-pronoun: Pronouns where the sub-classification is undefined +#名詞-代名詞 +# +# noun-pronoun-misc: miscellaneous pronouns: +# e.g. それ, ここ, あいつ, あなた, あちこち, いくつ, どこか, なに, みなさん, みんな, わたくし, われわれ +#名詞-代名詞-一般 +# +# noun-pronoun-contraction: Spoken language contraction made by combining a +# pronoun and the particle 'wa'. +# e.g. ありゃ, こりゃ, こりゃあ, そりゃ, そりゃあ +#名詞-代名詞-縮約 +# +# noun-adverbial: Temporal nouns such as names of days or months that behave +# like adverbs. Nouns that represent amount or ratios and can be used adverbially, +# e.g. 金曜, 一月, 午後, 少量 +#名詞-副詞可能 +# +# noun-verbal: Nouns that take arguments with case and can appear followed by +# 'suru' and related verbs (する, できる, なさる, くださる) +# e.g. インプット, 愛着, 悪化, 悪戦苦闘, 一安心, 下取り +#名詞-サ変接続 +# +# noun-adjective-base: The base form of adjectives, words that appear before な ("na") +# e.g. 健康, 安易, 駄目, だめ +#名詞-形容動詞語幹 +# +# noun-numeric: Arabic numbers, Chinese numerals, and counters like 何 (回), 数. +# e.g. 0, 1, 2, 何, 数, 幾 +#名詞-数 +# +# noun-affix: noun affixes where the sub-classification is undefined +#名詞-非自立 +# +# noun-affix-misc: Of adnominalizers, the case-marker の ("no"), and words that +# attach to the base form of inflectional words, words that cannot be classified +# into any of the other categories below. This category includes indefinite nouns. +# e.g. あかつき, 暁, かい, 甲斐, 気, きらい, 嫌い, くせ, 癖, こと, 事, ごと, 毎, しだい, 次第, +# 順, せい, 所為, ついで, 序で, つもり, 積もり, 点, どころ, の, はず, 筈, はずみ, 弾み, +# 拍子, ふう, ふり, 振り, ほう, 方, 旨, もの, 物, 者, ゆえ, 故, ゆえん, 所以, わけ, 訳, +# わり, 割り, 割, ん-口語/, もん-口語/ +#名詞-非自立-一般 +# +# noun-affix-adverbial: noun affixes that that can behave as adverbs. +# e.g. あいだ, 間, あげく, 挙げ句, あと, 後, 余り, 以外, 以降, 以後, 以上, 以前, 一方, うえ, +# 上, うち, 内, おり, 折り, かぎり, 限り, きり, っきり, 結果, ころ, 頃, さい, 際, 最中, さなか, +# 最中, じたい, 自体, たび, 度, ため, 為, つど, 都度, とおり, 通り, とき, 時, ところ, 所, +# とたん, 途端, なか, 中, のち, 後, ばあい, 場合, 日, ぶん, 分, ほか, 他, まえ, 前, まま, +# 儘, 侭, みぎり, 矢先 +#名詞-非自立-副詞可能 +# +# noun-affix-aux: noun affixes treated as 助動詞 ("auxiliary verb") in school grammars +# with the stem よう(だ) ("you(da)"). +# e.g. よう, やう, 様 (よう) +#名詞-非自立-助動詞語幹 +# +# noun-affix-adjective-base: noun affixes that can connect to the indeclinable +# connection form な (aux "da"). +# e.g. みたい, ふう +#名詞-非自立-形容動詞語幹 +# +# noun-special: special nouns where the sub-classification is undefined. +#名詞-特殊 +# +# noun-special-aux: The そうだ ("souda") stem form that is used for reporting news, is +# treated as 助動詞 ("auxiliary verb") in school grammars, and attach to the base +# form of inflectional words. +# e.g. そう +#名詞-特殊-助動詞語幹 +# +# noun-suffix: noun suffixes where the sub-classification is undefined. +#名詞-接尾 +# +# noun-suffix-misc: Of the nouns or stem forms of other parts of speech that connect +# to ガル or タイ and can combine into compound nouns, words that cannot be classified into +# any of the other categories below. In general, this category is more inclusive than +# 接尾語 ("suffix") and is usually the last element in a compound noun. +# e.g. おき, かた, 方, 甲斐 (がい), がかり, ぎみ, 気味, ぐるみ, (～した) さ, 次第, 済 (ず) み, +# よう, (でき)っこ, 感, 観, 性, 学, 類, 面, 用 +#名詞-接尾-一般 +# +# noun-suffix-person: Suffixes that form nouns and attach to person names more often +# than other nouns. +# e.g. 君, 様, 著 +#名詞-接尾-人名 +# +# noun-suffix-place: Suffixes that form nouns and attach to place names more often +# than other nouns. +# e.g. 町, 市, 県 +#名詞-接尾-地域 +# +# noun-suffix-verbal: Of the suffixes that attach to nouns and form nouns, those that +# can appear before スル ("suru"). +# e.g. 化, 視, 分け, 入り, 落ち, 買い +#名詞-接尾-サ変接続 +# +# noun-suffix-aux: The stem form of そうだ (様態) that is used to indicate conditions, +# is treated as 助動詞 ("auxiliary verb") in school grammars, and attach to the +# conjunctive form of inflectional words. +# e.g. そう +#名詞-接尾-助動詞語幹 +# +# noun-suffix-adjective-base: Suffixes that attach to other nouns or the conjunctive +# form of inflectional words and appear before the copula だ ("da"). +# e.g. 的, げ, がち +#名詞-接尾-形容動詞語幹 +# +# noun-suffix-adverbial: Suffixes that attach to other nouns and can behave as adverbs. +# e.g. 後 (ご), 以後, 以降, 以前, 前後, 中, 末, 上, 時 (じ) +#名詞-接尾-副詞可能 +# +# noun-suffix-classifier: Suffixes that attach to numbers and form nouns. This category +# is more inclusive than 助数詞 ("classifier") and includes common nouns that attach +# to numbers. +# e.g. 個, つ, 本, 冊, パーセント, cm, kg, カ月, か国, 区画, 時間, 時半 +#名詞-接尾-助数詞 +# +# noun-suffix-special: Special suffixes that mainly attach to inflecting words. +# e.g. (楽し) さ, (考え) 方 +#名詞-接尾-特殊 +# +# noun-suffix-conjunctive: Nouns that behave like conjunctions and join two words +# together. +# e.g. (日本) 対 (アメリカ), 対 (アメリカ), (3) 対 (5), (女優) 兼 (主婦) +#名詞-接続詞的 +# +# noun-verbal_aux: Nouns that attach to the conjunctive particle て ("te") and are +# semantically verb-like. +# e.g. ごらん, ご覧, 御覧, 頂戴 +#名詞-動詞非自立的 +# +# noun-quotation: text that cannot be segmented into words, proverbs, Chinese poetry, +# dialects, English, etc. Currently, the only entry for 名詞 引用文字列 ("noun quotation") +# is いわく ("iwaku"). +#名詞-引用文字列 +# +# noun-nai_adjective: Words that appear before the auxiliary verb ない ("nai") and +# behave like an adjective. +# e.g. 申し訳, 仕方, とんでも, 違い +#名詞-ナイ形容詞語幹 +# +##### +# prefix: unclassified prefixes +#接頭詞 +# +# prefix-nominal: Prefixes that attach to nouns (including adjective stem forms) +# excluding numerical expressions. +# e.g. お (水), 某 (氏), 同 (社), 故 (～氏), 高 (品質), お (見事), ご (立派) +#接頭詞-名詞接続 +# +# prefix-verbal: Prefixes that attach to the imperative form of a verb or a verb +# in conjunctive form followed by なる/なさる/くださる. +# e.g. お (読みなさい), お (座り) +#接頭詞-動詞接続 +# +# prefix-adjectival: Prefixes that attach to adjectives. +# e.g. お (寒いですねえ), バカ (でかい) +#接頭詞-形容詞接続 +# +# prefix-numerical: Prefixes that attach to numerical expressions. +# e.g. 約, およそ, 毎時 +#接頭詞-数接続 +# +##### +# verb: unclassified verbs +#動詞 +# +# verb-main: +#動詞-自立 +# +# verb-auxiliary: +#動詞-非自立 +# +# verb-suffix: +#動詞-接尾 +# +##### +# adjective: unclassified adjectives +#形容詞 +# +# adjective-main: +#形容詞-自立 +# +# adjective-auxiliary: +#形容詞-非自立 +# +# adjective-suffix: +#形容詞-接尾 +# +##### +# adverb: unclassified adverbs +#副詞 +# +# adverb-misc: Words that can be segmented into one unit and where adnominal +# modification is not possible. +# e.g. あいかわらず, 多分 +#副詞-一般 +# +# adverb-particle_conjunction: Adverbs that can be followed by の, は, に, +# な, する, だ, etc. +# e.g. こんなに, そんなに, あんなに, なにか, なんでも +#副詞-助詞類接続 +# +##### +# adnominal: Words that only have noun-modifying forms. +# e.g. この, その, あの, どの, いわゆる, なんらかの, 何らかの, いろんな, こういう, そういう, ああいう, +# どういう, こんな, そんな, あんな, どんな, 大きな, 小さな, おかしな, ほんの, たいした, +# 「(, も) さる (ことながら)」, 微々たる, 堂々たる, 単なる, いかなる, 我が」「同じ, 亡き +#連体詞 +# +##### +# conjunction: Conjunctions that can occur independently. +# e.g. が, けれども, そして, じゃあ, それどころか +接続詞 +# +##### +# particle: unclassified particles. +助詞 +# +# particle-case: case particles where the subclassification is undefined. +助詞-格助詞 +# +# particle-case-misc: Case particles. +# e.g. から, が, で, と, に, へ, より, を, の, にて +助詞-格助詞-一般 +# +# particle-case-quote: the "to" that appears after nouns, a person’s speech, +# quotation marks, expressions of decisions from a meeting, reasons, judgements, +# conjectures, etc. +# e.g. ( だ) と (述べた.), ( である) と (して執行猶予...) +助詞-格助詞-引用 +# +# particle-case-compound: Compounds of particles and verbs that mainly behave +# like case particles. +# e.g. という, といった, とかいう, として, とともに, と共に, でもって, にあたって, に当たって, に当って, +# にあたり, に当たり, に当り, に当たる, にあたる, において, に於いて,に於て, における, に於ける, +# にかけ, にかけて, にかんし, に関し, にかんして, に関して, にかんする, に関する, に際し, +# に際して, にしたがい, に従い, に従う, にしたがって, に従って, にたいし, に対し, にたいして, +# に対して, にたいする, に対する, について, につき, につけ, につけて, につれ, につれて, にとって, +# にとり, にまつわる, によって, に依って, に因って, により, に依り, に因り, による, に依る, に因る, +# にわたって, にわたる, をもって, を以って, を通じ, を通じて, を通して, をめぐって, をめぐり, をめぐる, +# って-口語/, ちゅう-関西弁「という」/, (何) ていう (人)-口語/, っていう-口語/, といふ, とかいふ +助詞-格助詞-連語 +# +# particle-conjunctive: +# e.g. から, からには, が, けれど, けれども, けど, し, つつ, て, で, と, ところが, どころか, とも, ども, +# ながら, なり, ので, のに, ば, ものの, や ( した), やいなや, (ころん) じゃ(いけない)-口語/, +# (行っ) ちゃ(いけない)-口語/, (言っ) たって (しかたがない)-口語/, (それがなく)ったって (平気)-口語/ +助詞-接続助詞 +# +# particle-dependency: +# e.g. こそ, さえ, しか, すら, は, も, ぞ +助詞-係助詞 +# +# particle-adverbial: +# e.g. がてら, かも, くらい, 位, ぐらい, しも, (学校) じゃ(これが流行っている)-口語/, +# (それ)じゃあ (よくない)-口語/, ずつ, (私) なぞ, など, (私) なり (に), (先生) なんか (大嫌い)-口語/, +# (私) なんぞ, (先生) なんて (大嫌い)-口語/, のみ, だけ, (私) だって-口語/, だに, +# (彼)ったら-口語/, (お茶) でも (いかが), 等 (とう), (今後) とも, ばかり, ばっか-口語/, ばっかり-口語/, +# ほど, 程, まで, 迄, (誰) も (が)([助詞-格助詞] および [助詞-係助詞] の前に位置する「も」) +助詞-副助詞 +# +# particle-interjective: particles with interjective grammatical roles. +# e.g. (松島) や +助詞-間投助詞 +# +# particle-coordinate: +# e.g. と, たり, だの, だり, とか, なり, や, やら +助詞-並立助詞 +# +# particle-final: +# e.g. かい, かしら, さ, ぜ, (だ)っけ-口語/, (とまってる) で-方言/, な, ナ, なあ-口語/, ぞ, ね, ネ, +# ねぇ-口語/, ねえ-口語/, ねん-方言/, の, のう-口語/, や, よ, ヨ, よぉ-口語/, わ, わい-口語/ +助詞-終助詞 +# +# particle-adverbial/conjunctive/final: The particle "ka" when unknown whether it is +# adverbial, conjunctive, or sentence final. For example: +# (a) 「A か B か」. Ex:「(国内で運用する) か,(海外で運用する) か (.)」 +# (b) Inside an adverb phrase. Ex:「(幸いという) か (, 死者はいなかった.)」 +# 「(祈りが届いたせい) か (, 試験に合格した.)」 +# (c) 「かのように」. Ex:「(何もなかった) か (のように振る舞った.)」 +# e.g. か +助詞-副助詞／並立助詞／終助詞 +# +# particle-adnominalizer: The "no" that attaches to nouns and modifies +# non-inflectional words. +助詞-連体化 +# +# particle-adnominalizer: The "ni" and "to" that appear following nouns and adverbs +# that are giongo, giseigo, or gitaigo. +# e.g. に, と +助詞-副詞化 +# +# particle-special: A particle that does not fit into one of the above classifications. +# This includes particles that are used in Tanka, Haiku, and other poetry. +# e.g. かな, けむ, ( しただろう) に, (あんた) にゃ(わからん), (俺) ん (家) +助詞-特殊 +# +##### +# auxiliary-verb: +助動詞 +# +##### +# interjection: Greetings and other exclamations. +# e.g. おはよう, おはようございます, こんにちは, こんばんは, ありがとう, どうもありがとう, ありがとうございます, +# いただきます, ごちそうさま, さよなら, さようなら, はい, いいえ, ごめん, ごめんなさい +#感動詞 +# +##### +# symbol: unclassified Symbols. +記号 +# +# symbol-misc: A general symbol not in one of the categories below. +# e.g. [○◎@$〒→+] +記号-一般 +# +# symbol-comma: Commas +# e.g. [,、] +記号-読点 +# +# symbol-period: Periods and full stops. +# e.g. [.．。] +記号-句点 +# +# symbol-space: Full-width whitespace. +記号-空白 +# +# symbol-open_bracket: +# e.g. [({‘“『【] +記号-括弧開 +# +# symbol-close_bracket: +# e.g. [)}’”』」】] +記号-括弧閉 +# +# symbol-alphabetic: +#記号-アルファベット +# +##### +# other: unclassified other +#その他 +# +# other-interjection: Words that are hard to classify as noun-suffixes or +# sentence-final particles. +# e.g. (だ)ァ +その他-間投 +# +##### +# filler: Aizuchi that occurs during a conversation or sounds inserted as filler. +# e.g. あの, うんと, えと +フィラー +# +##### +# non-verbal: non-verbal sound. +非言語音 +# +##### +# fragment: +#語断片 +# +##### +# unknown: unknown part of speech. +#未知語 +# +##### End of file diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_ar.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_ar.txt new file mode 100644 index 00000000..046829db --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_ar.txt @@ -0,0 +1,125 @@ +# This file was created by Jacques Savoy and is distributed under the BSD license. +# See http://members.unine.ch/jacques.savoy/clef/index.html. +# Also see http://www.opensource.org/licenses/bsd-license.html +# Cleaned on October 11, 2009 (not normalized, so use before normalization) +# This means that when modifying this list, you might need to add some +# redundant entries, for example containing forms with both أ and ا +من +ومن +منها +منه +في +وفي +فيها +فيه +و +ف +ثم +او +أو +ب +بها +به +ا +أ +اى +اي +أي +أى +لا +ولا +الا +ألا +إلا +لكن +ما +وما +كما +فما +عن +مع +اذا +إذا +ان +أن +إن +انها +أنها +إنها +انه +أنه +إنه +بان +بأن +فان +فأن +وان +وأن +وإن +التى +التي +الذى +الذي +الذين +الى +الي +إلى +إلي +على +عليها +عليه +اما +أما +إما +ايضا +أيضا +كل +وكل +لم +ولم +لن +ولن +هى +هي +هو +وهى +وهي +وهو +فهى +فهي +فهو +انت +أنت +لك +لها +له +هذه +هذا +تلك +ذلك +هناك +كانت +كان +يكون +تكون +وكانت +وكان +غير +بعض +قد +نحو +بين +بينما +منذ +ضمن +حيث +الان +الآن +خلال +بعد +قبل +حتى +عند +عندما +لدى +جميع diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_bg.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_bg.txt new file mode 100644 index 00000000..1ae4ba2a --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_bg.txt @@ -0,0 +1,193 @@ +# This file was created by Jacques Savoy and is distributed under the BSD license. +# See http://members.unine.ch/jacques.savoy/clef/index.html. +# Also see http://www.opensource.org/licenses/bsd-license.html +а +аз +ако +ала +бе +без +беше +би +бил +била +били +било +близо +бъдат +бъде +бяха +в +вас +ваш +ваша +вероятно +вече +взема +ви +вие +винаги +все +всеки +всички +всичко +всяка +във +въпреки +върху +г +ги +главно +го +д +да +дали +до +докато +докога +дори +досега +доста +е +едва +един +ето +за +зад +заедно +заради +засега +затова +защо +защото +и +из +или +им +има +имат +иска +й +каза +как +каква +какво +както +какъв +като +кога +когато +което +които +кой +който +колко +която +къде +където +към +ли +м +ме +между +мен +ми +мнозина +мога +могат +може +моля +момента +му +н +на +над +назад +най +направи +напред +например +нас +не +него +нея +ни +ние +никой +нито +но +някои +някой +няма +обаче +около +освен +особено +от +отгоре +отново +още +пак +по +повече +повечето +под +поне +поради +после +почти +прави +пред +преди +през +при +пък +първо +с +са +само +се +сега +си +скоро +след +сме +според +сред +срещу +сте +съм +със +също +т +тази +така +такива +такъв +там +твой +те +тези +ти +тн +то +това +тогава +този +той +толкова +точно +трябва +тук +тъй +тя +тях +у +харесва +ч +че +често +чрез +ще +щом +я diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_ca.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_ca.txt new file mode 100644 index 00000000..3da65dea --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_ca.txt @@ -0,0 +1,220 @@ +# Catalan stopwords from http://github.com/vcl/cue.language (Apache 2 Licensed) +a +abans +ací +ah +així +això +al +als +aleshores +algun +alguna +algunes +alguns +alhora +allà +allí +allò +altra +altre +altres +amb +ambdós +ambdues +apa +aquell +aquella +aquelles +aquells +aquest +aquesta +aquestes +aquests +aquí +baix +cada +cadascú +cadascuna +cadascunes +cadascuns +com +contra +d'un +d'una +d'unes +d'uns +dalt +de +del +dels +des +després +dins +dintre +donat +doncs +durant +e +eh +el +els +em +en +encara +ens +entre +érem +eren +éreu +es +és +esta +està +estàvem +estaven +estàveu +esteu +et +etc +ets +fins +fora +gairebé +ha +han +has +havia +he +hem +heu +hi +ho +i +igual +iguals +ja +l'hi +la +les +li +li'n +llavors +m'he +ma +mal +malgrat +mateix +mateixa +mateixes +mateixos +me +mentre +més +meu +meus +meva +meves +molt +molta +moltes +molts +mon +mons +n'he +n'hi +ne +ni +no +nogensmenys +només +nosaltres +nostra +nostre +nostres +o +oh +oi +on +pas +pel +pels +per +però +perquè +poc +poca +pocs +poques +potser +propi +qual +quals +quan +quant +que +què +quelcom +qui +quin +quina +quines +quins +s'ha +s'han +sa +semblant +semblants +ses +seu +seus +seva +seva +seves +si +sobre +sobretot +sóc +solament +sols +son +són +sons +sota +sou +t'ha +t'han +t'he +ta +tal +també +tampoc +tan +tant +tanta +tantes +teu +teus +teva +teves +ton +tons +tot +tota +totes +tots +un +una +unes +uns +us +va +vaig +vam +van +vas +veu +vosaltres +vostra +vostre +vostres diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_cz.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_cz.txt new file mode 100644 index 00000000..53c6097d --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_cz.txt @@ -0,0 +1,172 @@ +a +s +k +o +i +u +v +z +dnes +cz +tímto +budeš +budem +byli +jseš +můj +svým +ta +tomto +tohle +tuto +tyto +jej +zda +proč +máte +tato +kam +tohoto +kdo +kteří +mi +nám +tom +tomuto +mít +nic +proto +kterou +byla +toho +protože +asi +ho +naši +napište +re +což +tím +takže +svých +její +svými +jste +aj +tu +tedy +teto +bylo +kde +ke +pravé +ji +nad +nejsou +či +pod +téma +mezi +přes +ty +pak +vám +ani +když +však +neg +jsem +tento +článku +články +aby +jsme +před +pta +jejich +byl +ještě +až +bez +také +pouze +první +vaše +která +nás +nový +tipy +pokud +může +strana +jeho +své +jiné +zprávy +nové +není +vás +jen +podle +zde +už +být +více +bude +již +než +který +by +které +co +nebo +ten +tak +má +při +od +po +jsou +jak +další +ale +si +se +ve +to +jako +za +zpět +ze +do +pro +je +na +atd +atp +jakmile +přičemž +já +on +ona +ono +oni +ony +my +vy +jí +ji +mě +mne +jemu +tomu +těm +těmu +němu +němuž +jehož +jíž +jelikož +jež +jakož +načež diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_da.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_da.txt new file mode 100644 index 00000000..6e90e8f1 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_da.txt @@ -0,0 +1,110 @@ + | From https://snowballstem.org/algorithms/danish/stop.txt + | This file is distributed under the BSD License. + | See https://snowballstem.org/license.html + | Also see https://opensource.org/licenses/bsd-license.html + | - Encoding was converted to UTF-8. + | - This notice was added. + | + | NOTE: To use this file with StopFilterFactory, you must specify format="snowball" + + | A Danish stop word list. Comments begin with vertical bar. Each stop + | word is at the start of a line. + + | This is a ranked list (commonest to rarest) of stopwords derived from + | a large text sample. + + +og | and +i | in +jeg | I +det | that (dem. pronoun)/it (pers. pronoun) +at | that (in front of a sentence)/to (with infinitive) +en | a/an +den | it (pers. pronoun)/that (dem. pronoun) +til | to/at/for/until/against/by/of/into, more +er | present tense of "to be" +som | who, as +på | on/upon/in/on/at/to/after/of/with/for, on +de | they +med | with/by/in, along +han | he +af | of/by/from/off/for/in/with/on, off +for | at/for/to/from/by/of/ago, in front/before, because +ikke | not +der | who/which, there/those +var | past tense of "to be" +mig | me/myself +sig | oneself/himself/herself/itself/themselves +men | but +et | a/an/one, one (number), someone/somebody/one +har | present tense of "to have" +om | round/about/for/in/a, about/around/down, if +vi | we +min | my +havde | past tense of "to have" +ham | him +hun | she +nu | now +over | over/above/across/by/beyond/past/on/about, over/past +da | then, when/as/since +fra | from/off/since, off, since +du | you +ud | out +sin | his/her/its/one's +dem | them +os | us/ourselves +op | up +man | you/one +hans | his +hvor | where +eller | or +hvad | what +skal | must/shall etc. +selv | myself/yourself/herself/ourselves etc., even +her | here +alle | all/everyone/everybody etc. +vil | will (verb) +blev | past tense of "to stay/to remain/to get/to become" +kunne | could +ind | in +når | when +være | present tense of "to be" +dog | however/yet/after all +noget | something +ville | would +jo | you know/you see (adv), yes +deres | their/theirs +efter | after/behind/according to/for/by/from, later/afterwards +ned | down +skulle | should +denne | this +end | than +dette | this +mit | my/mine +også | also +under | under/beneath/below/during, below/underneath +have | have +dig | you +anden | other +hende | her +mine | my +alt | everything +meget | much/very, plenty of +sit | his, her, its, one's +sine | his, her, its, one's +vor | our +mod | against +disse | these +hvis | if +din | your/yours +nogle | some +hos | by/at +blive | be/become +mange | many +ad | by/through +bliver | present tense of "to be/to become" +hendes | her/hers +været | be +thi | for (conj) +jer | you +sådan | such, like this/like that diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_de.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_de.txt new file mode 100644 index 00000000..804bbbdb --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_de.txt @@ -0,0 +1,294 @@ + | From https://snowballstem.org/algorithms/german/stop.txt + | This file is distributed under the BSD License. + | See https://snowballstem.org/license.html + | Also see https://opensource.org/licenses/bsd-license.html + | - Encoding was converted to UTF-8. + | - This notice was added. + | + | NOTE: To use this file with StopFilterFactory, you must specify format="snowball" + + | A German stop word list. Comments begin with vertical bar. Each stop + | word is at the start of a line. + + | The number of forms in this list is reduced significantly by passing it + | through the German stemmer. + + +aber | but + +alle | all +allem +allen +aller +alles + +als | than, as +also | so +am | an + dem +an | at + +ander | other +andere +anderem +anderen +anderer +anderes +anderm +andern +anderr +anders + +auch | also +auf | on +aus | out of +bei | by +bin | am +bis | until +bist | art +da | there +damit | with it +dann | then + +der | the +den +des +dem +die +das + +daß | that + +derselbe | the same +derselben +denselben +desselben +demselben +dieselbe +dieselben +dasselbe + +dazu | to that + +dein | thy +deine +deinem +deinen +deiner +deines + +denn | because + +derer | of those +dessen | of him + +dich | thee +dir | to thee +du | thou + +dies | this +diese +diesem +diesen +dieser +dieses + + +doch | (several meanings) +dort | (over) there + + +durch | through + +ein | a +eine +einem +einen +einer +eines + +einig | some +einige +einigem +einigen +einiger +einiges + +einmal | once + +er | he +ihn | him +ihm | to him + +es | it +etwas | something + +euer | your +eure +eurem +euren +eurer +eures + +für | for +gegen | towards +gewesen | p.p. of sein +hab | have +habe | have +haben | have +hat | has +hatte | had +hatten | had +hier | here +hin | there +hinter | behind + +ich | I +mich | me +mir | to me + + +ihr | you, to her +ihre +ihrem +ihren +ihrer +ihres +euch | to you + +im | in + dem +in | in +indem | while +ins | in + das +ist | is + +jede | each, every +jedem +jeden +jeder +jedes + +jene | that +jenem +jenen +jener +jenes + +jetzt | now +kann | can + +kein | no +keine +keinem +keinen +keiner +keines + +können | can +könnte | could +machen | do +man | one + +manche | some, many a +manchem +manchen +mancher +manches + +mein | my +meine +meinem +meinen +meiner +meines + +mit | with +muss | must +musste | had to +nach | to(wards) +nicht | not +nichts | nothing +noch | still, yet +nun | now +nur | only +ob | whether +oder | or +ohne | without +sehr | very + +sein | his +seine +seinem +seinen +seiner +seines + +selbst | self +sich | herself + +sie | they, she +ihnen | to them + +sind | are +so | so + +solche | such +solchem +solchen +solcher +solches + +soll | shall +sollte | should +sondern | but +sonst | else +über | over +um | about, around +und | and + +uns | us +unse +unsem +unsen +unser +unses + +unter | under +viel | much +vom | von + dem +von | from +vor | before +während | while +war | was +waren | were +warst | wast +was | what +weg | away, off +weil | because +weiter | further + +welche | which +welchem +welchen +welcher +welches + +wenn | when +werde | will +werden | will +wie | how +wieder | again +will | want +wir | we +wird | will +wirst | willst +wo | where +wollen | want +wollte | wanted +würde | would +würden | would +zu | to +zum | zu + dem +zur | zu + der +zwar | indeed +zwischen | between + diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_el.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_el.txt new file mode 100644 index 00000000..232681f5 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_el.txt @@ -0,0 +1,78 @@ +# Lucene Greek Stopwords list +# Note: by default this file is used after GreekLowerCaseFilter, +# so when modifying this file use 'σ' instead of 'ς' +ο +η +το +οι +τα +του +τησ +των +τον +την +και +κι +κ +ειμαι +εισαι +ειναι +ειμαστε +ειστε +στο +στον +στη +στην +μα +αλλα +απο +για +προσ +με +σε +ωσ +παρα +αντι +κατα +μετα +θα +να +δε +δεν +μη +μην +επι +ενω +εαν +αν +τοτε +που +πωσ +ποιοσ +ποια +ποιο +ποιοι +ποιεσ +ποιων +ποιουσ +αυτοσ +αυτη +αυτο +αυτοι +αυτων +αυτουσ +αυτεσ +αυτα +εκεινοσ +εκεινη +εκεινο +εκεινοι +εκεινεσ +εκεινα +εκεινων +εκεινουσ +οπωσ +ομωσ +ισωσ +οσο +οτι diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_en.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_en.txt new file mode 100644 index 00000000..2c164c0b --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_en.txt @@ -0,0 +1,54 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# a couple of test stopwords to test that the words are really being +# configured from this file: +stopworda +stopwordb + +# Standard english stop words taken from Lucene's StopAnalyzer +a +an +and +are +as +at +be +but +by +for +if +in +into +is +it +no +not +of +on +or +such +that +the +their +then +there +these +they +this +to +was +will +with diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_es.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_es.txt new file mode 100644 index 00000000..48bd65ef --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_es.txt @@ -0,0 +1,356 @@ + | From https://snowballstem.org/algorithms/spanish/stop.txt + | This file is distributed under the BSD License. + | See https://snowballstem.org/license.html + | Also see https://opensource.org/licenses/bsd-license.html + | - Encoding was converted to UTF-8. + | - This notice was added. + | + | NOTE: To use this file with StopFilterFactory, you must specify format="snowball" + + | A Spanish stop word list. Comments begin with vertical bar. Each stop + | word is at the start of a line. + + + | The following is a ranked list (commonest to rarest) of stopwords + | deriving from a large sample of text. + + | Extra words have been added at the end. + +de | from, of +la | the, her +que | who, that +el | the +en | in +y | and +a | to +los | the, them +del | de + el +se | himself, from him etc +las | the, them +por | for, by, etc +un | a +para | for +con | with +no | no +una | a +su | his, her +al | a + el + | es from SER +lo | him +como | how +más | more +pero | pero +sus | su plural +le | to him, her +ya | already +o | or + | fue from SER +este | this + | ha from HABER +sí | himself etc +porque | because +esta | this + | son from SER +entre | between + | está from ESTAR +cuando | when +muy | very +sin | without +sobre | on + | ser from SER + | tiene from TENER +también | also +me | me +hasta | until +hay | there is/are +donde | where + | han from HABER +quien | whom, that + | están from ESTAR + | estado from ESTAR +desde | from +todo | all +nos | us +durante | during + | estados from ESTAR +todos | all +uno | a +les | to them +ni | nor +contra | against +otros | other + | fueron from SER +ese | that +eso | that + | había from HABER +ante | before +ellos | they +e | and (variant of y) +esto | this +mí | me +antes | before +algunos | some +qué | what? +unos | a +yo | I +otro | other +otras | other +otra | other +él | he +tanto | so much, many +esa | that +estos | these +mucho | much, many +quienes | who +nada | nothing +muchos | many +cual | who + | sea from SER +poco | few +ella | she +estar | to be + | haber from HABER +estas | these + | estaba from ESTAR + | estamos from ESTAR +algunas | some +algo | something +nosotros | we + + | other forms + +mi | me +mis | mi plural +tú | thou +te | thee +ti | thee +tu | thy +tus | tu plural +ellas | they +nosotras | we +vosotros | you +vosotras | you +os | you +mío | mine +mía | +míos | +mías | +tuyo | thine +tuya | +tuyos | +tuyas | +suyo | his, hers, theirs +suya | +suyos | +suyas | +nuestro | ours +nuestra | +nuestros | +nuestras | +vuestro | yours +vuestra | +vuestros | +vuestras | +esos | those +esas | those + + | forms of estar, to be (not including the infinitive): +estoy +estás +está +estamos +estáis +están +esté +estés +estemos +estéis +estén +estaré +estarás +estará +estaremos +estaréis +estarán +estaría +estarías +estaríamos +estaríais +estarían +estaba +estabas +estábamos +estabais +estaban +estuve +estuviste +estuvo +estuvimos +estuvisteis +estuvieron +estuviera +estuvieras +estuviéramos +estuvierais +estuvieran +estuviese +estuvieses +estuviésemos +estuvieseis +estuviesen +estando +estado +estada +estados +estadas +estad + + | forms of haber, to have (not including the infinitive): +he +has +ha +hemos +habéis +han +haya +hayas +hayamos +hayáis +hayan +habré +habrás +habrá +habremos +habréis +habrán +habría +habrías +habríamos +habríais +habrían +había +habías +habíamos +habíais +habían +hube +hubiste +hubo +hubimos +hubisteis +hubieron +hubiera +hubieras +hubiéramos +hubierais +hubieran +hubiese +hubieses +hubiésemos +hubieseis +hubiesen +habiendo +habido +habida +habidos +habidas + + | forms of ser, to be (not including the infinitive): +soy +eres +es +somos +sois +son +sea +seas +seamos +seáis +sean +seré +serás +será +seremos +seréis +serán +sería +serías +seríamos +seríais +serían +era +eras +éramos +erais +eran +fui +fuiste +fue +fuimos +fuisteis +fueron +fuera +fueras +fuéramos +fuerais +fueran +fuese +fueses +fuésemos +fueseis +fuesen +siendo +sido + | sed also means 'thirst' + + | forms of tener, to have (not including the infinitive): +tengo +tienes +tiene +tenemos +tenéis +tienen +tenga +tengas +tengamos +tengáis +tengan +tendré +tendrás +tendrá +tendremos +tendréis +tendrán +tendría +tendrías +tendríamos +tendríais +tendrían +tenía +tenías +teníamos +teníais +tenían +tuve +tuviste +tuvo +tuvimos +tuvisteis +tuvieron +tuviera +tuvieras +tuviéramos +tuvierais +tuvieran +tuviese +tuvieses +tuviésemos +tuvieseis +tuviesen +teniendo +tenido +tenida +tenidos +tenidas +tened + diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_et.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_et.txt new file mode 100644 index 00000000..1b06a134 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_et.txt @@ -0,0 +1,1603 @@ +# Estonian stopwords list +all +alla +allapoole +allpool +alt +altpoolt +eel +eespool +enne +hommikupoole +hoolimata +ilma +kaudu +keset +kesk +kohe +koos +kuhupoole +kuni +kuspool +kustpoolt +kõige +käsikäes +lappi +ligi +läbi +mööda +paitsi +peale +pealepoole +pealpool +pealt +pealtpoolt +piki +pikku +piku +pikuti +põiki +pärast +päri +risti +sealpool +sealtpoolt +seespool +seltsis +siiapoole +siinpool +siitpoolt +sinnapoole +sissepoole +taga +tagantpoolt +tagapidi +tagapool +taha +tahapoole +teispool +teispoole +tänu +tükkis +vaatamata +vastu +väljapoole +väljaspool +väljastpoolt +õhtupoole +ühes +ühestükis +ühestükkis +ülalpool +ülaltpoolt +üle +ülespoole +ülevalpool +ülevaltpoolt +ümber +ümbert +aegu +aegus +alguks +algul +algule +algult +alguni +all +alla +alt +alul +alutsi +arvel +asemel +asemele +eel +eeli +ees +eesotsas +eest +eestotsast +esitsi +ette +etteotsa +haaval +heaks +hoolimata +hulgas +hulgast +hulka +jalgu +jalus +jalust +jaoks +jooksul +juurde +juures +juurest +jälil +jälile +järel +järele +järelt +järgi +kaasas +kallal +kallale +kallalt +kamul +kannul +kannule +kannult +kaudu +kaupa +keskel +keskele +keskelt +keskis +keskpaiku +kestel +kestes +kilda +killas +killast +kimpu +kimpus +kiuste +kohal +kohale +kohalt +kohaselt +kohe +kohta +koos +korral +kukil +kukile +kukilt +kulul +kõrva +kõrval +kõrvale +kõrvalt +kõrvas +kõrvast +käekõrval +käekõrvale +käekõrvalt +käes +käest +kätte +külge +küljes +küljest +küüsi +küüsis +küüsist +ligi +ligidal +ligidale +ligidalt +aegu +aegus +alguks +algul +algule +algult +alguni +all +alla +alt +alul +alutsi +arvel +asemel +asemele +eel +eeli +ees +eesotsas +eest +eestotsast +esitsi +ette +etteotsa +haaval +heaks +hoolimata +hulgas +hulgast +hulka +jalgu +jalus +jalust +jaoks +jooksul +juurde +juures +juurest +jälil +jälile +järel +järele +järelt +järgi +kaasas +kallal +kallale +kallalt +kamul +kannul +kannule +kannult +kaudu +kaupa +keskel +keskele +keskelt +keskis +keskpaiku +kestel +kestes +kilda +killas +killast +kimpu +kimpus +kiuste +kohal +kohale +kohalt +kohaselt +kohe +kohta +koos +korral +kukil +kukile +kukilt +kulul +kõrva +kõrval +kõrvale +kõrvalt +kõrvas +kõrvast +käekõrval +käekõrvale +käekõrvalt +käes +käest +kätte +külge +küljes +küljest +küüsi +küüsis +küüsist +ligi +ligidal +ligidale +ligidalt +lool +läbi +lähedal +lähedale +lähedalt +man +mant +manu +meelest +mööda +nahas +nahka +nahkas +najal +najale +najalt +nõjal +nõjale +otsa +otsas +otsast +paigale +paigu +paiku +peal +peale +pealt +perra +perrä +pidi +pihta +piki +pikku +pool +poole +poolest +poolt +puhul +puksiiris +pähe +päralt +päras +pärast +päri +ringi +ringis +risust +saadetusel +saadik +saatel +saati +seas +seast +sees +seest +sekka +seljataga +seltsi +seltsis +seltsist +sisse +slepis +suhtes +šlepis +taga +tagant +tagantotsast +tagaotsas +tagaselja +tagasi +tagast +tagutsi +taha +tahaotsa +takka +tarvis +tasa +tuuri +tuuris +tõttu +tükkis +uhal +vaatamata +vahel +vahele +vahelt +vahepeal +vahepeale +vahepealt +vahetsi +varal +varale +varul +vastas +vastast +vastu +veerde +veeres +viisi +võidu +võrd +võrdki +võrra +võrragi +väel +väele +vältel +väärt +väärtki +äärde +ääre +ääres +äärest +ühes +üle +ümber +ümbert +a +abil +aina +ainult +alalt +alates +alati +alles +b +c +d +e +eales +ealeski +edasi +edaspidi +eelkõige +eemal +ei +eks +end +enda +enese +ennem +esialgu +f +g +h +hoopis +i +iganes +igatahes +igati +iial +iialgi +ikka +ikkagi +ilmaski +iseenda +iseenese +iseenesest +isegi +j +jah +ju +juba +juhul +just +järelikult +k +ka +kah +kas +kasvõi +keda +kestahes +kogu +koguni +kohati +kokku +kuhu +kuhugi +kuidagi +kuidas +kunagi +kus +kusagil +kusjuures +kuskil +kust +kõigepealt +küll +l +liiga +lisaks +m +miks +mil +millal +millalgi +mispärast +mistahes +mistõttu +mitte +muide +muidu +muidugi +muist +mujal +mujale +mujalt +mõlemad +mõnda +mõne +mõnikord +n +nii +niikaua +niimoodi +niipaljuke +niisama +niisiis +niivõrd +nõnda +nüüd +o +omaette +omakorda +omavahel +ometi +p +palju +paljuke +palju-palju +peaaegu +peagi +peamiselt +pigem +pisut +praegu +päris +r +rohkem +s +samas +samuti +seal +sealt +sedakorda +sedapuhku +seega +seejuures +seejärel +seekord +seepärast +seetõttu +sellepärast +seni +sestap +siia +siiani +siin +siinkohal +siis +siiski +siit +sinna +suht +š +z +ž +t +teel +teineteise +tõesti +täiesti +u +umbes +v +w +veel +veelgi +vist +võibolla +võib-olla +väga +vähemalt +välja +väljas +väljast +õ +ä +ära +ö +ü +ühtlasi +üksi +ükskõik +ülal +ülale +ülalt +üles +ülesse +üleval +ülevalt +ülimalt +üsna +x +y +aga +ega +ehk +ehkki +elik +ellik +enge +ennegu +ent +et +ja +justkui +kui +kuid +kuigi +kuivõrd +kuna +kuni +kut +mistab +muudkui +nagu +nigu +ning +olgugi +otsekui +otsenagu +selmet +sest +sestab +vaid +või +aa +adaa +adjöö +ae +ah +ahaa +ahah +ah-ah-ah +ah-haa +ahoi +ai +aidaa +aidu-raidu +aih +aijeh +aituma +aitäh +aitüma +ammuu +amps +ampsti +aptsih +ass +at +ata +at-at-at +atsih +atsihh +auh +bai-bai +bingo +braavo +brr +ee +eeh +eh +ehee +eheh +eh-eh-hee +eh-eh-ee +ehei +ehh +ehhee +einoh +ena +ennäe +ennäh +fuh +fui +fuih +haa +hah +hahaa +hah-hah-hah +halleluuja +hallo +halloo +hass +hee +heh +he-he-hee +hei +heldeke(ne) +heureka +hihii +hip-hip-hurraa +hmh +hmjah +hoh-hoh-hoo +hohoo +hoi +hollallaa +hoo +hoplaa +hopp +hops +hopsassaa +hopsti +hosianna +huh +huidii +huist +hurjah +hurjeh +hurjoh +hurjuh +hurraa +huu +hõhõh +hõi +hõissa +hõissassa +hõk +hõkk +häh +hä-hä-hää +hüvasti +ih-ah-haa +ih-ih-hii +ii-ha-ha +issake +issakene +isver +jaa-ah +ja-ah +jaah +janäe +jeeh +jeerum +jeever +jessas +jestas +juhhei +jumalaga +jumalime +jumaluke +jumalukene +jutas +kaaps +kaapsti +kaasike +kae +kalps +kalpsti +kannäe +kanäe +kappadi +kaps +kapsti +karkõmm +karkäuh +karkääks +karkääksti +karmauh +karmauhti +karnaps +karnapsti +karniuhti +karpartsaki +karpauh +karpauhti +karplauh +karplauhti +karprauh +karprauhti +karsumdi +karsumm +kartsumdi +kartsumm +karviuh +karviuhti +kaske +kassa +kauh +kauhti +keh +keksti +kepsti +khe +khm +kih +kiiks +kiiksti +kiis +kiiss +kikerii +kikerikii +kili +kilk +kilk-kõlk +kilks +kilks-kolks +kilks-kõlks +kill +killadi +killadi|-kolladi +killadi-kõlladi +killa-kolla +killa-kõlla +kill-kõll +kimps-komps +kipp +kips-kõps +kiriküüt +kirra-kõrra +kirr-kõrr +kirts +klaps +klapsti +klirdi +klirr +klonks +klops +klopsti +kluk +klu-kluu +klõks +klõksti +klõmdi +klõmm +klõmpsti +klõnks +klõnksti +klõps +klõpsti +kläu +kohva-kohva +kok +koks +koksti +kolaki +kolk +kolks +kolksti +koll +kolladi +komp +komps +kompsti +kop +kopp +koppadi +kops +kopsti +kossu +kotsu +kraa +kraak +kraaks +kraaps +kraapsti +krahh +kraks +kraksti +kraps +krapsti +krauh +krauhti +kriiks +kriiksti +kriips +kriips-kraaps +kripa-krõpa +krips-kraps +kriuh +kriuks +kriuksti +kromps +kronk +kronks +krooks +kruu +krõks +krõksti +krõpa +krõps +krõpsti +krõuh +kräu +kräuh +kräuhti +kräuks +kss +kukeleegu +kukku +kuku +kulu +kurluu +kurnäu +kuss +kussu +kõks +kõksti +kõldi +kõlks +kõlksti +kõll +kõmaki +kõmdi +kõmm +kõmps +kõpp +kõps +kõpsadi +kõpsat +kõpsti +kõrr +kõrra-kõrra +kõss +kõtt +kõõksti +kärr +kärts +kärtsti +käuks +käuksti +kääga +kääks +kääksti +köh +köki-möki +köksti +laks +laksti +lampsti +larts +lartsti +lats +latsti +leelo +legoo +lehva +liiri-lõõri +lika-lõka +likat-lõkat +limpsti +lips +lipsti +lirts +lirtsaki +lirtsti +lonksti +lops +lopsti +lorts +lortsti +luks +lups +lupsti +lurts +lurtsti +lõks +lõksti +lõmps +lõmpsti +lõnks +lõnksti +lärts +lärtsti +läts +lätsti +lörts +lörtsti +lötsti +lööps +lööpsti +marss +mats +matsti +mauh +mauhti +mh +mhh +mhmh +miau +mjaa +mkm +m-mh +mnjaa +mnjah +moens +mulks +mulksti +mull-mull +mull-mull-mull +muu +muuh +mõh +mõmm +mäh +mäts +mäu +mää +möh +möh-öh-ää +möö +müh-müh +mühüh +müks +müksti +müraki +mürr +mürts +mürtsaki +mürtsti +mütaku +müta-mäta +müta-müta +müt-müt +müt-müt-müt +müts +mütsti +mütt +naa +naah +nah +naks +naksti +nanuu +naps +napsti +nilpsti +nipsti +nirr +niuh +niuh-näuh +niuhti +noh +noksti +nolpsti +nonoh +nonoo +nonäh +noo +nooh +nooks +norr +nurr +nuuts +nõh +nõhh +nõka-nõka +nõks +nõksat-nõksat +nõks-nõks +nõksti +nõõ +nõõh +näeh +näh +nälpsti +nämm-nämm +näpsti +näts +nätsti +näu +näuh +näuhti +näuks +näuksti +nääh +nääks +nühkat-nühkat +oeh +oh +ohh +ohhh +oh-hoi +oh-hoo +ohoh +oh-oh-oo +oh-oh-hoo +ohoi +ohoo +oi +oih +oijee +oijeh +oo +ooh +oo-oh +oo-ohh +oot +ossa +ot +paa +pah +pahh +pakaa +pamm +pantsti +pardon +pardonks +parlartsti +parts +partsti +partsumdi +partsumm +pastoi +pats +patst +patsti +pau +pauh +pauhti +pele +pfui +phuh +phuuh +phäh +phähh +piiks +piip +piiri-pääri +pimm +pimm-pamm +pimm-pomm +pimm-põmm +piraki +piuks +piu-pau +plaks +plaksti +plarts +plartsti +plats +platsti +plauh +plauhh +plauhti +pliks +pliks-plaks +plinn +pliraki +plirts +plirtsti +pliu +pliuh +ploks +plotsti +plumps +plumpsti +plõks +plõksti +plõmdi +plõmm +plõnn +plärr +plärts +plärtsat +plärtsti +pläu +pläuh +plää +plörtsat +pomm +popp +pops +popsti +ports +pot +pots +potsti +pott +praks +praksti +prants +prantsaki +prantsti +prassai +prauh +prauhh +prauhti +priks +priuh +priuhh +priuh-prauh +proosit +proost +prr +prrr +prõks +prõksti +prõmdi +prõmm +prõntsti +prääk +prääks +pst +psst +ptrr +ptruu +ptüi +puh +puhh +puksti +pumm +pumps +pup-pup-pup +purts +puuh +põks +põksti +põmdi +põmm +põmmadi +põnks +põnn +põnnadi +põnt +põnts +põntsti +põraki +põrr +põrra-põrra +päh +pähh +päntsti +pää +pöörd +püh +raks +raksti +raps +rapsti +ratataa +rauh +riips +riipsti +riks +riks-raks +rips-raps +rivitult +robaki +rops +ropsaki +ropsti +ruik +räntsti +räts +röh +röhh +sah +sahh +sahkat +saps +sapsti +sauh +sauhti +servus +sihkadi-sahkadi +sihka-sahka +sihkat-sahkat +silks +silk-solk +sips +sipsti +sirr +sirr-sorr +sirts +sirtsti +siu +siuh +siuh-sauh +siuh-säuh +siuhti +siuks +siuts +skool +so +soh +solks +solksti +solpsti +soo +sooh +so-oh +soo-oh +sopp +sops +sopsti +sorr +sorts +sortsti +so-soo +soss +soss-soss +ss +sss +sst +stopp +suhkat-sahkat +sulk +sulks +sulksti +sull +sulla-sulla +sulpa-sulpa +sulps +sulpsti +sumaki +sumdi +summ +summat-summat +sups +supsaku +supsti +surts +surtsti +suss +susti +suts +sutsti +säh +sähke +särts +särtsti +säu +säuh +säuhti +taevake +taevakene +takk +tere +terekest +tibi-tibi +tikk-takk +tiks +tilk +tilks +till +tilla-talla +till-tall +tilulii +tinn +tip +tip-tap +tirr +tirtsti +tiu +tjaa +tjah +tohhoh +tohhoo +tohoh +tohoo +tok +tokk +toks +toksti +tonks +tonksti +tota +totsti +tot-tot +tprr +tpruu +trah +trahh +trallallaa +trill +trillallaa +trr +trrr +tsah +tsahh +tsilk +tsilk-tsolk +tsirr +tsiuh +tskae +tsolk +tss +tst +tsst +tsuhh +tsuk +tsumm +tsurr +tsäuh +tšao +tšš +tššš +tuk +tuks +turts +turtsti +tutki +tutkit +tutu-lutu +tutulutu +tuut +tuutu-luutu +tõks +tötsti +tümps +uh +uhh +uh-huu +uhtsa +uhtsaa +uhuh +uhuu +ui +uih +uih-aih +uijah +uijeh +uist +uit +uka +upsti +uraa +urjah +urjeh +urjoh +urjuh +urr +urraa +ust +utu +uu +uuh +vaak +vaat +vae +vaeh +vai +vat +vau +vhüüt +vidiit +viiks +vilks +vilksti +vinki-vinki +virdi +virr +viu +viudi +viuh +viuhti +voeh +voh +vohh +volks +volksti +vooh +vops +vopsti +vot +vuh +vuhti +vuih +vulks +vulksti +vull +vulpsti +vups +vupsaki +vupsaku +vupsti +vurdi +vurr +vurra-vurra +vurts +vurtsti +vutt +võe +võeh +või +võih +võrr +võts +võtt +vääks +õe +õits +õk +õkk +õrr +õss +õuh +äh +ähh +ähhähhää +äh-hää +äh-äh-hää +äiu +äiu-ää +äss +ää +ääh +äähh +öh +öhh +ök +üh +eelmine +eikeegi +eimiski +emb-kumb +enam +enim +iga +igasugune +igaüks +ise +isesugune +järgmine +keegi +kes +kumb +kumbki +kõik +meiesugune +meietaoline +midagi +mihuke +mihukene +milletaoline +milline +mina +minake +mingi +mingisugune +minusugune +minutaoline +mis +miski +miskisugune +missugune +misuke +mitmes +mitmesugune +mitu +mitu-mitu +mitu-setu +muu +mõlema +mõnesugune +mõni +mõningane +mõningas +mäherdune +määrane +naasugune +need +nemad +nendesugune +nendetaoline +nihuke +nihukene +niimitu +niisamasugune +niisugune +nisuke +nisukene +oma +omaenese +omasugune +omataoline +pool +praegune +sama +samasugune +samataoline +see +seesama +seesamane +seesamune +seesinane +seesugune +selline +sihuke +sihukene +sina +sinusugune +sinutaoline +siuke +siukene +säherdune +säärane +taoline +teiesugune +teine +teistsugune +tema +temake +temakene +temasugune +temataoline +too +toosama +toosamane +üks +üksteise +hakkama +minema +olema +pidama +saama +tegema +tulema +võima diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_eu.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_eu.txt new file mode 100644 index 00000000..25f1db93 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_eu.txt @@ -0,0 +1,99 @@ +# example set of basque stopwords +al +anitz +arabera +asko +baina +bat +batean +batek +bati +batzuei +batzuek +batzuetan +batzuk +bera +beraiek +berau +berauek +bere +berori +beroriek +beste +bezala +da +dago +dira +ditu +du +dute +edo +egin +ere +eta +eurak +ez +gainera +gu +gutxi +guzti +haiei +haiek +haietan +hainbeste +hala +han +handik +hango +hara +hari +hark +hartan +hau +hauei +hauek +hauetan +hemen +hemendik +hemengo +hi +hona +honek +honela +honetan +honi +hor +hori +horiei +horiek +horietan +horko +horra +horrek +horrela +horretan +horri +hortik +hura +izan +ni +noiz +nola +non +nondik +nongo +nor +nora +ze +zein +zen +zenbait +zenbat +zer +zergatik +ziren +zituen +zu +zuek +zuen +zuten diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_fa.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_fa.txt new file mode 100644 index 00000000..723641c6 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_fa.txt @@ -0,0 +1,313 @@ +# This file was created by Jacques Savoy and is distributed under the BSD license. +# See http://members.unine.ch/jacques.savoy/clef/index.html. +# Also see http://www.opensource.org/licenses/bsd-license.html +# Note: by default this file is used after normalization, so when adding entries +# to this file, use the arabic 'ي' instead of 'ی' +انان +نداشته +سراسر +خياه +ايشان +وي +تاكنون +بيشتري +دوم +پس +ناشي +وگو +يا +داشتند +سپس +هنگام +هرگز +پنج +نشان +امسال +ديگر +گروهي +شدند +چطور +ده +و +دو +نخستين +ولي +چرا +چه +وسط +ه +كدام +قابل +يك +رفت +هفت +همچنين +در +هزار +بله +بلي +شايد +اما +شناسي +گرفته +دهد +داشته +دانست +داشتن +خواهيم +ميليارد +وقتيكه +امد +خواهد +جز +اورده +شده +بلكه +خدمات +شدن +برخي +نبود +بسياري +جلوگيري +حق +كردند +نوعي +بعري +نكرده +نظير +نبايد +بوده +بودن +داد +اورد +هست +جايي +شود +دنبال +داده +بايد +سابق +هيچ +همان +انجا +كمتر +كجاست +گردد +كسي +تر +مردم +تان +دادن +بودند +سري +جدا +ندارند +مگر +يكديگر +دارد +دهند +بنابراين +هنگامي +سمت +جا +انچه +خود +دادند +زياد +دارند +اثر +بدون +بهترين +بيشتر +البته +به +براساس +بيرون +كرد +بعضي +گرفت +توي +اي +ميليون +او +جريان +تول +بر +مانند +برابر +باشيم +مدتي +گويند +اكنون +تا +تنها +جديد +چند +بي +نشده +كردن +كردم +گويد +كرده +كنيم +نمي +نزد +روي +قصد +فقط +بالاي +ديگران +اين +ديروز +توسط +سوم +ايم +دانند +سوي +استفاده +شما +كنار +داريم +ساخته +طور +امده +رفته +نخست +بيست +نزديك +طي +كنيد +از +انها +تمامي +داشت +يكي +طريق +اش +چيست +روب +نمايد +گفت +چندين +چيزي +تواند +ام +ايا +با +ان +ايد +ترين +اينكه +ديگري +راه +هايي +بروز +همچنان +پاعين +كس +حدود +مختلف +مقابل +چيز +گيرد +ندارد +ضد +همچون +سازي +شان +مورد +باره +مرسي +خويش +برخوردار +چون +خارج +شش +هنوز +تحت +ضمن +هستيم +گفته +فكر +بسيار +پيش +براي +روزهاي +انكه +نخواهد +بالا +كل +وقتي +كي +چنين +كه +گيري +نيست +است +كجا +كند +نيز +يابد +بندي +حتي +توانند +عقب +خواست +كنند +بين +تمام +همه +ما +باشند +مثل +شد +اري +باشد +اره +طبق +بعد +اگر +صورت +غير +جاي +بيش +ريزي +اند +زيرا +چگونه +بار +لطفا +مي +درباره +من +ديده +همين +گذاري +برداري +علت +گذاشته +هم +فوق +نه +ها +شوند +اباد +همواره +هر +اول +خواهند +چهار +نام +امروز +مان +هاي +قبل +كنم +سعي +تازه +را +هستند +زير +جلوي +عنوان +بود diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_fi.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_fi.txt new file mode 100644 index 00000000..c9ee2f16 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_fi.txt @@ -0,0 +1,96 @@ + | From https://snowballstem.org/algorithms/finnish/stop.txt + | This file is distributed under the BSD License. + | See https://snowballstem.org/license.html + | Also see https://opensource.org/licenses/bsd-license.html + | - Encoding was converted to UTF-8. + | - This notice was added. + | + | NOTE: To use this file with StopFilterFactory, you must specify format="snowball" + +| forms of BE + +olla +olen +olet +on +olemme +olette +ovat +ole | negative form + +oli +olisi +olisit +olisin +olisimme +olisitte +olisivat +olit +olin +olimme +olitte +olivat +ollut +olleet + +en | negation +et +ei +emme +ette +eivät + +|Nom Gen Acc Part Iness Elat Illat Adess Ablat Allat Ess Trans +minä minun minut minua minussa minusta minuun minulla minulta minulle | I +sinä sinun sinut sinua sinussa sinusta sinuun sinulla sinulta sinulle | you +hän hänen hänet häntä hänessä hänestä häneen hänellä häneltä hänelle | he she +me meidän meidät meitä meissä meistä meihin meillä meiltä meille | we +te teidän teidät teitä teissä teistä teihin teillä teiltä teille | you +he heidän heidät heitä heissä heistä heihin heillä heiltä heille | they + +tämä tämän tätä tässä tästä tähän tällä tältä tälle tänä täksi | this +tuo tuon tuota tuossa tuosta tuohon tuolla tuolta tuolle tuona tuoksi | that +se sen sitä siinä siitä siihen sillä siltä sille sinä siksi | it +nämä näiden näitä näissä näistä näihin näillä näiltä näille näinä näiksi | these +nuo noiden noita noissa noista noihin noilla noilta noille noina noiksi | those +ne niiden niitä niissä niistä niihin niillä niiltä niille niinä niiksi | they + +kuka kenen kenet ketä kenessä kenestä keneen kenellä keneltä kenelle kenenä keneksi| who +ketkä keiden ketkä keitä keissä keistä keihin keillä keiltä keille keinä keiksi | (pl) +mikä minkä minkä mitä missä mistä mihin millä miltä mille minä miksi | which what +mitkä | (pl) + +joka jonka jota jossa josta johon jolla jolta jolle jona joksi | who which +jotka joiden joita joissa joista joihin joilla joilta joille joina joiksi | (pl) + +| conjunctions + +että | that +ja | and +jos | if +koska | because +kuin | than +mutta | but +niin | so +sekä | and +sillä | for +tai | or +vaan | but +vai | or +vaikka | although + + +| prepositions + +kanssa | with +mukaan | according to +noin | about +poikki | across +yli | over, across + +| other + +kun | when +nyt | now +itse | self + diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_fr.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_fr.txt new file mode 100644 index 00000000..658ae9c9 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_fr.txt @@ -0,0 +1,186 @@ + | From https://snowballstem.org/algorithms/french/stop.txt + | This file is distributed under the BSD License. + | See https://snowballstem.org/license.html + | Also see https://opensource.org/licenses/bsd-license.html + | - Encoding was converted to UTF-8. + | - This notice was added. + | + | NOTE: To use this file with StopFilterFactory, you must specify format="snowball" + + | A French stop word list. Comments begin with vertical bar. Each stop + | word is at the start of a line. + +au | a + le +aux | a + les +avec | with +ce | this +ces | these +dans | with +de | of +des | de + les +du | de + le +elle | she +en | `of them' etc +et | and +eux | them +il | he +je | I +la | the +le | the +leur | their +lui | him +ma | my (fem) +mais | but +me | me +même | same; as in moi-même (myself) etc +mes | me (pl) +moi | me +mon | my (masc) +ne | not +nos | our (pl) +notre | our +nous | we +on | one +ou | where +par | by +pas | not +pour | for +qu | que before vowel +que | that +qui | who +sa | his, her (fem) +se | oneself +ses | his (pl) + | son | his, her (masc). Omitted because it is homonym of "sound" +sur | on +ta | thy (fem) +te | thee +tes | thy (pl) +toi | thee +ton | thy (masc) +tu | thou +un | a +une | a +vos | your (pl) +votre | your +vous | you + + | single letter forms + +c | c' +d | d' +j | j' +l | l' +à | to, at +m | m' +n | n' +s | s' +t | t' +y | there + + | forms of être (not including the infinitive): + | été - Omitted because it is homonym of "summer" +étée +étées + | étés - Omitted because it is homonym of "summers" +étant +suis +es + | est - Omitted because it is homonym of "east" + | sommes - Omitted because it is homonym of "sums" +êtes +sont +serai +seras +sera +serons +serez +seront +serais +serait +serions +seriez +seraient +étais +était +étions +étiez +étaient +fus +fut +fûmes +fûtes +furent +sois +soit +soyons +soyez +soient +fusse +fusses + | fût - Omitted because it is homonym of "tap", like in "beer on tap" +fussions +fussiez +fussent + + | forms of avoir (not including the infinitive): +ayant +eu +eue +eues +eus +ai + | as - Omitted because it is homonym of "ace" +avons +avez +ont +aurai + | auras - Omitted because it is also the name of a kind of wind + | aura - Omitted because it is also the name of a kind of wind and homonym of "aura" +aurons +aurez +auront +aurais +aurait +aurions +auriez +auraient +avais +avait + | avions - Omitted because it is homonym of "planes" +aviez +avaient +eut +eûmes +eûtes +eurent +aie +aies +ait +ayons +ayez +aient +eusse +eusses +eût +eussions +eussiez +eussent + + | Later additions (from Jean-Christophe Deschamps) +ceci | this +cela | that (added 11 Apr 2012. Omission reported by Adrien Grand) +celà | that (incorrect, though common) +cet | this +cette | this +ici | here +ils | they +les | the (pl) +leurs | their (pl) +quel | which +quels | which +quelle | which +quelles | which +sans | without +soi | oneself + diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_ga.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_ga.txt new file mode 100644 index 00000000..9ff88d74 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_ga.txt @@ -0,0 +1,110 @@ + +a +ach +ag +agus +an +aon +ar +arna +as +b' +ba +beirt +bhúr +caoga +ceathair +ceathrar +chomh +chtó +chuig +chun +cois +céad +cúig +cúigear +d' +daichead +dar +de +deich +deichniúr +den +dhá +do +don +dtí +dá +dár +dó +faoi +faoin +faoina +faoinár +fara +fiche +gach +gan +go +gur +haon +hocht +i +iad +idir +in +ina +ins +inár +is +le +leis +lena +lenár +m' +mar +mo +mé +na +nach +naoi +naonúr +ná +ní +níor +nó +nócha +ocht +ochtar +os +roimh +sa +seacht +seachtar +seachtó +seasca +seisear +siad +sibh +sinn +sna +sé +sí +tar +thar +thú +triúr +trí +trína +trínár +tríocha +tú +um +ár +é +éis +í +ó +ón +óna +ónár diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_gl.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_gl.txt new file mode 100644 index 00000000..d8760b12 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_gl.txt @@ -0,0 +1,161 @@ +# galican stopwords +a +aínda +alí +aquel +aquela +aquelas +aqueles +aquilo +aquí +ao +aos +as +así +á +ben +cando +che +co +coa +comigo +con +connosco +contigo +convosco +coas +cos +cun +cuns +cunha +cunhas +da +dalgunha +dalgunhas +dalgún +dalgúns +das +de +del +dela +delas +deles +desde +deste +do +dos +dun +duns +dunha +dunhas +e +el +ela +elas +eles +en +era +eran +esa +esas +ese +eses +esta +estar +estaba +está +están +este +estes +estiven +estou +eu +é +facer +foi +foron +fun +había +hai +iso +isto +la +las +lle +lles +lo +los +mais +me +meu +meus +min +miña +miñas +moi +na +nas +neste +nin +no +non +nos +nosa +nosas +noso +nosos +nós +nun +nunha +nuns +nunhas +o +os +ou +ó +ós +para +pero +pode +pois +pola +polas +polo +polos +por +que +se +senón +ser +seu +seus +sexa +sido +sobre +súa +súas +tamén +tan +te +ten +teñen +teño +ter +teu +teus +ti +tido +tiña +tiven +túa +túas +un +unha +unhas +uns +vos +vosa +vosas +voso +vosos +vós diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_hi.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_hi.txt new file mode 100644 index 00000000..86286bb0 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_hi.txt @@ -0,0 +1,235 @@ +# Also see http://www.opensource.org/licenses/bsd-license.html +# See http://members.unine.ch/jacques.savoy/clef/index.html. +# This file was created by Jacques Savoy and is distributed under the BSD license. +# Note: by default this file also contains forms normalized by HindiNormalizer +# for spelling variation (see section below), such that it can be used whether or +# not you enable that feature. When adding additional entries to this list, +# please add the normalized form as well. +अंदर +अत +अपना +अपनी +अपने +अभी +आदि +आप +इत्यादि +इन +इनका +इन्हीं +इन्हें +इन्हों +इस +इसका +इसकी +इसके +इसमें +इसी +इसे +उन +उनका +उनकी +उनके +उनको +उन्हीं +उन्हें +उन्हों +उस +उसके +उसी +उसे +एक +एवं +एस +ऐसे +और +कई +कर +करता +करते +करना +करने +करें +कहते +कहा +का +काफ़ी +कि +कितना +किन्हें +किन्हों +किया +किर +किस +किसी +किसे +की +कुछ +कुल +के +को +कोई +कौन +कौनसा +गया +घर +जब +जहाँ +जा +जितना +जिन +जिन्हें +जिन्हों +जिस +जिसे +जीधर +जैसा +जैसे +जो +तक +तब +तरह +तिन +तिन्हें +तिन्हों +तिस +तिसे +तो +था +थी +थे +दबारा +दिया +दुसरा +दूसरे +दो +द्वारा +न +नहीं +ना +निहायत +नीचे +ने +पर +पर +पहले +पूरा +पे +फिर +बनी +बही +बहुत +बाद +बाला +बिलकुल +भी +भीतर +मगर +मानो +मे +में +यदि +यह +यहाँ +यही +या +यिह +ये +रखें +रहा +रहे +ऱ्वासा +लिए +लिये +लेकिन +व +वर्ग +वह +वह +वहाँ +वहीं +वाले +वुह +वे +वग़ैरह +संग +सकता +सकते +सबसे +सभी +साथ +साबुत +साभ +सारा +से +सो +ही +हुआ +हुई +हुए +है +हैं +हो +होता +होती +होते +होना +होने +# additional normalized forms of the above +अपनि +जेसे +होति +सभि +तिंहों +इंहों +दवारा +इसि +किंहें +थि +उंहों +ओर +जिंहें +वहिं +अभि +बनि +हि +उंहिं +उंहें +हें +वगेरह +एसे +रवासा +कोन +निचे +काफि +उसि +पुरा +भितर +हे +बहि +वहां +कोइ +यहां +जिंहों +तिंहें +किसि +कइ +यहि +इंहिं +जिधर +इंहें +अदि +इतयादि +हुइ +कोनसा +इसकि +दुसरे +जहां +अप +किंहों +उनकि +भि +वरग +हुअ +जेसा +नहिं diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_hu.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_hu.txt new file mode 100644 index 00000000..3fa279ea --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_hu.txt @@ -0,0 +1,211 @@ + | From https://snowballstem.org/algorithms/hungarian/stop.txt + | This file is distributed under the BSD License. + | See https://snowballstem.org/license.html + | Also see https://opensource.org/licenses/bsd-license.html + | - Encoding was converted to UTF-8. + | - This notice was added. + | + | NOTE: To use this file with StopFilterFactory, you must specify format="snowball" + +| Hungarian stop word list +| prepared by Anna Tordai + +a +ahogy +ahol +aki +akik +akkor +alatt +által +általában +amely +amelyek +amelyekben +amelyeket +amelyet +amelynek +ami +amit +amolyan +amíg +amikor +át +abban +ahhoz +annak +arra +arról +az +azok +azon +azt +azzal +azért +aztán +azután +azonban +bár +be +belül +benne +cikk +cikkek +cikkeket +csak +de +e +eddig +egész +egy +egyes +egyetlen +egyéb +egyik +egyre +ekkor +el +elég +ellen +elő +először +előtt +első +én +éppen +ebben +ehhez +emilyen +ennek +erre +ez +ezt +ezek +ezen +ezzel +ezért +és +fel +felé +hanem +hiszen +hogy +hogyan +igen +így +illetve +ill. +ill +ilyen +ilyenkor +ison +ismét +itt +jó +jól +jobban +kell +kellett +keresztül +keressünk +ki +kívül +között +közül +legalább +lehet +lehetett +legyen +lenne +lenni +lesz +lett +maga +magát +majd +majd +már +más +másik +meg +még +mellett +mert +mely +melyek +mi +mit +míg +miért +milyen +mikor +minden +mindent +mindenki +mindig +mint +mintha +mivel +most +nagy +nagyobb +nagyon +ne +néha +nekem +neki +nem +néhány +nélkül +nincs +olyan +ott +össze +ő +ők +őket +pedig +persze +rá +s +saját +sem +semmi +sok +sokat +sokkal +számára +szemben +szerint +szinte +talán +tehát +teljes +tovább +továbbá +több +úgy +ugyanis +új +újabb +újra +után +utána +utolsó +vagy +vagyis +valaki +valami +valamint +való +vagyok +van +vannak +volt +voltam +voltak +voltunk +vissza +vele +viszont +volna diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_hy.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_hy.txt new file mode 100644 index 00000000..60c1c50f --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_hy.txt @@ -0,0 +1,46 @@ +# example set of Armenian stopwords. +այդ +այլ +այն +այս +դու +դուք +եմ +են +ենք +ես +եք +է +էի +էին +էինք +էիր +էիք +էր +ըստ +թ +ի +ին +իսկ +իր +կամ +համար +հետ +հետո +մենք +մեջ +մի +ն +նա +նաև +նրա +նրանք +որ +որը +որոնք +որպես +ու +ում +պիտի +վրա +և diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_id.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_id.txt new file mode 100644 index 00000000..4617f83a --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_id.txt @@ -0,0 +1,359 @@ +# from appendix D of: A Study of Stemming Effects on Information +# Retrieval in Bahasa Indonesia +ada +adanya +adalah +adapun +agak +agaknya +agar +akan +akankah +akhirnya +aku +akulah +amat +amatlah +anda +andalah +antar +diantaranya +antara +antaranya +diantara +apa +apaan +mengapa +apabila +apakah +apalagi +apatah +atau +ataukah +ataupun +bagai +bagaikan +sebagai +sebagainya +bagaimana +bagaimanapun +sebagaimana +bagaimanakah +bagi +bahkan +bahwa +bahwasanya +sebaliknya +banyak +sebanyak +beberapa +seberapa +begini +beginian +beginikah +beginilah +sebegini +begitu +begitukah +begitulah +begitupun +sebegitu +belum +belumlah +sebelum +sebelumnya +sebenarnya +berapa +berapakah +berapalah +berapapun +betulkah +sebetulnya +biasa +biasanya +bila +bilakah +bisa +bisakah +sebisanya +boleh +bolehkah +bolehlah +buat +bukan +bukankah +bukanlah +bukannya +cuma +percuma +dahulu +dalam +dan +dapat +dari +daripada +dekat +demi +demikian +demikianlah +sedemikian +dengan +depan +di +dia +dialah +dini +diri +dirinya +terdiri +dong +dulu +enggak +enggaknya +entah +entahlah +terhadap +terhadapnya +hal +hampir +hanya +hanyalah +harus +haruslah +harusnya +seharusnya +hendak +hendaklah +hendaknya +hingga +sehingga +ia +ialah +ibarat +ingin +inginkah +inginkan +ini +inikah +inilah +itu +itukah +itulah +jangan +jangankan +janganlah +jika +jikalau +juga +justru +kala +kalau +kalaulah +kalaupun +kalian +kami +kamilah +kamu +kamulah +kan +kapan +kapankah +kapanpun +dikarenakan +karena +karenanya +ke +kecil +kemudian +kenapa +kepada +kepadanya +ketika +seketika +khususnya +kini +kinilah +kiranya +sekiranya +kita +kitalah +kok +lagi +lagian +selagi +lah +lain +lainnya +melainkan +selaku +lalu +melalui +terlalu +lama +lamanya +selama +selama +selamanya +lebih +terlebih +bermacam +macam +semacam +maka +makanya +makin +malah +malahan +mampu +mampukah +mana +manakala +manalagi +masih +masihkah +semasih +masing +mau +maupun +semaunya +memang +mereka +merekalah +meski +meskipun +semula +mungkin +mungkinkah +nah +namun +nanti +nantinya +nyaris +oleh +olehnya +seorang +seseorang +pada +padanya +padahal +paling +sepanjang +pantas +sepantasnya +sepantasnyalah +para +pasti +pastilah +per +pernah +pula +pun +merupakan +rupanya +serupa +saat +saatnya +sesaat +saja +sajalah +saling +bersama +sama +sesama +sambil +sampai +sana +sangat +sangatlah +saya +sayalah +se +sebab +sebabnya +sebuah +tersebut +tersebutlah +sedang +sedangkan +sedikit +sedikitnya +segala +segalanya +segera +sesegera +sejak +sejenak +sekali +sekalian +sekalipun +sesekali +sekaligus +sekarang +sekarang +sekitar +sekitarnya +sela +selain +selalu +seluruh +seluruhnya +semakin +sementara +sempat +semua +semuanya +sendiri +sendirinya +seolah +seperti +sepertinya +sering +seringnya +serta +siapa +siapakah +siapapun +disini +disinilah +sini +sinilah +sesuatu +sesuatunya +suatu +sesudah +sesudahnya +sudah +sudahkah +sudahlah +supaya +tadi +tadinya +tak +tanpa +setelah +telah +tentang +tentu +tentulah +tentunya +tertentu +seterusnya +tapi +tetapi +setiap +tiap +setidaknya +tidak +tidakkah +tidaklah +toh +waduh +wah +wahai +sewaktu +walau +walaupun +wong +yaitu +yakni +yang diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_it.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_it.txt new file mode 100644 index 00000000..c74160e2 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_it.txt @@ -0,0 +1,303 @@ + | From https://snowballstem.org/algorithms/italian/stop.txt + | This file is distributed under the BSD License. + | See https://snowballstem.org/license.html + | Also see https://opensource.org/licenses/bsd-license.html + | - Encoding was converted to UTF-8. + | - This notice was added. + | + | NOTE: To use this file with StopFilterFactory, you must specify format="snowball" + + | An Italian stop word list. Comments begin with vertical bar. Each stop + | word is at the start of a line. + +ad | a (to) before vowel +al | a + il +allo | a + lo +ai | a + i +agli | a + gli +all | a + l' +agl | a + gl' +alla | a + la +alle | a + le +con | with +col | con + il +coi | con + i (forms collo, cogli etc are now very rare) +da | from +dal | da + il +dallo | da + lo +dai | da + i +dagli | da + gli +dall | da + l' +dagl | da + gll' +dalla | da + la +dalle | da + le +di | of +del | di + il +dello | di + lo +dei | di + i +degli | di + gli +dell | di + l' +degl | di + gl' +della | di + la +delle | di + le +in | in +nel | in + el +nello | in + lo +nei | in + i +negli | in + gli +nell | in + l' +negl | in + gl' +nella | in + la +nelle | in + le +su | on +sul | su + il +sullo | su + lo +sui | su + i +sugli | su + gli +sull | su + l' +sugl | su + gl' +sulla | su + la +sulle | su + le +per | through, by +tra | among +contro | against +io | I +tu | thou +lui | he +lei | she +noi | we +voi | you +loro | they +mio | my +mia | +miei | +mie | +tuo | +tua | +tuoi | thy +tue | +suo | +sua | +suoi | his, her +sue | +nostro | our +nostra | +nostri | +nostre | +vostro | your +vostra | +vostri | +vostre | +mi | me +ti | thee +ci | us, there +vi | you, there +lo | him, the +la | her, the +li | them +le | them, the +gli | to him, the +ne | from there etc +il | the +un | a +uno | a +una | a +ma | but +ed | and +se | if +perché | why, because +anche | also +come | how +dov | where (as dov') +dove | where +che | who, that +chi | who +cui | whom +non | not +più | more +quale | who, that +quanto | how much +quanti | +quanta | +quante | +quello | that +quelli | +quella | +quelle | +questo | this +questi | +questa | +queste | +si | yes +tutto | all +tutti | all + + | single letter forms: + +a | at +c | as c' for ce or ci +e | and +i | the +l | as l' +o | or + + | forms of avere, to have (not including the infinitive): + +ho +hai +ha +abbiamo +avete +hanno +abbia +abbiate +abbiano +avrò +avrai +avrà +avremo +avrete +avranno +avrei +avresti +avrebbe +avremmo +avreste +avrebbero +avevo +avevi +aveva +avevamo +avevate +avevano +ebbi +avesti +ebbe +avemmo +aveste +ebbero +avessi +avesse +avessimo +avessero +avendo +avuto +avuta +avuti +avute + + | forms of essere, to be (not including the infinitive): +sono +sei +è +siamo +siete +sia +siate +siano +sarò +sarai +sarà +saremo +sarete +saranno +sarei +saresti +sarebbe +saremmo +sareste +sarebbero +ero +eri +era +eravamo +eravate +erano +fui +fosti +fu +fummo +foste +furono +fossi +fosse +fossimo +fossero +essendo + + | forms of fare, to do (not including the infinitive, fa, fat-): +faccio +fai +facciamo +fanno +faccia +facciate +facciano +farò +farai +farà +faremo +farete +faranno +farei +faresti +farebbe +faremmo +fareste +farebbero +facevo +facevi +faceva +facevamo +facevate +facevano +feci +facesti +fece +facemmo +faceste +fecero +facessi +facesse +facessimo +facessero +facendo + + | forms of stare, to be (not including the infinitive): +sto +stai +sta +stiamo +stanno +stia +stiate +stiano +starò +starai +starà +staremo +starete +staranno +starei +staresti +starebbe +staremmo +stareste +starebbero +stavo +stavi +stava +stavamo +stavate +stavano +stetti +stesti +stette +stemmo +steste +stettero +stessi +stesse +stessimo +stessero +stando diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_ja.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_ja.txt new file mode 100644 index 00000000..d4321be6 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_ja.txt @@ -0,0 +1,127 @@ +# +# This file defines a stopword set for Japanese. +# +# This set is made up of hand-picked frequent terms from segmented Japanese Wikipedia. +# Punctuation characters and frequent kanji have mostly been left out. See LUCENE-3745 +# for frequency lists, etc. that can be useful for making your own set (if desired) +# +# Note that there is an overlap between these stopwords and the terms stopped when used +# in combination with the JapanesePartOfSpeechStopFilter. When editing this file, note +# that comments are not allowed on the same line as stopwords. +# +# Also note that stopping is done in a case-insensitive manner. Change your StopFilter +# configuration if you need case-sensitive stopping. Lastly, note that stopping is done +# using the same character width as the entries in this file. Since this StopFilter is +# normally done after a CJKWidthFilter in your chain, you would usually want your romaji +# entries to be in half-width and your kana entries to be in full-width. +# +の +に +は +を +た +が +で +て +と +し +れ +さ +ある +いる +も +する +から +な +こと +として +い +や +れる +など +なっ +ない +この +ため +その +あっ +よう +また +もの +という +あり +まで +られ +なる +へ +か +だ +これ +によって +により +おり +より +による +ず +なり +られる +において +ば +なかっ +なく +しかし +について +せ +だっ +その後 +できる +それ +う +ので +なお +のみ +でき +き +つ +における +および +いう +さらに +でも +ら +たり +その他 +に関する +たち +ます +ん +なら +に対して +特に +せる +及び +これら +とき +では +にて +ほか +ながら +うち +そして +とともに +ただし +かつて +それぞれ +または +お +ほど +ものの +に対する +ほとんど +と共に +といった +です +とも +ところ +ここ +##### End of file diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_lv.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_lv.txt new file mode 100644 index 00000000..e21a23c0 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_lv.txt @@ -0,0 +1,172 @@ +# Set of Latvian stopwords from A Stemming Algorithm for Latvian, Karlis Kreslins +# the original list of over 800 forms was refined: +# pronouns, adverbs, interjections were removed +# +# prepositions +aiz +ap +ar +apakš +ārpus +augšpus +bez +caur +dēļ +gar +iekš +iz +kopš +labad +lejpus +līdz +no +otrpus +pa +par +pār +pēc +pie +pirms +pret +priekš +starp +šaipus +uz +viņpus +virs +virspus +zem +apakšpus +# Conjunctions +un +bet +jo +ja +ka +lai +tomēr +tikko +turpretī +arī +kaut +gan +tādēļ +tā +ne +tikvien +vien +kā +ir +te +vai +kamēr +# Particles +ar +diezin +droši +diemžēl +nebūt +ik +it +taču +nu +pat +tiklab +iekšpus +nedz +tik +nevis +turpretim +jeb +iekam +iekām +iekāms +kolīdz +līdzko +tiklīdz +jebšu +tālab +tāpēc +nekā +itin +jā +jau +jel +nē +nezin +tad +tikai +vis +tak +iekams +vien +# modal verbs +būt +biju +biji +bija +bijām +bijāt +esmu +esi +esam +esat +būšu +būsi +būs +būsim +būsiet +tikt +tiku +tiki +tika +tikām +tikāt +tieku +tiec +tiek +tiekam +tiekat +tikšu +tiks +tiksim +tiksiet +tapt +tapi +tapāt +topat +tapšu +tapsi +taps +tapsim +tapsiet +kļūt +kļuvu +kļuvi +kļuva +kļuvām +kļuvāt +kļūstu +kļūsti +kļūst +kļūstam +kļūstat +kļūšu +kļūsi +kļūs +kļūsim +kļūsiet +# verbs +varēt +varēju +varējām +varēšu +varēsim +var +varēji +varējāt +varēsi +varēsiet +varat +varēja +varēs diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_nl.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_nl.txt new file mode 100644 index 00000000..48c55151 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_nl.txt @@ -0,0 +1,121 @@ + | From https://snowballstem.org/algorithms/dutch/stop.txt + | This file is distributed under the BSD License. + | See https://snowballstem.org/license.html + | Also see https://opensource.org/licenses/bsd-license.html + | - Encoding was converted to UTF-8. + | - This notice was added. + | + | NOTE: To use this file with StopFilterFactory, you must specify format="snowball" + + + | A Dutch stop word list. Comments begin with vertical bar. Each stop + | word is at the start of a line. + + | This is a ranked list (commonest to rarest) of stopwords derived from + | a large sample of Dutch text. + + | Dutch stop words frequently exhibit homonym clashes. These are indicated + | clearly below. + +de | the +en | and +van | of, from +ik | I, the ego +te | (1) chez, at etc, (2) to, (3) too +dat | that, which +die | that, those, who, which +in | in, inside +een | a, an, one +hij | he +het | the, it +niet | not, nothing, naught +zijn | (1) to be, being, (2) his, one's, its +is | is +was | (1) was, past tense of all persons sing. of 'zijn' (to be) (2) wax, (3) the washing, (4) rise of river +op | on, upon, at, in, up, used up +aan | on, upon, to (as dative) +met | with, by +als | like, such as, when +voor | (1) before, in front of, (2) furrow +had | had, past tense all persons sing. of 'hebben' (have) +er | there +maar | but, only +om | round, about, for etc +hem | him +dan | then +zou | should/would, past tense all persons sing. of 'zullen' +of | or, whether, if +wat | what, something, anything +mijn | possessive and noun 'mine' +men | people, 'one' +dit | this +zo | so, thus, in this way +door | through by +over | over, across +ze | she, her, they, them +zich | oneself +bij | (1) a bee, (2) by, near, at +ook | also, too +tot | till, until +je | you +mij | me +uit | out of, from +der | Old Dutch form of 'van der' still found in surnames +daar | (1) there, (2) because +haar | (1) her, their, them, (2) hair +naar | (1) unpleasant, unwell etc, (2) towards, (3) as +heb | present first person sing. of 'to have' +hoe | how, why +heeft | present third person sing. of 'to have' +hebben | 'to have' and various parts thereof +deze | this +u | you +want | (1) for, (2) mitten, (3) rigging +nog | yet, still +zal | 'shall', first and third person sing. of verb 'zullen' (will) +me | me +zij | she, they +nu | now +ge | 'thou', still used in Belgium and south Netherlands +geen | none +omdat | because +iets | something, somewhat +worden | to become, grow, get +toch | yet, still +al | all, every, each +waren | (1) 'were' (2) to wander, (3) wares, (3) +veel | much, many +meer | (1) more, (2) lake +doen | to do, to make +toen | then, when +moet | noun 'spot/mote' and present form of 'to must' +ben | (1) am, (2) 'are' in interrogative second person singular of 'to be' +zonder | without +kan | noun 'can' and present form of 'to be able' +hun | their, them +dus | so, consequently +alles | all, everything, anything +onder | under, beneath +ja | yes, of course +eens | once, one day +hier | here +wie | who +werd | imperfect third person sing. of 'become' +altijd | always +doch | yet, but etc +wordt | present third person sing. of 'become' +wezen | (1) to be, (2) 'been' as in 'been fishing', (3) orphans +kunnen | to be able +ons | us/our +zelf | self +tegen | against, towards, at +na | after, near +reeds | already +wil | (1) present tense of 'want', (2) 'will', noun, (3) fender +kon | could; past tense of 'to be able' +niets | nothing +uw | your +iemand | somebody +geweest | been; past participle of 'be' +andere | other + diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_no.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_no.txt new file mode 100644 index 00000000..f4276094 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_no.txt @@ -0,0 +1,190 @@ + | From https://snowballstem.org/algorithms/norwegian/stop.txt + | This file is distributed under the BSD License. + | See https://snowballstem.org/license.html + | Also see https://opensource.org/licenses/bsd-license.html + | - Encoding was converted to UTF-8. + | - This notice was added. + | + | NOTE: To use this file with StopFilterFactory, you must specify format="snowball" + + | A Norwegian stop word list. Comments begin with vertical bar. Each stop + | word is at the start of a line. + + | This stop word list is for the dominant bokmål dialect. Words unique + | to nynorsk are marked *. + + | Revised by Jan Bruusgaard , Jan 2005 + +og | and +i | in +jeg | I +det | it/this/that +at | to (w. inf.) +en | a/an +et | a/an +den | it/this/that +til | to +er | is/am/are +som | who/which/that +på | on +de | they / you(formal) +med | with +han | he +av | of +ikke | not +ikkje | not * +der | there +så | so +var | was/were +meg | me +seg | you +men | but +ett | one +har | have +om | about +vi | we +min | my +mitt | my +ha | have +hadde | had +hun | she +nå | now +over | over +da | when/as +ved | by/know +fra | from +du | you +ut | out +sin | your +dem | them +oss | us +opp | up +man | you/one +kan | can +hans | his +hvor | where +eller | or +hva | what +skal | shall/must +selv | self (reflective) +sjøl | self (reflective) +her | here +alle | all +vil | will +bli | become +ble | became +blei | became * +blitt | have become +kunne | could +inn | in +når | when +være | be +kom | come +noen | some +noe | some +ville | would +dere | you +deres | their/theirs +kun | only/just +ja | yes +etter | after +ned | down +skulle | should +denne | this +for | for/because +deg | you +si | hers/his +sine | hers/his +sitt | hers/his +mot | against +å | to +meget | much +hvorfor | why +dette | this +disse | these/those +uten | without +hvordan | how +ingen | none +din | your +ditt | your +blir | become +samme | same +hvilken | which +hvilke | which (plural) +sånn | such a +inni | inside/within +mellom | between +vår | our +hver | each +hvem | who +vors | us/ours +hvis | whose +både | both +bare | only/just +enn | than +fordi | as/because +før | before +mange | many +også | also +slik | just +vært | been +båe | both * +begge | both +siden | since +dykk | your * +dykkar | yours * +dei | they * +deira | them * +deires | theirs * +deim | them * +di | your (fem.) * +då | as/when * +eg | I * +ein | a/an * +eit | a/an * +eitt | a/an * +elles | or * +honom | he * +hjå | at * +ho | she * +hoe | she * +henne | her +hennar | her/hers +hennes | hers +hoss | how * +hossen | how * +ingi | noone * +inkje | noone * +korleis | how * +korso | how * +kva | what/which * +kvar | where * +kvarhelst | where * +kven | who/whom * +kvi | why * +kvifor | why * +me | we * +medan | while * +mi | my * +mine | my * +mykje | much * +no | now * +nokon | some (masc./neut.) * +noka | some (fem.) * +nokor | some * +noko | some * +nokre | some * +sia | since * +sidan | since * +so | so * +somt | some * +somme | some * +um | about* +upp | up * +vere | be * +vore | was * +verte | become * +vort | become * +varte | became * +vart | became * + diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_pt.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_pt.txt new file mode 100644 index 00000000..d03d7f23 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_pt.txt @@ -0,0 +1,253 @@ + | From https://snowballstem.org/algorithms/portuguese/stop.txt + | This file is distributed under the BSD License. + | See https://snowballstem.org/license.html + | Also see https://opensource.org/licenses/bsd-license.html + | - Encoding was converted to UTF-8. + | - This notice was added. + | + | NOTE: To use this file with StopFilterFactory, you must specify format="snowball" + + | A Portuguese stop word list. Comments begin with vertical bar. Each stop + | word is at the start of a line. + + + | The following is a ranked list (commonest to rarest) of stopwords + | deriving from a large sample of text. + + | Extra words have been added at the end. + +de | of, from +a | the; to, at; her +o | the; him +que | who, that +e | and +do | de + o +da | de + a +em | in +um | a +para | for + | é from SER +com | with +não | not, no +uma | a +os | the; them +no | em + o +se | himself etc +na | em + a +por | for +mais | more +as | the; them +dos | de + os +como | as, like +mas | but + | foi from SER +ao | a + o +ele | he +das | de + as + | tem from TER +à | a + a +seu | his +sua | her +ou | or + | ser from SER +quando | when +muito | much + | há from HAV +nos | em + os; us +já | already, now + | está from EST +eu | I +também | also +só | only, just +pelo | per + o +pela | per + a +até | up to +isso | that +ela | he +entre | between + | era from SER +depois | after +sem | without +mesmo | same +aos | a + os + | ter from TER +seus | his +quem | whom +nas | em + as +me | me +esse | that +eles | they + | estão from EST +você | you + | tinha from TER + | foram from SER +essa | that +num | em + um +nem | nor +suas | her +meu | my +às | a + as +minha | my + | têm from TER +numa | em + uma +pelos | per + os +elas | they + | havia from HAV + | seja from SER +qual | which + | será from SER +nós | we + | tenho from TER +lhe | to him, her +deles | of them +essas | those +esses | those +pelas | per + as +este | this + | fosse from SER +dele | of him + + | other words. There are many contractions such as naquele = em+aquele, + | mo = me+o, but they are rare. + | Indefinite article plural forms are also rare. + +tu | thou +te | thee +vocês | you (plural) +vos | you +lhes | to them +meus | my +minhas +teu | thy +tua +teus +tuas +nosso | our +nossa +nossos +nossas + +dela | of her +delas | of them + +esta | this +estes | these +estas | these +aquele | that +aquela | that +aqueles | those +aquelas | those +isto | this +aquilo | that + + | forms of estar, to be (not including the infinitive): +estou +está +estamos +estão +estive +esteve +estivemos +estiveram +estava +estávamos +estavam +estivera +estivéramos +esteja +estejamos +estejam +estivesse +estivéssemos +estivessem +estiver +estivermos +estiverem + + | forms of haver, to have (not including the infinitive): +hei +há +havemos +hão +houve +houvemos +houveram +houvera +houvéramos +haja +hajamos +hajam +houvesse +houvéssemos +houvessem +houver +houvermos +houverem +houverei +houverá +houveremos +houverão +houveria +houveríamos +houveriam + + | forms of ser, to be (not including the infinitive): +sou +somos +são +era +éramos +eram +fui +foi +fomos +foram +fora +fôramos +seja +sejamos +sejam +fosse +fôssemos +fossem +for +formos +forem +serei +será +seremos +serão +seria +seríamos +seriam + + | forms of ter, to have (not including the infinitive): +tenho +tem +temos +tém +tinha +tínhamos +tinham +tive +teve +tivemos +tiveram +tivera +tivéramos +tenha +tenhamos +tenham +tivesse +tivéssemos +tivessem +tiver +tivermos +tiverem +terei +terá +teremos +terão +teria +teríamos +teriam diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_ro.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_ro.txt new file mode 100644 index 00000000..4fdee90a --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_ro.txt @@ -0,0 +1,233 @@ +# This file was created by Jacques Savoy and is distributed under the BSD license. +# See http://members.unine.ch/jacques.savoy/clef/index.html. +# Also see http://www.opensource.org/licenses/bsd-license.html +acea +aceasta +această +aceea +acei +aceia +acel +acela +acele +acelea +acest +acesta +aceste +acestea +aceşti +aceştia +acolo +acum +ai +aia +aibă +aici +al +ăla +ale +alea +ălea +altceva +altcineva +am +ar +are +aş +aşadar +asemenea +asta +ăsta +astăzi +astea +ăstea +ăştia +asupra +aţi +au +avea +avem +aveţi +azi +bine +bucur +bună +ca +că +căci +când +care +cărei +căror +cărui +cât +câte +câţi +către +câtva +ce +cel +ceva +chiar +cînd +cine +cineva +cît +cîte +cîţi +cîtva +contra +cu +cum +cumva +curând +curînd +da +dă +dacă +dar +datorită +de +deci +deja +deoarece +departe +deşi +din +dinaintea +dintr +dintre +drept +după +ea +ei +el +ele +eram +este +eşti +eu +face +fără +fi +fie +fiecare +fii +fim +fiţi +iar +ieri +îi +îl +îmi +împotriva +în +înainte +înaintea +încât +încît +încotro +între +întrucât +întrucît +îţi +la +lângă +le +li +lîngă +lor +lui +mă +mâine +mea +mei +mele +mereu +meu +mi +mine +mult +multă +mulţi +ne +nicăieri +nici +nimeni +nişte +noastră +noastre +noi +noştri +nostru +nu +ori +oricând +oricare +oricât +orice +oricînd +oricine +oricît +oricum +oriunde +până +pe +pentru +peste +pînă +poate +pot +prea +prima +primul +prin +printr +sa +să +săi +sale +sau +său +se +şi +sînt +sîntem +sînteţi +spre +sub +sunt +suntem +sunteţi +ta +tăi +tale +tău +te +ţi +ţie +tine +toată +toate +tot +toţi +totuşi +tu +un +una +unde +undeva +unei +unele +uneori +unor +vă +vi +voastră +voastre +voi +voştri +vostru +vouă +vreo +vreun diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_ru.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_ru.txt new file mode 100644 index 00000000..65512d49 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_ru.txt @@ -0,0 +1,244 @@ + | From https://snowballstem.org/algorithms/russian/stop.txt + | This file is distributed under the BSD License. + | See https://snowballstem.org/license.html + | Also see https://opensource.org/licenses/bsd-license.html + | - Encoding was converted to UTF-8. + | - This notice was added. + | + | NOTE: To use this file with StopFilterFactory, you must specify format="snowball" + + + | a russian stop word list. comments begin with vertical bar. each stop + | word is at the start of a line. + + | this is a ranked list (commonest to rarest) of stopwords derived from + | a large text sample. + + | letter `ё' is translated to `е'. + +и | and +в | in/into +во | alternative form +не | not +что | what/that +он | he +на | on/onto +я | i +с | from +со | alternative form +как | how +а | milder form of `no' (but) +то | conjunction and form of `that' +все | all +она | she +так | so, thus +его | him +но | but +да | yes/and +ты | thou +к | towards, by +у | around, chez +же | intensifier particle +вы | you +за | beyond, behind +бы | conditional/subj. particle +по | up to, along +только | only +ее | her +мне | to me +было | it was +вот | here is/are, particle +от | away from +меня | me +еще | still, yet, more +нет | no, there isnt/arent +о | about +из | out of +ему | to him +теперь | now +когда | when +даже | even +ну | so, well +вдруг | suddenly +ли | interrogative particle +если | if +уже | already, but homonym of `narrower' +или | or +ни | neither +быть | to be +был | he was +него | prepositional form of его +до | up to +вас | you accusative +нибудь | indef. suffix preceded by hyphen +опять | again +уж | already, but homonym of `adder' +вам | to you +сказал | he said +ведь | particle `after all' +там | there +потом | then +себя | oneself +ничего | nothing +ей | to her +может | usually with `быть' as `maybe' +они | they +тут | here +где | where +есть | there is/are +надо | got to, must +ней | prepositional form of ей +для | for +мы | we +тебя | thee +их | them, their +чем | than +была | she was +сам | self +чтоб | in order to +без | without +будто | as if +человек | man, person, one +чего | genitive form of `what' +раз | once +тоже | also +себе | to oneself +под | beneath +жизнь | life +будет | will be +ж | short form of intensifer particle `же' +тогда | then +кто | who +этот | this +говорил | was saying +того | genitive form of `that' +потому | for that reason +этого | genitive form of `this' +какой | which +совсем | altogether +ним | prepositional form of `его', `они' +здесь | here +этом | prepositional form of `этот' +один | one +почти | almost +мой | my +тем | instrumental/dative plural of `тот', `то' +чтобы | full form of `in order that' +нее | her (acc.) +кажется | it seems +сейчас | now +были | they were +куда | where to +зачем | why +сказать | to say +всех | all (acc., gen. preposn. plural) +никогда | never +сегодня | today +можно | possible, one can +при | by +наконец | finally +два | two +об | alternative form of `о', about +другой | another +хоть | even +после | after +над | above +больше | more +тот | that one (masc.) +через | across, in +эти | these +нас | us +про | about +всего | in all, only, of all +них | prepositional form of `они' (they) +какая | which, feminine +много | lots +разве | interrogative particle +сказала | she said +три | three +эту | this, acc. fem. sing. +моя | my, feminine +впрочем | moreover, besides +хорошо | good +свою | ones own, acc. fem. sing. +этой | oblique form of `эта', fem. `this' +перед | in front of +иногда | sometimes +лучше | better +чуть | a little +том | preposn. form of `that one' +нельзя | one must not +такой | such a one +им | to them +более | more +всегда | always +конечно | of course +всю | acc. fem. sing of `all' +между | between + + + | b: some paradigms + | + | personal pronouns + | + | я меня мне мной [мною] + | ты тебя тебе тобой [тобою] + | он его ему им [него, нему, ним] + | она ее эи ею [нее, нэи, нею] + | оно его ему им [него, нему, ним] + | + | мы нас нам нами + | вы вас вам вами + | они их им ими [них, ним, ними] + | + | себя себе собой [собою] + | + | demonstrative pronouns: этот (this), тот (that) + | + | этот эта это эти + | этого эты это эти + | этого этой этого этих + | этому этой этому этим + | этим этой этим [этою] этими + | этом этой этом этих + | + | тот та то те + | того ту то те + | того той того тех + | тому той тому тем + | тем той тем [тою] теми + | том той том тех + | + | determinative pronouns + | + | (a) весь (all) + | + | весь вся все все + | всего всю все все + | всего всей всего всех + | всему всей всему всем + | всем всей всем [всею] всеми + | всем всей всем всех + | + | (b) сам (himself etc) + | + | сам сама само сами + | самого саму само самих + | самого самой самого самих + | самому самой самому самим + | самим самой самим [самою] самими + | самом самой самом самих + | + | stems of verbs `to be', `to have', `to do' and modal + | + | быть бы буд быв есть суть + | име + | дел + | мог мож мочь + | уме + | хоч хот + | долж + | можн + | нужн + | нельзя + diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_sv.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_sv.txt new file mode 100644 index 00000000..d1d0d100 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_sv.txt @@ -0,0 +1,133 @@ + | From https://snowballstem.org/algorithms/swedish/stop.txt + | This file is distributed under the BSD License. + | See https://snowballstem.org/license.html + | Also see https://opensource.org/licenses/bsd-license.html + | - Encoding was converted to UTF-8. + | - This notice was added. + | + | NOTE: To use this file with StopFilterFactory, you must specify format="snowball" + + | A Swedish stop word list. Comments begin with vertical bar. Each stop + | word is at the start of a line. + + | This is a ranked list (commonest to rarest) of stopwords derived from + | a large text sample. + + | Swedish stop words occasionally exhibit homonym clashes. For example + | så = so, but also seed. These are indicated clearly below. + +och | and +det | it, this/that +att | to (with infinitive) +i | in, at +en | a +jag | I +hon | she +som | who, that +han | he +på | on +den | it, this/that +med | with +var | where, each +sig | him(self) etc +för | for +så | so (also: seed) +till | to +är | is +men | but +ett | a +om | if; around, about +hade | had +de | they, these/those +av | of +icke | not, no +mig | me +du | you +henne | her +då | then, when +sin | his +nu | now +har | have +inte | inte någon = no one +hans | his +honom | him +skulle | 'sake' +hennes | her +där | there +min | my +man | one (pronoun) +ej | nor +vid | at, by, on (also: vast) +kunde | could +något | some etc +från | from, off +ut | out +när | when +efter | after, behind +upp | up +vi | we +dem | them +vara | be +vad | what +över | over +än | than +dig | you +kan | can +sina | his +här | here +ha | have +mot | towards +alla | all +under | under (also: wonder) +någon | some etc +eller | or (else) +allt | all +mycket | much +sedan | since +ju | why +denna | this/that +själv | myself, yourself etc +detta | this/that +åt | to +utan | without +varit | was +hur | how +ingen | no +mitt | my +ni | you +bli | to be, become +blev | from bli +oss | us +din | thy +dessa | these/those +några | some etc +deras | their +blir | from bli +mina | my +samma | (the) same +vilken | who, that +er | you, your +sådan | such a +vår | our +blivit | from bli +dess | its +inom | within +mellan | between +sådant | such a +varför | why +varje | each +vilka | who, that +ditt | thy +vem | who +vilket | who, that +sitt | his +sådana | such a +vart | each +dina | thy +vars | whose +vårt | our +våra | our +ert | your +era | your +vilkas | whose + diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_th.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_th.txt new file mode 100644 index 00000000..07f0fabe --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_th.txt @@ -0,0 +1,119 @@ +# Thai stopwords from: +# "Opinion Detection in Thai Political News Columns +# Based on Subjectivity Analysis" +# Khampol Sukhum, Supot Nitsuwat, and Choochart Haruechaiyasak +ไว้ +ไม่ +ไป +ได้ +ให้ +ใน +โดย +แห่ง +แล้ว +และ +แรก +แบบ +แต่ +เอง +เห็น +เลย +เริ่ม +เรา +เมื่อ +เพื่อ +เพราะ +เป็นการ +เป็น +เปิดเผย +เปิด +เนื่องจาก +เดียวกัน +เดียว +เช่น +เฉพาะ +เคย +เข้า +เขา +อีก +อาจ +อะไร +ออก +อย่าง +อยู่ +อยาก +หาก +หลาย +หลังจาก +หลัง +หรือ +หนึ่ง +ส่วน +ส่ง +สุด +สําหรับ +ว่า +วัน +ลง +ร่วม +ราย +รับ +ระหว่าง +รวม +ยัง +มี +มาก +มา +พร้อม +พบ +ผ่าน +ผล +บาง +น่า +นี้ +นํา +นั้น +นัก +นอกจาก +ทุก +ที่สุด +ที่ +ทําให้ +ทํา +ทาง +ทั้งนี้ +ทั้ง +ถ้า +ถูก +ถึง +ต้อง +ต่างๆ +ต่าง +ต่อ +ตาม +ตั้งแต่ +ตั้ง +ด้าน +ด้วย +ดัง +ซึ่ง +ช่วง +จึง +จาก +จัด +จะ +คือ +ความ +ครั้ง +คง +ขึ้น +ของ +ขอ +ขณะ +ก่อน +ก็ +การ +กับ +กัน +กว่า +กล่าว diff --git a/data-loading/configsets/name_lookup/conf/lang/stopwords_tr.txt b/data-loading/configsets/name_lookup/conf/lang/stopwords_tr.txt new file mode 100644 index 00000000..84d9408d --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/stopwords_tr.txt @@ -0,0 +1,212 @@ +# Turkish stopwords from LUCENE-559 +# merged with the list from "Information Retrieval on Turkish Texts" +# (http://www.users.muohio.edu/canf/papers/JASIST2008offPrint.pdf) +acaba +altmış +altı +ama +ancak +arada +aslında +ayrıca +bana +bazı +belki +ben +benden +beni +benim +beri +beş +bile +bin +bir +birçok +biri +birkaç +birkez +birşey +birşeyi +biz +bize +bizden +bizi +bizim +böyle +böylece +bu +buna +bunda +bundan +bunlar +bunları +bunların +bunu +bunun +burada +çok +çünkü +da +daha +dahi +de +defa +değil +diğer +diye +doksan +dokuz +dolayı +dolayısıyla +dört +edecek +eden +ederek +edilecek +ediliyor +edilmesi +ediyor +eğer +elli +en +etmesi +etti +ettiği +ettiğini +gibi +göre +halen +hangi +hatta +hem +henüz +hep +hepsi +her +herhangi +herkesin +hiç +hiçbir +için +iki +ile +ilgili +ise +işte +itibaren +itibariyle +kadar +karşın +katrilyon +kendi +kendilerine +kendini +kendisi +kendisine +kendisini +kez +ki +kim +kimden +kime +kimi +kimse +kırk +milyar +milyon +mu +mü +mı +nasıl +ne +neden +nedenle +nerde +nerede +nereye +niye +niçin +o +olan +olarak +oldu +olduğu +olduğunu +olduklarını +olmadı +olmadığı +olmak +olması +olmayan +olmaz +olsa +olsun +olup +olur +olursa +oluyor +on +ona +ondan +onlar +onlardan +onları +onların +onu +onun +otuz +oysa +öyle +pek +rağmen +sadece +sanki +sekiz +seksen +sen +senden +seni +senin +siz +sizden +sizi +sizin +şey +şeyden +şeyi +şeyler +şöyle +şu +şuna +şunda +şundan +şunları +şunu +tarafından +trilyon +tüm +üç +üzere +var +vardı +ve +veya +ya +yani +yapacak +yapılan +yapılması +yapıyor +yapmak +yaptı +yaptığı +yaptığını +yaptıkları +yedi +yerine +yetmiş +yine +yirmi +yoksa +yüz +zaten diff --git a/data-loading/configsets/name_lookup/conf/lang/userdict_ja.txt b/data-loading/configsets/name_lookup/conf/lang/userdict_ja.txt new file mode 100644 index 00000000..6f0368e4 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/lang/userdict_ja.txt @@ -0,0 +1,29 @@ +# +# This is a sample user dictionary for Kuromoji (JapaneseTokenizer) +# +# Add entries to this file in order to override the statistical model in terms +# of segmentation, readings and part-of-speech tags. Notice that entries do +# not have weights since they are always used when found. This is by-design +# in order to maximize ease-of-use. +# +# Entries are defined using the following CSV format: +# , ... , ... , +# +# Notice that a single half-width space separates tokens and readings, and +# that the number tokens and readings must match exactly. +# +# Also notice that multiple entries with the same is undefined. +# +# Whitespace only lines are ignored. Comments are not allowed on entry lines. +# + +# Custom segmentation for kanji compounds +日本経済新聞,日本 経済 新聞,ニホン ケイザイ シンブン,カスタム名詞 +関西国際空港,関西 国際 空港,カンサイ コクサイ クウコウ,カスタム名詞 + +# Custom segmentation for compound katakana +トートバッグ,トート バッグ,トート バッグ,かずカナ名詞 +ショルダーバッグ,ショルダー バッグ,ショルダー バッグ,かずカナ名詞 + +# Custom reading for former sumo wrestler +朝青龍,朝青龍,アサショウリュウ,カスタム人名 diff --git a/data-loading/configsets/name_lookup/conf/managed-schema.xml b/data-loading/configsets/name_lookup/conf/managed-schema.xml new file mode 100644 index 00000000..b5214376 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/managed-schema.xml @@ -0,0 +1,549 @@ + + + + id + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/data-loading/configsets/name_lookup/conf/protwords.txt b/data-loading/configsets/name_lookup/conf/protwords.txt new file mode 100644 index 00000000..1dfc0abe --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/protwords.txt @@ -0,0 +1,21 @@ +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +#----------------------------------------------------------------------- +# Use a protected word file to protect against the stemmer reducing two +# unrelated words to the same base word. + +# Some non-words that normally won't be encountered, +# just to test that they won't be stemmed. +dontstems +zwhacky + diff --git a/data-loading/configsets/name_lookup/conf/solrconfig.xml b/data-loading/configsets/name_lookup/conf/solrconfig.xml new file mode 100644 index 00000000..fb171905 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/solrconfig.xml @@ -0,0 +1,1032 @@ + + + + + + + + + 9.12 + + + ${solr.data.dir:} + + + + + + + + + + + + + + + + + + + + 512 + + + + + + + + + + + + + ${solr.lock.type:native} + + + + + + + + + + + + + + + + + + + + + ${solr.ulog.dir:} + + + + + ${solr.autoCommit.maxTime:15000} + false + + + + + + + ${solr.autoSoftCommit.maxTime:-1} + + + + + + + + + + + + + + ${solr.max.booleanClauses:1024} + + + ${solr.query.minPrefixLength:-1} + + + + + + + + + + + + + + + + + + + + + + + + + true + + + + + + 20 + + + 200 + + + + + + + + + + + + + + + + false + + + + + + + + + + + + + + + + + + + + + explicit + 10 + + + + + + + explicit + json + true + + + + + + + _text_ + + + + + + + text_general + + + + + + default + _text_ + solr.DirectSolrSpellChecker + + internal + + 0.5 + + 2 + + 1 + + 5 + + 4 + + 0.01 + + + + + + + + + + + + default + on + true + 10 + 5 + 5 + true + true + 10 + 5 + + + spellcheck + + + + + + + + + + + + 100 + + + + + + + + 70 + + 0.5 + + [-\w ,/\n\"']{20,200} + + + + + + + ]]> + ]]> + + + + + + + + + + + + + + + + + + + + + + + + ,, + ,, + ,, + ,, + ,]]> + ]]> + + + + + + 10 + .,!? + + + + + + + WORD + + + en + US + + + + + + + + + + + + [^\w-\.] + _ + + + 1000 + true + + + + + + + yyyy-MM-dd['T'[HH:mm[:ss[.SSS]][z + yyyy-MM-dd['T'[HH:mm[:ss[,SSS]][z + yyyy-MM-dd HH:mm[:ss[.SSS]][z + yyyy-MM-dd HH:mm[:ss[,SSS]][z + [EEE, ]dd MMM yyyy HH:mm[:ss] z + EEEE, dd-MMM-yy HH:mm:ss z + EEE MMM ppd HH:mm:ss [z ]yyyy + + + + + java.lang.String + text_general + + *_str + 256 + + + true + + + java.lang.Boolean + booleans + + + java.util.Date + pdates + + + java.lang.Long + java.lang.Integer + plongs + + + java.lang.Number + pdoubles + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/data-loading/configsets/name_lookup/conf/stopwords.txt b/data-loading/configsets/name_lookup/conf/stopwords.txt new file mode 100644 index 00000000..ae1e83ee --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/stopwords.txt @@ -0,0 +1,14 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/data-loading/configsets/name_lookup/conf/synonyms.txt b/data-loading/configsets/name_lookup/conf/synonyms.txt new file mode 100644 index 00000000..eab4ee87 --- /dev/null +++ b/data-loading/configsets/name_lookup/conf/synonyms.txt @@ -0,0 +1,29 @@ +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +#----------------------------------------------------------------------- +#some test synonym mappings unlikely to appear in real input text +aaafoo => aaabar +bbbfoo => bbbfoo bbbbar +cccfoo => cccbar cccbaz +fooaaa,baraaa,bazaaa + +# Some synonym groups specific to this example +GB,gib,gigabyte,gigabytes +MB,mib,megabyte,megabytes +Television, Televisions, TV, TVs +#notice we use "gib" instead of "GiB" so any WordDelimiterGraphFilter coming +#after us won't split it into two words. + +# Synonym mappings can be used for spelling correction too +pixima => pixma + diff --git a/data-loading/setup-and-load-solr.sh b/data-loading/setup-and-load-solr.sh index 2b0a9bec..290df0a6 100755 --- a/data-loading/setup-and-load-solr.sh +++ b/data-loading/setup-and-load-solr.sh @@ -1,35 +1,85 @@ #!/usr/bin/env bash +# +# Load Babel synonym files (JSON, one document per line) into a Solr core that +# already exists. The core is created from the checked-in configset with +# `solr create -c name_lookup -d configsets/name_lookup` (see the Makefile and +# the CI workflow) -- this script does NOT create it, so the schema lives in +# exactly one place: data-loading/configsets/name_lookup/conf. +# +# Speed: files are streamed to Solr in parallel with commit=false, and a single +# commit is issued at the end. There is no per-file commit and no sleeping. +# +# Safety: because the load is parallel and the data is huge, we count the input +# documents (lines) before loading and compare against Solr's document count +# afterwards. Any failed upload (curl --fail) or a count mismatch aborts with a +# non-zero exit code, so a partial/dropped load cannot pass silently. -# We don't use set -e because the loop test relies on failures being ignored. set -uo pipefail -# Configuration options -SOLR_SERVER="http://localhost:8983" +# Configuration (overridable from the environment). +SOLR_SERVER="${SOLR_SERVER:-http://localhost:8983}" +CORE="${SOLR_CORE:-name_lookup}" +PARALLELISM="${LOAD_PARALLELISM:-$(getconf _NPROCESSORS_ONLN 2>/dev/null || echo 4)}" -# Step 1. Make sure the Solr service is up and running. -HEALTH_ENDPOINT="${SOLR_SERVER}/solr/admin/cores?action=STATUS" -response=$(wget --spider --server-response ${HEALTH_ENDPOINT} 2>&1 | grep "HTTP/" | awk '{ print $2 }') >&2 -until [ "$response" = "200" ]; do - response=$(wget --spider --server-response ${HEALTH_ENDPOINT} 2>&1 | grep "HTTP/" | awk '{ print $2 }') >&2 - echo " -- SOLR is unavailable - sleeping" +GLOB="${1:?usage: setup-and-load-solr.sh \"data/synonyms/*.txt*\" (quote the glob!)}" + +# Step 1. Wait for the core to be available. +until curl -sf "${SOLR_SERVER}/solr/${CORE}/admin/ping" >/dev/null 2>&1; do + echo " -- Solr core '${CORE}' is unavailable - sleeping" sleep 3 done -echo "SOLR is up and running at ${SOLR_SERVER}." - -# Step 2. Create fields for search. -SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" -source "$SCRIPT_DIR/setup_solr.sh" -echo Solr database has been set up. - -# Step 3. Load specified files. -for f in $1; do - echo "Loading $f..." - # curl -d @$f needs to load the entire file into memory before uploading it, whereas - # curl -X POST -T $f will stream it. See https://github.com/TranslatorSRI/NameResolution/issues/194 - curl -H 'Content-Type: application/json' -X POST -T $f \ - "$SOLR_SERVER/solr/name_lookup/update/json/docs?processor=uuid&uuid.fieldName=id&commit=true" - sleep 60 +echo "Solr core '${CORE}' is up at ${SOLR_SERVER}." + +# Step 2. Expand the glob and count the input documents (one JSON doc per line). +# `grep -c ''` counts every line including a final one with no trailing newline, +# and we count per file so a missing newline never merges two files' records. +shopt -s nullglob +files=( $GLOB ) +if [ ${#files[@]} -eq 0 ]; then + echo "No files matched '${GLOB}'." >&2 + exit 1 +fi +echo "Counting documents in ${#files[@]} file(s)..." +expected=0 +for f in "${files[@]}"; do + n=$(grep -c '' "$f") + echo " ${n} docs in ${f}" + expected=$((expected + n)) done -echo "Check solr" -curl -s --negotiate -u: "$SOLR_SERVER/solr/name_lookup/query?q=*:*&rows=0" +echo "Expecting ${expected} documents in total." + +# Step 3. Load the files in parallel, streaming each one, without committing. +# curl -T streams the file rather than buffering it in memory (issue #194), and +# --fail turns any HTTP >=400 (e.g. malformed JSON) into a non-zero exit. +load_one() { + local f="$1" + echo "Loading ${f}..." + curl -sf --show-error -H 'Content-Type: application/json' -X POST -T "$f" \ + "${SOLR_SERVER}/solr/${CORE}/update/json/docs?processor=uuid&uuid.fieldName=id&commit=false" \ + || { echo "FAILED to load ${f}" >&2; return 1; } +} +export -f load_one +export SOLR_SERVER CORE + +printf '%s\0' "${files[@]}" \ + | xargs -0 -P "$PARALLELISM" -I{} bash -c 'load_one "$@"' _ {} +load_rc=$? +if [ "$load_rc" -ne 0 ]; then + echo "One or more files failed to load (xargs exit ${load_rc}). Aborting." >&2 + exit 1 +fi + +# Step 4. Commit once. +echo "Committing..." +curl -sf --show-error "${SOLR_SERVER}/solr/${CORE}/update?commit=true" >/dev/null \ + || { echo "Commit failed." >&2; exit 1; } +# Step 5. Verify the document count matches. +actual=$(curl -sf "${SOLR_SERVER}/solr/${CORE}/query?q=*:*&rows=0" \ + | grep -oE '"numFound":[0-9]+' | head -1 | grep -oE '[0-9]+') +echo "Solr reports ${actual} documents; expected ${expected}." +if [ "${actual:-0}" != "${expected}" ]; then + echo "DOCUMENT COUNT MISMATCH: loaded ${actual}, expected ${expected}. Aborting." >&2 + exit 1 +fi +echo "Load complete: ${actual} documents." diff --git a/data-loading/setup_solr.sh b/data-loading/setup_solr.sh deleted file mode 100644 index 0ea2842f..00000000 --- a/data-loading/setup_solr.sh +++ /dev/null @@ -1,147 +0,0 @@ -#!/usr/bin/env bash -# -# Set up the fields and types needed by NameRes. -# -# This file should be sourced, not called directly. - -# require sourcing -[[ "${BASH_SOURCE[0]}" != "$0" ]] || { - echo "Must be sourced: source $0" >&2 - exit 1 -} - -# require SOLR_SERVER -: "${SOLR_SERVER:?SOLR_SERVER must be set}" - -echo "Setting up Solr database with SOLR_SERVER='$SOLR_SERVER'" - -# add collection -curl -X POST "$SOLR_SERVER/solr/admin/collections?action=CREATE&name=name_lookup&numShards=1&replicationFactor=1" - -# do not autocreate fields -curl "$SOLR_SERVER/solr/name_lookup/config" -d '{"set-user-property": {"update.autoCreateFields": "false"}}' - -# add lowercase text type -curl -X POST -H 'Content-type:application/json' --data-binary '{ - "add-field-type" : { - "name": "LowerTextField", - "class": "solr.TextField", - "positionIncrementGap": "100", - "analyzer": { - "tokenizer": { - "class": "solr.StandardTokenizerFactory" - }, - "filters": [{ - "class": "solr.LowerCaseFilterFactory" - }] - } - } -}' "$SOLR_SERVER/solr/name_lookup/schema" - -# add exactish text type (as described at https://stackoverflow.com/a/29105025/27310) -curl -X POST -H 'Content-type:application/json' --data-binary '{ - "add-field-type" : { - "name": "exactish", - "class": "solr.TextField", - "positionIncrementGap": "100", - "analyzer": { - "tokenizer": { - "class": "solr.KeywordTokenizerFactory" - }, - "filters": [{ - "class": "solr.LowerCaseFilterFactory" - }] - } - } -}' "$SOLR_SERVER/solr/name_lookup/schema" - - - -# add fields -curl -X POST -H 'Content-type:application/json' --data-binary '{ - "add-field": [ - { - "name":"names", - "type":"LowerTextField", - "indexed":true, - "stored":true, - "multiValued":true - }, - { - "name":"names_exactish", - "type":"exactish", - "indexed":true, - "stored":false, - "multiValued":true - }, - { - "name":"curie", - "type":"string", - "stored":true - }, - { - "name":"preferred_name", - "type":"LowerTextField", - "stored":true - }, - { - "name":"preferred_name_exactish", - "type":"exactish", - "indexed":true, - "stored":false, - "multiValued":false - }, - { - "name":"types", - "type":"string", - "stored":true - "multiValued":true - }, - { - "name":"shortest_name_length", - "type":"pint", - "stored":true - }, - { - "name":"curie_suffix", - "type":"plong", - "docValues":true, - "stored":true, - "required":false, - "sortMissingLast":true - }, - { - "name":"taxa", - "type":"string", - "stored":true, - "multiValued":true - }, - { - "name":"taxon_specific", - "type":"boolean", - "stored":true, - "multiValued":false, - "sortMissingLast":true - }, - { - "name":"clique_identifier_count", - "type":"pint", - "stored":true - } - ] }' "$SOLR_SERVER/solr/name_lookup/schema" - -# Add a copy field to copy names into names_exactish. -curl -X POST -H 'Content-type:application/json' --data-binary '{ - "add-copy-field": { - "source": "names", - "dest": "names_exactish" - } -}' "$SOLR_SERVER/solr/name_lookup/schema" - -# Add a copy field to copy preferred_name into preferred_name_exactish. -curl -X POST -H 'Content-type:application/json' --data-binary '{ - "add-copy-field": { - "source": "preferred_name", - "dest": "preferred_name_exactish" - } -}' "$SOLR_SERVER/solr/name_lookup/schema" diff --git a/tests/create_test_synonyms.py b/tests/create_test_synonyms.py index d8a6c3f2..8783117b 100644 --- a/tests/create_test_synonyms.py +++ b/tests/create_test_synonyms.py @@ -13,7 +13,12 @@ smallsyns = list(filter( lambda x: x['curie'] in allcuries , syns )) +# Write as JSON-lines (one JSON document per line), matching the format of the +# Babel synonym files that are loaded in production. The loader counts lines to +# verify every document made it into Solr, so the format must match. with open('tests/data/test-synonyms.json','w') as outf: - json.dump(smallsyns,outf,indent=1) + for syn in smallsyns: + outf.write(json.dumps(syn)) + outf.write('\n') diff --git a/tests/data/test-synonyms.json b/tests/data/test-synonyms.json index bc7046a2..3566fe9b 100644 --- a/tests/data/test-synonyms.json +++ b/tests/data/test-synonyms.json @@ -1,2083 +1,89 @@ -[ - { - "curie": "CHEBI:48407", - "names": [ - "antiparkinson agent" - ], - "preferred_name": "antiparkinson agent", - "types": [ - "NamedThing" - ], - "shortest_name_length": 19, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "CHEBI:74925", - "names": [ - "BACE1 inhibitor", - "BACE1 inhibitors", - "memapsin 2 inhibitor", - "memapsin 2 inhibitors", - "EC 3.4.23.46 inhibitor", - "EC 3.4.23.46 inhibitors", - "beta-secretase inhibitor", - "beta-secretase inhibitors", - "gamma-secretase inhibitor", - "Gamma-secretase inhibitor", - "gamma-secretase inhibitors", - "Gamma-secretase inhibitors", - "memapsin 2 (EC 3.4.23.46) inhibitor", - "EC 3.4.23.46 (memapsin 2) inhibitors", - "memapsin 2 (EC 3.4.23.46) inhibitors", - "beta-site APP-cleaving enzyme 1 inhibitor", - "beta-site APP-cleaving enzyme 1 inhibitors", - "membrane-associated aspartic protease 2 inhibitor", - "membrane-associated aspartic protease 2 inhibitors", - "beta-site Alzheimer's amyloid precursor protein cleaving enzyme 1 inhibitor", - "beta-site Alzheimer's amyloid precursor protein cleaving enzyme 1 inhibitors", - "beta-site Alzheimer's amyloid precursor protein cleaving enzyme 1 (BACE1) inhibitor", - "beta-site Alzheimer's amyloid precursor protein cleaving enzyme 1 (BACE1) inhibitors" - ], - "preferred_name": "BACE1 inhibitor", - "types": [ - "NamedThing" - ], - "shortest_name_length": 15, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "HP:0001300", - "names": [ - "Parkinsonian disease" - ], - "preferred_name": "parkinsonian disorder", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 20, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "HP:0002322", - "names": [ - "Rest tremor", - "Resting tremor", - "Tremor at rest", - "Parkinsonian tremor" - ], - "preferred_name": "Resting tremor", - "types": [ - "PhenotypicFeature", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 11, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "HP:0002511", - "names": [ - "Alzheimer disease", - "Late-onset form of familial Alzheimer disease" - ], - "preferred_name": "Alzheimer disease", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 17, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "HP:0002548", - "names": [ - "Favorable response to levodopa", - "Parkinsonism with favourable response to dopaminergic medication" - ], - "preferred_name": "Parkinsonism with favorable response to dopaminergic medication", - "types": [ - "PhenotypicFeature", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 30, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0000828", - "names": [ - "juvenile-onset Parkinson's disease" - ], - "preferred_name": "juvenile-onset Parkinson disease", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 34, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0001945", - "names": [ - "postencephalitic parkinsonism", - "postencephalitic Parkinsonism" - ], - "preferred_name": "postencephalitic Parkinson disease", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 29, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0004975", - "names": [ - "AD", - "Alzheimer disease", - "Alzheimer dementia", - "Alzheimers disease", - "Alzheimers dementia", - "Alzheimer's disease", - "Alzheimer's dementia", - "Alzheimer disease, familial" - ], - "preferred_name": "Alzheimer disease", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 2, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0005180", - "names": [ - "paralysis agitans", - "Parkinson's disease" - ], - "preferred_name": "Parkinson disease", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 17, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0006966", - "names": [ - "secondary Parkinsonism", - "symptomatic parkinsonism", - "secondary parkinsonism, unspecified", - "secondary parkinsonism (disorder) [ambiguous]", - "disorder presenting primarily with parkinsonism" - ], - "preferred_name": "secondary Parkinson disease", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 22, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0007088", - "names": [ - "AD1", - "Alzheimer disease 1", - "Alzheimer disease; AD", - "presenile and senile dementia", - "Alzheimer disease, familial, 1", - "Alzheimer disease, protection against", - "early-onset familial form of Alzheimer disease", - "Alzheimer disease, early-onset, with cerebral amyloid angiopathy" - ], - "preferred_name": "Alzheimer disease type 1", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 3, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0007089", - "names": [ - "AD2", - "LOFAD", - "Alzheimer disease 2", - "Alzheimer disease-2", - "Alzheimer's disease 2", - "Alzheimer disease type 2", - "Alzheimer disease 2; AD2", - "Alzheimer's disease type 2", - "late onset Alzheimer disease", - "Alzheimer disease 2, late onset", - "Alzheimer disease 2, late-onset", - "late onset familial Alzheimer disease", - "Alzheimer disease associated with APOE4", - "Alzheimer disease associated with Apoe4", - "Alzheimer disease associated with APOE E4" - ], - "preferred_name": "Alzheimer disease 2", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 3, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0007104", - "names": [ - "PDALS", - "ALS-pDC", - "Guam disease", - "Lytico-Bodig disease", - "Lytigo-Bodig disease", - "Parkinsonism-dementia-ALS complex", - "amyotrophic lateral sclerosis-PARKINSONISM/dementia complex 1", - "amyotrophic lateral sclerosis-Parkinsonism/dementia Complex type 1", - "amyotrophic lateral sclerosis-Parkinsonism/dementia Complex of Guam", - "amyotrophic lateral sclerosis-parkinsonism-dementia of Guam syndrome", - "amyotrophic lateral sclerosis, Parkinsonism/dementia complex of Guam" - ], - "preferred_name": "amyotrophic lateral sclerosis-parkinsonism-dementia complex", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 5, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0007312", - "names": [ - "Charcot-Marie-Tooth disease with ptosis and parkinsonism" - ], - "preferred_name": "Charcot-Marie-Tooth disease with ptosis and parkinsonism", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 56, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0007433", - "names": [ - "dementia/parkinsonism with non-Alzheimer amyloid plaques" - ], - "preferred_name": "dementia/parkinsonism with non-Alzheimer amyloid plaques", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 56, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0007488", - "names": [ - "DLB", - "Lewy body disease", - "Lewy body dementia", - "dementia, Lewy body", - "dementia, Lewy body; DLB", - "dementia with Lewy bodies", - "diffuse Lewy body disease", - "Senile dementia of the Lewy body type", - "Lewy body variant of Alzheimer disease", - "diffuse Lewy body disease with gaze palsy" - ], - "preferred_name": "Lewy body dementia", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 3, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0007495", - "names": [ - "DRD", - "DYT5a", - "DYT-GCH1", - "dystonia 5", - "dystonia type 5", - "GTPCH1-deficient DRD", - "dystonia, DOPA-responsive", - "dystonia, DOPA-responsive; DRD", - "autosomal dominant Segawa syndrome", - "HPD with marked diurnal fluctuation", - "Segawa syndrome, autosomal dominant", - "GTPCH1-deficient dopa-responsive dystonia", - "autosomal dominant dopa-responsive dystonia", - "Dopa-responsive dystonia, autosomal dominant", - "Dopa-responsive dystonia; Segawa syndrome AD", - "dystonia, Dopa-responsive, autosomal dominant", - "dystonia, progressive, with diurnal variation", - "dystonia-Parkinsonism with diurnal fluctuation", - "GTP cyclohydrolase 1-deficient dopa-responsive dystonia", - "hereditary progressive dystonia with marked diurnal fluctuation" - ], - "preferred_name": "dystonia 5", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 3, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0007496", - "names": [ - "RDP", - "DYT12", - "DYT-ATP1A3", - "dystonia 12", - "dystonia type 12", - "dystonia 12; DYT12", - "ATP1A3 dystonic disorder", - "rapid-onset dystonia-parkinsonism", - "dystonia-Parkinsonism, rapid-onset", - "dystonic disorder caused by mutation in ATP1A3" - ], - "preferred_name": "dystonia 12", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 3, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0008193", - "names": [ - "paralysis agitans, juvenile, of Hunt", - "Parkinson disease, juvenile, of Hunt" - ], - "preferred_name": "paralysis agitans, juvenile, of Hunt", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 36, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0008199", - "names": [ - "PD", - "park", - "Parkinson disease, late-onset", - "Parkinson disease, late-onset; PD" - ], - "preferred_name": "late-onset Parkinson disease", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 2, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0008200", - "names": [ - "PARK1", - "atypical Parkinson disease", - "Parkinson disease 1, autosomal dominant", - "autosomal dominant Parkinson's disease 1", - "autosomal dominant Parkinson disease type 1", - "Parkinson disease 1, autosomal dominant; PARK1", - "Parkinson disease 1, autosomal dominant Lewy body" - ], - "preferred_name": "autosomal dominant Parkinson disease 1", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 5, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0008201", - "names": [ - "Perry syndrome", - "parkinsonism with alveolar hypoventilation and mental depression", - "Parkinsonism with alveolar hypoventilation and mental depression" - ], - "preferred_name": "Perry syndrome", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 14, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0008685", - "names": [ - "WPW", - "Wpw syndrome", - "preexcitation syndrome", - "anomalous A-V excitation", - "Wolff-Parkinson-White pattern", - "Wolff-Parkinson-White syndrome", - "WOLFF-Parkinson-WHITE syndrome", - "accessory atrioventricular pathways", - "anomalous atrioventricular excitation", - "Wolff-Parkinson-White pattern (finding)" - ], - "preferred_name": "Wolff-Parkinson-white syndrome", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 3, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0009497", - "names": [ - "Vitsala", - "Kifafa seizure disorder", - "Complex familial seizure disorder", - "parkinsonian features and neurologic abnormalities, mental retardation and transient psychotic episodes" - ], - "preferred_name": "Kifafa seizure disorder", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 7, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0009830", - "names": [ - "PARK15", - "pallidopyramidal syndrome", - "Pallidopyramidal syndrome", - "pallido-pyramidal disease", - "pallido-pyramidal syndrome", - "parkinsonian-pyramidal syndrome", - "autosomal recessive early-onset Parkinson disease 15", - "Parkinson disease 15, autosomal recessive early-onset", - "autosomal recessive early-onset Parkinson's disease 15", - "autosomal recessive early-onset Parkinson disease type 15", - "Parkinson disease 15, autosomal recessive early-onset; PARK15" - ], - "preferred_name": "parkinsonian-pyramidal syndrome", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 6, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0009839", - "names": [ - "PSP-p", - "atypical PSP", - "PSP-parkinsonism", - "Parkinson-dementia syndrome", - "progressive supranuclear palsy atypical", - "supranuclear palsy, progressive, 1, atypical", - "Steele-Richardson-Olszewski syndrome, atypical" - ], - "preferred_name": "progressive supranuclear palsy-parkinsonism syndrome", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 5, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0010360", - "names": [ - "PARK12", - "Parkinson disease 12", - "Parkinson disease, X-linked", - "Parkinson disease 12; PARK12" - ], - "preferred_name": "parkinson disease 12", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 6, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0010422", - "names": [ - "AD16", - "Alzheimer disease 16", - "Alzheimer's disease 16", - "Alzheimer disease 16; AD16", - "Alzheimer's disease type 16" - ], - "preferred_name": "Alzheimer disease 16", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 4, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0010482", - "names": [ - "XPDS", - "PARKINSONISM with spasticity, X-linked", - "PARKINSONISM with spasticity, X-linked; XPDS" - ], - "preferred_name": "X-linked parkinsonism-spasticity syndrome", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 4, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0010709", - "names": [ - "Wsn", - "BGMR", - "WSMN", - "Waisman syndrome", - "WAISMAN syndrome", - "Laxova-Opitz syndrome", - "WAISMAN syndrome; WSMN", - "Laxova Brown hogan syndrome", - "basal ganglia disorder with mental retardation", - "basal ganglion disorder with mental retardation", - "Parkinsonism, early onset with mental retardation", - "Parkinsonism, early-onset, with mental retardation", - "X-linked recessive basal ganglia disorder with mental retardation" - ], - "preferred_name": "early-onset parkinsonism-intellectual disability syndrome", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 3, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0010747", - "names": [ - "XDP", - "DYT3", - "Lubag", - "DYT-TAF1", - "Lubag syndrome", - "dystonia 3, torsion, X-linked", - "X-linked dystonia Parkinsonism", - "dystonia-Parkinsonism, X-linked", - "dystonia 3, torsion, X-linked; DYT3", - "X-linked dystonia-parkinsonism/Lubag", - "X-linked dystonia-Parkinsonism syndrome", - "torsion dystonia-Parkinsonism, Filipino type", - "X-linked torsion dystonia-Parkinsonism syndrome" - ], - "preferred_name": "X-linked dystonia-parkinsonism", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 3, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0010783", - "names": [ - "Alzheimer disease, susceptibility to, mitochondrial" - ], - "preferred_name": "Alzheimer disease, susceptibility to, mitochondrial", - "types": [ - "NamedThing" - ], - "shortest_name_length": 51, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0010796", - "names": [ - "Parkinson disease, mitochondrial" - ], - "preferred_name": "Parkinson disease, mitochondrial", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 32, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0010820", - "names": [ - "JP", - "PDJ", - "PARK2", - "Parkinson disease 2", - "juvenile parkinsonism", - "PRKN young-onset Parkinson disease", - "autosomal recessive juvenile Parkinson disease", - "Parkinson disease, juvenile, autosomal recessive", - "Parkinson disease 2, autosomal recessive juvenile", - "autosomal recessive juvenile Parkinson's disease 2", - "Parkinson disease autosomal recessive, early onset", - "Parkinsonism, early onset, with diurnal fluctuation", - "Parkinsonism, early-onset, with diurnal fluctuation", - "autosomal recessive juvenile Parkinson disease type 2", - "young-onset Parkinson disease caused by mutation in PRKN", - "Parkinson disease 2, autosomal recessive juvenile; PARK2" - ], - "preferred_name": "autosomal recessive juvenile Parkinson disease 2", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 2, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0010857", - "names": [ - "FTD", - "Ftdp17", - "Pick Complex", - "semantic variant PPA", - "frontotemporal dementia", - "Ftld with Tau inclusions", - "Wilhelmsen-Lynch disease", - "frontotemporal dementia; FTD", - "frontotemporal lobe dementia", - "Pallidopontonigral Degeneration", - "semantic primary progressive aphasia", - "frontotemporal dementia with Parkinsonism", - "dementia, frontotemporal, with Parkinsonism", - "multiple system tauopathy with presenile dementia", - "frontotemporal lobar Degeneration with Tau inclusions", - "disinhibition-dementia-Parkinsonism-amyotrophy Complex" - ], - "preferred_name": "semantic dementia", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 3, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "NCBIGene:378899", - "preferred_name": "FTL", - "names": [ - "FTL", - "FTD", - "ferritoid", - "ferritin light chain", - "ferritin, light polypeptide", - "Q8AYG9_CHICK Ferritin (trembl)" - ], - "taxa": [ - "NCBITaxon:9031" - ], - "taxon_specific": true, - "types": [ - "Gene", - "GeneOrGeneProduct", - "GenomicEntity", - "ChemicalEntityOrGeneOrGeneProduct", - "PhysicalEssence", - "OntologyClass", - "BiologicalEntity", - "ThingWithTaxon", - "NamedThing", - "Entity", - "PhysicalEssenceOrOccurrent", - "MacromolecularMachineMixin", - "Protein", - "GeneProductMixin", - "Polypeptide", - "ChemicalEntityOrProteinOrPolypeptide" - ], - "shortest_name_length": 3, - "clique_identifier_count": 3 - }, - { - "curie": "MONDO:0011194", - "names": [ - "AD5", - "Ad5", - "Alzheimer disease 5", - "Alzheimer's disease 5", - "Alzheimer disease type 5", - "Alzheimer's disease type 5", - "Alzheimer disease, familial 5", - "Alzheimer disease, familial, 5" - ], - "preferred_name": "Alzheimer disease 5", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 3, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0011220", - "names": [ - "PARK3", - "Parkinson disease type 3", - "autosomal dominant Parkinson disease", - "Parkinson disease 3, autosomal dominant", - "Parkinson disease 3, autosomal dominant; PARK3", - "Parkinson disease 3, autosomal dominant Lewy body" - ], - "preferred_name": "parkinson disease 3, autosomal dominant", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 5, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0011401", - "names": [ - "Alzheimer disease without neurofibrillary tangles", - "Alzheimer's disease without neurofibrillary tangles" - ], - "preferred_name": "Alzheimer disease without neurofibrillary tangles", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 49, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0011513", - "names": [ - "Alzheimer disease, familial early-onset, with Coexisting amyloid and prion pathology" - ], - "preferred_name": "Alzheimer disease, familial early-onset, with coexisting amyloid and prion pathology", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 84, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0011551", - "names": [ - "DYT5b", - "autosomal recessive Segawa syndrome", - "Segawa syndrome, autosomal recessive", - "autosomal recessive dopa-responsive dystonia", - "Parkinsonism, infantile, autosomal recessive", - "dopa-responsive dystonia, autosomal recessive", - "Dopa-responsive dystonia, autosomal recessive", - "Dopa-responsive dystonia, autosomal recessive", - "DOPA responsive dystonia, autosomal recessive", - "dystonia, Dopa-responsive, autosomal recessive", - "dystonia, DOPA responsive, autosomal recessive", - "tyrosine hydroxylase-deficient dopa-responsive dystonia" - ], - "preferred_name": "TH-deficient dopa-responsive dystonia", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 5, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0011561", - "names": [ - "AD6", - "AD6", - "Alzheimer disease 6", - "Alzheimer's disease 6", - "Alzheimer disease type 6", - "Alzheimer's disease type 6", - "Alzheimer disease 6, late onset", - "Alzheimer disease 6, late-onset", - "plasma Beta-amyloid-42 level quantitative trait locus" - ], - "preferred_name": "Alzheimer disease 6", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 3, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0011562", - "names": [ - "PARK4", - "Parkinson disease 4, autosomal dominant", - "autosomal dominant Parkinson's disease 4", - "autosomal dominant Parkinson disease type 4", - "Parkinson disease 4, autosomal dominant; PARK4", - "autosomal dominant Lewy body Parkinson disease 4", - "Parkinson disease 4, autosomal dominant Lewy body" - ], - "preferred_name": "autosomal dominant Parkinson disease 4", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 5, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0011613", - "names": [ - "PARK6", - "PINK1 Parkinson disease", - "early-onset Parkinson disease 6", - "Parkinson disease 6, early-onset", - "Parkinson disease caused by mutation in PINK1", - "Parkinson disease 6, late-onset, susceptibility to", - "Parkinson disease 6, autosomal recessive early-onset", - "autosomal recessive early-onset Parkinson's disease 6", - "autosomal recessive early-onset Parkinson disease type 6", - "Parkinson disease 6, autosomal recessive early-onset; PARK6", - "Parkinson disease, autosomal recessive early-onset, digenic, Pink1/Dj1" - ], - "preferred_name": "autosomal recessive early-onset Parkinson disease 6", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 5, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0011647", - "names": [ - "AD7", - "Ad7", - "Alzheimer disease 7", - "Alzheimer's disease 7", - "Alzheimer disease type 7", - "Alzheimer's disease type 7", - "Alzheimer disease, familial 7", - "Alzheimer disease, familial, 7" - ], - "preferred_name": "Alzheimer disease 7", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 3, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0011658", - "names": [ - "PARK7", - "PARK7 Parkinson disease", - "Parkinson disease caused by mutation in PARK7", - "Parkinson disease 7, autosomal recessive early-onset", - "autosomal recessive early-onset Parkinson's disease 7", - "autosomal recessive early-onset Parkinson disease type 7", - "Parkinson disease 7, autosomal recessive early-onset; PARK7" - ], - "preferred_name": "autosomal recessive early-onset Parkinson disease 7", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 5, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0011706", - "names": [ - "KRS", - "PARK9", - "KRPPD", - "park 9", - "Kufor-Rakeb syndrome", - "Parkinson disease type 9", - "Kufor-Rakeb syndrome; KRS", - "ceroid lipofuscinosis, neuronal, 12", - "autosomal recessive Parkinson disease 9", - "Parkinson disease 9, autosomal recessive", - "autosomal recessive juvenile onset Parkinson disease 9", - "Parkinson disease 9, autosomal recessive, juvenile-onset", - "Pallidopyramidal Degeneration with supranuclear upgaze paresis and dementia", - "Pallidopyramidal degeneration with supranuclear upgaze paresis, and dementia" - ], - "preferred_name": "Kufor-Rakeb syndrome", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 3, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0011737", - "names": [ - "PARK10", - "Parkinson disease 10", - "Parkinson disease 10; PARK10", - "Parkinson disease, Age at onset of" - ], - "preferred_name": "parkinson disease 10", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 6, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0011743", - "names": [ - "AD4", - "Ad4", - "Alzheimer disease 4", - "Alzheimer's disease 4", - "Alzheimer disease type 4", - "Alzheimer's disease type 4", - "Alzheimer disease, familial4", - "Alzheimer disease, familial, 4", - "Alzheimer disease familial type 4", - "familial Alzheimer disease, type 4", - "familial Alzheimer's disease, type 4" - ], - "preferred_name": "Alzheimer disease 4", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 3, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0011764", - "names": [ - "PARK8", - "LRRK2 Parkinson disease", - "Parkinson disease 8, autosomal dominant", - "autosomal dominant Parkinson's disease 8", - "autosomal dominant Parkinson disease type 8", - "Parkinson disease caused by mutation in LRRK2", - "Parkinson disease 8, autosomal dominant; PARK8" - ], - "preferred_name": "autosomal dominant Parkinson disease 8", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 5, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0011777", - "names": [ - "AD8", - "Ad8", - "Alzheimer disease 8", - "Alzheimer's disease 8", - "Alzheimer disease type 8", - "Alzheimer's disease type 8", - "Alzheimer disease, familial 8", - "Alzheimer disease, familial, 8" - ], - "preferred_name": "Alzheimer disease 8", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 3, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0011896", - "names": [ - "PARK11", - "GIGYF2 hereditary late onset Parkinson disease", - "susceptibility to autosomal dominant Parkinson disease 11", - "Parkinson disease 11, autosomal dominant, susceptibility to", - "Parkinson disease 11, autosomal dominant, susceptibility to; PARK11", - "hereditary late onset Parkinson disease caused by mutation in GIGYF2" - ], - "preferred_name": "Parkinson disease 11, autosomal dominant, susceptibility to", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 6, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0011913", - "names": [ - "AD3", - "Alzheimer disease 3", - "Alzheimer's disease 3", - "Alzheimer disease 3; AD", - "Alzheimer disease type 3", - "Alzheimer's disease type 3", - "Alzheimer disease familial 3", - "Alzheimer disease, familial, 3", - "Alzheimer disease 3, early onset", - "Alzheimer disease 3, early-onset", - "familial Alzheimer disease, type 3", - "familial Alzheimer's disease, type 3", - "Alzheimer disease early onset type 3", - "PSEN1 early-onset autosomal dominant Alzheimer disease", - "Alzheimer disease, familial, 3, with spastic paraparesis and apraxia", - "early-onset autosomal dominant Alzheimer disease caused by mutation in PSEN1", - "Alzheimer disease, familial, 3, with spastic paraparesis and unusual plaques" - ], - "preferred_name": "Alzheimer disease 3", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 3, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0012153", - "names": [ - "AD9", - "Alzheimer disease 9", - "Alzheimer disease 9; AD9", - "Alzheimer disease 9, late-onset", - "Alzheimer disease 9, susceptibility to", - "Alzheimer disease 9, susceptibility to; AD9" - ], - "preferred_name": "Alzheimer disease 9", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 3, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0012321", - "names": [ - "AD10", - "Ad10", - "Alzheimer disease 10", - "Alzheimer's disease 10", - "Alzheimer disease type 10", - "Alzheimer's disease type 10", - "Alzheimer disease familial 10", - "Alzheimer disease, familial, 10" - ], - "preferred_name": "Alzheimer disease 10", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 4, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0012344", - "names": [ - "AD11", - "Ad11", - "Alzheimer disease 11", - "Alzheimer's disease 11", - "Alzheimer disease type 11", - "Alzheimer's disease type 11", - "Alzheimer disease, familial, 11" - ], - "preferred_name": "Alzheimer disease 11", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 4, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0012466", - "names": [ - "PARK13", - "HTRA2 young-onset Parkinson disease", - "young-onset Parkinson disease caused by mutation in HTRA2", - "susceptibility to autosomal dominant Parkinson disease 13", - "Parkinson disease 13, autosomal dominant, susceptibility to", - "Parkinson disease 13, autosomal dominant, susceptibility to; PARK13" - ], - "preferred_name": "Parkinson disease 13, autosomal dominant, susceptibility to", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 6, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0012609", - "names": [ - "AD12", - "Ad12", - "Alzheimer disease 12", - "Alzheimer's disease 12", - "Alzheimer disease type 12", - "Alzheimer's disease type 12", - "Alzheimer disease familial 12", - "Alzheimer disease, familial, 12" - ], - "preferred_name": "Alzheimer disease 12", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 4, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0012630", - "names": [ - "AD13", - "Alzheimer disease 13", - "Alzheimer's disease 13", - "Alzheimer disease 13; AD13", - "Alzheimer's disease type 13" - ], - "preferred_name": "Alzheimer disease 13", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 4, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0012631", - "names": [ - "AD14", - "Alzheimer disease 14", - "Alzheimer's disease 14", - "Alzheimer disease 14; AD14", - "Alzheimer's disease type 14" - ], - "preferred_name": "Alzheimer disease 14", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 4, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0012632", - "names": [ - "AD15", - "Alzheimer disease 15", - "Alzheimer's disease 15", - "Alzheimer disease 15; AD15", - "Alzheimer's disease type 15" - ], - "preferred_name": "AD15", - "types": [ - "NamedThing" - ], - "shortest_name_length": 4, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0012789", - "names": [ - "DYT16", - "DYT-PRKRA", - "dystonia 16", - "dystonia type 16", - "dystonia 16; DYT16", - "PRKRA dystonic disorder", - "early-onset dystonia parkinsonism", - "Young-onset dystonia-(parkinsonism)", - "dystonic disorder caused by mutation in PRKRA" - ], - "preferred_name": "dystonia 16", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 5, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0013060", - "names": [ - "PARK14", - "dystonia-Parkinsonism Adult-onset", - "dystonia-Parkinsonism, adult-onset", - "adult-onset dystonia - parkinsonism", - "PLA2G6-related dystonia-parkinsonism", - "dystonia-parkinsonism, Paisan-Ruiz type", - "Parkinson disease 14, autosomal recessive", - "autosomal recessive Parkinson's disease 14", - "autosomal recessive Parkinson disease type 14", - "PLA2G6 hereditary late onset Parkinson disease", - "Parkinson disease 14, autosomal recessive; PARK14", - "hereditary late onset Parkinson disease caused by mutation in PLA2G6" - ], - "preferred_name": "autosomal recessive Parkinson disease 14", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 6, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0013150", - "names": [ - "IPD", - "PKDYS", - "infantile Parkinsonism-dystonia", - "Parkinsonism-dystonia infantile", - "PARKINSONISM-dystonia, infantile", - "PARKINSONISM-dystonia, infantile; PKDYS", - "dopamine transporter deficiency syndrome" - ], - "preferred_name": "parkinsonism-dystonia, infantile", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 3, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0013167", - "names": [ - "PARK16", - "Parkinson disease 16", - "Parkinson disease 16; PARK16" - ], - "preferred_name": "parkinson disease 16", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 6, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0013340", - "names": [ - "PARK5", - "UCHL1 young-onset Parkinson disease", - "susceptibility to autosomal dominant Parkinson disease 5", - "young-onset Parkinson disease caused by mutation in UCHL1", - "Parkinson disease 5, autosomal dominant, susceptibility to", - "Parkinson disease 5, autosomal dominant, susceptibility to; PARK5" - ], - "preferred_name": "Parkinson disease 5, autosomal dominant, susceptibility to", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 5, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0013625", - "names": [ - "PARK17", - "Parkinson disease 17", - "Parkinson's disease 17", - "VPS35 Parkinson disease", - "Parkinson disease type 17", - "Parkinson disease 17; PARK17", - "autosomal dominant Parkinson disease 17", - "Parkinson disease caused by mutation in VPS35" - ], - "preferred_name": "Parkinson disease 17", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 6, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0013653", - "names": [ - "PARK18", - "EIF4G1 hereditary late onset Parkinson disease", - "susceptibility to autosomal dominant Parkinson disease 18", - "Parkinson disease 18, autosomal dominant, susceptibility to", - "Parkinson disease 18, autosomal dominant, susceptibility to; PARK18", - "hereditary late onset Parkinson disease caused by mutation in EIF4G1" - ], - "preferred_name": "Parkinson disease 18, autosomal dominant, susceptibility to", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 6, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0014036", - "names": [ - "AD17", - "Alzheimer disease 17", - "Alzheimer's disease 17", - "Alzheimer disease 17; AD17", - "Alzheimer's disease type 17", - "Alzheimer disease 17, late onset", - "Alzheimer disease 17, late-onset" - ], - "preferred_name": "Alzheimer disease 17", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 4, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0014231", - "names": [ - "PARK19", - "PARK19A", - "Park19, formerly", - "DNAJC6 Parkinson disease", - "Parkinson disease 19B, early-onset", - "Parkinson disease 19, juvenile-onset", - "Parkinson disease 19A, juvenile-onset", - "juvenile onset Parkinson's disease 19A", - "juvenile onset Parkinson disease type 19A", - "Parkinson disease 19, juvenile-onset; PARK19", - "Parkinson disease caused by mutation in DNAJC6", - "Parkinson disease 19A, juvenile-onset; PARK19A" - ], - "preferred_name": "juvenile onset Parkinson disease 19A", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 6, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0014233", - "names": [ - "PARK20", - "SYNJ1 Parkinson disease", - "Parkinson disease 20, early-onset", - "early-onset Parkinson's disease 20", - "early-onset Parkinson disease type 20", - "Parkinson disease 20, early-onset; PARK20", - "Parkinson disease caused by mutation in SYNJ1" - ], - "preferred_name": "early-onset Parkinson disease 20", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 6, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0014265", - "names": [ - "AD18", - "Alzheimer disease 18", - "Alzheimer's disease 18", - "ADAM10 Alzheimer disease", - "Alzheimer disease type 18", - "Alzheimer disease 18; AD18", - "Alzheimer's disease type 18", - "Alzheimer disease 18, late-onset", - "Alzheimer disease caused by mutation in ADAM10" - ], - "preferred_name": "Alzheimer disease 18", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 4, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0014316", - "names": [ - "AD19", - "Alzheimer disease 19", - "Alzheimer's disease 19", - "PLD3 Alzheimer disease", - "Alzheimer disease type 19", - "Alzheimer disease 19; AD19", - "Alzheimer's disease type 19", - "Alzheimer disease 19 late onset", - "Alzheimer disease 19, late-onset", - "Alzheimer disease caused by mutation in PLD3" - ], - "preferred_name": "Alzheimer disease 19", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 4, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0014604", - "names": [ - "PARK21", - "Parkinson disease 21", - "Parkinson disease type 21", - "Parkinson disease 21; PARK21", - "DNAJC13 hereditary late onset Parkinson disease", - "hereditary late onset Parkinson disease caused by mutation in DNAJC13" - ], - "preferred_name": "Parkinson disease 21", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 6, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0014742", - "names": [ - "PARK22", - "CHCHD2 Parkinson disease", - "Parkinson disease 22, autosomal dominant", - "Parkinson disease caused by mutation in CHCHD2", - "Parkinson disease 22, autosomal dominant; PARK22" - ], - "preferred_name": "Parkinson disease 22, autosomal dominant", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 6, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0014796", - "names": [ - "PARK23", - "VPS13C young-onset Parkinson disease", - "autosomal recessive early-onset Parkinson disease 23", - "Parkinson disease 23, autosomal recessive early-onset", - "autosomal recessive early-onset Parkinson's disease 23", - "autosomal recessive early-onset Parksinson disease type 23", - "young-onset Parkinson disease caused by mutation in VPS13C", - "Parkinson disease 23, autosomal recessive early-onset; PARK23" - ], - "preferred_name": "autosomal recessive early-onset Parkinson disease 23", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 6, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0015140", - "names": [ - "EOFAD", - "early-onset, autosomal dominant Alzheimer disease", - "early-onset familial autosomal dominant Alzheimer disease" - ], - "preferred_name": "early-onset autosomal dominant Alzheimer disease", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 5, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0017279", - "names": [ - "YOPD", - "early-onset Parkinson disease", - "early-onset Parkinson's disease" - ], - "preferred_name": "young-onset Parkinson disease", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 4, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0017639", - "names": [ - "CO-induced parkinsonism" - ], - "preferred_name": "carbon monoxide-induced parkinsonism", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 23, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0018466", - "names": [ - "LOPD", - "late-onset Parkinson disease", - "late onset Parkinson disease", - "late onset Parkinson's disease", - "hereditary late-onset Parkinson disease", - "autosomal dominant late-onset Parkinson disease" - ], - "preferred_name": "late-onset Parkinson disease", - "types": [ - "NamedThing" - ], - "shortest_name_length": 4, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0018899", - "names": [ - "PCA", - "Benson syndrome", - "biparietal Alzheimer disease" - ], - "preferred_name": "posterior cortical atrophy", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 3, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0020352", - "names": [ - "MSA-p", - "MSA, parkinsonian type" - ], - "preferred_name": "multiple system atrophy, parkinsonian type", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 5, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0054835", - "names": [ - "PKDYS", - "PKDYS1", - "dopamine transporter deficiency syndrome", - "Parkinsonism-dystonia, infantile, 1; PKDYS1" - ], - "preferred_name": "parkinsonism-dystonia, infantile, 1", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 5, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0054836", - "names": [ - "PKDYS2", - "Parkinsonism-dystonia, infantile, 2; PKDYS2" - ], - "preferred_name": "parkinsonism-dystonia, infantile, 2", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 6, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0100065", - "names": [ - "tyrosine hydroxylase infantile parkinsonism and motor delay" - ], - "preferred_name": "TH-deficient infantile parkinsonism and motor delay", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 59, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0100087", - "names": [ - "FAD", - "GARD:0000632", - "Alzheimer disease, familial" - ], - "preferred_name": "familial Alzheimer disease", - "types": [ - "Disease", - "DiseaseOrPhenotypicFeature", - "ThingWithTaxon", - "BiologicalEntity", - "NamedThing", - "Entity" - ], - "shortest_name_length": 3, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "MONDO:0100088", - "names": [ - "Alzheimer disease type 2" - ], - "preferred_name": "Alzheimer disease type 2", - "types": [ - "NamedThing" - ], - "shortest_name_length": 24, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - }, - { - "curie": "www:bget?hsa:11315", - "names": [ - "DJ1", - "DJ-1", - "GATD2", - "HEL-S-67p; Parkinsonism associated deglycase", - "K05687 protein DJ-1 [EC:3.5.1.124] | (RefSeq) PARK7" - ], - "preferred_name": "DJ-1", - "types": [ - "NamedThing" - ], - "shortest_name_length": 3, - "clique_identifier_count": 1, - "taxa": [], "taxon_specific": false - } -] +{"curie": "CHEBI:48407", "names": ["antiparkinson agent"], "preferred_name": "antiparkinson agent", "types": ["NamedThing"], "shortest_name_length": 19, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "CHEBI:74925", "names": ["BACE1 inhibitor", "BACE1 inhibitors", "memapsin 2 inhibitor", "memapsin 2 inhibitors", "EC 3.4.23.46 inhibitor", "EC 3.4.23.46 inhibitors", "beta-secretase inhibitor", "beta-secretase inhibitors", "gamma-secretase inhibitor", "Gamma-secretase inhibitor", "gamma-secretase inhibitors", "Gamma-secretase inhibitors", "memapsin 2 (EC 3.4.23.46) inhibitor", "EC 3.4.23.46 (memapsin 2) inhibitors", "memapsin 2 (EC 3.4.23.46) inhibitors", "beta-site APP-cleaving enzyme 1 inhibitor", "beta-site APP-cleaving enzyme 1 inhibitors", "membrane-associated aspartic protease 2 inhibitor", "membrane-associated aspartic protease 2 inhibitors", "beta-site Alzheimer's amyloid precursor protein cleaving enzyme 1 inhibitor", "beta-site Alzheimer's amyloid precursor protein cleaving enzyme 1 inhibitors", "beta-site Alzheimer's amyloid precursor protein cleaving enzyme 1 (BACE1) inhibitor", "beta-site Alzheimer's amyloid precursor protein cleaving enzyme 1 (BACE1) inhibitors"], "preferred_name": "BACE1 inhibitor", "types": ["NamedThing"], "shortest_name_length": 15, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "HP:0001300", "names": ["Parkinsonian disease"], "preferred_name": "parkinsonian disorder", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 20, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "HP:0002322", "names": ["Rest tremor", "Resting tremor", "Tremor at rest", "Parkinsonian tremor"], "preferred_name": "Resting tremor", "types": ["PhenotypicFeature", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 11, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "HP:0002511", "names": ["Alzheimer disease", "Late-onset form of familial Alzheimer disease"], "preferred_name": "Alzheimer disease", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 17, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "HP:0002548", "names": ["Favorable response to levodopa", "Parkinsonism with favourable response to dopaminergic medication"], "preferred_name": "Parkinsonism with favorable response to dopaminergic medication", "types": ["PhenotypicFeature", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 30, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0000828", "names": ["juvenile-onset Parkinson's disease"], "preferred_name": "juvenile-onset Parkinson disease", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 34, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0001945", "names": ["postencephalitic parkinsonism", "postencephalitic Parkinsonism"], "preferred_name": "postencephalitic Parkinson disease", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 29, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0004975", "names": ["AD", "Alzheimer disease", "Alzheimer dementia", "Alzheimers disease", "Alzheimers dementia", "Alzheimer's disease", "Alzheimer's dementia", "Alzheimer disease, familial"], "preferred_name": "Alzheimer disease", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 2, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0005180", "names": ["paralysis agitans", "Parkinson's disease"], "preferred_name": "Parkinson disease", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 17, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0006966", "names": ["secondary Parkinsonism", "symptomatic parkinsonism", "secondary parkinsonism, unspecified", "secondary parkinsonism (disorder) [ambiguous]", "disorder presenting primarily with parkinsonism"], "preferred_name": "secondary Parkinson disease", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 22, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0007088", "names": ["AD1", "Alzheimer disease 1", "Alzheimer disease; AD", "presenile and senile dementia", "Alzheimer disease, familial, 1", "Alzheimer disease, protection against", "early-onset familial form of Alzheimer disease", "Alzheimer disease, early-onset, with cerebral amyloid angiopathy"], "preferred_name": "Alzheimer disease type 1", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 3, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0007089", "names": ["AD2", "LOFAD", "Alzheimer disease 2", "Alzheimer disease-2", "Alzheimer's disease 2", "Alzheimer disease type 2", "Alzheimer disease 2; AD2", "Alzheimer's disease type 2", "late onset Alzheimer disease", "Alzheimer disease 2, late onset", "Alzheimer disease 2, late-onset", "late onset familial Alzheimer disease", "Alzheimer disease associated with APOE4", "Alzheimer disease associated with Apoe4", "Alzheimer disease associated with APOE E4"], "preferred_name": "Alzheimer disease 2", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 3, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0007104", "names": ["PDALS", "ALS-pDC", "Guam disease", "Lytico-Bodig disease", "Lytigo-Bodig disease", "Parkinsonism-dementia-ALS complex", "amyotrophic lateral sclerosis-PARKINSONISM/dementia complex 1", "amyotrophic lateral sclerosis-Parkinsonism/dementia Complex type 1", "amyotrophic lateral sclerosis-Parkinsonism/dementia Complex of Guam", "amyotrophic lateral sclerosis-parkinsonism-dementia of Guam syndrome", "amyotrophic lateral sclerosis, Parkinsonism/dementia complex of Guam"], "preferred_name": "amyotrophic lateral sclerosis-parkinsonism-dementia complex", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 5, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0007312", "names": ["Charcot-Marie-Tooth disease with ptosis and parkinsonism"], "preferred_name": "Charcot-Marie-Tooth disease with ptosis and parkinsonism", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 56, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0007433", "names": ["dementia/parkinsonism with non-Alzheimer amyloid plaques"], "preferred_name": "dementia/parkinsonism with non-Alzheimer amyloid plaques", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 56, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0007488", "names": ["DLB", "Lewy body disease", "Lewy body dementia", "dementia, Lewy body", "dementia, Lewy body; DLB", "dementia with Lewy bodies", "diffuse Lewy body disease", "Senile dementia of the Lewy body type", "Lewy body variant of Alzheimer disease", "diffuse Lewy body disease with gaze palsy"], "preferred_name": "Lewy body dementia", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 3, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0007495", "names": ["DRD", "DYT5a", "DYT-GCH1", "dystonia 5", "dystonia type 5", "GTPCH1-deficient DRD", "dystonia, DOPA-responsive", "dystonia, DOPA-responsive; DRD", "autosomal dominant Segawa syndrome", "HPD with marked diurnal fluctuation", "Segawa syndrome, autosomal dominant", "GTPCH1-deficient dopa-responsive dystonia", "autosomal dominant dopa-responsive dystonia", "Dopa-responsive dystonia, autosomal dominant", "Dopa-responsive dystonia; Segawa syndrome AD", "dystonia, Dopa-responsive, autosomal dominant", "dystonia, progressive, with diurnal variation", "dystonia-Parkinsonism with diurnal fluctuation", "GTP cyclohydrolase 1-deficient dopa-responsive dystonia", "hereditary progressive dystonia with marked diurnal fluctuation"], "preferred_name": "dystonia 5", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 3, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0007496", "names": ["RDP", "DYT12", "DYT-ATP1A3", "dystonia 12", "dystonia type 12", "dystonia 12; DYT12", "ATP1A3 dystonic disorder", "rapid-onset dystonia-parkinsonism", "dystonia-Parkinsonism, rapid-onset", "dystonic disorder caused by mutation in ATP1A3"], "preferred_name": "dystonia 12", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 3, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0008193", "names": ["paralysis agitans, juvenile, of Hunt", "Parkinson disease, juvenile, of Hunt"], "preferred_name": "paralysis agitans, juvenile, of Hunt", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 36, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0008199", "names": ["PD", "park", "Parkinson disease, late-onset", "Parkinson disease, late-onset; PD"], "preferred_name": "late-onset Parkinson disease", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 2, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0008200", "names": ["PARK1", "atypical Parkinson disease", "Parkinson disease 1, autosomal dominant", "autosomal dominant Parkinson's disease 1", "autosomal dominant Parkinson disease type 1", "Parkinson disease 1, autosomal dominant; PARK1", "Parkinson disease 1, autosomal dominant Lewy body"], "preferred_name": "autosomal dominant Parkinson disease 1", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 5, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0008201", "names": ["Perry syndrome", "parkinsonism with alveolar hypoventilation and mental depression", "Parkinsonism with alveolar hypoventilation and mental depression"], "preferred_name": "Perry syndrome", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 14, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0008685", "names": ["WPW", "Wpw syndrome", "preexcitation syndrome", "anomalous A-V excitation", "Wolff-Parkinson-White pattern", "Wolff-Parkinson-White syndrome", "WOLFF-Parkinson-WHITE syndrome", "accessory atrioventricular pathways", "anomalous atrioventricular excitation", "Wolff-Parkinson-White pattern (finding)"], "preferred_name": "Wolff-Parkinson-white syndrome", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 3, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0009497", "names": ["Vitsala", "Kifafa seizure disorder", "Complex familial seizure disorder", "parkinsonian features and neurologic abnormalities, mental retardation and transient psychotic episodes"], "preferred_name": "Kifafa seizure disorder", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 7, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0009830", "names": ["PARK15", "pallidopyramidal syndrome", "Pallidopyramidal syndrome", "pallido-pyramidal disease", "pallido-pyramidal syndrome", "parkinsonian-pyramidal syndrome", "autosomal recessive early-onset Parkinson disease 15", "Parkinson disease 15, autosomal recessive early-onset", "autosomal recessive early-onset Parkinson's disease 15", "autosomal recessive early-onset Parkinson disease type 15", "Parkinson disease 15, autosomal recessive early-onset; PARK15"], "preferred_name": "parkinsonian-pyramidal syndrome", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 6, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0009839", "names": ["PSP-p", "atypical PSP", "PSP-parkinsonism", "Parkinson-dementia syndrome", "progressive supranuclear palsy atypical", "supranuclear palsy, progressive, 1, atypical", "Steele-Richardson-Olszewski syndrome, atypical"], "preferred_name": "progressive supranuclear palsy-parkinsonism syndrome", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 5, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0010360", "names": ["PARK12", "Parkinson disease 12", "Parkinson disease, X-linked", "Parkinson disease 12; PARK12"], "preferred_name": "parkinson disease 12", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 6, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0010422", "names": ["AD16", "Alzheimer disease 16", "Alzheimer's disease 16", "Alzheimer disease 16; AD16", "Alzheimer's disease type 16"], "preferred_name": "Alzheimer disease 16", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 4, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0010482", "names": ["XPDS", "PARKINSONISM with spasticity, X-linked", "PARKINSONISM with spasticity, X-linked; XPDS"], "preferred_name": "X-linked parkinsonism-spasticity syndrome", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 4, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0010709", "names": ["Wsn", "BGMR", "WSMN", "Waisman syndrome", "WAISMAN syndrome", "Laxova-Opitz syndrome", "WAISMAN syndrome; WSMN", "Laxova Brown hogan syndrome", "basal ganglia disorder with mental retardation", "basal ganglion disorder with mental retardation", "Parkinsonism, early onset with mental retardation", "Parkinsonism, early-onset, with mental retardation", "X-linked recessive basal ganglia disorder with mental retardation"], "preferred_name": "early-onset parkinsonism-intellectual disability syndrome", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 3, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0010747", "names": ["XDP", "DYT3", "Lubag", "DYT-TAF1", "Lubag syndrome", "dystonia 3, torsion, X-linked", "X-linked dystonia Parkinsonism", "dystonia-Parkinsonism, X-linked", "dystonia 3, torsion, X-linked; DYT3", "X-linked dystonia-parkinsonism/Lubag", "X-linked dystonia-Parkinsonism syndrome", "torsion dystonia-Parkinsonism, Filipino type", "X-linked torsion dystonia-Parkinsonism syndrome"], "preferred_name": "X-linked dystonia-parkinsonism", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 3, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0010783", "names": ["Alzheimer disease, susceptibility to, mitochondrial"], "preferred_name": "Alzheimer disease, susceptibility to, mitochondrial", "types": ["NamedThing"], "shortest_name_length": 51, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0010796", "names": ["Parkinson disease, mitochondrial"], "preferred_name": "Parkinson disease, mitochondrial", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 32, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0010820", "names": ["JP", "PDJ", "PARK2", "Parkinson disease 2", "juvenile parkinsonism", "PRKN young-onset Parkinson disease", "autosomal recessive juvenile Parkinson disease", "Parkinson disease, juvenile, autosomal recessive", "Parkinson disease 2, autosomal recessive juvenile", "autosomal recessive juvenile Parkinson's disease 2", "Parkinson disease autosomal recessive, early onset", "Parkinsonism, early onset, with diurnal fluctuation", "Parkinsonism, early-onset, with diurnal fluctuation", "autosomal recessive juvenile Parkinson disease type 2", "young-onset Parkinson disease caused by mutation in PRKN", "Parkinson disease 2, autosomal recessive juvenile; PARK2"], "preferred_name": "autosomal recessive juvenile Parkinson disease 2", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 2, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0010857", "names": ["FTD", "Ftdp17", "Pick Complex", "semantic variant PPA", "frontotemporal dementia", "Ftld with Tau inclusions", "Wilhelmsen-Lynch disease", "frontotemporal dementia; FTD", "frontotemporal lobe dementia", "Pallidopontonigral Degeneration", "semantic primary progressive aphasia", "frontotemporal dementia with Parkinsonism", "dementia, frontotemporal, with Parkinsonism", "multiple system tauopathy with presenile dementia", "frontotemporal lobar Degeneration with Tau inclusions", "disinhibition-dementia-Parkinsonism-amyotrophy Complex"], "preferred_name": "semantic dementia", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 3, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "NCBIGene:378899", "preferred_name": "FTL", "names": ["FTL", "FTD", "ferritoid", "ferritin light chain", "ferritin, light polypeptide", "Q8AYG9_CHICK Ferritin (trembl)"], "taxa": ["NCBITaxon:9031"], "taxon_specific": true, "types": ["Gene", "GeneOrGeneProduct", "GenomicEntity", "ChemicalEntityOrGeneOrGeneProduct", "PhysicalEssence", "OntologyClass", "BiologicalEntity", "ThingWithTaxon", "NamedThing", "Entity", "PhysicalEssenceOrOccurrent", "MacromolecularMachineMixin", "Protein", "GeneProductMixin", "Polypeptide", "ChemicalEntityOrProteinOrPolypeptide"], "shortest_name_length": 3, "clique_identifier_count": 3} +{"curie": "MONDO:0011194", "names": ["AD5", "Ad5", "Alzheimer disease 5", "Alzheimer's disease 5", "Alzheimer disease type 5", "Alzheimer's disease type 5", "Alzheimer disease, familial 5", "Alzheimer disease, familial, 5"], "preferred_name": "Alzheimer disease 5", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 3, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0011220", "names": ["PARK3", "Parkinson disease type 3", "autosomal dominant Parkinson disease", "Parkinson disease 3, autosomal dominant", "Parkinson disease 3, autosomal dominant; PARK3", "Parkinson disease 3, autosomal dominant Lewy body"], "preferred_name": "parkinson disease 3, autosomal dominant", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 5, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0011401", "names": ["Alzheimer disease without neurofibrillary tangles", "Alzheimer's disease without neurofibrillary tangles"], "preferred_name": "Alzheimer disease without neurofibrillary tangles", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 49, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0011513", "names": ["Alzheimer disease, familial early-onset, with Coexisting amyloid and prion pathology"], "preferred_name": "Alzheimer disease, familial early-onset, with coexisting amyloid and prion pathology", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 84, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0011551", "names": ["DYT5b", "autosomal recessive Segawa syndrome", "Segawa syndrome, autosomal recessive", "autosomal recessive dopa-responsive dystonia", "Parkinsonism, infantile, autosomal recessive", "dopa-responsive dystonia, autosomal recessive", "Dopa-responsive dystonia, autosomal recessive", "Dopa-responsive dystonia, autosomal recessive", "DOPA responsive dystonia, autosomal recessive", "dystonia, Dopa-responsive, autosomal recessive", "dystonia, DOPA responsive, autosomal recessive", "tyrosine hydroxylase-deficient dopa-responsive dystonia"], "preferred_name": "TH-deficient dopa-responsive dystonia", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 5, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0011561", "names": ["AD6", "AD6", "Alzheimer disease 6", "Alzheimer's disease 6", "Alzheimer disease type 6", "Alzheimer's disease type 6", "Alzheimer disease 6, late onset", "Alzheimer disease 6, late-onset", "plasma Beta-amyloid-42 level quantitative trait locus"], "preferred_name": "Alzheimer disease 6", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 3, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0011562", "names": ["PARK4", "Parkinson disease 4, autosomal dominant", "autosomal dominant Parkinson's disease 4", "autosomal dominant Parkinson disease type 4", "Parkinson disease 4, autosomal dominant; PARK4", "autosomal dominant Lewy body Parkinson disease 4", "Parkinson disease 4, autosomal dominant Lewy body"], "preferred_name": "autosomal dominant Parkinson disease 4", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 5, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0011613", "names": ["PARK6", "PINK1 Parkinson disease", "early-onset Parkinson disease 6", "Parkinson disease 6, early-onset", "Parkinson disease caused by mutation in PINK1", "Parkinson disease 6, late-onset, susceptibility to", "Parkinson disease 6, autosomal recessive early-onset", "autosomal recessive early-onset Parkinson's disease 6", "autosomal recessive early-onset Parkinson disease type 6", "Parkinson disease 6, autosomal recessive early-onset; PARK6", "Parkinson disease, autosomal recessive early-onset, digenic, Pink1/Dj1"], "preferred_name": "autosomal recessive early-onset Parkinson disease 6", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 5, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0011647", "names": ["AD7", "Ad7", "Alzheimer disease 7", "Alzheimer's disease 7", "Alzheimer disease type 7", "Alzheimer's disease type 7", "Alzheimer disease, familial 7", "Alzheimer disease, familial, 7"], "preferred_name": "Alzheimer disease 7", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 3, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0011658", "names": ["PARK7", "PARK7 Parkinson disease", "Parkinson disease caused by mutation in PARK7", "Parkinson disease 7, autosomal recessive early-onset", "autosomal recessive early-onset Parkinson's disease 7", "autosomal recessive early-onset Parkinson disease type 7", "Parkinson disease 7, autosomal recessive early-onset; PARK7"], "preferred_name": "autosomal recessive early-onset Parkinson disease 7", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 5, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0011706", "names": ["KRS", "PARK9", "KRPPD", "park 9", "Kufor-Rakeb syndrome", "Parkinson disease type 9", "Kufor-Rakeb syndrome; KRS", "ceroid lipofuscinosis, neuronal, 12", "autosomal recessive Parkinson disease 9", "Parkinson disease 9, autosomal recessive", "autosomal recessive juvenile onset Parkinson disease 9", "Parkinson disease 9, autosomal recessive, juvenile-onset", "Pallidopyramidal Degeneration with supranuclear upgaze paresis and dementia", "Pallidopyramidal degeneration with supranuclear upgaze paresis, and dementia"], "preferred_name": "Kufor-Rakeb syndrome", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 3, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0011737", "names": ["PARK10", "Parkinson disease 10", "Parkinson disease 10; PARK10", "Parkinson disease, Age at onset of"], "preferred_name": "parkinson disease 10", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 6, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0011743", "names": ["AD4", "Ad4", "Alzheimer disease 4", "Alzheimer's disease 4", "Alzheimer disease type 4", "Alzheimer's disease type 4", "Alzheimer disease, familial4", "Alzheimer disease, familial, 4", "Alzheimer disease familial type 4", "familial Alzheimer disease, type 4", "familial Alzheimer's disease, type 4"], "preferred_name": "Alzheimer disease 4", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 3, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0011764", "names": ["PARK8", "LRRK2 Parkinson disease", "Parkinson disease 8, autosomal dominant", "autosomal dominant Parkinson's disease 8", "autosomal dominant Parkinson disease type 8", "Parkinson disease caused by mutation in LRRK2", "Parkinson disease 8, autosomal dominant; PARK8"], "preferred_name": "autosomal dominant Parkinson disease 8", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 5, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0011777", "names": ["AD8", "Ad8", "Alzheimer disease 8", "Alzheimer's disease 8", "Alzheimer disease type 8", "Alzheimer's disease type 8", "Alzheimer disease, familial 8", "Alzheimer disease, familial, 8"], "preferred_name": "Alzheimer disease 8", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 3, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0011896", "names": ["PARK11", "GIGYF2 hereditary late onset Parkinson disease", "susceptibility to autosomal dominant Parkinson disease 11", "Parkinson disease 11, autosomal dominant, susceptibility to", "Parkinson disease 11, autosomal dominant, susceptibility to; PARK11", "hereditary late onset Parkinson disease caused by mutation in GIGYF2"], "preferred_name": "Parkinson disease 11, autosomal dominant, susceptibility to", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 6, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0011913", "names": ["AD3", "Alzheimer disease 3", "Alzheimer's disease 3", "Alzheimer disease 3; AD", "Alzheimer disease type 3", "Alzheimer's disease type 3", "Alzheimer disease familial 3", "Alzheimer disease, familial, 3", "Alzheimer disease 3, early onset", "Alzheimer disease 3, early-onset", "familial Alzheimer disease, type 3", "familial Alzheimer's disease, type 3", "Alzheimer disease early onset type 3", "PSEN1 early-onset autosomal dominant Alzheimer disease", "Alzheimer disease, familial, 3, with spastic paraparesis and apraxia", "early-onset autosomal dominant Alzheimer disease caused by mutation in PSEN1", "Alzheimer disease, familial, 3, with spastic paraparesis and unusual plaques"], "preferred_name": "Alzheimer disease 3", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 3, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0012153", "names": ["AD9", "Alzheimer disease 9", "Alzheimer disease 9; AD9", "Alzheimer disease 9, late-onset", "Alzheimer disease 9, susceptibility to", "Alzheimer disease 9, susceptibility to; AD9"], "preferred_name": "Alzheimer disease 9", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 3, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0012321", "names": ["AD10", "Ad10", "Alzheimer disease 10", "Alzheimer's disease 10", "Alzheimer disease type 10", "Alzheimer's disease type 10", "Alzheimer disease familial 10", "Alzheimer disease, familial, 10"], "preferred_name": "Alzheimer disease 10", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 4, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0012344", "names": ["AD11", "Ad11", "Alzheimer disease 11", "Alzheimer's disease 11", "Alzheimer disease type 11", "Alzheimer's disease type 11", "Alzheimer disease, familial, 11"], "preferred_name": "Alzheimer disease 11", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 4, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0012466", "names": ["PARK13", "HTRA2 young-onset Parkinson disease", "young-onset Parkinson disease caused by mutation in HTRA2", "susceptibility to autosomal dominant Parkinson disease 13", "Parkinson disease 13, autosomal dominant, susceptibility to", "Parkinson disease 13, autosomal dominant, susceptibility to; PARK13"], "preferred_name": "Parkinson disease 13, autosomal dominant, susceptibility to", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 6, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0012609", "names": ["AD12", "Ad12", "Alzheimer disease 12", "Alzheimer's disease 12", "Alzheimer disease type 12", "Alzheimer's disease type 12", "Alzheimer disease familial 12", "Alzheimer disease, familial, 12"], "preferred_name": "Alzheimer disease 12", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 4, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0012630", "names": ["AD13", "Alzheimer disease 13", "Alzheimer's disease 13", "Alzheimer disease 13; AD13", "Alzheimer's disease type 13"], "preferred_name": "Alzheimer disease 13", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 4, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0012631", "names": ["AD14", "Alzheimer disease 14", "Alzheimer's disease 14", "Alzheimer disease 14; AD14", "Alzheimer's disease type 14"], "preferred_name": "Alzheimer disease 14", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 4, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0012632", "names": ["AD15", "Alzheimer disease 15", "Alzheimer's disease 15", "Alzheimer disease 15; AD15", "Alzheimer's disease type 15"], "preferred_name": "AD15", "types": ["NamedThing"], "shortest_name_length": 4, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0012789", "names": ["DYT16", "DYT-PRKRA", "dystonia 16", "dystonia type 16", "dystonia 16; DYT16", "PRKRA dystonic disorder", "early-onset dystonia parkinsonism", "Young-onset dystonia-(parkinsonism)", "dystonic disorder caused by mutation in PRKRA"], "preferred_name": "dystonia 16", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 5, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0013060", "names": ["PARK14", "dystonia-Parkinsonism Adult-onset", "dystonia-Parkinsonism, adult-onset", "adult-onset dystonia - parkinsonism", "PLA2G6-related dystonia-parkinsonism", "dystonia-parkinsonism, Paisan-Ruiz type", "Parkinson disease 14, autosomal recessive", "autosomal recessive Parkinson's disease 14", "autosomal recessive Parkinson disease type 14", "PLA2G6 hereditary late onset Parkinson disease", "Parkinson disease 14, autosomal recessive; PARK14", "hereditary late onset Parkinson disease caused by mutation in PLA2G6"], "preferred_name": "autosomal recessive Parkinson disease 14", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 6, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0013150", "names": ["IPD", "PKDYS", "infantile Parkinsonism-dystonia", "Parkinsonism-dystonia infantile", "PARKINSONISM-dystonia, infantile", "PARKINSONISM-dystonia, infantile; PKDYS", "dopamine transporter deficiency syndrome"], "preferred_name": "parkinsonism-dystonia, infantile", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 3, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0013167", "names": ["PARK16", "Parkinson disease 16", "Parkinson disease 16; PARK16"], "preferred_name": "parkinson disease 16", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 6, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0013340", "names": ["PARK5", "UCHL1 young-onset Parkinson disease", "susceptibility to autosomal dominant Parkinson disease 5", "young-onset Parkinson disease caused by mutation in UCHL1", "Parkinson disease 5, autosomal dominant, susceptibility to", "Parkinson disease 5, autosomal dominant, susceptibility to; PARK5"], "preferred_name": "Parkinson disease 5, autosomal dominant, susceptibility to", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 5, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0013625", "names": ["PARK17", "Parkinson disease 17", "Parkinson's disease 17", "VPS35 Parkinson disease", "Parkinson disease type 17", "Parkinson disease 17; PARK17", "autosomal dominant Parkinson disease 17", "Parkinson disease caused by mutation in VPS35"], "preferred_name": "Parkinson disease 17", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 6, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0013653", "names": ["PARK18", "EIF4G1 hereditary late onset Parkinson disease", "susceptibility to autosomal dominant Parkinson disease 18", "Parkinson disease 18, autosomal dominant, susceptibility to", "Parkinson disease 18, autosomal dominant, susceptibility to; PARK18", "hereditary late onset Parkinson disease caused by mutation in EIF4G1"], "preferred_name": "Parkinson disease 18, autosomal dominant, susceptibility to", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 6, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0014036", "names": ["AD17", "Alzheimer disease 17", "Alzheimer's disease 17", "Alzheimer disease 17; AD17", "Alzheimer's disease type 17", "Alzheimer disease 17, late onset", "Alzheimer disease 17, late-onset"], "preferred_name": "Alzheimer disease 17", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 4, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0014231", "names": ["PARK19", "PARK19A", "Park19, formerly", "DNAJC6 Parkinson disease", "Parkinson disease 19B, early-onset", "Parkinson disease 19, juvenile-onset", "Parkinson disease 19A, juvenile-onset", "juvenile onset Parkinson's disease 19A", "juvenile onset Parkinson disease type 19A", "Parkinson disease 19, juvenile-onset; PARK19", "Parkinson disease caused by mutation in DNAJC6", "Parkinson disease 19A, juvenile-onset; PARK19A"], "preferred_name": "juvenile onset Parkinson disease 19A", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 6, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0014233", "names": ["PARK20", "SYNJ1 Parkinson disease", "Parkinson disease 20, early-onset", "early-onset Parkinson's disease 20", "early-onset Parkinson disease type 20", "Parkinson disease 20, early-onset; PARK20", "Parkinson disease caused by mutation in SYNJ1"], "preferred_name": "early-onset Parkinson disease 20", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 6, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0014265", "names": ["AD18", "Alzheimer disease 18", "Alzheimer's disease 18", "ADAM10 Alzheimer disease", "Alzheimer disease type 18", "Alzheimer disease 18; AD18", "Alzheimer's disease type 18", "Alzheimer disease 18, late-onset", "Alzheimer disease caused by mutation in ADAM10"], "preferred_name": "Alzheimer disease 18", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 4, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0014316", "names": ["AD19", "Alzheimer disease 19", "Alzheimer's disease 19", "PLD3 Alzheimer disease", "Alzheimer disease type 19", "Alzheimer disease 19; AD19", "Alzheimer's disease type 19", "Alzheimer disease 19 late onset", "Alzheimer disease 19, late-onset", "Alzheimer disease caused by mutation in PLD3"], "preferred_name": "Alzheimer disease 19", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 4, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0014604", "names": ["PARK21", "Parkinson disease 21", "Parkinson disease type 21", "Parkinson disease 21; PARK21", "DNAJC13 hereditary late onset Parkinson disease", "hereditary late onset Parkinson disease caused by mutation in DNAJC13"], "preferred_name": "Parkinson disease 21", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 6, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0014742", "names": ["PARK22", "CHCHD2 Parkinson disease", "Parkinson disease 22, autosomal dominant", "Parkinson disease caused by mutation in CHCHD2", "Parkinson disease 22, autosomal dominant; PARK22"], "preferred_name": "Parkinson disease 22, autosomal dominant", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 6, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0014796", "names": ["PARK23", "VPS13C young-onset Parkinson disease", "autosomal recessive early-onset Parkinson disease 23", "Parkinson disease 23, autosomal recessive early-onset", "autosomal recessive early-onset Parkinson's disease 23", "autosomal recessive early-onset Parksinson disease type 23", "young-onset Parkinson disease caused by mutation in VPS13C", "Parkinson disease 23, autosomal recessive early-onset; PARK23"], "preferred_name": "autosomal recessive early-onset Parkinson disease 23", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 6, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0015140", "names": ["EOFAD", "early-onset, autosomal dominant Alzheimer disease", "early-onset familial autosomal dominant Alzheimer disease"], "preferred_name": "early-onset autosomal dominant Alzheimer disease", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 5, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0017279", "names": ["YOPD", "early-onset Parkinson disease", "early-onset Parkinson's disease"], "preferred_name": "young-onset Parkinson disease", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 4, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0017639", "names": ["CO-induced parkinsonism"], "preferred_name": "carbon monoxide-induced parkinsonism", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 23, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0018466", "names": ["LOPD", "late-onset Parkinson disease", "late onset Parkinson disease", "late onset Parkinson's disease", "hereditary late-onset Parkinson disease", "autosomal dominant late-onset Parkinson disease"], "preferred_name": "late-onset Parkinson disease", "types": ["NamedThing"], "shortest_name_length": 4, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0018899", "names": ["PCA", "Benson syndrome", "biparietal Alzheimer disease"], "preferred_name": "posterior cortical atrophy", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 3, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0020352", "names": ["MSA-p", "MSA, parkinsonian type"], "preferred_name": "multiple system atrophy, parkinsonian type", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 5, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0054835", "names": ["PKDYS", "PKDYS1", "dopamine transporter deficiency syndrome", "Parkinsonism-dystonia, infantile, 1; PKDYS1"], "preferred_name": "parkinsonism-dystonia, infantile, 1", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 5, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0054836", "names": ["PKDYS2", "Parkinsonism-dystonia, infantile, 2; PKDYS2"], "preferred_name": "parkinsonism-dystonia, infantile, 2", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 6, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0100065", "names": ["tyrosine hydroxylase infantile parkinsonism and motor delay"], "preferred_name": "TH-deficient infantile parkinsonism and motor delay", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 59, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0100087", "names": ["FAD", "GARD:0000632", "Alzheimer disease, familial"], "preferred_name": "familial Alzheimer disease", "types": ["Disease", "DiseaseOrPhenotypicFeature", "ThingWithTaxon", "BiologicalEntity", "NamedThing", "Entity"], "shortest_name_length": 3, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "MONDO:0100088", "names": ["Alzheimer disease type 2"], "preferred_name": "Alzheimer disease type 2", "types": ["NamedThing"], "shortest_name_length": 24, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} +{"curie": "www:bget?hsa:11315", "names": ["DJ1", "DJ-1", "GATD2", "HEL-S-67p; Parkinsonism associated deglycase", "K05687 protein DJ-1 [EC:3.5.1.124] | (RefSeq) PARK7"], "preferred_name": "DJ-1", "types": ["NamedThing"], "shortest_name_length": 3, "clique_identifier_count": 1, "taxa": [], "taxon_specific": false} From bb59c418da691a4e28ceca3809506d924327278d Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 16 Jul 2026 10:24:49 -0400 Subject: [PATCH 05/46] Switch Solr to standalone mode and self-contained core backups Run Solr in standalone mode everywhere (no ZooKeeper for a one-node job): drop -cloud from the loader Dockerfile and -DzkRun from docker-compose and CI. The Makefile now creates the core from the checked-in configset, loads in parallel, optimizes the index before export (closes #256), and tars the whole core -- config, schema and index -- into a self-contained snapshot.backup.tar.gz. Restoring is now just "untar into the Solr home and start Solr", so the old replication-backup targets and the solr-restore/ helper are removed. CI creates the core from the configset and runs the parallel loader end to end. Co-Authored-By: Claude Opus 4.8 --- .github/workflows/tester.yml | 10 +++- data-loading/Dockerfile | 8 ++-- data-loading/Makefile | 92 ++++++++++++++++++++---------------- docker-compose.yml | 6 ++- solr-restore/README.md | 12 ----- solr-restore/restore.sh | 64 ------------------------- 6 files changed, 69 insertions(+), 123 deletions(-) delete mode 100644 solr-restore/README.md delete mode 100644 solr-restore/restore.sh diff --git a/.github/workflows/tester.yml b/.github/workflows/tester.yml index 9902ed0b..ea4da134 100644 --- a/.github/workflows/tester.yml +++ b/.github/workflows/tester.yml @@ -22,8 +22,14 @@ jobs: with: python-version: ${{ matrix.python-version }} - - name: Run SOLR - run: docker run --name name_lookup -d -p 8983:8983 -t solr -DzkRun + - name: Run SOLR (standalone) + run: docker run --name name_lookup -d -p 8983:8983 solr:9.10.0 + + - name: Create the name_lookup core from the checked-in configset + run: | + until curl -sf http://localhost:8983/solr/admin/cores?action=STATUS >/dev/null; do sleep 2; done + docker cp data-loading/configsets/name_lookup name_lookup:/tmp/name_lookup + docker exec name_lookup solr create -c name_lookup -d /tmp/name_lookup - name: Install dependencies run: | diff --git a/data-loading/Dockerfile b/data-loading/Dockerfile index 331b9fcc..5d9246b9 100644 --- a/data-loading/Dockerfile +++ b/data-loading/Dockerfile @@ -67,12 +67,14 @@ ENV PATH="$VIRTUAL_ENV/bin:$PATH" COPY --chown=nru requirements.txt ${ROOT} RUN pip3 install -r requirements.txt -# Copy necessary files. +# Copy necessary files. The configset carries the Solr schema/config; the core is +# created from it with `solr create -c name_lookup -d configsets/name_lookup`. COPY --chown=nru setup-and-load-solr.sh ${ROOT} COPY --chown=nru README.md ${ROOT} COPY --chown=nru Makefile ${ROOT} +COPY --chown=nru configsets ${ROOT}/configsets -# On entry, start the Solr instance. +# On entry, start the Solr instance (standalone -- no ZooKeeper for a one-node load). ENV SOLR_EXEC="${ROOT}/solr/solr-${SOLR_VERSION}/bin/solr" ENV SOLR_DIR="$SOLR_DIR" -ENTRYPOINT ${SOLR_EXEC} -cloud -f -p 8983 -m 64G -s ${SOLR_DIR} +ENTRYPOINT ${SOLR_EXEC} start -f --user-managed -p 8983 -m 64G -s ${SOLR_DIR} diff --git a/data-loading/Makefile b/data-loading/Makefile index e0028bfd..93eaa0ef 100644 --- a/data-loading/Makefile +++ b/data-loading/Makefile @@ -1,8 +1,11 @@ # This Makefile contains all the instructions necessary to -# download Babel synonym files from a web location, create -# and load them into a Solr dataset and generate a Solr backup -# that can be used to start a NameRes instance. +# download Babel synonym files from a web location, load them into a +# standalone Solr core (using the checked-in configset under +# configsets/name_lookup) and generate a self-contained Solr backup that can be +# used to start a NameRes instance. # +# The backup is the whole Solr core -- config, schema AND index -- so restoring +# it is just "untar into the Solr home and start Solr" (see ../data-loading/README.md). # Configuration SYNONYMS_URL=https://stars.renci.org/var/babel_outputs/2025sep1/synonyms/ @@ -10,74 +13,83 @@ SYNONYMS_URL=https://stars.renci.org/var/babel_outputs/2025sep1/synonyms/ # How much memory should Solr use. SOLR_MEM=220G -# SOLR_DIR should be set up to point to the Solr data directory (usually /var/solr) -# and SOLR_EXEC should be set up to point to the Solr executable. -# These will both be set up by the Dockerfile. +# The checked-in configset used to create the core (contains conf/). +CONFIGSET=configsets/name_lookup + +# The Solr core name. Do not change without also changing NameRes and the configset. +CORE=name_lookup + +# Split any synonym file larger than SPLIT_SIZE into chunks of SPLIT_LINES lines. +# Splitting only helps parallelism, so we only bother with the big files; JSON is +# one document per line, so we must split on line boundaries, never on bytes. +SPLIT_SIZE=2G +SPLIT_LINES=10000000 + +# SOLR_DIR should be set up to point to the Solr data directory (usually /var/solr, +# which is also the Solr home) and SOLR_EXEC should be set up to point to the Solr +# executable. These will both be set up by the Dockerfile. # All and clean targets. .PHONY: all clean all: data/setup.done echo Solr has now been set up and loaded with the synonym data. - echo Run 'make start-solr-backup' to start a backup. Run 'make check-solr-backup' to check - echo if the backup has completed. Once that has completed, run 'make data/backup.done' to + echo Run 'make data/backup.done' to optimize the index, shut down Solr and echo generate a snapshot.backup.tar.gz file that can be used in NameRes. clean: rm -rf data/* mkdir data -# This is a three step process. -# -# Step 1. Download an uncompress synonym files. +# Step 1. Download and uncompress synonym files, then split the large ones so the +# parallel loader has more units of work. data/synonyms/done: mkdir -p data/synonyms wget -c -r -l1 -nd -P data/synonyms ${SYNONYMS_URL} gunzip data/synonyms/*.txt.gz echo Downloaded synonyms from ${SYNONYMS_URL} - # split -d -l 10000000 data/synonyms/SmallMolecule.txt data/synonyms/SmallMolecule.txt. && rm data/synonyms/SmallMolecule.txt - split -d -l 10000000 data/synonyms/DrugChemicalConflated.txt data/synonyms/DrugChemicalConflated.txt. && rm data/synonyms/DrugChemicalConflated.txt - split -d -l 10000000 data/synonyms/GeneProteinConflated.txt data/synonyms/GeneProteinConflated.txt. && rm data/synonyms/GeneProteinConflated.txt - echo Split DrugChemicalConflated.txt and GeneProteinConflated.txt, and deleted the original files. + find data/synonyms -maxdepth 1 -type f -name '*.txt' -size +${SPLIT_SIZE} -print | while read -r f; do \ + echo "Splitting large file $$f (larger than ${SPLIT_SIZE})..."; \ + split -d -l ${SPLIT_LINES} "$$f" "$$f." && rm "$$f"; \ + done touch $@ -# Step 3. Start Solr server. +# Step 2. Start the Solr server (standalone -- no ZooKeeper needed for one node). +# --user-managed keeps standalone mode explicit (Solr 10 will default to cloud). data/solr.pid: mkdir -p ${SOLR_DIR}/logs - ${SOLR_EXEC} -cloud -p 8983 -v -m ${SOLR_MEM} -s ${SOLR_DIR} >> ${SOLR_DIR}/logs/solr.txt 2>> ${SOLR_DIR}/logs/solr.err.txt + ${SOLR_EXEC} start -p 8983 --user-managed -v -m ${SOLR_MEM} -s ${SOLR_DIR} >> ${SOLR_DIR}/logs/solr.txt 2>> ${SOLR_DIR}/logs/solr.err.txt while [ ! -s $@ ]; do \ ${SOLR_EXEC} status | grep -Po 'Solr process \K([0-9]+)' > $@; \ done $(info Solr started with PID file at $@) cat $@ -# Step 4. Load JSON files into Solr server. -data/setup.done: data/synonyms/done data/solr.pid +# Step 3. Create the core from the checked-in configset (schema lives there). +data/core.done: data/solr.pid + ${SOLR_EXEC} create -c ${CORE} -d ${CONFIGSET} + touch $@ + +# Step 4. Load the synonym files into the core (parallel; with a doc-count guard). +data/setup.done: data/synonyms/done data/core.done mkdir -p data/logs bash setup-and-load-solr.sh "data/synonyms/*.txt*" >> data/logs/setup-and-load-solr.sh.log 2>> data/logs/setup-and-load-solr.sh.err.log && touch $@ -# Step 5. Start a Solr backup. -.PHONY: start-solr-backup -start-solr-backup: data/setup.done - curl 'http://localhost:8983/solr/name_lookup/replication?command=backup&name=backup' - -# Step 6. Wait for the backup to complete. -.PHONY: check-solr-backup -check-solr-backup: - curl 'http://localhost:8983/solr/name_lookup/replication?command=details' - -# Step 6. Shutdown the Solr instance. -### data/stop-solr: -### docker exec name_lookup solr stop -p 8983 -verbose - -# Step 7. Generate the backup tarball. -data/backup.done: - mkdir -p data/var/solr/data - mv /var/solr/name_lookup_shard1_replica_n1/data/snapshot.backup data/var/solr/data - cd data && tar zcvf snapshot.backup.tar.gz var && touch backup.done +# Step 5. Optimize the index, shut down Solr, and tar the whole core into a +# self-contained backup. Extracting snapshot.backup.tar.gz into the Solr home +# yields a ready-to-serve name_lookup/ core -- no restore-time schema setup. +data/backup.done: data/setup.done + @echo "Optimizing the index before export (this can take a while on a large index)..." + curl -sf --show-error 'http://localhost:8983/solr/${CORE}/update?optimize=true' + # The optimize above commits, so the on-disk index is consistent. `solr stop` + # can exit non-zero under Docker even on a clean stop (SOLR-16463), so ignore it. + ${SOLR_EXEC} stop -p 8983 || true + rm -f data/solr.pid + tar -C ${SOLR_DIR} -czvf data/snapshot.backup.tar.gz ${CORE} + touch $@ .PHONY: stop-solr stop-solr: - rm data/solr.pid - ${SOLR_EXEC} stop + rm -f data/solr.pid + ${SOLR_EXEC} stop -p 8983 || true $(info Solr stopped.) diff --git a/docker-compose.yml b/docker-compose.yml index ba710749..d20d90a3 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,7 +1,8 @@ services: nameres_solr: container_name: nameres_solr - image: solr:9.1 + # Must be >= the Solr version that built the backup (see data-loading/Dockerfile). + image: solr:9.10.0 mem_limit: 18G environment: # Change this setting to control how much memory you would like your Solr setup to have. @@ -9,7 +10,8 @@ services: SOLR_JAVA_MEM: '-Xmx16G' ports: - '8983:8983' - command: ['-DzkRun'] + # Standalone mode (no ZooKeeper). To load data, extract a NameRes backup into + # ./data/solr so the name_lookup/ core is auto-discovered on startup. # Solr needs to store its data somewhere. It defaults to `./data`, but you can reconfigure this to any # directory you want. diff --git a/solr-restore/README.md b/solr-restore/README.md deleted file mode 100644 index 9da72dfd..00000000 --- a/solr-restore/README.md +++ /dev/null @@ -1,12 +0,0 @@ -# solr-restore - -This directory contains a script that can be used to restore a local Apache Solr backup to a Solr database -in Docker along with the indexes needed to query them from NameRes. It assumes that the backup is present -on the Solr server in the Solr data directory (by default `./data/solr`) and is -named `snapshot.backup.tar.gz`. If you follow the instructions in [the main README file](../README.md), -this script will be used automatically. - -It is essentially the same script as is included in -[the name-lookup Helm chart](https://github.com/helxplatform/translator-devops/tree/develop/helm/name-lookup) -of the `translator-devops` repository, but with some modifications allowing the script to be used -locally. diff --git a/solr-restore/restore.sh b/solr-restore/restore.sh deleted file mode 100644 index 4bc6133c..00000000 --- a/solr-restore/restore.sh +++ /dev/null @@ -1,64 +0,0 @@ -#!/usr/bin/env bash -# -# restore.sh -# -# Restores a Solr backup located in the Solr data directory (`$SOLR_DATA/var/solr/data/snapshot.backup`). -# -# To do this, it must: -# - Initiate the restore. -# - Wait until the restore has completed. -# - Create the necessary fields (hopefully we can make this unnecessary, see https://github.com/TranslatorSRI/NameResolution/issues/185) -# -# This script should only require the `wget` program. -# -# TODO: This script does not currently implement any Blocklists. - -# We don't use set -e because the loop test relies on failures being ignored. -set -uo pipefail - -# Configuration options -SOLR_SERVER="http://localhost:8983" -SLEEP_INTERVAL=60 - -# Please don't change these values unless you change NameRes appropriately! -COLLECTION_NAME="name_lookup" -BACKUP_NAME="backup" - -# Step 0. Make sure the Solr data directory looks like it contains the uncompressed backup. -if [ ! -d "./data/solr/var" ]; then - echo 'WARNING: No ./data/solr/var directory found; are you sure you uncompressed the NameRes backup into the Solr data directory?' >&2 -fi - -# Step 1. Make sure the Solr service is up and running. -HEALTH_ENDPOINT="${SOLR_SERVER}/solr/admin/cores?action=STATUS" -response=$(wget --spider --server-response ${HEALTH_ENDPOINT} 2>&1 | grep "HTTP/" | awk '{ print $2 }') >&2 -until [ "$response" = "200" ]; do - response=$(wget --spider --server-response ${HEALTH_ENDPOINT} 2>&1 | grep "HTTP/" | awk '{ print $2 }') >&2 - echo " -- SOLR is unavailable - sleeping" - sleep 3 -done -echo "SOLR is up and running at ${SOLR_SERVER}." - -# Step 2. Create fields for search. -SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" -source "$SCRIPT_DIR/../data-loading/setup_solr.sh" -echo Solr database has been set up. - -# Step 3. Restore the data -CORE_NAME="${COLLECTION_NAME}_shard1_replica_n1" -echo "Starting Solr restore on core ${CORE_NAME}, with status at ${SOLR_SERVER}/solr/${CORE_NAME}/replication?command=restorestatus" -RESTORE_URL="${SOLR_SERVER}/solr/${CORE_NAME}/replication?command=restore&location=/var/solr/data/var/solr/data/&name=${BACKUP_NAME}" -wget -O - "$RESTORE_URL" -sleep "$SLEEP_INTERVAL" -RESTORE_STATUS_URL="${SOLR_SERVER}/solr/${CORE_NAME}/replication?command=restorestatus" -RESTORE_STATUS=$(wget -q -O - "$RESTORE_STATUS_URL" 2>&1 | grep "success") -RESTORE_STATUS="" -until [ -n "$RESTORE_STATUS" ] ; do - echo "Solr restore in progress. If this takes longer than 30 minutes, please visit ${SOLR_SERVER} with your browser to check Solr." - RESTORE_STATUS=$(wget -q -O - "$RESTORE_STATUS_URL" 2>&1 | grep "success") - sleep "$SLEEP_INTERVAL" -done -echo "Solr restore complete!" - -echo "Solr contents:" -curl -s --negotiate -u: "$SOLR_SERVER/solr/name_lookup/query?q=*:*&rows=0" From 625f8cc6ce71b76d0f0779ea4d4847a56613fec3 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 16 Jul 2026 10:25:06 -0400 Subject: [PATCH 06/46] Document the new data-loading process Rewrite data-loading/README.md around the self-contained standalone-core flow, with an "Options considered" section explaining why we chose it over the status quo and PR #249, and the list of issues it closes. Update the test-run commands in CLAUDE.md to create the core from the configset first. Co-Authored-By: Claude Opus 4.8 --- CLAUDE.md | 7 +- data-loading/README.md | 214 ++++++++++++++++++++++------------------- 2 files changed, 120 insertions(+), 101 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 3cce0fb5..7f652130 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,7 +10,12 @@ NameRes (Name Resolver) is a biomedical entity name resolution service that maps ### Running Tests ```bash -# Load test data into Solr first (requires Solr running) +# Start standalone Solr and create the name_lookup core from the checked-in configset +docker run --name name_lookup -d -p 8983:8983 solr:9.10.0 +docker cp data-loading/configsets/name_lookup name_lookup:/tmp/name_lookup +docker exec name_lookup solr create -c name_lookup -d /tmp/name_lookup + +# Load test data into the core (parallel load, with a document-count guard) ./data-loading/setup-and-load-solr.sh tests/data/test-synonyms.json # Run all tests diff --git a/data-loading/README.md b/data-loading/README.md index 376c9e9f..161e3b1c 100644 --- a/data-loading/README.md +++ b/data-loading/README.md @@ -1,105 +1,119 @@ # Loading NameResolution data -NameResolution data needs to be loaded as a compressed [Apache Solr](https://solr.apache.org/) database. -To create this dataset is a three-step process: synonym data generated by Babel must first be loaded into -Solr, a Solr backup created, and then compressed with a particular directory structure to be used in NameRes. +NameResolution answers queries out of an [Apache Solr](https://solr.apache.org/) index. +This directory builds that index from the synonym files produced by +[Babel](https://github.com/NCATSTranslator/Babel) and packages it as a +**self-contained Solr backup** that a NameRes instance can download and serve directly. + +"Self-contained" is the important part: the backup is a whole Solr *core* -- +its configuration, its schema **and** its built index -- tarred up together. +Restoring it is just "untar into the Solr home and start Solr". There is no +ZooKeeper, no collection to create, and no schema to re-apply at restore time. +The schema lives in exactly one place, the checked-in configset at +[`configsets/name_lookup/conf`](configsets/name_lookup/conf). + +## How it works + +Solr runs in **standalone** mode (a single node, one core -- ZooKeeper adds +nothing for a one-shard job). The core is created from the checked-in configset +with `solr create -c name_lookup -d configsets/name_lookup`. Data is loaded by +streaming each Babel synonym file to Solr **in parallel**, with a single commit +at the end; there is no per-file commit and no sleeping. Finally the index is +optimized and the core directory is tarred into `snapshot.backup.tar.gz`. + +Because a parallel bulk load of a ~130 GB dataset would be painful to debug if +it silently dropped documents, `setup-and-load-solr.sh` **counts the input +documents before loading and compares against Solr's document count afterward**, +and every upload uses `curl --fail`. A failed upload or a count mismatch aborts +with a non-zero exit code. ## Using the Makefile -This directory includes a Makefile that can be used to run most of these steps -automatically. This is a seven-step process: - -1. Edit the Makefile to choose the directory containing Babel synonym files. Note that - all files in that directory will be used, and any files named `.txt.gz` will uncompressed. -2. Run `make all` to download the synonym data, uncompress Gzipped files, split the larger files - and delete the split files to avoid duplicate loading. `make all` will also start the Solr server -- - you can check this by looking for a PID file Solr in `data/solr.pid`. -3. (Optional) Access the Solr server and confirm that all the data has been loaded. -4. Run `make start-solr-backup` to start the Solr backup. -5. Run `make check-solr-backup` to check on the Solr backup. Look for `"status":"success"` to confirm that the - backup has completed. -6. Run `make data/backup.done` to move the backup into the `data/` directory, place it in the correct directory - structure for NameRes, and create a `snapshot.backup.tar.gz` file. -7. Copy the `snapshot.backup.tar.gz` file to a web server so that it can be loaded from NameRes. - -## Step-by-step instructions - -1. Set up a Solr server locally. The easiest way to do this is via Docker: - - ```shell - $ docker run -v "$PWD/data/solrdata:/var/solr" --name name_lookup -p 8983:8983 -t solr -cloud -p 8983 -m 12G - ``` - - You can adjust the `12G` to increase the amount of memory available to Solr. You can also add `-d` to the - Docker arguments if you would like to run this node in the background. - -2. Copy the synonym files into the `data/synonyms` directory. Synonym files that are too large will - need to split it into smaller files. (`gsplit` is the GNU version of `split`, which includes support - for adding an additional suffix to files) - - ```shell - $ gsplit -l 5000000 -d --additional-suffix .txt SmallMolecule.txt SmallMolecule - $ gsplit -l 5000000 -d --additional-suffix .txt MolecularMixture.txt MolecularMixture - ``` - -3. Download all the synonym text files into the `data/json` folder. You can download this by running `make`. - - ```shell - $ pip install -r requirements.txt - $ make - ``` - -4. Load the JSON files into the Solr database by running: - - ```shell - $ ./setup-and-load-solr.sh "data/json/*.json" - ``` - - Note the double-quotes: setup-and-load-solr.sh requires a glob pattern as its first argument, not a list of files to process! - -5. Generate a backup of the Solr instance. The first command will create a directory at - `solrdata/data/name_lookup_shard1_repical_n1/data/snapshot.backup` -- you can track its progress by comparing the - number of files in that directory to the number of files in `../data/index` (as I write this, it has 513 files). - - ```shell - $ curl 'http://localhost:8983/solr/name_lookup/replication?command=backup&name=backup' - $ curl 'http://localhost:8983/solr/name_lookup/replication?command=details' - ``` - - Once the backup is complete, you'll see a part of the `details` response that looks like this: - - ```json - "backup":{ - "startTime":"2022-09-13T18:42:43.678219123Z", - "fileCount":512, - "indexFileCount":512, - "status":"success", - "snapshotCompletedAt":"2022-09-13T19:36:00.599797304Z", - "endTime":"2022-09-13T19:36:00.599797304Z", - "snapshotName":"backup", - "directoryName":"snapshot.backup" - } - ``` - -6. Shutdown the Solr instance. - - ```shell - $ docker exec name_lookup solr stop -p 8983 -verbose - ``` - -7. Generate the backup tarball. At the moment, this is expected to be in the format - `var/solr/data/snapshot.backup/[index files]`. The easiest way to generate this tarball correctly is to run: - - ```shell - $ mkdir -p data/var/solr/data - $ mv /var/solr/name_lookup_shard1_replica_n1/data/snapshot.backup data/var/solr/data - $ cd data - $ tar zcvf snapshot.backup.tar.gz var - ``` - -8. Publish `snapshot.backup.tar.gz` to a publicly-accessible URL. - -9. Use the instructions at https://github.com/helxplatform/translator-devops/tree/develop/helm/name-lookup to set up an - instance of NameRes that downloads snapshot.backup.tar.gz from this publicly-accessible URL. - -The Makefile included in this directory contains targets for more of these steps. +The Makefile runs the whole pipeline. It expects `SOLR_DIR` (the Solr home, +usually `/var/solr`) and `SOLR_EXEC` (the `bin/solr` executable) to be set -- the +[`Dockerfile`](Dockerfile) sets both. + +1. Edit `SYNONYMS_URL` in the Makefile to point at the Babel synonyms directory + you want to load. +2. Run `make all`. This downloads and uncompresses the synonym files, splits any + file larger than `SPLIT_SIZE` (so the parallel loader has more units of work), + starts Solr, creates the `name_lookup` core from the configset, and loads the + data with the document-count guard. +3. (Optional) Query the Solr server at to confirm the + data looks right. +4. Run `make data/backup.done`. This optimizes the index, shuts down Solr and + writes `data/snapshot.backup.tar.gz`. +5. Publish `snapshot.backup.tar.gz` to a publicly-accessible URL and point a + NameRes instance at it (see below). + +Tuning knobs: `SOLR_MEM` (Solr heap for the load), `LOAD_PARALLELISM` +(concurrent uploads; defaults to the number of CPUs), `SPLIT_SIZE` / `SPLIT_LINES`. + +## Restoring the backup + +The backup extracts to a ready-to-serve `name_lookup/` core, so restoring it is +trivial in every environment: + +- **Locally (docker-compose):** extract the tarball into the Solr data directory + (`./data/solr` by default) so you get `./data/solr/name_lookup/`, then + `docker-compose up`. Solr auto-discovers the core on startup. +- **In Kubernetes (Helm):** the [`name-lookup` chart](../data/name-lookup) does + this automatically -- an init container downloads and extracts the backup into + the Solr volume, and the Solr StatefulSet auto-discovers the core. The restore + Job no longer creates a collection or re-applies the schema; its only remaining + job is deleting blocklisted CURIEs. + +The Solr version used to serve the backup must be **>=** the version that built +it (an older Solr cannot read a newer Lucene index). Both are 9.10.x today. + +## Options considered + +We looked at three approaches before settling on the current one. + +### A. Status quo (cloud mode, replication backup, schema re-applied on restore) +Solr ran in cloud mode (`-DzkRun`) everywhere. The backup contained only the +Lucene index, so the restore code had to recreate the collection and re-declare +every field/type/copy-field over the Schema API. **Rejected:** ZooKeeper is +overhead for one node; the schema was defined twice (loader *and* restore) and +had already drifted; the backup was not self-contained; and the load committed +after every file with a fixed `sleep` between files. + +### B. PR [#249](https://github.com/NCATSTranslator/NameResolution/pull/249) (standalone + config in the backup) +Switched to standalone mode and started shipping `solrconfig.xml` / +`managed-schema.xml` in the backup. A real improvement, but it still created the +core through the API and did not tackle load speed. **Superseded** by C. + +### C. Chosen: standalone + checked-in configset + self-contained core backup + parallel load +The backup is the whole core (config + schema + index), so restore is just +"untar and start". The schema has a single source of truth (the checked-in +configset). The load is parallel with a single deferred commit, guarded by a +document count. This is the most maintainable *and* the fastest option. + +Sub-choices within C: + +- **Parallel load vs. cheap wins only.** We parallelize (the real speed lever) + and protect against dropped/corrupt data with the pre/post document-count + check and `curl --fail`, rather than only removing the per-file commit. +- **`tar` the stopped core vs. the replication backup API.** We `tar` the core + directory after a clean commit -- simpler, and the result is self-contained. +- **Optimize before export.** We run `optimize=true` before tarring: one segment + makes the index smaller and faster to serve (fewer files to keep in the OS + page cache), at the cost of a one-time forced merge during the build. + +## Issues addressed + +- Closes [#238](https://github.com/NCATSTranslator/NameResolution/issues/238) and + [#185](https://github.com/NCATSTranslator/NameResolution/issues/185): the backup + now includes the config and schema, so restore no longer recreates fields. +- Closes [#256](https://github.com/NCATSTranslator/NameResolution/issues/256): + the index is optimized before the snapshot is exported. +- Closes [#266](https://github.com/NCATSTranslator/NameResolution/issues/266): + `queryResultCache` is sized in the checked-in `solrconfig.xml`. +- Addresses [translator-devops#609](https://github.com/helxplatform/translator-devops/issues/609): + the download init container is idempotent, so a pod restart no longer wipes a + good core and leaves Solr empty. +- Makes progress on [#265](https://github.com/NCATSTranslator/NameResolution/issues/265): + the Helm chart's Solr heap is lowered to leave room for the OS page cache, and + optimizing shrinks the index. Final heap/memory sizing and the GC-flag rework + ([#272](https://github.com/NCATSTranslator/NameResolution/issues/272)) still + need to be validated under query load and are tracked separately. From c7f6221d3bdd47f70d788d162a7b0a6756e99564 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 03:14:47 -0400 Subject: [PATCH 07/46] Apply suggestion from @gaurav --- .github/workflows/tester.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/tester.yml b/.github/workflows/tester.yml index ea4da134..8abdf741 100644 --- a/.github/workflows/tester.yml +++ b/.github/workflows/tester.yml @@ -22,7 +22,7 @@ jobs: with: python-version: ${{ matrix.python-version }} - - name: Run SOLR (standalone) + - name: Run Solr (standalone) run: docker run --name name_lookup -d -p 8983:8983 solr:9.10.0 - name: Create the name_lookup core from the checked-in configset From 5ba11789568e8f9db2924794e0d4f0248705bfc6 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 03:40:01 -0400 Subject: [PATCH 08/46] Fix /status when Solr runs in standalone mode /status looked for a core called name_lookup_shard1_replica_n1, which is the name SolrCloud gives it. In standalone mode the core is just name_lookup, so /status reported "Expected core not found." with no document counts -- and because the endpoint still answers 200, the Helm probes stayed green while it did. tests/test_status.py caught this. The core name now comes from SOLR_CORE (default name_lookup) and is used for the query URLs too, so there is one place to change it. If that name is not present but Solr reports exactly one core, we report on that one, so backups built by the old cloud pipeline keep working. Co-Authored-By: Claude Opus 4.8 --- api/server.py | 19 ++++++++++++++----- .../name_lookup/conf/configoverlay.json | 1 - 2 files changed, 14 insertions(+), 6 deletions(-) delete mode 100644 data-loading/configsets/name_lookup/conf/configoverlay.json diff --git a/api/server.py b/api/server.py index 67296bfe..45084f9a 100755 --- a/api/server.py +++ b/api/server.py @@ -22,6 +22,9 @@ SOLR_HOST = os.getenv("SOLR_HOST", "localhost") SOLR_PORT = os.getenv("SOLR_PORT", "8983") +# The Solr core to query. In standalone mode this is the core name; the cloud-mode +# backups we used to ship called it name_lookup_shard1_replica_n1 instead (see status()). +SOLR_CORE = os.getenv("SOLR_CORE", "name_lookup") app = FastAPI(**get_app_info()) logger = logging.getLogger(__name__) @@ -83,10 +86,16 @@ async def status() -> Dict: if 'version' in app_info and app_info['version']: nameres_version = 'v' + app_info['version'] - # We should have a status for name_lookup_shard1_replica_n1. - if 'status' in result and 'name_lookup_shard1_replica_n1' in result['status']: - core = result['status']['name_lookup_shard1_replica_n1'] + # We should have a status for our core. Standalone Solr calls it ${SOLR_CORE} + # (name_lookup); the older cloud-mode backups called it + # name_lookup_shard1_replica_n1. A NameRes Solr only ever has one core, so if the + # expected name isn't there but there is exactly one core, report on that one. + cores = result.get('status', {}) + core = cores.get(SOLR_CORE) + if core is None and len(cores) == 1: + core = next(iter(cores.values())) + if core is not None: index = {} if 'index' in core: index = core['index'] @@ -219,7 +228,7 @@ async def synonyms_post( async def name_lookup(curies) -> Dict[str, Dict]: """Returns a list of synonyms for a particular CURIE.""" time_start = time.time_ns() - query = f"http://{SOLR_HOST}:{SOLR_PORT}/solr/name_lookup/select" + query = f"http://{SOLR_HOST}:{SOLR_PORT}/solr/{SOLR_CORE}/select" curie_filter = " OR ".join( f"curie:\"{curie}\"" for curie in curies @@ -534,7 +543,7 @@ async def lookup(string: str, logger.debug(f"Query: {json.dumps(params, indent=2)}") time_solr_start = time.time_ns() - query_url = f"http://{SOLR_HOST}:{SOLR_PORT}/solr/name_lookup/select" + query_url = f"http://{SOLR_HOST}:{SOLR_PORT}/solr/{SOLR_CORE}/select" async with httpx.AsyncClient(timeout=None) as client: response = await client.post(query_url, json=params) if response.status_code >= 300: diff --git a/data-loading/configsets/name_lookup/conf/configoverlay.json b/data-loading/configsets/name_lookup/conf/configoverlay.json deleted file mode 100644 index 271cd0a4..00000000 --- a/data-loading/configsets/name_lookup/conf/configoverlay.json +++ /dev/null @@ -1 +0,0 @@ -{"userProps":{"update.autoCreateFields":"false"}} \ No newline at end of file From 6eb52a28471726222a5e4624cdb7679c59be9104 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 03:40:19 -0400 Subject: [PATCH 09/46] Harden the load guard: count deltas, bound the wait, survive odd filenames Three problems with the guard, in rough order of how much they would hurt: - It compared the input line count against Solr's *total* document count. Every load assigns fresh UUIDs, so loading is additive: a second load against a non-empty core would double the data and then blame the count. Compare the increase instead, and say so when the core is not empty. - The wait for Solr was unbounded, so a Solr that never starts hung the build (six hours in CI) instead of failing it. - The glob was expanded with default IFS, so a path containing a space split into two non-existent files, which nullglob then dropped -- the count would quietly come up short. Empty IFS during expansion fixes it. Blank lines are no longer counted as documents, since Solr ignores them. Also corrected a comment: --fail does not catch malformed JSON. Solr answers 200 and indexes nothing for a file of plain text; only the count notices. Co-Authored-By: Claude Opus 4.8 --- data-loading/setup-and-load-solr.sh | 77 ++++++++++++++++++++++------- 1 file changed, 60 insertions(+), 17 deletions(-) diff --git a/data-loading/setup-and-load-solr.sh b/data-loading/setup-and-load-solr.sh index 290df0a6..5b3f71d2 100755 --- a/data-loading/setup-and-load-solr.sh +++ b/data-loading/setup-and-load-solr.sh @@ -10,9 +10,14 @@ # commit is issued at the end. There is no per-file commit and no sleeping. # # Safety: because the load is parallel and the data is huge, we count the input -# documents (lines) before loading and compare against Solr's document count -# afterwards. Any failed upload (curl --fail) or a count mismatch aborts with a -# non-zero exit code, so a partial/dropped load cannot pass silently. +# documents (lines) before loading and compare against the *increase* in Solr's +# document count afterwards. Any failed upload (curl --fail) or a count mismatch +# aborts with a non-zero exit code, so a partial/dropped load cannot pass silently. +# +# The count is the important half of that: Solr rejects some bad input outright +# (an unknown field is a 400, which --fail catches), but it answers 200 and +# indexes nothing for other junk -- a file of plain text, for instance. Only the +# count notices that. set -uo pipefail @@ -20,37 +25,75 @@ set -uo pipefail SOLR_SERVER="${SOLR_SERVER:-http://localhost:8983}" CORE="${SOLR_CORE:-name_lookup}" PARALLELISM="${LOAD_PARALLELISM:-$(getconf _NPROCESSORS_ONLN 2>/dev/null || echo 4)}" +# How long to wait for Solr to come up, in 3-second increments. +STARTUP_TRIES="${SOLR_STARTUP_TRIES:-60}" GLOB="${1:?usage: setup-and-load-solr.sh \"data/synonyms/*.txt*\" (quote the glob!)}" -# Step 1. Wait for the core to be available. -until curl -sf "${SOLR_SERVER}/solr/${CORE}/admin/ping" >/dev/null 2>&1; do +# Number of documents currently in the core. +solr_count() { + curl -sf "${SOLR_SERVER}/solr/${CORE}/query?q=*:*&rows=0" \ + | grep -oE '"numFound":[0-9]+' | head -1 | grep -oE '[0-9]+' +} + +# Step 1. Wait for the core to be available (bounded: a Solr that never comes up +# should fail the build, not hang it). +ready="" +for _ in $(seq "${STARTUP_TRIES}"); do + if curl -sf "${SOLR_SERVER}/solr/${CORE}/admin/ping" >/dev/null 2>&1; then + ready="yes" + break + fi echo " -- Solr core '${CORE}' is unavailable - sleeping" sleep 3 done +if [ -z "$ready" ]; then + echo "Solr core '${CORE}' at ${SOLR_SERVER} did not come up after $((STARTUP_TRIES * 3))s. Aborting." >&2 + exit 1 +fi echo "Solr core '${CORE}' is up at ${SOLR_SERVER}." # Step 2. Expand the glob and count the input documents (one JSON doc per line). -# `grep -c ''` counts every line including a final one with no trailing newline, -# and we count per file so a missing newline never merges two files' records. +# Emptying IFS stops the shell from splitting the glob on spaces, so filenames +# containing spaces survive; pathname expansion still yields one word per file. +saved_ifs="$IFS" +IFS= shopt -s nullglob files=( $GLOB ) +shopt -u nullglob +IFS="$saved_ifs" if [ ${#files[@]} -eq 0 ]; then echo "No files matched '${GLOB}'." >&2 exit 1 fi +# `grep -c '[^[:space:]]'` counts every non-blank line, including a final one with +# no trailing newline. Blank lines are skipped because Solr ignores them, and we +# count per file so a missing newline never merges two files' records. This assumes +# one document per line -- pretty-printed JSON would be counted wrong. echo "Counting documents in ${#files[@]} file(s)..." expected=0 for f in "${files[@]}"; do - n=$(grep -c '' "$f") + n=$(grep -c '[^[:space:]]' "$f") echo " ${n} docs in ${f}" expected=$((expected + n)) done -echo "Expecting ${expected} documents in total." +echo "Expecting to add ${expected} documents in total." + +# The core may already contain documents (every load assigns fresh UUIDs, so +# loading is additive, never idempotent). Compare the delta, not the total. +before=$(solr_count) +if [ -z "${before}" ]; then + echo "Could not read the current document count from Solr. Aborting." >&2 + exit 1 +fi +if [ "${before}" -ne 0 ]; then + echo "NOTE: core '${CORE}' already contains ${before} documents; this load adds to them." +fi # Step 3. Load the files in parallel, streaming each one, without committing. # curl -T streams the file rather than buffering it in memory (issue #194), and -# --fail turns any HTTP >=400 (e.g. malformed JSON) into a non-zero exit. +# --fail turns any HTTP >=400 (e.g. an unknown field) into a non-zero exit. Input +# that Solr accepts but does not index is caught by the count check in step 5. load_one() { local f="$1" echo "Loading ${f}..." @@ -74,12 +117,12 @@ echo "Committing..." curl -sf --show-error "${SOLR_SERVER}/solr/${CORE}/update?commit=true" >/dev/null \ || { echo "Commit failed." >&2; exit 1; } -# Step 5. Verify the document count matches. -actual=$(curl -sf "${SOLR_SERVER}/solr/${CORE}/query?q=*:*&rows=0" \ - | grep -oE '"numFound":[0-9]+' | head -1 | grep -oE '[0-9]+') -echo "Solr reports ${actual} documents; expected ${expected}." -if [ "${actual:-0}" != "${expected}" ]; then - echo "DOCUMENT COUNT MISMATCH: loaded ${actual}, expected ${expected}. Aborting." >&2 +# Step 5. Verify that the number of documents added matches the input. +after=$(solr_count) +added=$(( ${after:-0} - before )) +echo "Solr reports ${after:-0} documents (${added} added); expected to add ${expected}." +if [ "${added}" != "${expected}" ]; then + echo "DOCUMENT COUNT MISMATCH: added ${added}, expected ${expected}. Aborting." >&2 exit 1 fi -echo "Load complete: ${actual} documents." +echo "Load complete: ${after} documents in '${CORE}'." From 6c7ffc0f451346049f61fccac77b10a9d0f710cd Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 03:40:19 -0400 Subject: [PATCH 10/46] Make the backup build restart-safe and stop gzipping on one core data/solr.pid is now an order-only prerequisite of core.done and backup.done. Optimizing needs a running Solr, and the stamp files outlive a restarted container while the server does not, so `make data/backup.done` after a restart used to curl into nothing. Order-only, because a restarted Solr must not make the core look stale: re-running `solr create` on an existing core is an error. The tarball is compressed with pigz when it is available (added to the image). A 130 GB Lucene index is already largely compressed and gzip is single-threaded, so this was hours of CPU for a few percent. Dropped tar -v, which wrote a line per index file into the build log, and added pipefail so a tar failure is not hidden by a successful gzip. Uncompressing now uses find -exec, so rerunning the download step after a partial run is a no-op rather than a "no such file" failure, and the wait for the PID file sleeps and gives up rather than spinning a CPU forever. Solr 9 wants Java 11 at a minimum but ships and tests on 17 (the official 9.10 image runs 17), so the loader image uses 17 as well. Co-Authored-By: Claude Opus 4.8 --- data-loading/Dockerfile | 12 +++++++++--- data-loading/Makefile | 38 ++++++++++++++++++++++++++++++++------ 2 files changed, 41 insertions(+), 9 deletions(-) diff --git a/data-loading/Dockerfile b/data-loading/Dockerfile index 5d9246b9..64cbbddd 100644 --- a/data-loading/Dockerfile +++ b/data-loading/Dockerfile @@ -15,7 +15,9 @@ FROM ghcr.io/translatorsri/renci-python-image:3.11.5 # Configuration options: # - ${ROOT} is where the source code will be copied. ARG ROOT=/code/nameres-data-loading -# - ${SOLR_VERSION} is the SOLR version to install. +# - ${SOLR_VERSION} is the SOLR version to install. This has to be an exact version +# (it names a release tarball), and it is the floor for every Solr that serves the +# resulting backup -- see "Restoring the backup" in README.md. # You can find the latest version at https://solr.apache.org/downloads.html ARG SOLR_VERSION=9.10.0 # - ${SOLR_DIR} is the SOLR directory to use. @@ -29,12 +31,16 @@ RUN apt -y upgrade RUN apt install -y python3-venv RUN pip3 install --upgrade pip -# We need Java 11 to run SOLR. -RUN apt install -y openjdk-11-jre +# Solr 9 needs Java 11 at a minimum, but is built and tested against 17; use 17. +RUN apt install -y openjdk-17-jre # SOLR uses lsof to check on its status. RUN apt install -y lsof +# pigz (parallel gzip) is used by the Makefile to compress the backup; gzip alone +# takes hours on a 130 GB index. +RUN apt install -y pigz + # The following packages are useful in debugging, but can be # removed once this container is working properly. RUN apt install -y htop diff --git a/data-loading/Makefile b/data-loading/Makefile index 93eaa0ef..df6f1e52 100644 --- a/data-loading/Makefile +++ b/data-loading/Makefile @@ -7,12 +7,22 @@ # The backup is the whole Solr core -- config, schema AND index -- so restoring # it is just "untar into the Solr home and start Solr" (see ../data-loading/README.md). +# Recipes below use pipefail, which /bin/sh (dash on Debian) does not support. +SHELL := /bin/bash + # Configuration SYNONYMS_URL=https://stars.renci.org/var/babel_outputs/2025sep1/synonyms/ -# How much memory should Solr use. +# How much memory should Solr use during the load. This is the heap of the +# *loading* Solr on a large build machine, not of the Solr that serves queries -- +# serving wants a small heap so the OS page cache can hold the index instead. SOLR_MEM=220G +# A 130 GB Lucene index is already largely compressed, and gzip is single-threaded, +# so compressing the backup with gzip takes hours for a few percent. Use pigz +# (parallel gzip, same format) when it is available. +GZIP_CMD := $(shell command -v pigz > /dev/null 2>&1 && echo pigz || echo gzip) + # The checked-in configset used to create the core (contains conf/). CONFIGSET=configsets/name_lookup @@ -46,7 +56,9 @@ clean: data/synonyms/done: mkdir -p data/synonyms wget -c -r -l1 -nd -P data/synonyms ${SYNONYMS_URL} - gunzip data/synonyms/*.txt.gz + # -exec (rather than a glob) so that a rerun where everything is already + # uncompressed is a no-op instead of a "no such file" failure. + find data/synonyms -maxdepth 1 -type f -name '*.txt.gz' -exec gunzip {} + echo Downloaded synonyms from ${SYNONYMS_URL} find data/synonyms -maxdepth 1 -type f -name '*.txt' -size +${SPLIT_SIZE} -print | while read -r f; do \ echo "Splitting large file $$f (larger than ${SPLIT_SIZE})..."; \ @@ -59,14 +71,25 @@ data/synonyms/done: data/solr.pid: mkdir -p ${SOLR_DIR}/logs ${SOLR_EXEC} start -p 8983 --user-managed -v -m ${SOLR_MEM} -s ${SOLR_DIR} >> ${SOLR_DIR}/logs/solr.txt 2>> ${SOLR_DIR}/logs/solr.err.txt - while [ ! -s $@ ]; do \ + # Bounded, with a sleep: a Solr that fails to start should stop the build rather + # than spin a CPU forever. + @tries=0; while [ ! -s $@ ]; do \ + tries=$$((tries + 1)); \ + if [ $$tries -gt 60 ]; then \ + echo "Solr did not start after two minutes; see ${SOLR_DIR}/logs/solr.err.txt" >&2; \ + exit 1; \ + fi; \ + sleep 2; \ ${SOLR_EXEC} status | grep -Po 'Solr process \K([0-9]+)' > $@; \ done $(info Solr started with PID file at $@) cat $@ # Step 3. Create the core from the checked-in configset (schema lives there). -data/core.done: data/solr.pid +# solr.pid is order-only (after the |): the core needs a running Solr, but restarting +# Solr must not make the core look stale -- re-running `solr create` on a core that +# already exists is an error. +data/core.done: | data/solr.pid ${SOLR_EXEC} create -c ${CORE} -d ${CONFIGSET} touch $@ @@ -78,14 +101,17 @@ data/setup.done: data/synonyms/done data/core.done # Step 5. Optimize the index, shut down Solr, and tar the whole core into a # self-contained backup. Extracting snapshot.backup.tar.gz into the Solr home # yields a ready-to-serve name_lookup/ core -- no restore-time schema setup. -data/backup.done: data/setup.done +# data/solr.pid is an order-only prerequisite because optimizing needs a *running* +# Solr: the stamp files survive a container restart, but the server does not, so a +# restarted container should start Solr again rather than fail here. +data/backup.done: data/setup.done | data/solr.pid @echo "Optimizing the index before export (this can take a while on a large index)..." curl -sf --show-error 'http://localhost:8983/solr/${CORE}/update?optimize=true' # The optimize above commits, so the on-disk index is consistent. `solr stop` # can exit non-zero under Docker even on a clean stop (SOLR-16463), so ignore it. ${SOLR_EXEC} stop -p 8983 || true rm -f data/solr.pid - tar -C ${SOLR_DIR} -czvf data/snapshot.backup.tar.gz ${CORE} + set -o pipefail; tar -C ${SOLR_DIR} -cf - ${CORE} | ${GZIP_CMD} > data/snapshot.backup.tar.gz touch $@ .PHONY: stop-solr From cb327584d6ee12fedb05b3fae2d83c3a23a4f38e Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 03:40:30 -0400 Subject: [PATCH 11/46] Keep the schema's single source of truth out of configoverlay.json configoverlay.json carried one setting, update.autoCreateFields=false. That file is written by Solr's Config API, so shipping it in the configset invites exactly the drift this configset exists to prevent. Set the default on the add-unknown-fields chain in solrconfig.xml instead -- same behaviour, verified by posting a document with an unknown field and getting a 400. Also dropped size/initialSize from queryResultCache: as the comment two lines above it in the same file says, both are ignored once maxRamMB is set. Co-Authored-By: Claude Opus 4.8 --- .../configsets/name_lookup/conf/solrconfig.xml | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/data-loading/configsets/name_lookup/conf/solrconfig.xml b/data-loading/configsets/name_lookup/conf/solrconfig.xml index fb171905..150a2ce9 100644 --- a/data-loading/configsets/name_lookup/conf/solrconfig.xml +++ b/data-loading/configsets/name_lookup/conf/solrconfig.xml @@ -363,11 +363,10 @@ maxRamMB - the maximum amount of RAM (in MB) that this cache is allowed to occupy --> - - + - + From fb83dbb0383de73265223032357f50543dcc6d30 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 03:40:30 -0400 Subject: [PATCH 12/46] Check the load guard and the backup roundtrip in CI The guard is what stands between us and a silently half-loaded 130 GB index, and the backup is the actual product of data-loading/, but neither was tested anywhere automated. Two steps, a few seconds each on the 89-document test core: - the loader must exit non-zero on a file that does not load - tar the core, drop it into a fresh Solr, and serve it: 89 documents with no collection creation and no schema setup The wait for Solr is also bounded now, and dumps the container log when it gives up, rather than hanging the job until the six-hour timeout. Co-Authored-By: Claude Opus 4.8 --- .github/workflows/tester.yml | 44 ++++++++++++++++++++++++++++++++++-- 1 file changed, 42 insertions(+), 2 deletions(-) diff --git a/.github/workflows/tester.yml b/.github/workflows/tester.yml index 8abdf741..c7a426a0 100644 --- a/.github/workflows/tester.yml +++ b/.github/workflows/tester.yml @@ -23,11 +23,16 @@ jobs: python-version: ${{ matrix.python-version }} - name: Run Solr (standalone) - run: docker run --name name_lookup -d -p 8983:8983 solr:9.10.0 + run: docker run --name name_lookup -d -p 8983:8983 solr:9.10 - name: Create the name_lookup core from the checked-in configset run: | - until curl -sf http://localhost:8983/solr/admin/cores?action=STATUS >/dev/null; do sleep 2; done + for _ in $(seq 60); do + curl -sf 'http://localhost:8983/solr/admin/cores?action=STATUS' > /dev/null && break + sleep 2 + done + curl -sf 'http://localhost:8983/solr/admin/cores?action=STATUS' > /dev/null \ + || { echo "Solr did not come up:"; docker logs name_lookup; exit 1; } docker cp data-loading/configsets/name_lookup name_lookup:/tmp/name_lookup docker exec name_lookup solr create -c name_lookup -d /tmp/name_lookup @@ -44,3 +49,38 @@ jobs: - name: Run the tests run: | python -m pytest tests/ + + # The loader's guard is what stands between us and a silently half-loaded + # 130 GB index, so check that a bad file actually stops it. + - name: The loader must abort on a file that does not load + run: | + echo 'this is not json' > /tmp/malformed.json + if ./data-loading/setup-and-load-solr.sh /tmp/malformed.json; then + echo "Expected the loader to fail on a malformed file, but it succeeded." + exit 1 + fi + echo "Loader aborted as expected." + + # The backup is the actual product of data-loading/, so check that a core can be + # tarred up, dropped into a fresh Solr, and served without any restore-time setup. + - name: Verify the backup roundtrip (tar the core, restore it into a fresh Solr) + run: | + docker exec name_lookup tar -C /var/solr/data -cf - name_lookup > snapshot.backup.tar + docker run --name name_lookup_restored -d -p 8984:8983 solr:9.10 + for _ in $(seq 60); do + curl -sf 'http://localhost:8984/solr/admin/cores?action=STATUS' > /dev/null && break + sleep 2 + done + docker cp snapshot.backup.tar name_lookup_restored:/tmp/snapshot.backup.tar + docker exec -u root name_lookup_restored tar -C /var/solr/data -xf /tmp/snapshot.backup.tar + docker exec -u root name_lookup_restored chown -R solr:solr /var/solr/data/name_lookup + docker restart name_lookup_restored + for _ in $(seq 60); do + curl -sf 'http://localhost:8984/solr/name_lookup/admin/ping' > /dev/null && break + sleep 2 + done + count=$(curl -sf 'http://localhost:8984/solr/name_lookup/query?q=*:*&rows=0' \ + | grep -oE '"numFound":[0-9]+' | head -1 | grep -oE '[0-9]+') + expected=$(grep -c '[^[:space:]]' tests/data/test-synonyms.json) + echo "Restored core reports ${count:-0} documents; expected ${expected}." + [ "${count:-0}" = "${expected}" ] From 4eec268b387f0ec9f07f1e93e4f1e905cd6e037a Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 03:50:39 -0400 Subject: [PATCH 13/46] Track the Solr 9.10 line when serving, and refresh the restore docs Pin by role. The builder has to name an exact release tarball, so SOLR_VERSION stays at 9.10.0, and it is the floor for anything that serves the backup. The servers -- docker-compose, CI, the Helm chart -- now track 9.10, so they pick up patch releases (any 9.10.x reads a 9.10.x index) and stay at or above the builder without anyone remembering to bump them. Deployment.md still told people to run solr-restore/restore.sh, which this branch deletes. Restoring is now "extract the backup into ./data/solr and start Solr". Fixed the mount description while there: ./data/solr is mounted at /var/solr/data, the Solr home, which is why the core is found at all. Co-Authored-By: Claude Opus 4.8 --- CLAUDE.md | 4 ++-- data-loading/README.md | 6 +++++- docker-compose.yml | 6 ++++-- documentation/Deployment.md | 40 ++++++++++++++++++++----------------- 4 files changed, 33 insertions(+), 23 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 7f652130..ec5676a8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -11,7 +11,7 @@ NameRes (Name Resolver) is a biomedical entity name resolution service that maps ### Running Tests ```bash # Start standalone Solr and create the name_lookup core from the checked-in configset -docker run --name name_lookup -d -p 8983:8983 solr:9.10.0 +docker run --name name_lookup -d -p 8983:8983 solr:9.10 docker cp data-loading/configsets/name_lookup name_lookup:/tmp/name_lookup docker exec name_lookup solr create -c name_lookup -d /tmp/name_lookup @@ -85,7 +85,7 @@ Solr documents contain: `curie`, `preferred_name`, `names` (synonym list), and b ### Infrastructure - **Stateless API container** - Python 3.11.5/FastAPI -- **Persistent Solr 9.1** - Data in volume-mounted `./data/solr` +- **Persistent Solr 9.10 (standalone)** - Data in volume-mounted `./data/solr` - **Data loading** - Separate pipeline in `data-loading/` (Makefile-driven, also has Kubernetes configs) - **CI/CD** - GitHub Actions: runs tests on push, publishes Docker image to GitHub Packages on release diff --git a/data-loading/README.md b/data-loading/README.md index 161e3b1c..45cf443b 100644 --- a/data-loading/README.md +++ b/data-loading/README.md @@ -64,7 +64,11 @@ trivial in every environment: job is deleting blocklisted CURIEs. The Solr version used to serve the backup must be **>=** the version that built -it (an older Solr cannot read a newer Lucene index). Both are 9.10.x today. +it (an older Solr cannot read a newer Lucene index). The builder is pinned exactly +(`SOLR_VERSION` in the [`Dockerfile`](Dockerfile), 9.10.0 today) because it names a +release tarball; the servers (`docker-compose.yml`, CI, the Helm chart) track the +`9.10` line, so they float forward onto patch releases and stay at or above the +builder without anyone having to remember to bump them. ## Options considered diff --git a/docker-compose.yml b/docker-compose.yml index d20d90a3..7c1c426c 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,8 +1,10 @@ services: nameres_solr: container_name: nameres_solr - # Must be >= the Solr version that built the backup (see data-loading/Dockerfile). - image: solr:9.10.0 + # Pinned to the 9.10 line rather than an exact patch: a serving Solr must be >= the + # Solr that built the backup (data-loading pins that exactly), and any 9.10.x can + # read a 9.10.x index, so floating within the line only picks up fixes. + image: solr:9.10 mem_limit: 18G environment: # Change this setting to control how much memory you would like your Solr setup to have. diff --git a/documentation/Deployment.md b/documentation/Deployment.md index d4ba344e..f44d95ce 100644 --- a/documentation/Deployment.md +++ b/documentation/Deployment.md @@ -23,9 +23,15 @@ instance or from Translator. Apache Solr databases. 3. Download the Solr backup URL you want to use and save it in `./data/solr`. It should be approximately 104G in size. -4. Uncompress the Solr backup file. It should produce a `var/solr/data/snapshot.backup` directory - in the Solr data (by default, `./data/solr/var/solr/data/snapshot.backup`). You can delete - the downloaded file (`snapshot.backup.tar.gz`) once it has been decompressed. +4. Uncompress the Solr backup file into the Solr data directory. Backups produced by + [data-loading](../data-loading/README.md) are whole Solr cores -- configuration, schema + and index -- so this produces a ready-to-serve `name_lookup/` directory (by default, + `./data/solr/name_lookup`). You can delete the downloaded file + (`snapshot.backup.tar.gz`) once it has been decompressed. + + ```shell + $ tar -C ./data/solr -xzvf snapshot.backup.tar.gz + ``` 5. Check the [docker-compose.yml](./docker-compose.yml) file to ensure that it is as you expect. * The Docker Compose file will use the latest released version of NameRes @@ -35,32 +41,29 @@ instance or from Translator. If you want to run many Solr queries, you might want to increase this. To do this, you will need to change BOTH the `mem_limit` setting in the `nameres_solr` service in `docker-compose.yml` and the `SOLR_JAVA_MEM` setting. - * The `docker-compose.yml` file also mounts the local `data/` directory into the Solr - container as `/var/solr`. This will allow you to start a new NameRes from the same + * The `docker-compose.yml` file also mounts the local `data/solr` directory into the Solr + container as `/var/solr/data` (the Solr home, which is where Solr looks for cores). + This will allow you to start a new NameRes from the same directory in the future. If you want to use a different directory, please change the `volumes` setting in the `nameres_solr` service in `docker-compose.yml`. Removing the binding will cause the Solr data to be stored in the Docker instance, and the data will be lost when the container is stopped. 6. Start the Solr and NameRes pods by running `docker compose up`. By default, Docker Compose will download and start the relevant pods and show you logs from both sources. You may - press `Ctrl+C` to stop the pods. -7. Trigger the Solr restore by running the restore script using `bash`, i.e. - `bash solr-restore/restore.sh`. This script assumes that the Solr pod is available on `localhost:8983` - and contains a `var/solr/data/snapshot.backup` directory with the data to restore. It will set up - some data types needed by NameRes and then triggering a restore of a backup. It will then go into a - sleep loop until the restore is complete, which should take 15-20 minutes. -8. Check that the script ended properly (`Solr restore complete!`). Look up http://localhost:2433/status - to ensure that the database has been loaded as expected. You can now delete the uncompressed database - backup in `$SOLR_DATA/var` to save disk space. -9. With the default settings, NameRes should be running on localhost on port 2433 (i.e. http://localhost:2433/). + press `Ctrl+C` to stop the pods. Solr discovers the `name_lookup` core on startup, so + there is no restore step: no collection to create and no schema to apply. +7. Look up http://localhost:2433/status to ensure that the database has been loaded as + expected -- `numDocs` should match the number of documents in the backup. +8. With the default settings, NameRes should be running on localhost on port 2433 (i.e. http://localhost:2433/). You should see a message in the NameRes pod log saying something like `Uvicorn running on http://0.0.0.0:2433 (Press CTRL+C to quit)` to confirm this. * By default, the web frontend (http://0.0.0.0:2433/docs) defaults to using the [NameRes RENCI Dev](https://name-resolution-sri.renci.org/docs) — you will need to change the "Servers" setting to use your local NameRes instance. - * If you try this before the restore has finished, looking up http://0.0.0.0:2433/status will give you an error - (`Expected core not found.`). This is because the Solr database and indexes have not yet been loaded. - Once this is finished, the NameRes instance should be ready to use. + * If http://0.0.0.0:2433/status reports an error (`Expected core not found.`), Solr has not + found the `name_lookup` core: check that the backup was extracted into the Solr data + directory (you should have a `name_lookup/core.properties` file in there) and that Solr + can read it. #### Loading from synonyms files @@ -93,6 +96,7 @@ curl -X POST "http://localhost:2433/lookup?string=oxycod&offset=0&limit=10" -H " NameRes can be configured by setting environmental variables: * `SOLR_HOST` and `SOLR_PORT`: Hostname and port for the Solr database containing NameRes information. +* `SOLR_CORE`: The Solr core to query (defaults to `name_lookup`). * `SERVER_NAME`: The name of this server (defaults to `infores:sri-name-resolver`) * `SERVER_ROOT`: The server root (defaults to `/`) * `MATURITY_VALUE`: How mature is this NameRes (defaults to `maturity`, e.g. `development`) From 5e86aacaedf390c4f9b6efaf871e57e9dfde2210 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 04:15:01 -0400 Subject: [PATCH 14/46] Activated publishing NameRes Loading on PR for testing. --- .github/workflows/release-nameres-loading.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/release-nameres-loading.yml b/.github/workflows/release-nameres-loading.yml index cc7e3753..8c3000fb 100644 --- a/.github/workflows/release-nameres-loading.yml +++ b/.github/workflows/release-nameres-loading.yml @@ -1,6 +1,7 @@ name: 'Release a new version of NameResolution Data Loading to Github Packages' on: + pull_request: release: types: [published] From 12f49c08e0993c450b1cea0758af14583ae88ea4 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 04:20:57 -0400 Subject: [PATCH 15/46] Size the load to the CPUs we actually have, and stop over-sizing the heap Two settings were being chosen from numbers that do not describe the container they run in. Parallelism came from getconf _NPROCESSORS_ONLN, and pigz defaulted to one thread per core. Both report the *node's* CPU count inside a container: in the loading pod, 8 CPUs' worth of quota on a 64-core node meant 64 parallel uploads and 64 compression threads, all throttled. available-cpus.sh reads the cgroup quota instead (v2 and v1, falling back to the node count when unlimited), and both the loader and pigz now use it, so raising the pod's cpu limit is picked up without a second setting to remember. SOLR_MEM was 220G, which `solr -m` turns into -Xms220G -Xmx220G -- committed, not a ceiling. A bulk load does not want that: Lucene buffers into ramBufferSizeMB (512 MB) and streams merges through the OS page cache, so the heap was taking memory away from the only thing that would have used it. It is now 31G, which also stays under the compressed-oops threshold, and the serving side already measured 11-13Gi of RSS against 111Gi of page cache for the same index. The tunables use ?= so a pod spec can override them from the environment. Co-Authored-By: Claude Opus 4.8 --- data-loading/Dockerfile | 1 + data-loading/Makefile | 33 +++++++++++++++++++--------- data-loading/available-cpus.sh | 34 +++++++++++++++++++++++++++++ data-loading/setup-and-load-solr.sh | 9 ++++++-- 4 files changed, 65 insertions(+), 12 deletions(-) create mode 100755 data-loading/available-cpus.sh diff --git a/data-loading/Dockerfile b/data-loading/Dockerfile index 64cbbddd..3808eccf 100644 --- a/data-loading/Dockerfile +++ b/data-loading/Dockerfile @@ -76,6 +76,7 @@ RUN pip3 install -r requirements.txt # Copy necessary files. The configset carries the Solr schema/config; the core is # created from it with `solr create -c name_lookup -d configsets/name_lookup`. COPY --chown=nru setup-and-load-solr.sh ${ROOT} +COPY --chown=nru available-cpus.sh ${ROOT} COPY --chown=nru README.md ${ROOT} COPY --chown=nru Makefile ${ROOT} COPY --chown=nru configsets ${ROOT}/configsets diff --git a/data-loading/Makefile b/data-loading/Makefile index df6f1e52..f4bf6a0e 100644 --- a/data-loading/Makefile +++ b/data-loading/Makefile @@ -10,18 +10,31 @@ # Recipes below use pipefail, which /bin/sh (dash on Debian) does not support. SHELL := /bin/bash -# Configuration -SYNONYMS_URL=https://stars.renci.org/var/babel_outputs/2025sep1/synonyms/ +# Configuration. These use ?= so that they can be overridden from the environment +# (e.g. from a Kubernetes pod spec) without editing this file. +SYNONYMS_URL ?= https://stars.renci.org/var/babel_outputs/2025sep1/synonyms/ -# How much memory should Solr use during the load. This is the heap of the -# *loading* Solr on a large build machine, not of the Solr that serves queries -- -# serving wants a small heap so the OS page cache can hold the index instead. -SOLR_MEM=220G +# Solr's heap during the load. `solr -m` sets BOTH -Xms and -Xmx, so this is +# committed, not just a ceiling. +# +# Indexing does not want a big heap: Lucene buffers documents in ramBufferSizeMB +# (512 MB, set in the configset) and streams merges through the OS page cache, so +# heap beyond a few GB is memory taken away from that cache. Staying under 32G also +# keeps compressed object pointers, which a larger heap silently gives up. +# See data-loading/kubernetes/README.md for how to check this against Grafana. +SOLR_MEM ?= 31G + +# How many CPUs we may actually use -- the cgroup limit in a container, not the +# node's core count. Used for the parallel compress below; setup-and-load-solr.sh +# works this out for itself. +NCPU := $(shell bash ./available-cpus.sh 2>/dev/null || getconf _NPROCESSORS_ONLN 2>/dev/null || echo 4) # A 130 GB Lucene index is already largely compressed, and gzip is single-threaded, # so compressing the backup with gzip takes hours for a few percent. Use pigz -# (parallel gzip, same format) when it is available. -GZIP_CMD := $(shell command -v pigz > /dev/null 2>&1 && echo pigz || echo gzip) +# (parallel gzip, same format) when it is available. -p keeps pigz inside our CPU +# quota; left alone it starts one thread per *node* core and spends the difference +# being throttled. +GZIP_CMD := $(shell command -v pigz > /dev/null 2>&1 && echo "pigz -p $(NCPU)" || echo gzip) # The checked-in configset used to create the core (contains conf/). CONFIGSET=configsets/name_lookup @@ -32,8 +45,8 @@ CORE=name_lookup # Split any synonym file larger than SPLIT_SIZE into chunks of SPLIT_LINES lines. # Splitting only helps parallelism, so we only bother with the big files; JSON is # one document per line, so we must split on line boundaries, never on bytes. -SPLIT_SIZE=2G -SPLIT_LINES=10000000 +SPLIT_SIZE ?= 2G +SPLIT_LINES ?= 10000000 # SOLR_DIR should be set up to point to the Solr data directory (usually /var/solr, # which is also the Solr home) and SOLR_EXEC should be set up to point to the Solr diff --git a/data-loading/available-cpus.sh b/data-loading/available-cpus.sh new file mode 100755 index 00000000..2298320e --- /dev/null +++ b/data-loading/available-cpus.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env bash +# +# Print the number of CPUs this process is actually allowed to use. +# +# getconf and nproc report the *node's* CPU count even inside a container with a +# CPU limit: in a pod limited to 8 CPUs on a 64-core node, both say 64. Sizing a +# thread pool from that number means 64 workers fighting over 8 CPUs' worth of +# quota, which is slower than 8 workers, not faster. So read the cgroup quota and +# only fall back to the node count when there isn't one. + +set -uo pipefail + +fallback=$(getconf _NPROCESSORS_ONLN 2>/dev/null || echo 4) + +if [ -r /sys/fs/cgroup/cpu.max ]; then + # cgroup v2: " ", where quota is the string "max" if unlimited. + read -r quota period < /sys/fs/cgroup/cpu.max +elif [ -r /sys/fs/cgroup/cpu/cpu.cfs_quota_us ]; then + # cgroup v1: separate files, quota is -1 if unlimited. + quota=$(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us) + period=$(cat /sys/fs/cgroup/cpu/cpu.cfs_period_us) +fi + +# Left to right, and [ stops at the first true test: "max" is never compared +# numerically, which would be an error. +if [ "${quota:-max}" = "max" ] || [ "${quota:-0}" -le 0 ] || [ "${period:-0}" -le 0 ]; then + echo "$fallback" + exit 0 +fi + +# Round down (a 1.5-CPU quota gets one worker), but never below one. +cpus=$((quota / period)) +[ "$cpus" -lt 1 ] && cpus=1 +echo "$cpus" diff --git a/data-loading/setup-and-load-solr.sh b/data-loading/setup-and-load-solr.sh index 5b3f71d2..a0ba607a 100755 --- a/data-loading/setup-and-load-solr.sh +++ b/data-loading/setup-and-load-solr.sh @@ -21,10 +21,15 @@ set -uo pipefail +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" + # Configuration (overridable from the environment). SOLR_SERVER="${SOLR_SERVER:-http://localhost:8983}" CORE="${SOLR_CORE:-name_lookup}" -PARALLELISM="${LOAD_PARALLELISM:-$(getconf _NPROCESSORS_ONLN 2>/dev/null || echo 4)}" +# available-cpus.sh reads the cgroup quota, so in a container this is the pod's CPU +# limit rather than the node's core count. See data-loading/kubernetes/README.md. +PARALLELISM="${LOAD_PARALLELISM:-$(bash "${SCRIPT_DIR}/available-cpus.sh" 2>/dev/null \ + || getconf _NPROCESSORS_ONLN 2>/dev/null || echo 4)}" # How long to wait for Solr to come up, in 3-second increments. STARTUP_TRIES="${SOLR_STARTUP_TRIES:-60}" @@ -51,7 +56,7 @@ if [ -z "$ready" ]; then echo "Solr core '${CORE}' at ${SOLR_SERVER} did not come up after $((STARTUP_TRIES * 3))s. Aborting." >&2 exit 1 fi -echo "Solr core '${CORE}' is up at ${SOLR_SERVER}." +echo "Solr core '${CORE}' is up at ${SOLR_SERVER}; loading with ${PARALLELISM} parallel uploads." # Step 2. Expand the glob and count the input documents (one JSON doc per line). # Emptying IFS stops the shell from splitting the glob on spaces, so filenames From 95b79ba837138ae4869b04be04f7762adbda76ae Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 04:20:57 -0400 Subject: [PATCH 16/46] Resize the data-loading pod and document how to tune it The pod asked for 220G/256G of memory and 6-8 CPUs, which was the right shape for the old serial loader and the old 220G heap: almost all of the memory went to a heap that could not use it, while the parallel loader is short of the CPU it now knows how to use. It asks for 96Gi and 16 CPUs instead, with requests == limits so a multi-hour load is not evicted partway through. The PVC comments described the pipeline as it was in 2022-2023. The Solr volume needs 2-3x the finished index because optimize writes the new segment before dropping the old ones, and the data volume no longer stages an uncompressed ~119G copy of the backup, since it is tarred straight out of the Solr volume. Added a README covering what actually costs time in a load (CPU while indexing, disk while merging and optimizing), how each resource should be sized, which Grafana panels answer "would more of this help", and what every knob does. Also replaced the keep-alive loop with sleep infinity and said why the pod overrides the image's entrypoint. Co-Authored-By: Claude Opus 4.8 --- data-loading/kubernetes/README.md | 130 ++++++++++++++++++ .../kubernetes/nameres-loading-data.k8s.yaml | 23 ++-- .../kubernetes/nameres-loading-solr.k8s.yaml | 16 ++- .../kubernetes/nameres-loading.k8s.yaml | 34 +++-- 4 files changed, 183 insertions(+), 20 deletions(-) create mode 100644 data-loading/kubernetes/README.md diff --git a/data-loading/kubernetes/README.md b/data-loading/kubernetes/README.md new file mode 100644 index 00000000..0c00b414 --- /dev/null +++ b/data-loading/kubernetes/README.md @@ -0,0 +1,130 @@ +# Loading NameRes data on Kubernetes + +These three files create a pod on a Kubernetes cluster big enough to load a whole +Babel release into Solr and produce the `snapshot.backup.tar.gz` that NameRes +instances restore. The pipeline itself is described in +[`../README.md`](../README.md); this file is about the resources it needs and how to +tell, from Grafana, whether it has the right ones. + +| File | What it is | +| --- | --- | +| `nameres-loading.k8s.yaml` | The pod. A workspace you exec into, not a job. | +| `nameres-loading-solr.k8s.yaml` | PVC mounted at `/var/solr` -- the Solr home, i.e. the index. | +| `nameres-loading-data.k8s.yaml` | PVC mounted at `data/` -- synonym files in, tarball out. | + +## Running a load + +```shell +$ kubectl apply -f nameres-loading-solr.k8s.yaml +$ kubectl apply -f nameres-loading-data.k8s.yaml +$ kubectl apply -f nameres-loading.k8s.yaml +$ kubectl exec -it nameres-loading -- /bin/bash + +# Inside the pod: +$ cd /code/nameres-data-loading +$ make all SYNONYMS_URL=https://stars.renci.org/var/babel_outputs//synonyms/ +$ make data/backup.done +``` + +`make all` downloads and splits the synonym files, starts Solr, creates the +`name_lookup` core from the checked-in configset and loads it. `make +data/backup.done` optimizes the index, stops Solr and writes +`data/snapshot.backup.tar.gz`. Copy that out (`kubectl cp`, or push it straight to +wherever `dataUrl` will point) and delete the pod and both PVCs. + +Run it under `screen` or `tmux`: a full load is hours long, and `kubectl exec` does +not survive a dropped connection. + +## What actually costs time + +In order, the load is: + +1. **Download** (`wget`), network-bound. One connection, ~130G. +2. **Index**, CPU-bound. Solr analyses every synonym as it arrives -- tokenizing and + lowercasing each name for `names` and again for the `names_exactish` copy field. + This is where the parallel uploads earn their keep. +3. **Merge**, I/O-bound, overlapping the above. +4. **Optimize**, I/O-bound. One forced merge of the entire index into a single + segment: read ~130G, write ~130G. +5. **Compress**, CPU-bound, parallelised with `pigz`. + +So the two levers are **CPU** (steps 2 and 5) and **disk throughput** (steps 3 and +4). Memory beyond Solr's heap only helps as page cache, and disk *space* buys +nothing but the absence of failure. + +## Sizing + +**CPU -- the main lever, and the one to try first.** The loader runs one upload per +CPU and `pigz` gets the same number, so raising `cpu` in the pod spec is picked up +automatically; there is no second setting to keep in sync. Note that both numbers +come from the cgroup *limit* via [`../available-cpus.sh`](../available-cpus.sh), not +from `nproc`: inside a container `nproc` reports the node's cores, so on a 64-core +node a pod limited to 8 CPUs would otherwise start 64 uploads and spend the +difference being throttled. If you need to override it, set `LOAD_PARALLELISM`. + +**Memory -- less than you would think.** `SOLR_MEM` (default `31G`) becomes +`solr -m`, which sets *both* `-Xms` and `-Xmx`, so it is committed rather than a +ceiling. Indexing does not want a large heap: Lucene buffers documents in +`ramBufferSizeMB` (512 MB, in the configset) and streams merges through the OS page +cache, so heap past a few GB is memory taken away from that cache. Staying under 32G +also keeps compressed object pointers, which a bigger heap silently gives up. The +pod's remaining memory is not wasted -- the kernel uses it to cache index files, and +page cache is charged to the container's limit, so the limit is what decides how much +of the index can be cached during merges. + +This is the same lesson as the serving side, where measurements showed 11-13Gi of +Solr RSS against 111Gi of page cache (see `solr.resources` in the chart's +`values.yaml`). + +**Disk space** is a floor, not a lever: + +- `/var/solr` needs **2-3x the finished index**, because `optimize=true` writes the + new single segment before deleting the old ones. 400Gi against a ~127Gi index is + about 3x; if the index passes ~140Gi, raise it. +- `data/` needs the uncompressed synonyms plus the tarball. It no longer needs room + for an uncompressed copy of the backup -- that staging step is gone. + +**Disk speed** is a lever, and possibly the biggest one. Both PVCs use +`storageClassName: basic`. If the cluster offers a faster (SSD/NVMe-backed) class, +the Solr PVC is the one that would benefit: it takes the whole write load of the +indexing run and then the read-and-write of the optimize. That is worth more than +any amount of tuning in the Makefile. + +## Watching a load in Grafana + +Take a look at these while a load is running; each one tells you what to change next +time. + +| What to look at | What it means | What to do | +| --- | --- | --- | +| **CPU throttling** (`container_cpu_cfs_throttled_seconds_total`, or the "throttling" panel) | Sustained throttling means the pod wants more CPU than its limit. | Raise `cpu`. The loader picks it up on its own. | +| **CPU usage vs. limit** | Flat at the limit during the load = CPU-bound, and more will help. Well below it = something else is the constraint, probably disk. | If it is below the limit, look at disk before adding CPU. | +| **Memory usage (RSS/WSS)** | This is roughly the JVM. It should sit near `SOLR_MEM` and be stable. | If it is far below `SOLR_MEM`, lower `SOLR_MEM` and give the memory back as cache. | +| **Memory usage (cache)** | Page cache: the index data the kernel is holding. Rising to fill the headroom is healthy. | If it is pinned at (limit - heap) for the whole run, more memory may speed up merges. | +| **Disk read/write throughput on the Solr PVC** | Flat-topped during merges and the optimize = saturated volume. | A faster storage class, not more CPU. | +| **Wall-clock time of each `make` step** | The logs in `data/logs/` are timestamped per step. | Tells you which of the five phases above to attack at all. | + +Two things worth checking specifically, since they are new: + +- **Does parallelism help linearly?** Note the load's wall-clock time and the CPU + usage at the current setting. If CPU sits at the limit and throttling is high, the + next load with more CPU should be faster in proportion. If it does not improve, + the bottleneck has moved to disk. +- **Is 31G of heap right?** Watch RSS during the load. If it plateaus well below 31G + (likely), drop `SOLR_MEM` further -- 16G is probably enough -- and let the page + cache have the difference. If Solr spends its time in GC or dies with an + OutOfMemoryError, raise it, but suspect something else first: a bulk load into a + single core should not need tens of gigabytes of live heap. + +## Knobs + +All of these can be set on the `make` command line, or as environment variables in +the pod spec (the Makefile uses `?=`). + +| Setting | Default | What it does | +| --- | --- | --- | +| `SYNONYMS_URL` | Babel 2025sep1 | Which Babel release to load. | +| `SOLR_MEM` | `31G` | Solr's heap during the load (`-Xms` and `-Xmx`). | +| `LOAD_PARALLELISM` | CPU limit | Concurrent uploads. Only set this to override the cgroup-derived default. | +| `SPLIT_SIZE` / `SPLIT_LINES` | `2G` / 10M | How the big synonym files are split. More, smaller chunks give the parallel loader a shorter tail at the end of the run; fewer, larger ones mean less `split` time up front. | +| `SOLR_STARTUP_TRIES` | 60 | How long the loader waits for Solr (3s each) before giving up. | diff --git a/data-loading/kubernetes/nameres-loading-data.k8s.yaml b/data-loading/kubernetes/nameres-loading-data.k8s.yaml index 9929c66c..17c0a405 100644 --- a/data-loading/kubernetes/nameres-loading-data.k8s.yaml +++ b/data-loading/kubernetes/nameres-loading-data.k8s.yaml @@ -1,13 +1,18 @@ -# Kubernetes file for setting up a PVC to use for nameres-Loading. -# nameres-loading-data is a directory for storing synonym files, -# the generated Solr back and its compressed version. +# Kubernetes file for setting up a PVC to use for nameres-loading. +# nameres-loading-data is the working directory (data/): the downloaded synonym +# files and the finished backup tarball. # -# As of 2023jun1, this directory needs to contain: -# - 129G of synonym files (in JSON) -# - 119G of snapshot.backup files moved here from Solr -# - 85G of snapshot.backup.tar.gz after compressing -# Therefore it needs to be a minimum of 350G. I'm going to set a -# size of 400G in case we need some extra space. +# It needs to hold, at once: +# - the uncompressed synonym files (129G as of Babel 2023jun1; check the file sizes +# in the SYNONYMS_URL directory listing for the release you are actually loading) +# - snapshot.backup.tar.gz (85G as of 2023jun1) +# +# It no longer needs room for an uncompressed copy of the backup. The old pipeline +# moved a ~119G snapshot.backup directory here before compressing it; the backup is +# now tarred straight out of the Solr volume into the tarball, which also means the +# two volumes are read and written in parallel rather than one after the other. +# +# 500Gi therefore leaves comfortable headroom for a couple of Babel releases' growth. apiVersion: v1 kind: PersistentVolumeClaim diff --git a/data-loading/kubernetes/nameres-loading-solr.k8s.yaml b/data-loading/kubernetes/nameres-loading-solr.k8s.yaml index 72ddc10a..f359292c 100644 --- a/data-loading/kubernetes/nameres-loading-solr.k8s.yaml +++ b/data-loading/kubernetes/nameres-loading-solr.k8s.yaml @@ -1,7 +1,17 @@ # Kubernetes file for setting up a PVC to use for nameres-loading. -# -# As of 2022dec11, this seems to come to 37G for files + 30G for snapshot.backup. -# I'm going to set the size to 150Gi so we have a bit of spare space if needed. +# +# This is the Solr home (/var/solr): it holds the name_lookup core, which is the +# index itself and, at the end, the thing that gets tarred into the backup. +# +# Size it at 3x the finished index, not 1x. `optimize=true` merges everything into +# one segment by writing the new segment before dropping the old ones, so peak usage +# during that step is roughly double the index, and Lucene's own guidance is to +# budget 2-3x. As of Babel 2025nov4 the index is ~127Gi, so 400Gi is about 3x and +# still the right order of magnitude -- but if the index passes ~140Gi, raise this. +# +# storageClassName is worth a thought: this volume takes the whole write load of the +# indexing run plus the read+write of the optimize, so a faster class here shortens +# the load more reliably than any tuning in the Makefile. See README.md. apiVersion: v1 kind: PersistentVolumeClaim diff --git a/data-loading/kubernetes/nameres-loading.k8s.yaml b/data-loading/kubernetes/nameres-loading.k8s.yaml index 9460c284..32667b37 100644 --- a/data-loading/kubernetes/nameres-loading.k8s.yaml +++ b/data-loading/kubernetes/nameres-loading.k8s.yaml @@ -1,4 +1,9 @@ -# Kubernetes file for Nameres-loading +# Kubernetes file for Nameres-loading. +# +# This pod is a workspace, not a job: it starts, does nothing, and waits for you to +# exec in and drive the load by hand (see README.md in this directory). The command +# below deliberately replaces the image's entrypoint, which would otherwise start a +# Solr of its own -- the Makefile starts Solr itself, using SOLR_MEM as its heap. apiVersion: v1 kind: Pod @@ -12,9 +17,8 @@ spec: - name: nameres-loading image: ghcr.io/ncatstranslator/nameresolution-data-loading:latest imagePullPolicy: Always - # I just need something to run while I figure out how to make this work - command: [ "/bin/bash", "-c", "--" ] - args: [ "while true; echo Running; do sleep 30; done;" ] + # Idle until someone execs in. `make all` is run by hand, not by the pod. + command: ["sleep", "infinity"] ports: - containerPort: 8983 volumeMounts: @@ -22,15 +26,29 @@ spec: name: nameres-loading-solr - mountPath: "/code/nameres-data-loading/data" name: nameres-loading-data + # Sized for the parallel loader. The load is CPU-bound (Solr analyses every + # synonym on the way in), then I/O-bound (merges and the final optimize), so CPU + # and disk speed are the levers; memory beyond the heap only helps as page cache. + # requests == limits keeps this Guaranteed: a load runs for hours and should not + # be evicted halfway through. + # + # The loader reads the CPU *limit* out of the cgroup and runs that many parallel + # uploads, and pigz is given the same number, so raising cpu here is picked up + # automatically -- there is no separate parallelism setting to remember. + # + # Memory: SOLR_MEM (31G by default) is the heap; the rest is JVM overhead and + # page cache for merges. This used to be 220G/256G with a 220G heap, which spent + # almost all of it in the one place a large heap does not help. See README.md + # before changing either number. resources: requests: ephemeral-storage: "1G" - memory: "220G" - cpu: "6" + memory: "96Gi" + cpu: "16" limits: ephemeral-storage: "1G" - memory: "256G" - cpu: "8" + memory: "96Gi" + cpu: "16" volumes: - name: nameres-loading-solr persistentVolumeClaim: From 1c82b3d5dc1c1b4b197b65e8648adfcce8f705f6 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 04:46:08 -0400 Subject: [PATCH 17/46] Keep each stamp file on the volume whose state it describes data/core.done and data/setup.done recorded facts about the Solr volume ("the core exists", "it has been loaded") but lived on the data volume. That was harmless while both were persistent PVCs with the same lifetime. It stops being harmless the moment the Solr volume becomes ephemeral: a recreated pod would find an empty index and a pair of stamps swearing it was loaded, skip straight to optimizing a core that no longer exists, and fail somewhere confusing. The three stamps that describe the Solr volume now live on it, so they vanish with it and make sees the truth. data/synonyms/done and data/backup.done stay where they are, because they describe the data volume. Logs move the other way, from the Solr volume to data/logs, since they are most useful exactly when the run has died and the Solr volume has gone with it. That includes Solr's own logs, via SOLR_LOGS_DIR. Co-Authored-By: Claude Opus 4.8 --- data-loading/Makefile | 46 ++++++++++++++++++++++++++++++------------- 1 file changed, 32 insertions(+), 14 deletions(-) diff --git a/data-loading/Makefile b/data-loading/Makefile index f4bf6a0e..1f08b7f8 100644 --- a/data-loading/Makefile +++ b/data-loading/Makefile @@ -51,18 +51,36 @@ SPLIT_LINES ?= 10000000 # SOLR_DIR should be set up to point to the Solr data directory (usually /var/solr, # which is also the Solr home) and SOLR_EXEC should be set up to point to the Solr # executable. These will both be set up by the Dockerfile. +SOLR_DIR ?= /var/solr + +# Stamp files live on the volume whose state they describe. These three describe +# the Solr volume -- a core exists, it has been loaded, a server is running -- and +# on Kubernetes that volume may be ephemeral NVMe (see kubernetes/README.md), so +# they have to disappear along with it. Keeping them in data/, which is a separate +# and persistent volume, would mean a recreated pod found an empty index and a set +# of stamps swearing it had been loaded. +SOLR_PID = ${SOLR_DIR}/solr.pid +CORE_DONE = ${SOLR_DIR}/core.done +SETUP_DONE = ${SOLR_DIR}/setup.done + +# Logs go to data/, which outlives the pod: they are most useful precisely when the +# run has died and the Solr volume has gone with it. +LOG_DIR = ${CURDIR}/data/logs # All and clean targets. .PHONY: all clean -all: data/setup.done +all: ${SETUP_DONE} echo Solr has now been set up and loaded with the synonym data. echo Run 'make data/backup.done' to optimize the index, shut down Solr and echo generate a snapshot.backup.tar.gz file that can be used in NameRes. +# Leaves the index itself alone -- to start over from an empty index, delete +# ${SOLR_DIR}/${CORE} (or, on Kubernetes, just recreate the pod). clean: rm -rf data/* mkdir data + rm -f ${SOLR_PID} ${CORE_DONE} ${SETUP_DONE} # Step 1. Download and uncompress synonym files, then split the large ones so the # parallel loader has more units of work. @@ -81,15 +99,15 @@ data/synonyms/done: # Step 2. Start the Solr server (standalone -- no ZooKeeper needed for one node). # --user-managed keeps standalone mode explicit (Solr 10 will default to cloud). -data/solr.pid: - mkdir -p ${SOLR_DIR}/logs - ${SOLR_EXEC} start -p 8983 --user-managed -v -m ${SOLR_MEM} -s ${SOLR_DIR} >> ${SOLR_DIR}/logs/solr.txt 2>> ${SOLR_DIR}/logs/solr.err.txt +${SOLR_PID}: + mkdir -p ${LOG_DIR} + SOLR_LOGS_DIR=${LOG_DIR} ${SOLR_EXEC} start -p 8983 --user-managed -v -m ${SOLR_MEM} -s ${SOLR_DIR} >> ${LOG_DIR}/solr.txt 2>> ${LOG_DIR}/solr.err.txt # Bounded, with a sleep: a Solr that fails to start should stop the build rather # than spin a CPU forever. @tries=0; while [ ! -s $@ ]; do \ tries=$$((tries + 1)); \ if [ $$tries -gt 60 ]; then \ - echo "Solr did not start after two minutes; see ${SOLR_DIR}/logs/solr.err.txt" >&2; \ + echo "Solr did not start after two minutes; see ${LOG_DIR}/solr.err.txt" >&2; \ exit 1; \ fi; \ sleep 2; \ @@ -102,33 +120,33 @@ data/solr.pid: # solr.pid is order-only (after the |): the core needs a running Solr, but restarting # Solr must not make the core look stale -- re-running `solr create` on a core that # already exists is an error. -data/core.done: | data/solr.pid +${CORE_DONE}: | ${SOLR_PID} ${SOLR_EXEC} create -c ${CORE} -d ${CONFIGSET} touch $@ # Step 4. Load the synonym files into the core (parallel; with a doc-count guard). -data/setup.done: data/synonyms/done data/core.done - mkdir -p data/logs - bash setup-and-load-solr.sh "data/synonyms/*.txt*" >> data/logs/setup-and-load-solr.sh.log 2>> data/logs/setup-and-load-solr.sh.err.log && touch $@ +${SETUP_DONE}: data/synonyms/done ${CORE_DONE} + mkdir -p ${LOG_DIR} + bash setup-and-load-solr.sh "data/synonyms/*.txt*" >> ${LOG_DIR}/setup-and-load-solr.sh.log 2>> ${LOG_DIR}/setup-and-load-solr.sh.err.log && touch $@ # Step 5. Optimize the index, shut down Solr, and tar the whole core into a # self-contained backup. Extracting snapshot.backup.tar.gz into the Solr home # yields a ready-to-serve name_lookup/ core -- no restore-time schema setup. -# data/solr.pid is an order-only prerequisite because optimizing needs a *running* -# Solr: the stamp files survive a container restart, but the server does not, so a +# The PID file is an order-only prerequisite because optimizing needs a *running* +# Solr: the stamps survive a container restart, but the server does not, so a # restarted container should start Solr again rather than fail here. -data/backup.done: data/setup.done | data/solr.pid +data/backup.done: ${SETUP_DONE} | ${SOLR_PID} @echo "Optimizing the index before export (this can take a while on a large index)..." curl -sf --show-error 'http://localhost:8983/solr/${CORE}/update?optimize=true' # The optimize above commits, so the on-disk index is consistent. `solr stop` # can exit non-zero under Docker even on a clean stop (SOLR-16463), so ignore it. ${SOLR_EXEC} stop -p 8983 || true - rm -f data/solr.pid + rm -f ${SOLR_PID} set -o pipefail; tar -C ${SOLR_DIR} -cf - ${CORE} | ${GZIP_CMD} > data/snapshot.backup.tar.gz touch $@ .PHONY: stop-solr stop-solr: - rm -f data/solr.pid + rm -f ${SOLR_PID} ${SOLR_EXEC} stop -p 8983 || true $(info Solr stopped.) From 541a913481ff690614c162626544805427daf2d7 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 04:46:14 -0400 Subject: [PATCH 18/46] Put the loading index on a node-local NVMe ephemeral volume The index is the volume that wants speed -- it takes the whole write load of the indexing run and then the read-and-write of the optimize -- and it is also the one volume we can afford to lose, since once the tarball exists the index is worthless. That makes it an exact fit for nvme-ephemeral: created with the pod, destroyed with it. data/ stays on a persistent PVC. It holds the ~130G download and the backup tarball, which is the only thing the whole exercise produces; that does not belong on a volume that dies with its pod. So deleting the pod now throws away the fast, rebuildable half and keeps the slow, irreplaceable half. fsGroup 1000 matches the image's nru user, without which the volume arrives root-owned and Solr cannot write to it. The old Solr PVC is kept as a documented fallback for when no node has enough local NVMe, since 400Gi of local disk is a real scheduling constraint in a way that network storage is not. README.md covers the split, the fallback, and what survives a pod restart. Co-Authored-By: Claude Opus 4.8 --- data-loading/kubernetes/README.md | 75 ++++++++++++++++--- .../kubernetes/nameres-loading-solr.k8s.yaml | 7 ++ .../kubernetes/nameres-loading.k8s.yaml | 37 ++++++++- 3 files changed, 105 insertions(+), 14 deletions(-) diff --git a/data-loading/kubernetes/README.md b/data-loading/kubernetes/README.md index 0c00b414..0be3671e 100644 --- a/data-loading/kubernetes/README.md +++ b/data-loading/kubernetes/README.md @@ -8,16 +8,39 @@ tell, from Grafana, whether it has the right ones. | File | What it is | | --- | --- | -| `nameres-loading.k8s.yaml` | The pod. A workspace you exec into, not a job. | -| `nameres-loading-solr.k8s.yaml` | PVC mounted at `/var/solr` -- the Solr home, i.e. the index. | +| `nameres-loading.k8s.yaml` | The pod. A workspace you exec into, not a job. Declares the NVMe volume inline. | | `nameres-loading-data.k8s.yaml` | PVC mounted at `data/` -- synonym files in, tarball out. | +| `nameres-loading-solr.k8s.yaml` | Fallback PVC for `/var/solr`, for when the NVMe volume does not fit. Normally unused. | + +## Two volumes, and why they differ + +The pod mounts two volumes, and the difference between them is the whole design: + +- **`/var/solr`, the index** is on a **node-local NVMe ephemeral volume**, declared + inline in the pod. It absorbs the entire write load of the indexing run and then + the read-and-write of the optimize, so it is the volume that benefits most from + being fast -- and it is pure intermediate state. Once the tarball exists, the index + is worthless. Losing it with the pod costs only the time to rebuild it from the + synonym files sitting next door. +- **`data/`, the working directory** stays on a **persistent PVC**. It holds the + ~130G download and, at the end, the backup tarball -- the only thing the whole + exercise actually produces. That does not belong on a volume that dies with its + pod. + +So deleting the pod throws away the fast, rebuildable half and keeps the slow, +irreplaceable half. That is deliberate. + +The Makefile knows about this split: the stamp files that record "the core exists" +and "the data is loaded" live on the Solr volume, not in `data/`, so a recreated pod +starts from a correct picture of an empty index rather than from stamps insisting the +work was already done. Logs go the other way, to `data/logs/`, because logs matter +most exactly when the run has died and the Solr volume has gone with it. ## Running a load ```shell -$ kubectl apply -f nameres-loading-solr.k8s.yaml $ kubectl apply -f nameres-loading-data.k8s.yaml -$ kubectl apply -f nameres-loading.k8s.yaml +$ kubectl apply -f nameres-loading.k8s.yaml # creates the NVMe volume with it $ kubectl exec -it nameres-loading -- /bin/bash # Inside the pod: @@ -30,11 +53,16 @@ $ make data/backup.done `name_lookup` core from the checked-in configset and loads it. `make data/backup.done` optimizes the index, stops Solr and writes `data/snapshot.backup.tar.gz`. Copy that out (`kubectl cp`, or push it straight to -wherever `dataUrl` will point) and delete the pod and both PVCs. +wherever `dataUrl` will point) **before deleting the pod**, then delete the pod and +the data PVC. The NVMe volume goes away with the pod on its own. Run it under `screen` or `tmux`: a full load is hours long, and `kubectl exec` does not survive a dropped connection. +If the pod dies partway through, recreate it and run the same commands. The synonym +files are still there and will not be downloaded again; the index will be rebuilt +from scratch, because it went with the NVMe volume. + ## What actually costs time In order, the load is: @@ -80,15 +108,38 @@ Solr RSS against 111Gi of page cache (see `solr.resources` in the chart's - `/var/solr` needs **2-3x the finished index**, because `optimize=true` writes the new single segment before deleting the old ones. 400Gi against a ~127Gi index is - about 3x; if the index passes ~140Gi, raise it. + about 3x; if the index passes ~140Gi, raise it. This now has to fit on a single + node's local NVMe, which is the one new constraint the fast volume brings. - `data/` needs the uncompressed synonyms plus the tarball. It no longer needs room for an uncompressed copy of the backup -- that staging step is gone. -**Disk speed** is a lever, and possibly the biggest one. Both PVCs use -`storageClassName: basic`. If the cluster offers a faster (SSD/NVMe-backed) class, -the Solr PVC is the one that would benefit: it takes the whole write load of the -indexing run and then the read-and-write of the optimize. That is worth more than -any amount of tuning in the Makefile. +**Disk speed** is a lever, and possibly the biggest one, which is why the index sits +on `storageClassName: nvme-ephemeral`. Steps 3 and 4 above are bounded by how fast +this volume can be written and re-read; no amount of Makefile tuning substitutes for +that. `data/` stays on `basic`, because it is written once by `wget` at network speed +and read once, sequentially, by the loader. + +### If the NVMe volume does not fit + +Local NVMe is a slice of a real disk on a real node, so a 400Gi request may not be +schedulable anywhere. If the pod stays `Pending` with a message about the volume, +either lower the request (it cannot go below roughly 2x the finished index) or fall +back to network storage: + +```shell +$ kubectl apply -f nameres-loading-solr.k8s.yaml +``` + +and in `nameres-loading.k8s.yaml`, replace the `ephemeral:` block on the +`nameres-loading-solr` volume with: + +```yaml + persistentVolumeClaim: + claimName: nameres-loading-solr +``` + +The load then works exactly as before, just slower through the merge and optimize +steps. Nothing else needs changing -- the stamp files follow `SOLR_DIR` either way. ## Watching a load in Grafana @@ -101,7 +152,7 @@ time. | **CPU usage vs. limit** | Flat at the limit during the load = CPU-bound, and more will help. Well below it = something else is the constraint, probably disk. | If it is below the limit, look at disk before adding CPU. | | **Memory usage (RSS/WSS)** | This is roughly the JVM. It should sit near `SOLR_MEM` and be stable. | If it is far below `SOLR_MEM`, lower `SOLR_MEM` and give the memory back as cache. | | **Memory usage (cache)** | Page cache: the index data the kernel is holding. Rising to fill the headroom is healthy. | If it is pinned at (limit - heap) for the whole run, more memory may speed up merges. | -| **Disk read/write throughput on the Solr PVC** | Flat-topped during merges and the optimize = saturated volume. | A faster storage class, not more CPU. | +| **Disk read/write throughput on the Solr volume** | Flat-topped during merges and the optimize = saturated volume. On NVMe this should no longer be what you are waiting for; if it still is, the CPU is not the constraint and neither is the heap. | Check the volume really is NVMe (`kubectl get pvc` while the pod runs, and look at the storage class). | | **Wall-clock time of each `make` step** | The logs in `data/logs/` are timestamped per step. | Tells you which of the five phases above to attack at all. | Two things worth checking specifically, since they are new: diff --git a/data-loading/kubernetes/nameres-loading-solr.k8s.yaml b/data-loading/kubernetes/nameres-loading-solr.k8s.yaml index f359292c..e3459204 100644 --- a/data-loading/kubernetes/nameres-loading-solr.k8s.yaml +++ b/data-loading/kubernetes/nameres-loading-solr.k8s.yaml @@ -1,5 +1,12 @@ # Kubernetes file for setting up a PVC to use for nameres-loading. # +# FALLBACK ONLY. The pod normally puts the Solr home on a node-local NVMe ephemeral +# volume, declared inline in nameres-loading.k8s.yaml -- it is faster and the index +# is disposable. Use this PVC instead only when no node has enough local NVMe for +# the whole index; see "If the NVMe volume does not fit" in README.md. Do not apply +# it otherwise, or you will reserve several hundred GB of network storage that +# nothing mounts. +# # This is the Solr home (/var/solr): it holds the name_lookup core, which is the # index itself and, at the end, the thing that gets tarred into the backup. # diff --git a/data-loading/kubernetes/nameres-loading.k8s.yaml b/data-loading/kubernetes/nameres-loading.k8s.yaml index 32667b37..a12466c9 100644 --- a/data-loading/kubernetes/nameres-loading.k8s.yaml +++ b/data-loading/kubernetes/nameres-loading.k8s.yaml @@ -4,6 +4,11 @@ # exec in and drive the load by hand (see README.md in this directory). The command # below deliberately replaces the image's entrypoint, which would otherwise start a # Solr of its own -- the Makefile starts Solr itself, using SOLR_MEM as its heap. +# +# The index lives on a node-local NVMe volume that is created and destroyed with the +# pod; the synonym files and the finished backup live on a persistent volume that +# does not. Deleting this pod therefore throws away the index and keeps the work +# that was expensive to fetch. See README.md. apiVersion: v1 kind: Pod @@ -49,10 +54,38 @@ spec: ephemeral-storage: "1G" memory: "96Gi" cpu: "16" + # The image runs as nru, uid/gid 1000. fsGroup makes the kubelet hand the NVMe + # volume over to that group; without it the volume is root-owned and Solr cannot + # write a thing. runAsUser only restates what the image already does. + securityContext: + runAsUser: 1000 + fsGroup: 1000 volumes: + # The index goes on a node-local NVMe volume. It is the right thing to put there + # twice over: it takes the whole write load of the indexing run and then the + # read-and-write of the optimize, so it is the volume that benefits most from the + # speed -- and it is pure intermediate state, so losing it with the pod costs + # nothing that cannot be rebuilt from the synonym files next door. + # + # This is a *generic ephemeral volume*: created with the pod and deleted with it. + # Not to be confused with the `ephemeral-storage` resource above, which is the + # node's own scratch space for the container filesystem. + # + # Sizing is the same 2-3x-the-index rule as before (optimize writes the new + # segment before dropping the old ones), so this has to fit on the node's local + # NVMe. If no node has that much, use nameres-loading-solr.k8s.yaml instead -- + # see README.md, "If the NVMe volume does not fit". - name: nameres-loading-solr - persistentVolumeClaim: - claimName: nameres-loading-solr + ephemeral: + volumeClaimTemplate: + spec: + accessModes: [ "ReadWriteOnce" ] + storageClassName: nvme-ephemeral + resources: + requests: + storage: 400Gi + # This one stays persistent: it holds the ~130G download and, at the end, the + # backup tarball, which is the only thing the whole exercise produces. - name: nameres-loading-data persistentVolumeClaim: claimName: nameres-loading-data From c43ca90087cc24e745906473ac978c85ba4753e7 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 04:48:30 -0400 Subject: [PATCH 19/46] Give the loading pod 32 CPUs and 128Gi The namespace has roughly 90 CPUs and 1000Gi spare, so 16/96Gi was leaving most of the available speed unused for a job that runs once and finishes. Stopping at 32 rather than taking the lot: the request has to fit on a single node, alongside the NVMe volume, and namespace quota says nothing about whether any one node can do that. Raising it further is a matter of checking node capacity -- the loader picks up the new CPU count on its own. Co-Authored-By: Claude Opus 4.8 --- data-loading/kubernetes/README.md | 7 ++++++- data-loading/kubernetes/nameres-loading.k8s.yaml | 12 ++++++++---- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/data-loading/kubernetes/README.md b/data-loading/kubernetes/README.md index 0be3671e..589f14f5 100644 --- a/data-loading/kubernetes/README.md +++ b/data-loading/kubernetes/README.md @@ -84,7 +84,11 @@ nothing but the absence of failure. **CPU -- the main lever, and the one to try first.** The loader runs one upload per CPU and `pigz` gets the same number, so raising `cpu` in the pod spec is picked up -automatically; there is no second setting to keep in sync. Note that both numbers +automatically; there is no second setting to keep in sync. The pod asks for 32, +which is deliberately short of what the namespace has spare: the real limit is that +the request has to fit on a **single node**, with the NVMe volume, at the same time. +Namespace quota is the easy test to pass and the wrong one to plan against -- check +`kubectl describe node` before raising this. Note that both numbers come from the cgroup *limit* via [`../available-cpus.sh`](../available-cpus.sh), not from `nproc`: inside a container `nproc` reports the node's cores, so on a 64-core node a pod limited to 8 CPUs would otherwise start 64 uploads and spend the @@ -178,4 +182,5 @@ the pod spec (the Makefile uses `?=`). | `SOLR_MEM` | `31G` | Solr's heap during the load (`-Xms` and `-Xmx`). | | `LOAD_PARALLELISM` | CPU limit | Concurrent uploads. Only set this to override the cgroup-derived default. | | `SPLIT_SIZE` / `SPLIT_LINES` | `2G` / 10M | How the big synonym files are split. More, smaller chunks give the parallel loader a shorter tail at the end of the run; fewer, larger ones mean less `split` time up front. | +| `ramBufferSizeMB` | 512 (in the configset) | Not an environment variable -- it lives in `solrconfig.xml`. With many parallel uploads, Lucene flushes a segment per indexing thread, so a bigger buffer means fewer and larger segments and less merging afterwards. Worth trying at 1-2G if the Grafana disk panels show the run still merging long after the uploads finish; there is heap to spare for it. | | `SOLR_STARTUP_TRIES` | 60 | How long the loader waits for Solr (3s each) before giving up. | diff --git a/data-loading/kubernetes/nameres-loading.k8s.yaml b/data-loading/kubernetes/nameres-loading.k8s.yaml index a12466c9..f59a90f3 100644 --- a/data-loading/kubernetes/nameres-loading.k8s.yaml +++ b/data-loading/kubernetes/nameres-loading.k8s.yaml @@ -45,15 +45,19 @@ spec: # page cache for merges. This used to be 220G/256G with a 220G heap, which spent # almost all of it in the one place a large heap does not help. See README.md # before changing either number. + # + # These have to fit on ONE node, which is a stricter test than the namespace + # quota -- and the same node has to have the NVMe volume free. Check node + # capacity before raising them, not just what the namespace has left. resources: requests: ephemeral-storage: "1G" - memory: "96Gi" - cpu: "16" + memory: "128Gi" + cpu: "32" limits: ephemeral-storage: "1G" - memory: "96Gi" - cpu: "16" + memory: "128Gi" + cpu: "32" # The image runs as nru, uid/gid 1000. fsGroup makes the kubelet hand the NVMe # volume over to that group; without it the volume is root-owned and Solr cannot # write a thing. runAsUser only restates what the image already does. From c4cad93cf080230b81e135c60396f4d6817901ca Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 04:51:32 -0400 Subject: [PATCH 20/46] Raise ramBufferSizeMB to 2G for the parallel load The buffer is a budget shared across indexing threads, so what it sets in practice is how large a segment grows before it is flushed: roughly the budget divided by the number of concurrent uploads. At 512 MB and 32 parallel uploads that is ~16 MB a segment; at 2 GB it is ~64 MB, so four times fewer segments for the merges to chew through afterwards. Safe at both ends. It is a budget rather than a reservation, so serving pods (which never index) allocate nothing from it, and 2 GB is a small fraction of the 31G load heap and well under Lucene's 1945 MB per-thread hard limit. Co-Authored-By: Claude Opus 4.8 --- .../configsets/name_lookup/conf/solrconfig.xml | 14 ++++++++++++-- data-loading/kubernetes/README.md | 4 ++-- 2 files changed, 14 insertions(+), 4 deletions(-) diff --git a/data-loading/configsets/name_lookup/conf/solrconfig.xml b/data-loading/configsets/name_lookup/conf/solrconfig.xml index 150a2ce9..22e8a971 100644 --- a/data-loading/configsets/name_lookup/conf/solrconfig.xml +++ b/data-loading/configsets/name_lookup/conf/solrconfig.xml @@ -104,8 +104,18 @@ before flushing. If both ramBufferSizeMB and maxBufferedDocs is set, then Lucene will flush based on whichever limit is hit first. --> - - 512 + + 2048 - + + + + id + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + - + + - - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + - \ No newline at end of file + + diff --git a/data-loading/configsets/name_lookup/conf/protwords.txt b/data-loading/configsets/name_lookup/conf/protwords.txt deleted file mode 100644 index 1dfc0abe..00000000 --- a/data-loading/configsets/name_lookup/conf/protwords.txt +++ /dev/null @@ -1,21 +0,0 @@ -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -#----------------------------------------------------------------------- -# Use a protected word file to protect against the stemmer reducing two -# unrelated words to the same base word. - -# Some non-words that normally won't be encountered, -# just to test that they won't be stemmed. -dontstems -zwhacky - diff --git a/data-loading/configsets/name_lookup/conf/solrconfig.xml b/data-loading/configsets/name_lookup/conf/solrconfig.xml index 22e8a971..39f2c5d5 100644 --- a/data-loading/configsets/name_lookup/conf/solrconfig.xml +++ b/data-loading/configsets/name_lookup/conf/solrconfig.xml @@ -1,111 +1,50 @@ + This ships inside the backup along with the index, so it is the configuration that + serves production queries as well as the one used to build the index. - - - + 9.12 - - ${solr.data.dir:} - - - + + false + managed-schema.xml + - One can force a particular implementation via solr.MMapDirectoryFactory - or solr.NIOFSDirectoryFactory. - - solr.RAMDirectoryFactory is memory based and not persistent. - --> - + - - - - - - - - - - 2048 - - - - - - - - - - - - - ${solr.lock.type:native} - - - - - - - - - - - - - - - - + ${solr.ulog.dir:} - + ${solr.autoCommit.maxTime:15000} false - - - + ${solr.autoSoftCommit.maxTime:-1} - - - - - - - + - - ${solr.max.booleanClauses:1024} - - - ${solr.query.minPrefixLength:-1} - - - - - - - - - - - - - - - - - - - - + - Example of a generic cache. These caches may be accessed by - name through SolrIndexSearcher.getCache(),cacheLookup(), and - cacheInsert(). The purpose is to enable easy caching of - user/application level data. The regenerator argument should - be specified as an implementation of solr.CacheRegenerator - if autowarming is desired. - --> - + - - - true - - - - - - 20 - - - 200 - - - - - - - - - - - - - - - + false - - - - - - - - - - + - - - + explicit @@ -632,7 +106,9 @@ - + explicit @@ -641,405 +117,14 @@ - - - - _text_ - - - - - - - text_general - - - - - - default - _text_ - solr.DirectSolrSpellChecker - - internal - - 0.5 - - 2 - - 1 - - 5 - - 4 - - 0.01 - - - - - - - - - - - - default - on - true - 10 - 5 - 5 - true - true - 10 - 5 - - - spellcheck - - - - - - - - - - - - 100 - - + - - - - 70 - - 0.5 - - [-\w ,/\n\"']{20,200} - - - - - - - ]]> - ]]> - - - - - - - - - - - - - - - - - - - - - - - - ,, - ,, - ,, - ,, - ,]]> - ]]> - - - - - - 10 - .,!? - - - - - - - WORD - - - en - US - - - - - - - - + This is the only update processor we declare. Solr's _default configset also + ships an "add-unknown-fields-to-the-schema" chain that invents fields for + unrecognised keys; we deliberately do not, so an unexpected key in a Babel file + is an error rather than a silent schema change. --> - - - [^\w-\.] - _ - - - 1000 - true - - - - - - - yyyy-MM-dd['T'[HH:mm[:ss[.SSS]][z - yyyy-MM-dd['T'[HH:mm[:ss[,SSS]][z - yyyy-MM-dd HH:mm[:ss[.SSS]][z - yyyy-MM-dd HH:mm[:ss[,SSS]][z - [EEE, ]dd MMM yyyy HH:mm[:ss] z - EEEE, dd-MMM-yy HH:mm:ss z - EEE MMM ppd HH:mm:ss [z ]yyyy - - - - - java.lang.String - text_general - - *_str - 256 - - - true - - - java.lang.Boolean - booleans - - - java.util.Date - pdates - - - java.lang.Long - java.lang.Integer - plongs - - - java.lang.Number - pdoubles - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/data-loading/configsets/name_lookup/conf/stopwords.txt b/data-loading/configsets/name_lookup/conf/stopwords.txt deleted file mode 100644 index ae1e83ee..00000000 --- a/data-loading/configsets/name_lookup/conf/stopwords.txt +++ /dev/null @@ -1,14 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. diff --git a/data-loading/configsets/name_lookup/conf/synonyms.txt b/data-loading/configsets/name_lookup/conf/synonyms.txt deleted file mode 100644 index eab4ee87..00000000 --- a/data-loading/configsets/name_lookup/conf/synonyms.txt +++ /dev/null @@ -1,29 +0,0 @@ -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -#----------------------------------------------------------------------- -#some test synonym mappings unlikely to appear in real input text -aaafoo => aaabar -bbbfoo => bbbfoo bbbbar -cccfoo => cccbar cccbaz -fooaaa,baraaa,bazaaa - -# Some synonym groups specific to this example -GB,gib,gigabyte,gigabytes -MB,mib,megabyte,megabytes -Television, Televisions, TV, TVs -#notice we use "gib" instead of "GiB" so any WordDelimiterGraphFilter coming -#after us won't split it into two words. - -# Synonym mappings can be used for spelling correction too -pixima => pixma - From a9387ac398d7b41b989a3b89351ca3bc4c50d12a Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 06:08:44 -0400 Subject: [PATCH 26/46] Take Solr from the official image instead of downloading it The build fetched Solr through https://www.apache.org/dyn/closer.lua, which redirects to a randomly chosen Apache mirror. Mirrors range from fast to barely moving, and ADD-from-a-URL has no timeout, no retry and no resume, so a bad draw hung the build until the job gave up -- 25 minutes on one run, still going at 18 on another, against 2-3 minutes historically. Same Dockerfile every time; the only variable was which mirror answered. Nothing verified the download either. Copying /opt/solr out of the official image removes the whole failure mode: it arrives over a CDN, the daemon verifies it by digest, and it caches like any other layer. The JDK comes with it, so the openjdk-17-jre install goes too, and there is no 250MB tarball to extract. The Solr that builds a backup is now bit-for-bit the Solr that serves it. The stage is pinned to linux/amd64 to match the RENCI base image, which is published for amd64 only. Without that, a build on an arm64 machine resolves this stage to arm64 while the final stage falls back to amd64, and every java call fails with "No such file or directory" -- found the hard way. Co-Authored-By: Claude Opus 4.8 --- data-loading/Dockerfile | 46 ++++++++++++++++++++++++++++------------- 1 file changed, 32 insertions(+), 14 deletions(-) diff --git a/data-loading/Dockerfile b/data-loading/Dockerfile index 3808eccf..2fd1622f 100644 --- a/data-loading/Dockerfile +++ b/data-loading/Dockerfile @@ -7,6 +7,29 @@ # # [1] https://github.com/helxplatform/translator-devops/blob/affcf34cf103230d25bdb859098d2a5ac81a49fb/helm/name-lookup/templates/scripts-config-map.yaml#L8-L105 +# Solr is lifted out of the official Solr image rather than downloaded. +# +# It used to be fetched with `ADD https://www.apache.org/dyn/closer.lua/...`, which +# redirects to a randomly chosen Apache mirror. Mirrors vary from fast to barely +# moving, and ADD-from-a-URL has no timeout, no retry and no resume, so a bad draw +# hung the build until the job timed out. Nothing verified the download either. +# +# Pulling the official image instead removes all of that: it comes over a CDN, the +# daemon verifies it by digest, and it is cached like any other layer. It also means +# the Solr that builds a backup is bit-for-bit the Solr that serves it. +# +# Pinned exactly, on purpose: this version is the floor for every Solr that serves the +# resulting backup, while the servers track the 9.10 line and float up. See +# "Restoring the backup" in README.md. Latest at https://solr.apache.org/downloads.html +# --platform is pinned because the RENCI base image below is published for amd64 only, +# so that is the only architecture this image can be. Without it, a build on an arm64 +# machine resolves this stage to arm64 (solr is multi-arch) while the final stage falls +# back to amd64, and the copied JDK is silently the wrong architecture: every java call +# then fails with "No such file or directory", which is the dynamic loader's unhelpful +# way of saying the ELF header is foreign. +ARG SOLR_VERSION=9.10.1 +FROM --platform=linux/amd64 solr:${SOLR_VERSION} AS solr + # Use the RENCI Python image to make it easier to work with other # RENCI Docker packages and to make sure we have an up to date image. # (https://github.com/TranslatorSRI/RENCI-Python-image) @@ -15,11 +38,6 @@ FROM ghcr.io/translatorsri/renci-python-image:3.11.5 # Configuration options: # - ${ROOT} is where the source code will be copied. ARG ROOT=/code/nameres-data-loading -# - ${SOLR_VERSION} is the SOLR version to install. This has to be an exact version -# (it names a release tarball), and it is the floor for every Solr that serves the -# resulting backup -- see "Restoring the backup" in README.md. -# You can find the latest version at https://solr.apache.org/downloads.html -ARG SOLR_VERSION=9.10.0 # - ${SOLR_DIR} is the SOLR directory to use. ARG SOLR_DIR=/var/solr @@ -31,10 +49,7 @@ RUN apt -y upgrade RUN apt install -y python3-venv RUN pip3 install --upgrade pip -# Solr 9 needs Java 11 at a minimum, but is built and tested against 17; use 17. -RUN apt install -y openjdk-17-jre - -# SOLR uses lsof to check on its status. +# SOLR uses lsof to check on its status. (It also needs ps, which the base image has.) RUN apt install -y lsof # pigz (parallel gzip) is used by the Makefile to compress the backup; gzip alone @@ -59,10 +74,13 @@ RUN chown nru:nru ${ROOT} WORKDIR ${ROOT} USER nru -# Download Solr into the Solr directory. -RUN mkdir -p "${ROOT}/solr" -ADD --chown=nru:nru https://www.apache.org/dyn/closer.lua/solr/solr/${SOLR_VERSION}/solr-${SOLR_VERSION}.tgz?action=download "${ROOT}/solr" -RUN tar zxvf "${ROOT}/solr/solr-${SOLR_VERSION}.tgz" --directory "${ROOT}/solr" +# Solr itself, and the JDK it ships with, taken straight from the official image. +# Owned by nru so `bin/solr` can write its own logs when SOLR_LOGS_DIR is not set +# (the Makefile does set it; the ENTRYPOINT below does not). +COPY --from=solr --chown=nru:nru /opt/solr /opt/solr +COPY --from=solr /opt/java /opt/java +ENV JAVA_HOME=/opt/java/openjdk +ENV PATH="${JAVA_HOME}/bin:${PATH}" # Set up VENV. ENV VIRTUAL_ENV=${ROOT}/venv @@ -82,6 +100,6 @@ COPY --chown=nru Makefile ${ROOT} COPY --chown=nru configsets ${ROOT}/configsets # On entry, start the Solr instance (standalone -- no ZooKeeper for a one-node load). -ENV SOLR_EXEC="${ROOT}/solr/solr-${SOLR_VERSION}/bin/solr" +ENV SOLR_EXEC="/opt/solr/bin/solr" ENV SOLR_DIR="$SOLR_DIR" ENTRYPOINT ${SOLR_EXEC} start -f --user-managed -p 8983 -m 64G -s ${SOLR_DIR} From 0ead5979b4a3186ccdcb5f92a67cfac396f3106f Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 06:23:07 -0400 Subject: [PATCH 27/46] Fix two bugs that would have failed the load at step 2 and step 3 Building the image locally and running `make all` inside it -- which nothing had done before, since CI exercises the loader against a stock Solr rather than through the Makefile -- turned up two failures, both fatal to a real load. `solr create -d configsets/name_lookup` cannot work: a *relative* -d is resolved against Solr's own server/solr/configsets directory, not the working directory, so core creation died with "Can't find resource 'solrconfig.xml'". CONFIGSET is now absolute via $(CURDIR). The start target waited for `solr status` to print a PID. In a container Solr reports "No Solr nodes are running" while serving happily on 8983 -- the stock solr image does the same, so this is not something about our image -- and the loop ran to its limit and failed. Nothing ever used that PID: stop-solr stops Solr by port, and the file is only a make stamp. It now waits for the admin endpoint to answer, which is the actual question being asked, and the stamp is renamed solr.started rather than solr.pid because it no longer holds a PID. Verified in the built image: make all loads 89 documents, make data/backup.done optimizes and writes a tarball through pigz, and that tarball restores into a stock Solr and serves all 89 with /status reporting ok. Co-Authored-By: Claude Opus 4.8 --- data-loading/Makefile | 34 ++++++++++++++++++++-------------- 1 file changed, 20 insertions(+), 14 deletions(-) diff --git a/data-loading/Makefile b/data-loading/Makefile index 2e14d678..80dab127 100644 --- a/data-loading/Makefile +++ b/data-loading/Makefile @@ -37,7 +37,10 @@ NCPU := $(shell bash ./available-cpus.sh 2>/dev/null || getconf _NPROCESSORS_ONL GZIP_CMD := $(shell command -v pigz > /dev/null 2>&1 && echo "pigz -p $(NCPU)" || echo gzip) # The checked-in configset used to create the core (contains conf/). -CONFIGSET=configsets/name_lookup +# Absolute, and it has to be: `solr create -d` resolves a *relative* path against +# Solr's own server/solr/configsets directory, not the working directory, so a +# relative path here fails with "Can't find resource 'solrconfig.xml'". +CONFIGSET ?= $(CURDIR)/configsets/name_lookup # The Solr core name. Do not change without also changing NameRes and the configset. CORE=name_lookup @@ -59,7 +62,7 @@ SOLR_DIR ?= /var/solr # they have to disappear along with it. Keeping them in data/, which is a separate # and persistent volume, would mean a recreated pod found an empty index and a set # of stamps swearing it had been loaded. -SOLR_PID = ${SOLR_DIR}/solr.pid +SOLR_STARTED = ${SOLR_DIR}/solr.started CORE_DONE = ${SOLR_DIR}/core.done SETUP_DONE = ${SOLR_DIR}/setup.done @@ -80,7 +83,7 @@ all: ${SETUP_DONE} clean: rm -rf data/* mkdir data - rm -f ${SOLR_PID} ${CORE_DONE} ${SETUP_DONE} + rm -f ${SOLR_STARTED} ${CORE_DONE} ${SETUP_DONE} # Step 1. Download and uncompress synonym files, then split the large ones so the # parallel loader has more units of work. @@ -99,28 +102,31 @@ data/synonyms/done: # Step 2. Start the Solr server (standalone -- no ZooKeeper needed for one node). # --user-managed keeps standalone mode explicit (Solr 10 will default to cloud). -${SOLR_PID}: +${SOLR_STARTED}: mkdir -p ${LOG_DIR} SOLR_LOGS_DIR=${LOG_DIR} ${SOLR_EXEC} start -p 8983 --user-managed -v -m ${SOLR_MEM} -s ${SOLR_DIR} >> ${LOG_DIR}/solr.txt 2>> ${LOG_DIR}/solr.err.txt - # Bounded, with a sleep: a Solr that fails to start should stop the build rather - # than spin a CPU forever. - @tries=0; while [ ! -s $@ ]; do \ + # Wait for Solr to actually answer, then stamp. This used to parse `solr status` + # for a PID, which reports "No Solr nodes are running" against a perfectly healthy + # Solr in a container -- the stock solr image does it too -- so the loop never + # terminated. Nothing ever used the PID (stop-solr stops Solr by port), so asking + # the admin endpoint whether Solr is up is both simpler and the actual question. + # Bounded, so a Solr that fails to start stops the build instead of hanging it. + @tries=0; until curl -sf 'http://localhost:8983/solr/admin/cores?action=STATUS' > /dev/null 2>&1; do \ tries=$$((tries + 1)); \ if [ $$tries -gt 60 ]; then \ echo "Solr did not start after two minutes; see ${LOG_DIR}/solr.err.txt" >&2; \ exit 1; \ fi; \ sleep 2; \ - ${SOLR_EXEC} status | grep -Po 'Solr process \K([0-9]+)' > $@; \ done - $(info Solr started with PID file at $@) - cat $@ + @date -u '+Solr started at %Y-%m-%dT%H:%M:%SZ on port 8983' > $@ + @cat $@ # Step 3. Create the core from the checked-in configset (schema lives there). # solr.pid is order-only (after the |): the core needs a running Solr, but restarting # Solr must not make the core look stale -- re-running `solr create` on a core that # already exists is an error. -${CORE_DONE}: | ${SOLR_PID} +${CORE_DONE}: | ${SOLR_STARTED} ${SOLR_EXEC} create -c ${CORE} -d ${CONFIGSET} touch $@ @@ -135,18 +141,18 @@ ${SETUP_DONE}: data/synonyms/done ${CORE_DONE} # The PID file is an order-only prerequisite because optimizing needs a *running* # Solr: the stamps survive a container restart, but the server does not, so a # restarted container should start Solr again rather than fail here. -data/backup.done: ${SETUP_DONE} | ${SOLR_PID} +data/backup.done: ${SETUP_DONE} | ${SOLR_STARTED} @echo "Optimizing the index before export (this can take a while on a large index)..." curl -sf --show-error 'http://localhost:8983/solr/${CORE}/update?optimize=true' # The optimize above commits, so the on-disk index is consistent. `solr stop` # can exit non-zero under Docker even on a clean stop (SOLR-16463), so ignore it. ${SOLR_EXEC} stop -p 8983 || true - rm -f ${SOLR_PID} + rm -f ${SOLR_STARTED} set -o pipefail; tar -C ${SOLR_DIR} -cf - ${CORE} | ${GZIP_CMD} > data/snapshot.backup.tar.gz touch $@ .PHONY: stop-solr stop-solr: - rm -f ${SOLR_PID} + rm -f ${SOLR_STARTED} ${SOLR_EXEC} stop -p 8983 || true $(info Solr stopped.) From 56d361f945b2fa950a3dd2da537a5c3fd3b05910 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 06:35:52 -0400 Subject: [PATCH 28/46] Go back to a persistent Solr volume until the namespace can do NVMe The namespace cannot create nvme-ephemeral generic ephemeral volumes, so the loading pod could not be scheduled. Issue #280 asks for that; the NVMe version is preserved on nvme-ephemeral-loading and will come back as its own PR rather than blocking a load that is otherwise ready to run. Sized up from 400Gi to 600Gi while reverting. The 400Gi figure was 3x the ~127Gi index measured for Babel 2025nov4, but the Makefile now points at 2026jul22 and releases grow. optimize=true writes the new single segment before dropping the old ones, so peak usage is 2-3x the finished index, and at a 180Gi index 3x is 540Gi. Running out of space happens during the optimize, which is the last step of a multi-hour load, and takes the whole load with it -- cheap insurance. fsGroup stays, because a freshly provisioned PVC is root-owned and Solr runs as nru, but it now uses fsGroupChangePolicy: OnRootMismatch. The default chowns every file on every mount, which on a volume holding a ~130G index is minutes of pod startup for no reason. Co-Authored-By: Claude Opus 4.8 --- data-loading/kubernetes/README.md | 99 ++++++++----------- .../kubernetes/nameres-loading-solr.k8s.yaml | 44 ++++++--- .../kubernetes/nameres-loading.k8s.yaml | 55 +++++------ 3 files changed, 94 insertions(+), 104 deletions(-) diff --git a/data-loading/kubernetes/README.md b/data-loading/kubernetes/README.md index 40601fe2..dd00e40c 100644 --- a/data-loading/kubernetes/README.md +++ b/data-loading/kubernetes/README.md @@ -8,39 +8,39 @@ tell, from Grafana, whether it has the right ones. | File | What it is | | --- | --- | -| `nameres-loading.k8s.yaml` | The pod. A workspace you exec into, not a job. Declares the NVMe volume inline. | +| `nameres-loading.k8s.yaml` | The pod. A workspace you exec into, not a job. | +| `nameres-loading-solr.k8s.yaml` | PVC mounted at `/var/solr` -- the Solr home, i.e. the index. | | `nameres-loading-data.k8s.yaml` | PVC mounted at `data/` -- synonym files in, tarball out. | -| `nameres-loading-solr.k8s.yaml` | Fallback PVC for `/var/solr`, for when the NVMe volume does not fit. Normally unused. | -## Two volumes, and why they differ +## Two volumes -The pod mounts two volumes, and the difference between them is the whole design: +- **`/var/solr`, the index.** Takes the entire write load of the indexing run and then + the read-and-write of the optimize, so its throughput is what decides how long a + load takes. Pure intermediate state: once the tarball exists the index is worthless. +- **`data/`, the working directory.** Holds the ~130G download and, at the end, the + backup tarball -- the only thing the whole exercise actually produces. -- **`/var/solr`, the index** is on a **node-local NVMe ephemeral volume**, declared - inline in the pod. It absorbs the entire write load of the indexing run and then - the read-and-write of the optimize, so it is the volume that benefits most from - being fast -- and it is pure intermediate state. Once the tarball exists, the index - is worthless. Losing it with the pod costs only the time to rebuild it from the - synonym files sitting next door. -- **`data/`, the working directory** stays on a **persistent PVC**. It holds the - ~130G download and, at the end, the backup tarball -- the only thing the whole - exercise actually produces. That does not belong on a volume that dies with its - pod. +Both are persistent PVCs today, so **both outlive the pod and both need deleting by +hand** once the tarball is somewhere safe. -So deleting the pod throws away the fast, rebuildable half and keeps the slow, -irreplaceable half. That is deliberate. +The index would be much better off on node-local NVMe: it is the volume that most +wants speed, and it is the one we could afford to lose, since it can be rebuilt from +the synonym files next door. That change is written and waiting in a separate PR, +blocked on [issue #280](https://github.com/NCATSTranslator/NameResolution/issues/280), +which asks for the namespace to support `nvme-ephemeral` volumes. -The Makefile knows about this split: the stamp files that record "the core exists" -and "the data is loaded" live on the Solr volume, not in `data/`, so a recreated pod -starts from a correct picture of an empty index rather than from stamps insisting the -work was already done. Logs go the other way, to `data/logs/`, because logs matter -most exactly when the run has died and the Solr volume has gone with it. +Being persistent does buy one thing in the meantime. The Makefile's stamp files -- +"the core exists", "the data is loaded" -- live on the Solr volume rather than in +`data/`, so if the pod dies partway through, a replacement resumes exactly where it +left off instead of reindexing from scratch. Logs go the other way, to `data/logs/`, +because logs matter most when a run has died. ## Running a load ```shell +$ kubectl apply -f nameres-loading-solr.k8s.yaml $ kubectl apply -f nameres-loading-data.k8s.yaml -$ kubectl apply -f nameres-loading.k8s.yaml # creates the NVMe volume with it +$ kubectl apply -f nameres-loading.k8s.yaml $ kubectl exec -it nameres-loading -- /bin/bash # Inside the pod: @@ -72,15 +72,14 @@ effect. `name_lookup` core from the checked-in configset and loads it. `make data/backup.done` optimizes the index, stops Solr and writes `data/snapshot.backup.tar.gz`. Copy that out (`kubectl cp`, or push it straight to -wherever `dataUrl` will point) **before deleting the pod**, then delete the pod and -the data PVC. The NVMe volume goes away with the pod on its own. +wherever `dataUrl` will point), then delete the pod and **both** PVCs -- they are +persistent, and between them they hold around a terabyte. Run it under `screen` or `tmux`: a full load is hours long, and `kubectl exec` does not survive a dropped connection. -If the pod dies partway through, recreate it and run the same commands. The synonym -files are still there and will not be downloaded again; the index will be rebuilt -from scratch, because it went with the NVMe volume. +If the pod dies partway through, recreate it and run the same commands: both volumes +survive, so make picks up from the last completed step rather than starting over. ## What actually costs time @@ -105,7 +104,7 @@ nothing but the absence of failure. CPU and `pigz` gets the same number, so raising `cpu` in the pod spec is picked up automatically; there is no second setting to keep in sync. The pod asks for 32, which is deliberately short of what the namespace has spare: the real limit is that -the request has to fit on a **single node**, with the NVMe volume, at the same time. +the request has to fit on a **single node**. Namespace quota is the easy test to pass and the wrong one to plan against -- check `kubectl describe node` before raising this. Note that both numbers come from the cgroup *limit* via [`../available-cpus.sh`](../available-cpus.sh), not @@ -130,39 +129,23 @@ Solr RSS against 111Gi of page cache (see `solr.resources` in the chart's **Disk space** is a floor, not a lever: - `/var/solr` needs **2-3x the finished index**, because `optimize=true` writes the - new single segment before deleting the old ones. 400Gi against a ~127Gi index is - about 3x; if the index passes ~140Gi, raise it. This now has to fit on a single - node's local NVMe, which is the one new constraint the fast volume brings. + new single segment before deleting the old ones. It is 600Gi, sized for an index + well above the ~127Gi measured for Babel 2025nov4, because releases grow and running + out of room happens during the optimize -- the last step of a multi-hour load. - `data/` needs the uncompressed synonyms plus the tarball. It no longer needs room for an uncompressed copy of the backup -- that staging step is gone. -**Disk speed** is a lever, and possibly the biggest one, which is why the index sits -on `storageClassName: nvme-ephemeral`. Steps 3 and 4 above are bounded by how fast -this volume can be written and re-read; no amount of Makefile tuning substitutes for -that. `data/` stays on `basic`, because it is written once by `wget` at network speed -and read once, sequentially, by the loader. - -### If the NVMe volume does not fit - -Local NVMe is a slice of a real disk on a real node, so a 400Gi request may not be -schedulable anywhere. If the pod stays `Pending` with a message about the volume, -either lower the request (it cannot go below roughly 2x the finished index) or fall -back to network storage: - -```shell -$ kubectl apply -f nameres-loading-solr.k8s.yaml -``` - -and in `nameres-loading.k8s.yaml`, replace the `ephemeral:` block on the -`nameres-loading-solr` volume with: - -```yaml - persistentVolumeClaim: - claimName: nameres-loading-solr -``` +**Disk speed** is a lever, and probably the biggest one left. Both PVCs use +`storageClassName: basic`, so the index sits on network storage and the merge and +optimize phases are bounded by it -- no amount of Makefile tuning substitutes for the +volume being fast. -The load then works exactly as before, just slower through the merge and optimize -steps. Nothing else needs changing -- the stamp files follow `SOLR_DIR` either way. +Moving the index onto a node-local NVMe ephemeral volume is written and ready in a +separate PR, blocked on +[issue #280](https://github.com/NCATSTranslator/NameResolution/issues/280): the +namespace cannot currently create `nvme-ephemeral` volumes. When that lands, the swap +is one block in `nameres-loading.k8s.yaml` and nothing else changes -- the stamp files +follow `SOLR_DIR` either way. ## Watching a load in Grafana @@ -175,7 +158,7 @@ time. | **CPU usage vs. limit** | Flat at the limit during the load = CPU-bound, and more will help. Well below it = something else is the constraint, probably disk. | If it is below the limit, look at disk before adding CPU. | | **Memory usage (RSS/WSS)** | This is roughly the JVM. It should sit near `SOLR_MEM` and be stable. | If it is far below `SOLR_MEM`, lower `SOLR_MEM` and give the memory back as cache. | | **Memory usage (cache)** | Page cache: the index data the kernel is holding. Rising to fill the headroom is healthy. | If it is pinned at (limit - heap) for the whole run, more memory may speed up merges. | -| **Disk read/write throughput on the Solr volume** | Flat-topped during merges and the optimize = saturated volume. On NVMe this should no longer be what you are waiting for; if it still is, the CPU is not the constraint and neither is the heap. | Check the volume really is NVMe (`kubectl get pvc` while the pod runs, and look at the storage class). | +| **Disk read/write throughput on the Solr volume** | Flat-topped during merges and the optimize = saturated volume. Expected on `basic` storage, and the main argument for issue #280. | Nothing to tune here; it is the storage class. | | **Wall-clock time of each `make` step** | The logs in `data/logs/` are timestamped per step. | Tells you which of the five phases above to attack at all. | Two things worth checking specifically, since they are new: diff --git a/data-loading/kubernetes/nameres-loading-solr.k8s.yaml b/data-loading/kubernetes/nameres-loading-solr.k8s.yaml index e3459204..1a5ee8e4 100644 --- a/data-loading/kubernetes/nameres-loading-solr.k8s.yaml +++ b/data-loading/kubernetes/nameres-loading-solr.k8s.yaml @@ -1,24 +1,36 @@ # Kubernetes file for setting up a PVC to use for nameres-loading. # -# FALLBACK ONLY. The pod normally puts the Solr home on a node-local NVMe ephemeral -# volume, declared inline in nameres-loading.k8s.yaml -- it is faster and the index -# is disposable. Use this PVC instead only when no node has enough local NVMe for -# the whole index; see "If the NVMe volume does not fit" in README.md. Do not apply -# it otherwise, or you will reserve several hundred GB of network storage that -# nothing mounts. -# # This is the Solr home (/var/solr): it holds the name_lookup core, which is the # index itself and, at the end, the thing that gets tarred into the backup. # -# Size it at 3x the finished index, not 1x. `optimize=true` merges everything into -# one segment by writing the new segment before dropping the old ones, so peak usage -# during that step is roughly double the index, and Lucene's own guidance is to -# budget 2-3x. As of Babel 2025nov4 the index is ~127Gi, so 400Gi is about 3x and -# still the right order of magnitude -- but if the index passes ~140Gi, raise this. +# ## Why 600Gi for a ~130Gi index +# +# Not padding. `optimize=true` merges everything into one segment by writing the new +# segment before dropping the old ones, so peak usage during that step is roughly +# double the finished index, and Lucene's guidance is to budget 2-3x. +# +# The finished index was ~127Gi for Babel 2025nov4. Newer releases are bigger, and +# the sizing has to hold for the release actually being loaded (SYNONYMS_URL in the +# Makefile), not the last one measured: at a 180Gi index, 3x is 540Gi and 400Gi would +# not be enough. The failure mode is the reason for the headroom -- running out of +# space happens during the optimize, which is the *last* step of a multi-hour load, +# and it takes the whole load with it. +# +# If storage quota is tight this can come down, but not below roughly 2x the finished +# index, and check the index size from the previous load before trimming it. +# +# ## Speed +# +# This volume takes the whole write load of the indexing run plus the read-and-write +# of the optimize, so its throughput is what decides how long a load takes. On +# `basic` (network storage) those two phases dominate. Moving the index to a +# node-local NVMe ephemeral volume is ready in a separate PR, blocked on issue #280, +# which asks for the namespace to support that storage class. +# +# ## Cleanup # -# storageClassName is worth a thought: this volume takes the whole write load of the -# indexing run plus the read+write of the optimize, so a faster class here shortens -# the load more reliably than any tuning in the Makefile. See README.md. +# This is a PVC, so it outlives the pod: delete it by hand once the backup tarball +# has been copied somewhere safe, or it keeps several hundred GB reserved. apiVersion: v1 kind: PersistentVolumeClaim @@ -31,5 +43,5 @@ spec: - ReadWriteOnce resources: requests: - storage: 400Gi + storage: 600Gi storageClassName: basic diff --git a/data-loading/kubernetes/nameres-loading.k8s.yaml b/data-loading/kubernetes/nameres-loading.k8s.yaml index 0111f1ee..5454424c 100644 --- a/data-loading/kubernetes/nameres-loading.k8s.yaml +++ b/data-loading/kubernetes/nameres-loading.k8s.yaml @@ -5,10 +5,11 @@ # below deliberately replaces the image's entrypoint, which would otherwise start a # Solr of its own -- the Makefile starts Solr itself, using SOLR_MEM as its heap. # -# The index lives on a node-local NVMe volume that is created and destroyed with the -# pod; the synonym files and the finished backup live on a persistent volume that -# does not. Deleting this pod therefore throws away the index and keeps the work -# that was expensive to fetch. See README.md. +# Both volumes are persistent PVCs, so both outlive the pod and both have to be +# deleted by hand when the load is finished. Putting the index on a node-local NVMe +# ephemeral volume instead would make the merge and optimize phases substantially +# faster; that is ready to go in a separate PR and is blocked on issue #280, which +# asks for the namespace to support it. See README.md. apiVersion: v1 kind: Pod @@ -56,8 +57,8 @@ spec: # before changing either number. # # These have to fit on ONE node, which is a stricter test than the namespace - # quota -- and the same node has to have the NVMe volume free. Check node - # capacity before raising them, not just what the namespace has left. + # quota. Check node capacity before raising them, not just what the namespace has + # left. resources: requests: ephemeral-storage: "1G" @@ -67,36 +68,30 @@ spec: ephemeral-storage: "1G" memory: "128Gi" cpu: "32" - # The image runs as nru, uid/gid 1000. fsGroup makes the kubelet hand the NVMe - # volume over to that group; without it the volume is root-owned and Solr cannot - # write a thing. runAsUser only restates what the image already does. + # The image runs as nru, uid/gid 1000. fsGroup hands the volumes to that group; + # without it a freshly provisioned volume is root-owned and Solr cannot write a + # thing. runAsUser only restates what the image already does. + # + # OnRootMismatch matters here because these are persistent volumes holding a + # ~130G index: the default (Always) makes the kubelet chown every file on every + # mount, which on a volume this size is minutes of pod startup for nothing. With + # OnRootMismatch it checks the top directory and skips the walk once it is right. securityContext: runAsUser: 1000 fsGroup: 1000 + fsGroupChangePolicy: OnRootMismatch volumes: - # The index goes on a node-local NVMe volume. It is the right thing to put there - # twice over: it takes the whole write load of the indexing run and then the - # read-and-write of the optimize, so it is the volume that benefits most from the - # speed -- and it is pure intermediate state, so losing it with the pod costs - # nothing that cannot be rebuilt from the synonym files next door. + # The Solr home: the index, and at the end the thing that gets tarred into the + # backup. This takes the entire write load of the indexing run and then the + # read-and-write of the optimize, so it is the volume whose speed decides how long + # a load takes -- see issue #280 for moving it onto node-local NVMe. # - # This is a *generic ephemeral volume*: created with the pod and deleted with it. - # Not to be confused with the `ephemeral-storage` resource above, which is the - # node's own scratch space for the container filesystem. - # - # Sizing is the same 2-3x-the-index rule as before (optimize writes the new - # segment before dropping the old ones), so this has to fit on the node's local - # NVMe. If no node has that much, use nameres-loading-solr.k8s.yaml instead -- - # see README.md, "If the NVMe volume does not fit". + # Being persistent has one consolation prize: the Makefile's stamp files live on + # this volume, so if the pod dies partway through, a replacement picks up exactly + # where it left off instead of reindexing from scratch. - name: nameres-loading-solr - ephemeral: - volumeClaimTemplate: - spec: - accessModes: [ "ReadWriteOnce" ] - storageClassName: nvme-ephemeral - resources: - requests: - storage: 400Gi + persistentVolumeClaim: + claimName: nameres-loading-solr # This one stays persistent: it holds the ~130G download and, at the end, the # backup tarball, which is the only thing the whole exercise produces. - name: nameres-loading-data From 92a3a28be469ff8a9a88cbd25db6b1358f88ce64 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 07:03:07 -0400 Subject: [PATCH 29/46] Address review: backup ownership, additive-load guard, and doc drift The two that would have bitten in production: Backup ownership. Solr writes to the core it serves -- it takes a write lock and refuses to load the core if it cannot. The loading image runs as nru (1000) and every Solr that serves the backup runs as solr (8983), so a restored core was owned by a user that does not exist on the serving side. Verified against a real restore: without this the core fails with SolrException: /var/solr/data/name_lookup/data/index/write.lock at the end of a multi-hour restore. CI papered over it with a chown, and Deployment.md did not mention it at all. tar now stamps BACKUP_UID/GID onto every member, which fixes every restore path at once instead of asking each one to remember. The CI roundtrip now chowns the source core to 1000 first -- standing in for the loading image -- and drops the chown on the restore side, so it exercises the fix rather than hiding it. tlog is excluded while we are here: the optimize commits, so there is nothing to replay. Additive loads. Loading assigns fresh UUIDs, so re-running a load over the same files indexes everything twice, and the document-count guard cannot see it because the delta is still exactly right. That is the one way the loader can hand back a corrupt index while reporting success, so a non-empty core is now a hard error (LOAD_APPEND=1 to override). It fires in the two cases that happen: a load re-run after one died partway through, and make re-running the load because a stamp file went missing. Both READMEs claimed a dead pod "resumes exactly where it left off"; resumption is per step and the load is one step, so they now say to clear the core. Smaller fixes: - /status used core['startTime'] where every field around it uses .get(). - `make clean` ran `mkdir data` on a directory that still existed. - wget now accepts only *.txt/*.txt.gz. Everything in data/synonyms is loaded, so a stray robots.txt became a file the loader tried to index -- caught by the count guard, but only after a multi-hour load. - The split loop swallowed a failed split that was not the last one. - The loading image's entrypoint still asked for a 64G heap. Doc drift: ramBufferSizeMB was described as 512 MB (it is 2 GB), SOLR_VERSION as 9.10.0 (it is 9.10.1), LOAD_PARALLELISM as defaulting to the CPU count (it is the cgroup limit, which is the entire point), and the useColdSearcher comment promised warming that nothing performs. Checked against a real Solr 9.10: core creates from the configset, 89 docs load, the malformed-file and non-empty-core guards both abort, the backup round-trips into a fresh Solr with no chown and serves queries, and pytest is 28/28. --- .github/workflows/release-nameres-loading.yml | 7 ++++ .github/workflows/tester.yml | 34 +++++++++++++--- api/server.py | 6 ++- data-loading/Dockerfile | 9 ++++- data-loading/Makefile | 40 ++++++++++++++++--- data-loading/README.md | 29 +++++++++++++- .../name_lookup/conf/solrconfig.xml | 7 +++- data-loading/kubernetes/README.md | 24 +++++++++-- data-loading/setup-and-load-solr.sh | 21 +++++++++- documentation/Deployment.md | 10 ++++- 10 files changed, 164 insertions(+), 23 deletions(-) diff --git a/.github/workflows/release-nameres-loading.yml b/.github/workflows/release-nameres-loading.yml index 3e3abb4b..d3049e37 100644 --- a/.github/workflows/release-nameres-loading.yml +++ b/.github/workflows/release-nameres-loading.yml @@ -1,6 +1,13 @@ name: 'Release a new version of NameResolution Data Loading to Github Packages' on: + # TEMPORARY, delete before merging (NameResolution#278). Publishes a `pr-NNN` image + # so a load can be run against this branch's pipeline without merging it -- the + # Makefile, loader and configset all live inside the image, so the image tag is the + # only way to run a change. Not fit to keep: it publishes a package on every push + # to every PR, and a PR from a fork gets a read-only GITHUB_TOKEN (which + # `permissions:` cannot widen), so the push fails and the contributor sees a red X + # they can do nothing about. pull_request: release: types: [published] diff --git a/.github/workflows/tester.yml b/.github/workflows/tester.yml index c7a426a0..4626d861 100644 --- a/.github/workflows/tester.yml +++ b/.github/workflows/tester.yml @@ -51,29 +51,53 @@ jobs: python -m pytest tests/ # The loader's guard is what stands between us and a silently half-loaded - # 130 GB index, so check that a bad file actually stops it. + # 130 GB index, so check that a bad file actually stops it. LOAD_APPEND=1 + # because the core is loaded by now, and without it the loader would refuse + # for that reason instead -- the test would pass without testing anything. - name: The loader must abort on a file that does not load run: | echo 'this is not json' > /tmp/malformed.json - if ./data-loading/setup-and-load-solr.sh /tmp/malformed.json; then + if LOAD_APPEND=1 ./data-loading/setup-and-load-solr.sh /tmp/malformed.json; then echo "Expected the loader to fail on a malformed file, but it succeeded." exit 1 fi echo "Loader aborted as expected." + # The other half of that guard: loading is additive and assigns fresh UUIDs, so + # a re-run over the same files doubles the index and the document-count check + # cannot see it (the delta is still exactly right). Refusing a non-empty core is + # the only thing that catches it. + - name: The loader must refuse a core that already has documents + run: | + if ./data-loading/setup-and-load-solr.sh tests/data/test-synonyms.json; then + echo "Expected the loader to refuse a non-empty core, but it loaded again." + exit 1 + fi + echo "Loader refused the non-empty core as expected." + # The backup is the actual product of data-loading/, so check that a core can be # tarred up, dropped into a fresh Solr, and served without any restore-time setup. + # + # The tar flags are the ones the Makefile uses, and the restore deliberately does + # NOT chown afterwards: the loading image runs as uid 1000 and the serving image + # as 8983, so if the tarball did not stamp the ownership the restored core would + # be unwritable and Solr would fail to take its write lock. chown-ing here would + # hide exactly the bug this is meant to catch. The chown to 1000 below stands in + # for the loading image, so the source ownership is wrong before we start. - name: Verify the backup roundtrip (tar the core, restore it into a fresh Solr) run: | - docker exec name_lookup tar -C /var/solr/data -cf - name_lookup > snapshot.backup.tar + docker exec -u root name_lookup chown -R 1000:1000 /var/solr/data/name_lookup + docker exec -u root name_lookup tar -C /var/solr/data \ + --numeric-owner --owner=8983 --group=8983 \ + --exclude='name_lookup/data/tlog' \ + -cf - name_lookup > snapshot.backup.tar docker run --name name_lookup_restored -d -p 8984:8983 solr:9.10 for _ in $(seq 60); do curl -sf 'http://localhost:8984/solr/admin/cores?action=STATUS' > /dev/null && break sleep 2 done docker cp snapshot.backup.tar name_lookup_restored:/tmp/snapshot.backup.tar - docker exec -u root name_lookup_restored tar -C /var/solr/data -xf /tmp/snapshot.backup.tar - docker exec -u root name_lookup_restored chown -R solr:solr /var/solr/data/name_lookup + docker exec -u root name_lookup_restored tar -C /var/solr/data --numeric-owner -xf /tmp/snapshot.backup.tar docker restart name_lookup_restored for _ in $(seq 60); do curl -sf 'http://localhost:8984/solr/name_lookup/admin/ping' > /dev/null && break diff --git a/api/server.py b/api/server.py index 45084f9a..df8c30c6 100755 --- a/api/server.py +++ b/api/server.py @@ -111,7 +111,11 @@ async def status() -> Dict: 'download_url': biolink_model_download_url, }, 'nameres_version': nameres_version, - 'startTime': core['startTime'], + # .get() rather than [], like every field below it: Solr's core STATUS + # returns a sparse entry for a core that is still initializing, and + # /status is what the Kubernetes probes call. A KeyError here would turn + # a startup window into a 500 instead of a report with blank counts. + 'startTime': core.get('startTime', ''), 'numDocs': index.get('numDocs', ''), 'maxDoc': index.get('maxDoc', ''), 'deletedDocs': index.get('deletedDocs', ''), diff --git a/data-loading/Dockerfile b/data-loading/Dockerfile index 2fd1622f..208a09f0 100644 --- a/data-loading/Dockerfile +++ b/data-loading/Dockerfile @@ -100,6 +100,13 @@ COPY --chown=nru Makefile ${ROOT} COPY --chown=nru configsets ${ROOT}/configsets # On entry, start the Solr instance (standalone -- no ZooKeeper for a one-node load). +# +# SOLR_MEM matches the Makefile's default, and for the same reason: `solr -m` sets +# both -Xms and -Xmx, and indexing wants the memory as page cache rather than heap +# (see the SOLR_MEM comment in the Makefile). Note that the Kubernetes pod replaces +# this command entirely and lets the Makefile start Solr, so this entrypoint only +# matters when the image is run directly. ENV SOLR_EXEC="/opt/solr/bin/solr" ENV SOLR_DIR="$SOLR_DIR" -ENTRYPOINT ${SOLR_EXEC} start -f --user-managed -p 8983 -m 64G -s ${SOLR_DIR} +ENV SOLR_MEM="31G" +ENTRYPOINT ${SOLR_EXEC} start -f --user-managed -p 8983 -m ${SOLR_MEM} -s ${SOLR_DIR} diff --git a/data-loading/Makefile b/data-loading/Makefile index 80dab127..173a3ebe 100644 --- a/data-loading/Makefile +++ b/data-loading/Makefile @@ -18,7 +18,7 @@ SYNONYMS_URL ?= https://stars.renci.org/var/babel_outputs/2026jul22/synonyms/ # committed, not just a ceiling. # # Indexing does not want a big heap: Lucene buffers documents in ramBufferSizeMB -# (512 MB, set in the configset) and streams merges through the OS page cache, so +# (2 GB, set in the configset) and streams merges through the OS page cache, so # heap beyond a few GB is memory taken away from that cache. Staying under 32G also # keeps compressed object pointers, which a larger heap silently gives up. # See data-loading/kubernetes/README.md for how to check this against Grafana. @@ -45,6 +45,16 @@ CONFIGSET ?= $(CURDIR)/configsets/name_lookup # The Solr core name. Do not change without also changing NameRes and the configset. CORE=name_lookup +# The uid/gid the backup's files are owned by. This is `solr` in the official Solr +# image, which is what serves the backup everywhere: docker-compose, the Helm chart +# and CI. The loading image runs as `nru` (1000) instead, so without this the +# restored core is owned by a user that does not exist on the serving side and Solr +# cannot take its write lock -- a core that fails to load with a permissions error at +# the end of a multi-hour restore. Stamping the ownership at tar time fixes it for +# every restore path at once, rather than asking each one to remember a chown. +BACKUP_UID ?= 8983 +BACKUP_GID ?= 8983 + # Split any synonym file larger than SPLIT_SIZE into chunks of SPLIT_LINES lines. # Splitting only helps parallelism, so we only bother with the big files; JSON is # one document per line, so we must split on line boundaries, never on bytes. @@ -82,21 +92,29 @@ all: ${SETUP_DONE} # ${SOLR_DIR}/${CORE} (or, on Kubernetes, just recreate the pod). clean: rm -rf data/* - mkdir data + mkdir -p data rm -f ${SOLR_STARTED} ${CORE_DONE} ${SETUP_DONE} # Step 1. Download and uncompress synonym files, then split the large ones so the # parallel loader has more units of work. data/synonyms/done: mkdir -p data/synonyms - wget -c -r -l1 -nd -P data/synonyms ${SYNONYMS_URL} + # -A restricts what is saved to the synonym files themselves. Everything in + # data/synonyms is loaded into Solr (the glob in step 4 is *.txt*), so anything + # else wget leaves behind -- a robots.txt, most obviously -- becomes a file the + # loader tries to index. Solr answers 200 and indexes nothing for it, so the + # document-count guard catches it, but only at the end of a multi-hour load. + wget -c -r -l1 -nd -A '*.txt,*.txt.gz' -P data/synonyms ${SYNONYMS_URL} # -exec (rather than a glob) so that a rerun where everything is already # uncompressed is a no-op instead of a "no such file" failure. find data/synonyms -maxdepth 1 -type f -name '*.txt.gz' -exec gunzip {} + echo Downloaded synonyms from ${SYNONYMS_URL} - find data/synonyms -maxdepth 1 -type f -name '*.txt' -size +${SPLIT_SIZE} -print | while read -r f; do \ + # pipefail plus `|| exit 1` so that a failed split fails the target. Without both, + # the loop's status is whichever iteration ran last and a file that failed to + # split halfway through the set is silently loaded whole (or not at all). + set -o pipefail; find data/synonyms -maxdepth 1 -type f -name '*.txt' -size +${SPLIT_SIZE} -print | while read -r f; do \ echo "Splitting large file $$f (larger than ${SPLIT_SIZE})..."; \ - split -d -l ${SPLIT_LINES} "$$f" "$$f." && rm "$$f"; \ + split -d -l ${SPLIT_LINES} "$$f" "$$f." && rm "$$f" || exit 1; \ done touch $@ @@ -148,7 +166,17 @@ data/backup.done: ${SETUP_DONE} | ${SOLR_STARTED} # can exit non-zero under Docker even on a clean stop (SOLR-16463), so ignore it. ${SOLR_EXEC} stop -p 8983 || true rm -f ${SOLR_STARTED} - set -o pipefail; tar -C ${SOLR_DIR} -cf - ${CORE} | ${GZIP_CMD} > data/snapshot.backup.tar.gz + # --owner/--group stamp the serving Solr's uid onto every member (see BACKUP_UID + # above); --numeric-owner makes the restoring tar use those numbers rather than + # looking up whatever "solr" means on the restoring host. + # + # tlog is left out: the optimize above committed, so there is nothing uncommitted + # to replay, and Solr recreates the directory on startup. It is dead weight in a + # tarball this size. + set -o pipefail; tar -C ${SOLR_DIR} \ + --numeric-owner --owner=${BACKUP_UID} --group=${BACKUP_GID} \ + --exclude='${CORE}/data/tlog' \ + -cf - ${CORE} | ${GZIP_CMD} > data/snapshot.backup.tar.gz touch $@ .PHONY: stop-solr diff --git a/data-loading/README.md b/data-loading/README.md index 77c8c0ef..9b16710f 100644 --- a/data-loading/README.md +++ b/data-loading/README.md @@ -98,7 +98,22 @@ usually `/var/solr`) and `SOLR_EXEC` (the `bin/solr` executable) to be set -- th NameRes instance at it (see below). Tuning knobs: `SOLR_MEM` (Solr heap for the load), `LOAD_PARALLELISM` -(concurrent uploads; defaults to the number of CPUs), `SPLIT_SIZE` / `SPLIT_LINES`. +(concurrent uploads; defaults to the cgroup CPU limit, which inside a container is +the pod's limit rather than the node's core count — see +[`available-cpus.sh`](available-cpus.sh)), `SPLIT_SIZE` / `SPLIT_LINES`. + +Loading is **additive**, not idempotent: every document is given a fresh UUID, so +loading the same files twice puts every document in twice. The loader therefore +refuses to load into a core that already has documents. If a load dies partway +through, do not just re-run it — delete the core and start it again: + +```shell +$ make stop-solr && rm -rf "${SOLR_DIR:-/var/solr}/name_lookup" "${SOLR_DIR:-/var/solr}"/*.done +$ make all +``` + +(`LOAD_APPEND=1` overrides the refusal, for the genuine case of loading a second, +distinct set of files into an existing core.) ## Restoring the backup @@ -114,9 +129,19 @@ trivial in every environment: Job no longer creates a collection or re-applies the schema; its only remaining job is deleting blocklisted CURIEs. +Solr needs to *write* to the core it serves — it takes a write lock, and refuses to +load the core if it cannot. The loading image runs as uid 1000 and the official Solr +image runs as uid 8983, so the Makefile stamps 8983 onto every file as it builds the +tarball (`BACKUP_UID` / `BACKUP_GID`) and the restore needs no `chown`. Extract as +root, or as a user that can set ownership; extracting as an unprivileged user gives +the files to whoever extracted them, and Solr will not be able to write. + +A backup built before this (anything whose files come out owned by uid 1000) needs +`chown -R 8983:8983` on the extracted core. + The Solr version used to serve the backup must be **>=** the version that built it (an older Solr cannot read a newer Lucene index). The builder is pinned exactly -(`SOLR_VERSION` in the [`Dockerfile`](Dockerfile), 9.10.0 today) because it names a +(`SOLR_VERSION` in the [`Dockerfile`](Dockerfile), 9.10.1 today) because it names a release tarball; the servers (`docker-compose.yml`, CI, the Helm chart) track the `9.10` line, so they float forward onto patch releases and stay at or above the builder without anyone having to remember to bump them. diff --git a/data-loading/configsets/name_lookup/conf/solrconfig.xml b/data-loading/configsets/name_lookup/conf/solrconfig.xml index 39f2c5d5..2920398f 100644 --- a/data-loading/configsets/name_lookup/conf/solrconfig.xml +++ b/data-loading/configsets/name_lookup/conf/solrconfig.xml @@ -88,8 +88,11 @@ - + false diff --git a/data-loading/kubernetes/README.md b/data-loading/kubernetes/README.md index dd00e40c..bd1ddd18 100644 --- a/data-loading/kubernetes/README.md +++ b/data-loading/kubernetes/README.md @@ -31,9 +31,21 @@ which asks for the namespace to support `nvme-ephemeral` volumes. Being persistent does buy one thing in the meantime. The Makefile's stamp files -- "the core exists", "the data is loaded" -- live on the Solr volume rather than in -`data/`, so if the pod dies partway through, a replacement resumes exactly where it -left off instead of reindexing from scratch. Logs go the other way, to `data/logs/`, -because logs matter most when a run has died. +`data/`, so a replacement pod skips the steps that already finished instead of +starting from nothing. Logs go the other way, to `data/logs/`, because logs matter +most when a run has died. + +That resumption is per *step*, though, and the load is one step. A pod that dies +mid-load leaves a partly-filled core and no stamp, and the load cannot be continued: +it is additive and assigns fresh UUIDs, so re-running it would index the already- +loaded files a second time. The loader refuses a non-empty core for that reason. The +recovery is to throw the index away and load again -- the download, which is the part +worth keeping, is on the other volume and survives: + +```shell +$ make stop-solr && rm -rf /var/solr/name_lookup /var/solr/*.done +$ make all +``` ## Running a load @@ -79,7 +91,9 @@ Run it under `screen` or `tmux`: a full load is hours long, and `kubectl exec` d not survive a dropped connection. If the pod dies partway through, recreate it and run the same commands: both volumes -survive, so make picks up from the last completed step rather than starting over. +survive, so make picks up from the last completed step rather than starting over. The +exception is dying *during* the load, which cannot be resumed -- see "Two volumes" +above for how to clear the core and restart it. ## What actually costs time @@ -186,3 +200,5 @@ the pod spec (the Makefile uses `?=`). | `SPLIT_SIZE` / `SPLIT_LINES` | `2G` / 10M | How the big synonym files are split. More, smaller chunks give the parallel loader a shorter tail at the end of the run; fewer, larger ones mean less `split` time up front. | | `ramBufferSizeMB` | 2048 (in the configset) | Not an environment variable -- it lives in `solrconfig.xml`. It is a budget shared across indexing threads, so the segment size at flush is roughly this divided by the number of concurrent uploads: ~64 MB at 32 uploads. Raise it further if the Grafana disk panels show the run still merging long after the uploads have finished; there is heap to spare. | | `SOLR_STARTUP_TRIES` | 60 | How long the loader waits for Solr (3s each) before giving up. | +| `LOAD_APPEND` | unset | Set to `1` to load into a core that already has documents. The loader refuses by default, because loading is additive and a re-run would duplicate every document. Only correct when you are deliberately adding a second, distinct set of files. | +| `BACKUP_UID` / `BACKUP_GID` | `8983` | The uid/gid stamped onto the backup's files, so the serving Solr can write to the core it restores without a `chown`. | diff --git a/data-loading/setup-and-load-solr.sh b/data-loading/setup-and-load-solr.sh index a0ba607a..6fb76aaf 100755 --- a/data-loading/setup-and-load-solr.sh +++ b/data-loading/setup-and-load-solr.sh @@ -91,8 +91,27 @@ if [ -z "${before}" ]; then echo "Could not read the current document count from Solr. Aborting." >&2 exit 1 fi + +# Refuse to load into a non-empty core by default. Loading is additive and every +# document gets a fresh UUID, so a second load over the same files silently doubles +# the index -- and the count guard below cannot see it, because the *delta* is still +# exactly the number of input documents. That is the one way this script can hand +# back a corrupt index while reporting success, so it is a hard error rather than a +# warning. It fires in the two cases that actually happen: a load re-run after the +# previous one died partway through, and make re-running the load because a stamp +# file went missing (see the stamp-file comment in the Makefile). if [ "${before}" -ne 0 ]; then - echo "NOTE: core '${CORE}' already contains ${before} documents; this load adds to them." + if [ "${LOAD_APPEND:-0}" = "1" ]; then + echo "NOTE: core '${CORE}' already contains ${before} documents; LOAD_APPEND=1, so this load adds to them." + else + echo "Core '${CORE}' already contains ${before} documents, and loading is additive:" >&2 + echo "re-running the load would duplicate every document rather than resume it." >&2 + echo >&2 + echo "To start over, delete the core's index and let make recreate it:" >&2 + echo " make stop-solr && rm -rf \"\${SOLR_DIR:-/var/solr}/${CORE}\" \"\${SOLR_DIR:-/var/solr}\"/*.done" >&2 + echo "To append anyway (you are loading a second, distinct set of files), set LOAD_APPEND=1." >&2 + exit 1 + fi fi # Step 3. Load the files in parallel, streaming each one, without committing. diff --git a/documentation/Deployment.md b/documentation/Deployment.md index f44d95ce..45424e30 100644 --- a/documentation/Deployment.md +++ b/documentation/Deployment.md @@ -30,8 +30,16 @@ instance or from Translator. (`snapshot.backup.tar.gz`) once it has been decompressed. ```shell - $ tar -C ./data/solr -xzvf snapshot.backup.tar.gz + $ sudo tar -C ./data/solr --numeric-owner -xzvf snapshot.backup.tar.gz ``` + + Solr writes to the core it serves (it takes a write lock, and will not load the core + if it cannot), and it runs as uid 8983 inside the container. The tarball records that + ownership, so extract it as root — `--numeric-owner` then applies the recorded uid + rather than looking up what `solr` happens to mean on your machine. Extracting as an + ordinary user gives the files to you instead, and Solr will fail to load the core. + For a backup built before this was the case, or if you see permission errors in the + Solr log, fix it with `sudo chown -R 8983:8983 ./data/solr/name_lookup`. 5. Check the [docker-compose.yml](./docker-compose.yml) file to ensure that it is as you expect. * The Docker Compose file will use the latest released version of NameRes From c563ded49c63a24eb2be6f50c5fa17e88e1e1705 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 14:52:59 -0400 Subject: [PATCH 30/46] Removed on:pull_request trigger. --- .github/workflows/release-nameres-loading.yml | 8 -------- 1 file changed, 8 deletions(-) diff --git a/.github/workflows/release-nameres-loading.yml b/.github/workflows/release-nameres-loading.yml index d3049e37..27c62c99 100644 --- a/.github/workflows/release-nameres-loading.yml +++ b/.github/workflows/release-nameres-loading.yml @@ -1,14 +1,6 @@ name: 'Release a new version of NameResolution Data Loading to Github Packages' on: - # TEMPORARY, delete before merging (NameResolution#278). Publishes a `pr-NNN` image - # so a load can be run against this branch's pipeline without merging it -- the - # Makefile, loader and configset all live inside the image, so the image tag is the - # only way to run a change. Not fit to keep: it publishes a package on every push - # to every PR, and a PR from a fork gets a read-only GITHUB_TOKEN (which - # `permissions:` cannot widen), so the push fails and the contributor sees a red X - # they can do nothing about. - pull_request: release: types: [published] From b17e04b220a4be290de4f4df1b609b32b8c20abb Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 15:04:40 -0400 Subject: [PATCH 31/46] Correct index-size rationale for 600Gi Solr PVC The measured index for Babel 2026jul22 (the release SYNONYMS_URL loads) is ~111Gi -- smaller than 2025nov4's ~127Gi, because it has fewer proteins. That falsifies the "newer releases are bigger" claim the 600Gi sizing rested on, so reword both places to note release size moves both ways and the headroom is for a possibly-larger future release. No size change: 600Gi stays comfortably above 2-3x either figure. Co-Authored-By: Claude Opus 4.8 --- data-loading/kubernetes/README.md | 7 ++++--- data-loading/kubernetes/nameres-loading-solr.k8s.yaml | 9 +++++---- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/data-loading/kubernetes/README.md b/data-loading/kubernetes/README.md index bd1ddd18..9722a3c3 100644 --- a/data-loading/kubernetes/README.md +++ b/data-loading/kubernetes/README.md @@ -143,9 +143,10 @@ Solr RSS against 111Gi of page cache (see `solr.resources` in the chart's **Disk space** is a floor, not a lever: - `/var/solr` needs **2-3x the finished index**, because `optimize=true` writes the - new single segment before deleting the old ones. It is 600Gi, sized for an index - well above the ~127Gi measured for Babel 2025nov4, because releases grow and running - out of room happens during the optimize -- the last step of a multi-hour load. + new single segment before deleting the old ones. It is 600Gi, sized well above the + finished index (~127Gi for Babel 2025nov4, ~111Gi for 2026jul22 -- release size + moves both ways) to leave room for a larger future one, because running out of room + happens during the optimize -- the last step of a multi-hour load. - `data/` needs the uncompressed synonyms plus the tarball. It no longer needs room for an uncompressed copy of the backup -- that staging step is gone. diff --git a/data-loading/kubernetes/nameres-loading-solr.k8s.yaml b/data-loading/kubernetes/nameres-loading-solr.k8s.yaml index 1a5ee8e4..2ff30c7b 100644 --- a/data-loading/kubernetes/nameres-loading-solr.k8s.yaml +++ b/data-loading/kubernetes/nameres-loading-solr.k8s.yaml @@ -3,15 +3,16 @@ # This is the Solr home (/var/solr): it holds the name_lookup core, which is the # index itself and, at the end, the thing that gets tarred into the backup. # -# ## Why 600Gi for a ~130Gi index +# ## Why 600Gi for a ~110-130Gi index # # Not padding. `optimize=true` merges everything into one segment by writing the new # segment before dropping the old ones, so peak usage during that step is roughly # double the finished index, and Lucene's guidance is to budget 2-3x. # -# The finished index was ~127Gi for Babel 2025nov4. Newer releases are bigger, and -# the sizing has to hold for the release actually being loaded (SYNONYMS_URL in the -# Makefile), not the last one measured: at a 180Gi index, 3x is 540Gi and 400Gi would +# Release size moves both ways: the finished index was ~127Gi for Babel 2025nov4 but +# ~111Gi for 2026jul22 (fewer proteins). The sizing has to hold for whichever release +# is actually being loaded (SYNONYMS_URL in the Makefile) and leave room for a larger +# one, not track the last one measured: at a 180Gi index, 3x is 540Gi and 400Gi would # not be enough. The failure mode is the reason for the headroom -- running out of # space happens during the optimize, which is the *last* step of a multi-hour load, # and it takes the whole load with it. From b6458cc16e5e1cc6f650c42a210ad148433a73c0 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 15:05:48 -0400 Subject: [PATCH 32/46] Updated NameRes Data Loading tag to latest. --- data-loading/kubernetes/nameres-loading.k8s.yaml | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/data-loading/kubernetes/nameres-loading.k8s.yaml b/data-loading/kubernetes/nameres-loading.k8s.yaml index 5454424c..bb582c8a 100644 --- a/data-loading/kubernetes/nameres-loading.k8s.yaml +++ b/data-loading/kubernetes/nameres-loading.k8s.yaml @@ -21,16 +21,7 @@ spec: restartPolicy: Never containers: - name: nameres-loading - # TEMPORARY, until NameResolution#278 is merged and released. The publish - # workflow only tags `latest` on a release, so `latest` is still the image from - # before this work: old Makefile, old loader, 220G heap, no available-cpus.sh. - # The Makefile, the loader and the configset all live inside the image, so the - # tag decides which pipeline actually runs -- editing this file is not enough. - # Switch back to `latest` once the release is cut. - # - # Check you got the image you meant, once you are in the pod: - # ls available-cpus.sh && grep ramBufferSizeMB configsets/name_lookup/conf/solrconfig.xml - image: ghcr.io/ncatstranslator/nameresolution-data-loading:pr-278 + image: ghcr.io/ncatstranslator/nameresolution-data-loading:latest imagePullPolicy: Always # Idle until someone execs in. `make all` is run by hand, not by the pod. command: ["sleep", "infinity"] From 210ff54a484308d49c5309c9ebf1f30ca24f7beb Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 15:08:52 -0400 Subject: [PATCH 33/46] Pin queryResultCache to CaffeineCache maxRamMB is a CaffeineCache feature. CaffeineCache is Solr's default cache today, so this is a no-op now, but if a later release changed that default to an impl without RAM accounting, maxRamMB would be silently ignored and the cache would grow unbounded. Pin the class so the configset's intent survives a Solr upgrade. Verified the core still creates from the configset on solr:9.10. Addresses a Copilot review comment on #278. Co-Authored-By: Claude Opus 4.8 --- .../configsets/name_lookup/conf/solrconfig.xml | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/data-loading/configsets/name_lookup/conf/solrconfig.xml b/data-loading/configsets/name_lookup/conf/solrconfig.xml index 2920398f..a40ab99a 100644 --- a/data-loading/configsets/name_lookup/conf/solrconfig.xml +++ b/data-loading/configsets/name_lookup/conf/solrconfig.xml @@ -83,8 +83,14 @@ - + are not set here. See NameResolution#266. + + class is pinned because maxRamMB is a CaffeineCache feature. CaffeineCache is + Solr's default today, but if a later release changed that default to an impl + without RAM accounting, maxRamMB would be silently ignored and this cache + would grow unbounded. The size-based caches above do not pin it, since an + entry count is honoured by any cache implementation. --> + From 60c18476bbfb46c86a9c130461c2b15a48bac150 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 15:08:52 -0400 Subject: [PATCH 34/46] Tolerate whitespace when extracting Solr's numFound The load guard and the CI backup-roundtrip check both scrape numFound out of /query with a regex that assumed "numFound": with no space. Solr 9.10 emits exactly that today (verified: "numFound":1), so this is defensive, but /query runs with indent=true and a JSON-writer change to "numFound": 1 would turn the guard into a false failure. Allow optional whitespace after the colon; head -1 still skips the sibling numFoundExact key. Addresses two Copilot review comments on #278. Co-Authored-By: Claude Opus 4.8 --- .github/workflows/tester.yml | 2 +- data-loading/setup-and-load-solr.sh | 6 ++++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/.github/workflows/tester.yml b/.github/workflows/tester.yml index 4626d861..546e5653 100644 --- a/.github/workflows/tester.yml +++ b/.github/workflows/tester.yml @@ -104,7 +104,7 @@ jobs: sleep 2 done count=$(curl -sf 'http://localhost:8984/solr/name_lookup/query?q=*:*&rows=0' \ - | grep -oE '"numFound":[0-9]+' | head -1 | grep -oE '[0-9]+') + | grep -oE '"numFound":[[:space:]]*[0-9]+' | head -1 | grep -oE '[0-9]+') expected=$(grep -c '[^[:space:]]' tests/data/test-synonyms.json) echo "Restored core reports ${count:-0} documents; expected ${expected}." [ "${count:-0}" = "${expected}" ] diff --git a/data-loading/setup-and-load-solr.sh b/data-loading/setup-and-load-solr.sh index 6fb76aaf..d8340a22 100755 --- a/data-loading/setup-and-load-solr.sh +++ b/data-loading/setup-and-load-solr.sh @@ -35,10 +35,12 @@ STARTUP_TRIES="${SOLR_STARTUP_TRIES:-60}" GLOB="${1:?usage: setup-and-load-solr.sh \"data/synonyms/*.txt*\" (quote the glob!)}" -# Number of documents currently in the core. +# Number of documents currently in the core. The regex tolerates optional whitespace +# after the colon: /query runs with indent=true, so Solr's JSON writer may format this +# as `"numFound": 89`. head -1 guards against sibling keys like numFoundExact. solr_count() { curl -sf "${SOLR_SERVER}/solr/${CORE}/query?q=*:*&rows=0" \ - | grep -oE '"numFound":[0-9]+' | head -1 | grep -oE '[0-9]+' + | grep -oE '"numFound":[[:space:]]*[0-9]+' | head -1 | grep -oE '[0-9]+' } # Step 1. Wait for the core to be available (bounded: a Solr that never comes up From 6ddb07d2d5a1268a21815e22c16a9941519875a9 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 15:27:08 -0400 Subject: [PATCH 35/46] Trim image-fingerprint checks from the k8s loading README The per-setting checks (ls available-cpus.sh, grep ramBufferSizeMB/SOLR_MEM) were a crutch for the unversioned transition image; with released version tags the image tag itself identifies the pipeline. Drop them, keep the durable facts (pipeline is baked into the image, `latest` is the previous pipeline until a release is cut, files are editable in the pod before make), and point at the org.opencontainers.image.revision label as the reliable tag-to-commit check. Co-Authored-By: Claude Opus 4.8 --- data-loading/kubernetes/README.md | 32 ++++++++++++++----------------- 1 file changed, 14 insertions(+), 18 deletions(-) diff --git a/data-loading/kubernetes/README.md b/data-loading/kubernetes/README.md index 9722a3c3..f11405a6 100644 --- a/data-loading/kubernetes/README.md +++ b/data-loading/kubernetes/README.md @@ -61,24 +61,20 @@ $ make all SYNONYMS_URL=https://stars.renci.org/var/babel_outputs//syno $ make data/backup.done ``` -### Check you have the image you think you have - -The Makefile, the loader script and the configset are all **baked into the image**. -Editing them in a checkout does nothing; the `image:` tag in the pod spec is what -decides which version of the pipeline actually runs. The publish workflow only tags -`latest` on a published release, so between merging a change and cutting a release, -`latest` is the *previous* pipeline. Before starting an eight-hour load: - -```shell -$ ls available-cpus.sh # missing => pre-2026-07 image -$ grep ramBufferSizeMB configsets/name_lookup/conf/solrconfig.xml -$ grep '^SOLR_MEM' Makefile -``` - -If the image is close enough but one setting is stale, you can edit these files -inside the pod before running `make`. The configset in particular is only read when -the core is created, so changing `solrconfig.xml` before `make all` takes full -effect. +### Which pipeline runs: the image tag decides + +The Makefile, loader script and configset are all **baked into the image** -- editing +them in a checkout does nothing, and the `image:` tag in `nameres-loading.k8s.yaml` is +what decides which version of the pipeline runs. So run a load from a released version +tag, not `latest`: the publish workflow only tags `latest` on a published release, so +between merging a change and cutting a release, `latest` is still the *previous* +pipeline. To resolve any tag to an exact commit, the image carries its git SHA in the +`org.opencontainers.image.revision` label (stamped by `docker/metadata-action` at +publish), which `docker inspect` or the ghcr UI will show. + +If you do need to run a not-yet-released change, you can edit these files inside the +pod before `make`. The configset in particular is only read when the core is created, +so changing `solrconfig.xml` before `make all` takes full effect. `make all` downloads and splits the synonym files, starts Solr, creates the `name_lookup` core from the checked-in configset and loads it. `make From a48765bf29d116ca80b38999e6602aa181f682a7 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Thu, 23 Jul 2026 15:33:22 -0400 Subject: [PATCH 36/46] Incremented version to v1.7.0. --- api/resources/openapi.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api/resources/openapi.yml b/api/resources/openapi.yml index 07313170..b06f18fd 100644 --- a/api/resources/openapi.yml +++ b/api/resources/openapi.yml @@ -1,7 +1,7 @@ openapi: 3.0.2 info: title: Name Resolver - version: 1.6.2 + version: 1.7.0 email: bizon@renci.org name: Chris Bizon x-id: https://github.com/cbizon From 733ff78d768f7bce8b8fc8fb7c478d8f955617c5 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Mon, 27 Jul 2026 02:46:39 -0400 Subject: [PATCH 37/46] Fix four broken relative links in documentation/ The links in documentation/Deployment.md are written as if the file sat at the repository root, so ./docker-compose.yml, ./data-loading/README.md and ./Dockerfile all 404 from inside documentation/. documentation/Scoring.md pointed at a Babel README anchor (?tab=readme-ov-file#how-does-babel-choose-a-preferred-label-for-a-clique) that no longer exists: that section moved into Babel's docs/Understanding.md. Point at the file rather than at an anchor within it, so a future reorganization of that document does not break the link again. Co-Authored-By: Claude Opus 5 --- documentation/Deployment.md | 8 ++++---- documentation/Scoring.md | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/documentation/Deployment.md b/documentation/Deployment.md index 45424e30..7b39f9fc 100644 --- a/documentation/Deployment.md +++ b/documentation/Deployment.md @@ -17,7 +17,7 @@ instance or from Translator. with [Docker Compose](https://docs.docker.com/compose/install/). 2. Create the local directory where your Solr data will be stored -- by default, this is `./data/solr` in this directory, but you can change this in - [docker-compose.yml](./docker-compose.yml). This directory will need to have a maximum + [docker-compose.yml](../docker-compose.yml). This directory will need to have a maximum storage of approx 400G: 104G of the downloaded file (which can be deleted once decompressed), 147G of uncompressed backup (both of which can be deleted once restored) and 147G of Apache Solr databases. @@ -40,7 +40,7 @@ instance or from Translator. ordinary user gives the files to you instead, and Solr will fail to load the core. For a backup built before this was the case, or if you see permission errors in the Solr log, fix it with `sudo chown -R 8983:8983 ./data/solr/name_lookup`. -5. Check the [docker-compose.yml](./docker-compose.yml) file to ensure that it is +5. Check the [docker-compose.yml](../docker-compose.yml) file to ensure that it is as you expect. * The Docker Compose file will use the latest released version of NameRes as the frontend. To use the source code in this repository, you will need to change @@ -75,11 +75,11 @@ instance or from Translator. #### Loading from synonyms files -The best way to do this is by using the [data-loading Docker image](./data-loading/README.md). +The best way to do this is by using the [data-loading Docker image](../data-loading/README.md). ### Python packaging -Currently, NameRes is only packaged as a Docker image (see [Dockerfile](./Dockerfile)), but you can +Currently, NameRes is only packaged as a Docker image (see [Dockerfile](../Dockerfile)), but you can also run it directly via Uvicorn. ```bash diff --git a/documentation/Scoring.md b/documentation/Scoring.md index 3da11b7b..1bf89686 100644 --- a/documentation/Scoring.md +++ b/documentation/Scoring.md @@ -15,7 +15,7 @@ If multiple terms are matched in the same document, the sum of the score for eac The TF*IDF score will be multiplied by [several boosts](https://github.com/NCATSTranslator/NameResolution/blob/56e2151bb9e6fd120644cebdf4ff45b3bc47da05/api/server.py#L436-L461) that depend on four factors: * We index two fields: the "preferred name" of every clique and the "synonyms" of every clique. The [preferred name - is chosen by Babel](https://github.com/NCATSTranslator/Babel?tab=readme-ov-file#how-does-babel-choose-a-preferred-label-for-a-clique), + is chosen by Babel](https://github.com/NCATSTranslator/Babel/blob/master/docs/Understanding.md), while the synonyms are collected from all the different Babel sources. * We set up two indexes: a [StandardTokenizer](https://solr.apache.org/guide/solr/latest/indexing-guide/tokenizers.html#standard-tokenizer) that splits the field into tokens at whitespace and punctuation characters, and a From 97046503c86c7c246dfac699dc2df3b90ee4627e Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Mon, 27 Jul 2026 03:01:45 -0400 Subject: [PATCH 38/46] Add documentation/Babel.md and route cross-repo links through it NameRes inherits a lot of behaviour from Babel that is visible through this API but explained nowhere here: which Babel synonym files an instance is actually built from, why conflation cannot be turned off per query, what the leftover-UMLS singletons in the results are, why `label` may not be the label of the CURIE you looked up, why `synonyms[0]` is not the best synonym, what `clique_identifier_count` really counts, why an empty `taxa` is not the same as "not taxon-specific", and why there are no descriptions. Collect all of that in one document, along with where to file a bug (concept wrong -> Babel, service wrong -> NameRes) and the fact that an index is a snapshot of a single Babel release rather than live output. Babel.md is now the only file in this repository that links to a specific file inside the Babel repository, so a reorganization there is a one-file fix here rather than a hunt through five. Bare links to the Babel repository root stay where they read naturally -- those cannot rot. Also fixes two sentences that were broken where they sat: the conflation section of API.md ended mid-example ('if you search for ""'), and the OpenAPI description read "Note that the returned by this service". Co-Authored-By: Claude Opus 5 --- README.md | 16 ++++- api/resources/openapi.yml | 5 +- documentation/API.md | 18 +++-- documentation/Babel.md | 146 ++++++++++++++++++++++++++++++++++++++ documentation/Scoring.md | 8 ++- 5 files changed, 183 insertions(+), 10 deletions(-) create mode 100644 documentation/Babel.md diff --git a/README.md b/README.md index 79c60fe0..b53db2cf 100644 --- a/README.md +++ b/README.md @@ -6,9 +6,23 @@ Name Resolver (Name Lookup or NameRes) takes lexical strings and attempts to map along with many other options. Given a preferred CURIE, the known synonyms of that CURIE can also be retrieved. Multiple results may be returned representing possible conceptual matches, but all of the identifiers have been correctly normalized using the [Node Normalization](https://github.com/NCATSTranslator/NodeNormalization) service. -Note that the results returned by this service have been conflated using both GeneProtein and DrugChemical conflation; you can read more about this at the [Conflation documentation](https://github.com/NCATSTranslator/Babel/blob/master/docs/Conflation.md). +The concepts NameRes searches are built by the [Babel](https://github.com/NCATSTranslator/Babel) +pipeline. Note that the results returned by this service have been conflated using both GeneProtein +and DrugChemical conflation; you can read more about this, and about the other Babel behaviour +visible through this API, at [Where NameRes data comes from](documentation/Babel.md). * See this [Jupyter Notebook](documentation/NameResolution.ipynb) for examples of use. * See the [API documentation](documentation/API.md) for information about the NameRes API. * See [Scoring](documentation/Scoring.md) for information about the scoring algorithm used by NameRes. +* See [Where NameRes data comes from](documentation/Babel.md) for how Babel builds the concepts + NameRes searches, and the gotchas that follow from that. * See [Deployment](documentation/Deployment.md) for instructions on deploying NameRes. + +## Reporting a problem + +If a *concept* is wrong -- identifiers merged that shouldn't be, a wrong Biolink type or preferred +label, missing synonyms -- report it to [Babel](https://github.com/NCATSTranslator/Babel/issues), +which builds the concepts. If the *service* is wrong -- errors, badly ranked results, a parameter +not behaving as documented -- report it +[here](https://github.com/NCATSTranslator/NameResolution/issues). If you're not sure, file it in +Babel. See [Where NameRes data comes from](documentation/Babel.md#reporting-a-problem) for details. diff --git a/api/resources/openapi.yml b/api/resources/openapi.yml index b06f18fd..cfea8e04 100644 --- a/api/resources/openapi.yml +++ b/api/resources/openapi.yml @@ -13,8 +13,9 @@ info: have been correctly normalized using the Node Normalization service.

You can read more about this API on the NameResolution GitHub repository.

- Note that the returned by this service have been conflated using both GeneProtein and DrugChemical conflation; - you can read more about this at the Conflation documentation.' + Note that the results returned by this service have been conflated using both GeneProtein and DrugChemical conflation; + you can read more about this, and about the other Babel behaviour visible through this API, at + Where NameRes data comes from.' license: name: MIT url: https://opensource.org/licenses/MIT diff --git a/documentation/API.md b/documentation/API.md index 5311a2d5..6c11f895 100644 --- a/documentation/API.md +++ b/documentation/API.md @@ -3,8 +3,9 @@ The Name Resolver API is intended to provide an [Apache Solr](https://solr.apache.org/)-based interface to the [Babel](https://github.com/NCATSTranslator/Babel) cliques of equivalent identifiers. Apache Solr is a document-based search engine: the documents in this case are descriptions of cliques as generated by the -[Babel](https://github.com/NCATSTranslator/Babel) pipeline in its -[Synonyms format](https://github.com/NCATSTranslator/Babel/blob/master/docs/DataFormats.md#synonym-files), including lists of all known synonyms. +[Babel](https://github.com/NCATSTranslator/Babel) pipeline in its Synonyms format, including lists of all known synonyms. +See [Where NameRes data comes from](./Babel.md) for what Babel puts in these documents, which Babel files a NameRes +instance is built from, and the behaviour that follows from both. Here is an example document for [NCBIGene:1756](https://name-resolution-sri.renci.org/synonyms?preferred_curies=NCBIGene%3A1756) (compared with the same CURIE [on NodeNorm](https://nodenormalization-sri.renci.org/get_normalized_nodes?curie=NCBIGene:1756)): @@ -91,11 +92,15 @@ The Name Resolver largely consists of two [search endpoints](#search-endpoints): ## Conflation Unlike the Node Normalizer, the Name Resolution Service does not currently support on-the-fly conflation. Instead, -all the [Babel conflations](https://github.com/NCATSTranslator/Babel/blob/master/docs/Conflation.md) are turned on when Solr database is built. At the moment, this includes: +all the Babel conflations are turned on when the Solr database is built. At the moment, this includes: + * GeneProtein conflation: protein-encoding genes are conflated with the protein(s) they encode, and the gene identifier - is used to identify this concept. Therefore, if you search for "" + is used to identify this concept. The `NCBIGene:1756` document shown above is an example: it is a gene, but its + `names` include protein synonyms such as `dystrophin` and `DMD_HUMAN Dystrophin (sprot)`, and there is no separate + document for the protein itself. * DrugChemical conflation: drugs are conflated with their active ingredient, and the identifier for the active ingredient is used to identify this concept. + This means that -- for example -- protein-encoding genes will include the synonyms found for the protein they encode, and that no separate entry will be available for those proteins. @@ -103,6 +108,9 @@ Once you have an identifier from Name Resolver, you can use the [Node Normalizer to look up the equivalent identifiers for that CURIE with and without conflation. Please use the Node Normalizer corresponding to the instance of Name Resolver you are using. +See [Where NameRes data comes from](./Babel.md#geneproteinconflated-and-drugchemicalconflated-replace-the-per-type-files) +for why conflation is baked into the index rather than applied per query, and what each conflation merges. + We are currently working on supporting [on-the-fly conflation](https://github.com/NCATSTranslator/NameResolution/issues/224) in Name Resolver. @@ -302,6 +310,8 @@ POST /synonyms with body: - This endpoint provides a complete view of all synonyms and properties for a given concept. - CURIEs are conflated with both GeneProtein and DrugChemical conflation. For example, when looking up a protein CURIE, the synonyms for the gene that encodes the protein will also be included. See the [Conflation section](#conflation) for more information. - If a CURIE is not found in the database, it will still appear in the response dictionary with an empty object as its value. +- Concept descriptions are not included: Babel collects them, but they are not carried into the synonym files NameRes + indexes. Use [NodeNorm](https://nodenormalization-sri.renci.org/)'s `description` flag if you need them. - For large batches of CURIEs, consider using this endpoint instead of making individual requests, as it is optimized for batch lookups. ## Health endpoints diff --git a/documentation/Babel.md b/documentation/Babel.md new file mode 100644 index 00000000..d9c5b220 --- /dev/null +++ b/documentation/Babel.md @@ -0,0 +1,146 @@ +# Where NameRes data comes from + +NameRes does not decide what a concept is. It is a search interface over concepts built by +[Babel](https://github.com/NCATSTranslator/Babel), the pipeline that merges identifiers from +dozens of biomedical vocabularies into "cliques" of equivalent identifiers and collects every +synonym for each one. + +The division of labour: + +- **Babel decides** which identifiers belong together, which one leads the clique, what the + preferred name is, which Biolink type it gets, and which synonyms it has. +- **NameRes decides** how those concepts are indexed, matched and ranked (see + [Scoring](./Scoring.md)), and what the API looks like (see [API](./API.md)). + +This document covers the Babel behaviour that is *visible through the NameRes API* — the things +that will otherwise look like NameRes bugs. It is also the only place in this repository that +links into the Babel repository, so that a reorganization there is a one-file fix here. + +## What is actually loaded + +NameRes is built from Babel's published `synonyms/` directory for one release +(`SYNONYMS_URL` in [`data-loading/Makefile`](../data-loading/Makefile)); the loader takes every +`*.txt`/`*.txt.gz` file in it. As of the `2025sep1` and `2026jul22` releases that is: + +```text +AnatomicalEntity BiologicalProcess Cell CellLine CellularComponent Disease +DrugChemicalConflated GeneFamily GeneProteinConflated GrossAnatomicalStructure +MacromolecularComplex MolecularActivity OrganismTaxon Pathway PhenotypicFeature +Publication umls +``` + +The set of searchable Biolink types is therefore Babel's decision, not NameRes's: if a Babel +release adds a synonym file, the next NameRes load picks it up with no change to this repository. + +Two entries in that list are worth reading twice. + +### `GeneProteinConflated` and `DrugChemicalConflated` replace the per-type files + +There is no `Gene.txt`, `Protein.txt` or `SmallMolecule.txt` in the published directory — the +conflated files stand in for them. This is why NameRes results are conflated and why, unlike +NodeNorm, NameRes cannot turn conflation off on a per-query basis: it is baked into the index at +load time. + +The practical consequence is that **there is no separate document to find for the conflated-away +member.** A protein is searchable, but the result you get back is identified by the gene that +encodes it and carries the union of both sets of synonyms; likewise a drug is identified by its +active ingredient. What each conflation merges, and in what order, is described in +[Babel's Conflation documentation](https://github.com/NCATSTranslator/Babel/blob/master/docs/Conflation.md). + +Once you have an identifier from NameRes you can use +[NodeNorm](https://nodenormalization-sri.renci.org/) to see the clique with and without each +conflation. Use the NodeNorm instance that corresponds to the NameRes instance you are querying. + +### `umls` is Babel's leftover-UMLS compendium + +Babel's last pipeline step sweeps up every valid UMLS concept that no other compendium claimed and +writes each one out as a **single-identifier clique**. These are in your search results. They are +usually the reason for a result that has a `UMLS:` CURIE, a `clique_identifier_count` of 1, and a +broader Biolink type than you expected — Babel derives the type from the concept's UMLS semantic +type, falling back to `biolink:NamedThing` where UMLS asserts none. + +This is deliberate coverage, not a defect: it means a UMLS concept still resolves to *something* +even when Babel could not merge it into a richer clique. See +[Babel's leftover-UMLS documentation](https://github.com/NCATSTranslator/Babel/blob/master/docs/sources/UMLS/Leftover.md). + +## Gotchas in the returned fields + +Every field NameRes returns from `/lookup` and `/synonyms` comes from Babel's +[synonym file format](https://github.com/NCATSTranslator/Babel/blob/master/docs/DataFormats.md). +Four of them behave in ways that surprise people. + +### `label` is not necessarily the label of the CURIE you looked up + +`label` is Babel's `preferred_name` for the whole clique, and Babel may deliberately choose a name +that is not the label of the clique leader — to disambiguate the concept, or to give the +Translator UI something better to display. Looking up a CURIE and getting back a different-looking +name is expected. How the preferred name is chosen (including the chemical-specific boost prefixes +and the length demotion) is in +[Understanding Babel outputs](https://github.com/NCATSTranslator/Babel/blob/master/docs/Understanding.md). + +### `synonyms` is not ordered by quality + +Babel writes `names` shortest-first — *except* for conflated cliques, where it writes all the +synonyms of the first clique, then all the synonyms of the second, and so on. Since almost +everything NameRes loads is conflated, `synonyms[0]` is not reliably the best or shortest name. +Use `label` when you want one name for a concept. + +### `clique_identifier_count` measures coverage, not importance + +NameRes multiplies each search score by the logarithm of this count (see +[Scoring](./Scoring.md)), on the theory that a widely cross-referenced concept is the one a user +more likely meant. What it literally counts is how many identifiers Babel merged into the clique, +which tracks how many source vocabularies happen to cover the concept. It is a good proxy in +practice, but it is why a leftover-UMLS singleton ranks below a heavily cross-referenced chemical +even when the two match the query equally well. + +### `taxa` is a union, and an empty list means "nobody said" + +Babel collects taxa per identifier and the clique's `taxa` is the union of those. An empty `taxa` +means no source asserted a taxon for any member, not that the concept is taxon-agnostic. This is +why `only_taxa` filtering keeps results that have *no* taxon alongside results with a matching +one — excluding them would drop concepts that are simply unannotated. + +### There are no descriptions + +Babel collects descriptions (from [UberGraph](https://github.com/INCATools/ubergraph/)) and +NodeNorm can return them, but they are not carried into the synonym files and so are not in the +NameRes index. Use NodeNorm's `description` flag if you need them. + +## The index is a snapshot of one Babel release + +A NameRes instance serves a fixed build, not live Babel output. `/status` reports which one: + +```json +{ + "babel_version": "2025sep1", + "babel_version_url": "https://github.com/ncatstranslator/Babel/blob/master/releases/2025sep1.md", + "biolink_model": { "tag": "v4.2.6-rc5" } +} +``` + +A concept that Babel has since fixed will still be wrong here until the instance is rebuilt and +redeployed. Check `/status` before reporting a stale clique, and see +[Babel's releases](https://github.com/NCATSTranslator/Babel/blob/master/releases/README.md) for +what changed between builds. + +## Reporting a problem + +Babel and NameRes have separate issue trackers, and the split is by *what is wrong*: + +- **The concept itself is wrong** — wrong identifiers merged or not merged, wrong Biolink type, + wrong preferred label, missing or bogus synonyms → the + [Babel issue tracker](https://github.com/NCATSTranslator/Babel/issues). +- **The service is wrong** — errors, results ranked badly for a query, a parameter not behaving as + documented → the + [NameRes issue tracker](https://github.com/NCATSTranslator/NameResolution/issues). + +If you are not sure, file it in Babel and it will be sorted out. +[Babel's guide to filing an issue](https://github.com/NCATSTranslator/Babel/blob/master/docs/NewIssue.md) +describes what to include and how the priority/impact/size fields are used. + +## Going deeper + +Everything above is what a NameRes user needs. If you want to know how the cliques are actually +built — the per-source ingestion, the concord files, the union-find merge, running the pipeline +yourself — start at [Babel's documentation index](https://github.com/NCATSTranslator/Babel/blob/master/docs/). diff --git a/documentation/Scoring.md b/documentation/Scoring.md index 1bf89686..e1a04dbe 100644 --- a/documentation/Scoring.md +++ b/documentation/Scoring.md @@ -14,9 +14,9 @@ If multiple terms are matched in the same document, the sum of the score for eac The TF*IDF score will be multiplied by [several boosts](https://github.com/NCATSTranslator/NameResolution/blob/56e2151bb9e6fd120644cebdf4ff45b3bc47da05/api/server.py#L436-L461) that depend on four factors: -* We index two fields: the "preferred name" of every clique and the "synonyms" of every clique. The [preferred name - is chosen by Babel](https://github.com/NCATSTranslator/Babel/blob/master/docs/Understanding.md), - while the synonyms are collected from all the different Babel sources. +* We index two fields: the "preferred name" of every clique and the "synonyms" of every clique. The preferred name + is chosen by Babel, while the synonyms are collected from all the different Babel sources; see + [Where NameRes data comes from](./Babel.md) for what that means for the values you get back. * We set up two indexes: a [StandardTokenizer](https://solr.apache.org/guide/solr/latest/indexing-guide/tokenizers.html#standard-tokenizer) that splits the field into tokens at whitespace and punctuation characters, and a [KeywordTokenizer](https://solr.apache.org/guide/solr/latest/indexing-guide/tokenizers.html#keyword-tokenizer) that @@ -27,6 +27,8 @@ that depend on four factors: (NOTE: this might be removed soon.) * We use the number of identifiers in the clique as a measure of how widely used a clique is. Since some cliques share the same preferred name or label, we can use this to promote the clique most likely to be useful. + (What this count literally measures, and where the proxy breaks down, is described in + [Where NameRes data comes from](./Babel.md#clique_identifier_count-measures-coverage-not-importance).) We combine these factors in this way in a standard query matches: From 6774fc48078198da46e7e38b9e404ba1c8e9e84b Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Mon, 27 Jul 2026 03:09:56 -0400 Subject: [PATCH 39/46] Correct documentation that had drifted from the code - Six of the OpenAPI endpoint descriptions linked into documentation/API.md with anchors that do not exist. Five used "#Conflation" where GitHub generates "#conflation", and GET /synonyms pointed at "#bulk-lookup". - documentation/Deployment.md's configuration list omitted BABEL_VERSION, BABEL_VERSION_URL, BIOLINK_MODEL_TAG and LOGLEVEL. The first three are the ones an operator most needs: they are what /status reports about the data an instance is serving, and there was no way to learn they existed. - CLAUDE.md listed two endpoints that do not exist (/lookup-curies, and /reverse-lookup rather than the actual deprecated /reverse_lookup), named api/resources/openapi.yml with a leading dot, and pinned a line count for api/server.py that no longer matched. The endpoint list now matches the five paths FastAPI actually generates. Co-Authored-By: Claude Opus 5 --- CLAUDE.md | 22 ++++++++++++++-------- api/server.py | 12 ++++++------ documentation/Deployment.md | 10 ++++++++++ 3 files changed, 30 insertions(+), 14 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index ec5676a8..e5a2c39a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -59,26 +59,29 @@ pip install -r requirements.txt 4. Results are scored, normalized, and returned as JSON ### Key Files -- `api/server.py` - Core FastAPI application (~717 lines): all endpoints, Pydantic models, Solr query construction, environment config +- `api/server.py` - Core FastAPI application: all endpoints, Pydantic models, Solr query construction, environment config - `api/apidocs.py` - Custom OpenAPI schema construction -- `api/resources/.openapi.yml` - OpenAPI 3.0.2 spec with service metadata +- `api/resources/openapi.yml` - OpenAPI 3.0.2 spec with service metadata - `main.py` / `main.sh` - WSGI/ASGI entry points (port 2433) - `tests/test_service.py` - Integration tests using FastAPI `TestClient` - `tests/data/test-synonyms.json` - Test dataset for Solr ### Environment Variables -- `SOLR_HOST` / `SOLR_PORT` - Solr connection (default: `localhost:8983`) +See `documentation/Deployment.md` for the full list. The main ones: +- `SOLR_HOST` / `SOLR_PORT` / `SOLR_CORE` - Solr connection (default: `localhost:8983`, core `name_lookup`) - `LOGLEVEL` - Logging level -- `SERVER_ROOT` - API root path prefix +- `SERVER_NAME` / `SERVER_ROOT` - Infores ID and API root path prefix - `MATURITY_VALUE` / `LOCATION_VALUE` - TRAPI metadata fields +- `BABEL_VERSION` / `BABEL_VERSION_URL` / `BIOLINK_MODEL_TAG` - reported by `/status`; describe the + data the index was built from +- `OTEL_ENABLED` / `JAEGER_*` - OpenTelemetry ### API Endpoints - `GET/POST /lookup` - Primary name-to-CURIE lookup with scoring - `POST /bulk-lookup` - Batch queries via `NameResQuery` model -- `GET /reverse-lookup` - CURIE-to-names lookup -- `POST /synonyms` - Get synonyms for a list of CURIEs -- `POST /lookup-curies` - Filter existing CURIEs with type subsetting -- `GET /status` - Health check with Solr document counts +- `GET/POST /synonyms` - Get synonyms for a list of preferred CURIEs +- `GET/POST /reverse_lookup` - Deprecated alias for `/synonyms` +- `GET /status` - Health check with Solr document counts, plus the Babel and Biolink versions ### Data Model Solr documents contain: `curie`, `preferred_name`, `names` (synonym list), and biolink type information. Lookup results are `LookupResult` objects with scoring fields. Results are conflated using GeneProtein and DrugChemical conflation rules. @@ -91,6 +94,9 @@ Solr documents contain: `curie`, `preferred_name`, `names` (synonym list), and b ## Documentation - `documentation/API.md` - Endpoint reference +- `documentation/Babel.md` - Where the data comes from, and the Babel behaviour visible through this + API. This is the only file that should link to a *specific file* inside the Babel repository; keep + new cross-repo links here so a reorganization there is a one-file fix. - `documentation/Deployment.md` - Docker/Kubernetes deployment guide - `documentation/Scoring.md` - Scoring algorithm details - `documentation/NameResolution.ipynb` - Interactive usage examples \ No newline at end of file diff --git a/api/server.py b/api/server.py index df8c30c6..74fc1987 100755 --- a/api/server.py +++ b/api/server.py @@ -179,8 +179,8 @@ async def reverse_lookup_get( "/synonyms", summary="Look up synonyms for a CURIE.", description="

Returns a list of synonyms for a particular preferred CURIE. You can normalize a CURIE to a preferred CURIE using NodeNorm.

" - "

You can find out more about this endpoint in the API documentation.

" - "

Note that CURIEs are conflated with both GeneProtein and DrugChemical conflation, so that e.g. when searching for a protein, the identifier of the gene that encodes the protein will be returned itself. See Conflation documentation for more information.

", + "

You can find out more about this endpoint in the API documentation.

" + "

Note that CURIEs are conflated with both GeneProtein and DrugChemical conflation, so that e.g. when searching for a protein, the identifier of the gene that encodes the protein will be returned itself. See Conflation documentation for more information.

", response_model=Dict[str, Dict], tags=["lookup"], ) @@ -216,7 +216,7 @@ async def lookup_names_post( summary="Look up synonyms for a CURIE.", description="

Returns a list of synonyms for a particular preferred CURIE. You can normalize a CURIE to a preferred CURIE using NodeNorm.

" "

You can find out more about this endpoint in the API documentation.

" - "

Note that CURIEs are conflated with both GeneProtein and DrugChemical conflation, so that e.g. a protein that encodes a gene can be looked up with the gene's CURIE, not the protein's CURIE. See Conflation documentation for more information.

", + "

Note that CURIEs are conflated with both GeneProtein and DrugChemical conflation, so that e.g. a protein that encodes a gene can be looked up with the gene's CURIE, not the protein's CURIE. See Conflation documentation for more information.

", response_model=Dict[str, Dict], tags=["lookup"], ) @@ -274,7 +274,7 @@ class LookupResult(BaseModel): summary="Look up cliques for a fragment of a name or synonym.", description="

Returns cliques with a name or synonym that contains a specified string.

" "

You can find out more about this endpoint in the API documentation.

" - "

Note that CURIEs are conflated with both GeneProtein and DrugChemical conflation, so that e.g. when searching for a protein, the identifier of the gene that encodes the protein will be returned itself. See Conflation documentation for more information.

", + "

Note that CURIEs are conflated with both GeneProtein and DrugChemical conflation, so that e.g. when searching for a protein, the identifier of the gene that encodes the protein will be returned itself. See Conflation documentation for more information.

", response_model=List[LookupResult], tags=["lookup"] ) @@ -338,7 +338,7 @@ async def lookup_curies_get( summary="Look up cliques for a fragment of a name or synonym.", description="

Returns cliques with a name or synonym that contains a specified string.

" "

You can find out more about this endpoint in the API documentation.

" - "

Note that CURIEs are conflated with both GeneProtein and DrugChemical conflation, so that e.g. when searching for a protein, the identifier of the gene that encodes the protein will be returned itself. See Conflation documentation for more information.

", + "

Note that CURIEs are conflated with both GeneProtein and DrugChemical conflation, so that e.g. when searching for a protein, the identifier of the gene that encodes the protein will be returned itself. See Conflation documentation for more information.

", response_model=List[LookupResult], tags=["lookup"] ) @@ -692,7 +692,7 @@ class NameResQuery(BaseModel): summary="Look up cliques for a fragment of multiple names or synonyms.", description="

Returns cliques for each query.

" "

You can find out more about this endpoint in the API documentation.

" - "

Note that CURIEs are conflated with both GeneProtein and DrugChemical conflation, so that e.g. a protein that encodes a gene can be looked up with the gene's CURIE, not the protein's CURIE. See Conflation documentation for more information.

", + "

Note that CURIEs are conflated with both GeneProtein and DrugChemical conflation, so that e.g. a protein that encodes a gene can be looked up with the gene's CURIE, not the protein's CURIE. See Conflation documentation for more information.

", response_model=Dict[str, List[LookupResult]], tags=["lookup"] ) diff --git a/documentation/Deployment.md b/documentation/Deployment.md index 7b39f9fc..5f779767 100644 --- a/documentation/Deployment.md +++ b/documentation/Deployment.md @@ -109,6 +109,16 @@ NameRes can be configured by setting environmental variables: * `SERVER_ROOT`: The server root (defaults to `/`) * `MATURITY_VALUE`: How mature is this NameRes (defaults to `maturity`, e.g. `development`) * `LOCATION_VALUE`: Where is this NameRes setup (defaults to `location`, e.g. `RENCI`) +* `LOGLEVEL`: The Python logging level (defaults to `INFO`). +* `BABEL_VERSION`: The [Babel](https://github.com/NCATSTranslator/Babel) release this Solr index was + built from, reported by `/status` (defaults to `unknown`, e.g. `2025sep1`). Set this when you load + an index -- it is the only record of which data an instance is serving, and `/status` is where + users look for it. See [Where NameRes data comes from](./Babel.md). +* `BABEL_VERSION_URL`: URL of the changelog for that Babel release, also reported by `/status` + (defaults to empty). +* `BIOLINK_MODEL_TAG`: The [Biolink Model](https://github.com/biolink/biolink-model) tag the Solr + index was built against (defaults to `master`, e.g. `v4.2.6-rc5`). `/status` derives the model URL + and download URL from it. * `OTEL_ENABLED`: Turn on Open TELemetry (default: `'false'`) -- only `'true'` will turn this on. * `JAEGER_HOST` and `JAEGER_PORT`: Hostname and port for the Jaegar instance to provide telemetry to. * `JAEGER_SERVICE_NAME`: The name of this service (defaults to the value of `SERVER_NAME`) From b8ce87ac33d5ec2dcc0313d310fd083d8fb12d6c Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Mon, 27 Jul 2026 03:16:50 -0400 Subject: [PATCH 40/46] Sharpen two field descriptions against a live index Verified against a local Solr loaded with tests/data/test-synonyms.json: - "No taxa" is not represented the same way by both endpoints. /lookup builds a LookupResult and so always emits taxa, defaulting to []; /synonyms returns the raw Solr document, which omits the field. Say so, and point at Babel's taxon_specific flag for code that wants the distinction as a boolean. - The /synonyms return list omitted taxon_specific and shortest_name_length, both of which are in every document. Co-Authored-By: Claude Opus 5 --- documentation/API.md | 5 ++++- documentation/Babel.md | 5 +++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/documentation/API.md b/documentation/API.md index 6c11f895..2f6ab1f7 100644 --- a/documentation/API.md +++ b/documentation/API.md @@ -258,8 +258,11 @@ Look up all synonyms and information for a CURIE. - `preferred_name`: The preferred name for this concept. - `names`: A list of all known synonyms for this concept. - `types`: A list of Biolink types for this concept. -- `taxa`: A list of taxa associated with this concept. +- `taxa`: A list of taxa associated with this concept. Unlike `/lookup`, this key is omitted + entirely rather than returned as an empty list when the concept has no taxa. +- `taxon_specific`: Whether this concept is associated with one or more specific taxa. - `clique_identifier_count`: The number of identifiers in this clique. +- `shortest_name_length`: The length of the shortest synonym in `names`. - Additional metadata fields such as `curie_suffix`, `id`, and `_version_`. **Example requests:** diff --git a/documentation/Babel.md b/documentation/Babel.md index d9c5b220..cb4796f0 100644 --- a/documentation/Babel.md +++ b/documentation/Babel.md @@ -101,6 +101,11 @@ means no source asserted a taxon for any member, not that the concept is taxon-a why `only_taxa` filtering keeps results that have *no* taxon alongside results with a matching one — excluding them would drop concepts that are simply unannotated. +Note that "no taxon" looks different between the two endpoints: `/lookup` always returns a `taxa` +key and gives it an empty list, while `/synonyms` returns the raw Solr document, which simply omits +the field. Babel's own `taxon_specific` flag is carried through to `/synonyms` if you want the +distinction as a boolean. + ### There are no descriptions Babel collects descriptions (from [UberGraph](https://github.com/INCATools/ubergraph/)) and From 2ff973cdba89e8cf341b295826d9528c6eb07c9e Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Mon, 27 Jul 2026 03:20:02 -0400 Subject: [PATCH 41/46] Link the /lookup vs /synonyms divergences to issue #291 The response-shape differences between the two endpoints are documented as current behaviour rather than as intent, so point at the issue that tracks removing them. Co-Authored-By: Claude Opus 5 --- documentation/API.md | 4 ++++ documentation/Babel.md | 3 ++- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/documentation/API.md b/documentation/API.md index 2f6ab1f7..7fc8693c 100644 --- a/documentation/API.md +++ b/documentation/API.md @@ -315,6 +315,10 @@ POST /synonyms with body: - If a CURIE is not found in the database, it will still appear in the response dictionary with an empty object as its value. - Concept descriptions are not included: Babel collects them, but they are not carried into the synonym files NameRes indexes. Use [NodeNorm](https://nodenormalization-sri.renci.org/)'s `description` flag if you need them. +- This endpoint returns the underlying Solr document, so it describes a concept differently from `/lookup`: it uses + `preferred_name`/`names` rather than `label`/`synonyms`, returns `types` without the `biolink:` prefix, and includes + index metadata. We intend to + [harmonize the two response shapes](https://github.com/NCATSTranslator/NameResolution/issues/291). - For large batches of CURIEs, consider using this endpoint instead of making individual requests, as it is optimized for batch lookups. ## Health endpoints diff --git a/documentation/Babel.md b/documentation/Babel.md index cb4796f0..c00bd4cf 100644 --- a/documentation/Babel.md +++ b/documentation/Babel.md @@ -104,7 +104,8 @@ one — excluding them would drop concepts that are simply unannotated. Note that "no taxon" looks different between the two endpoints: `/lookup` always returns a `taxa` key and gives it an empty list, while `/synonyms` returns the raw Solr document, which simply omits the field. Babel's own `taxon_specific` flag is carried through to `/synonyms` if you want the -distinction as a boolean. +distinction as a boolean. That inconsistency is ours, not Babel's, and we intend to +[remove it](https://github.com/NCATSTranslator/NameResolution/issues/291). ### There are no descriptions From 223af96f3771eeada2e68b093e25bafedca5014d Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Mon, 27 Jul 2026 03:35:51 -0400 Subject: [PATCH 42/46] Point every cross-repo link at main, not master Babel, NameResolution and NodeNormalization all use main as their default branch; Babel's was renamed from master relatively recently. NameResolution has no master branch at all. Links to /blob/master/ still resolve today only because GitHub keeps a redirect after a default-branch rename. That redirect is exactly the kind of thing that disappears quietly, which makes these the link rot this branch is supposed to be removing rather than an acceptable alias. This rewrites 24 links: the 12 pre-existing NameResolution self-links in api/server.py, the seven Babel links in documentation/Babel.md, the conflation_url reported by /status, the two Colab badge URLs, and the sample /status payloads in the docs. Reverses the normalization direction taken in the previous merge commit, which went to master on the strength of a stale origin/HEAD ref in a local Babel clone rather than the actual remote default. Co-Authored-By: Claude Opus 5 --- README.md | 2 +- api/resources/openapi.yml | 2 +- api/server.py | 24 ++++++++++++------------ documentation/API.md | 4 ++-- documentation/Babel.md | 16 ++++++++-------- documentation/NameResolution.ipynb | 2 +- 6 files changed, 25 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index fa24f8be..2eaa89a9 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ visible through this API, at [Where NameRes data comes from](documentation/Babel The best place to start is the Jupyter Notebook, which walks through the most common use cases with live examples: -* [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NCATSTranslator/NameResolution/blob/master/documentation/NameResolution.ipynb) [Jupyter Notebook](documentation/NameResolution.ipynb) — interactive examples covering lookup, filtering, autocomplete, bulk lookup, and synonyms +* [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NCATSTranslator/NameResolution/blob/main/documentation/NameResolution.ipynb) [Jupyter Notebook](documentation/NameResolution.ipynb) — interactive examples covering lookup, filtering, autocomplete, bulk lookup, and synonyms ## Documentation diff --git a/api/resources/openapi.yml b/api/resources/openapi.yml index b3db8295..3a2e0d2a 100644 --- a/api/resources/openapi.yml +++ b/api/resources/openapi.yml @@ -17,7 +17,7 @@ info: conflation; the active conflations for any deployment can be discovered via the /status endpoint. You can read more about this, and about the other Babel behaviour visible through this API, at - Where NameRes data comes from.

' + Where NameRes data comes from.

' license: name: MIT url: https://opensource.org/licenses/MIT diff --git a/api/server.py b/api/server.py index 606fa18c..4cf31c95 100755 --- a/api/server.py +++ b/api/server.py @@ -51,7 +51,7 @@ async def docs_redirect(): @app.get("/status", summary="Get status and counts for this NameRes instance.", description="

This endpoint will return status information and a list of counts from the underlying Solr database instance for this NameRes instance.

" - "

You can find out more about this endpoint in the API documentation.

" + "

You can find out more about this endpoint in the API documentation.

" ) async def status_get() -> Dict: """ Return status and count information from the underyling Solr instance. """ @@ -77,7 +77,7 @@ async def status() -> Dict: # Which conflations are active in this deployment? Baked in at data-loading time. conflations_raw = os.environ.get("CONFLATIONS", "GeneProtein,DrugChemical") conflations = [c.strip() for c in conflations_raw.split(",") if c.strip()] - conflation_url = "https://github.com/NCATSTranslator/Babel/blob/master/docs/Conflation.md" + conflation_url = "https://github.com/NCATSTranslator/Babel/blob/main/docs/Conflation.md" # Look up the BIOLINK_MODEL_TAG. # Note: this should be a tag from the Biolink Model repo, e.g. "master" or "v4.3.6". @@ -188,8 +188,8 @@ async def reverse_lookup_get( "/synonyms", summary="Look up synonyms for a CURIE.", description="

Returns a list of synonyms for a particular preferred CURIE. You can normalize a CURIE to a preferred CURIE using NodeNorm.

" - "

You can find out more about this endpoint in the API documentation.

" - "

Note that CURIEs are conflated with both GeneProtein and DrugChemical conflation, so that e.g. when searching for a protein, the identifier of the gene that encodes the protein will be returned itself. See Conflation documentation for more information.

", + "

You can find out more about this endpoint in the API documentation.

" + "

Note that CURIEs are conflated with both GeneProtein and DrugChemical conflation, so that e.g. when searching for a protein, the identifier of the gene that encodes the protein will be returned itself. See Conflation documentation for more information.

", response_model=Dict[str, Dict], tags=["lookup"], ) @@ -224,8 +224,8 @@ async def lookup_names_post( "/synonyms", summary="Look up synonyms for a CURIE.", description="

Returns a list of synonyms for a particular preferred CURIE. You can normalize a CURIE to a preferred CURIE using NodeNorm.

" - "

You can find out more about this endpoint in the API documentation.

" - "

Note that CURIEs are conflated with both GeneProtein and DrugChemical conflation, so that e.g. a protein that encodes a gene can be looked up with the gene's CURIE, not the protein's CURIE. See Conflation documentation for more information.

", + "

You can find out more about this endpoint in the API documentation.

" + "

Note that CURIEs are conflated with both GeneProtein and DrugChemical conflation, so that e.g. a protein that encodes a gene can be looked up with the gene's CURIE, not the protein's CURIE. See Conflation documentation for more information.

", response_model=Dict[str, Dict], tags=["lookup"], ) @@ -282,8 +282,8 @@ class LookupResult(BaseModel): @app.get("/lookup", summary="Look up cliques for a fragment of a name or synonym.", description="

Returns cliques with a name or synonym that contains a specified string.

" - "

You can find out more about this endpoint in the API documentation.

" - "

Note that CURIEs are conflated with both GeneProtein and DrugChemical conflation, so that e.g. when searching for a protein, the identifier of the gene that encodes the protein will be returned itself. See Conflation documentation for more information.

", + "

You can find out more about this endpoint in the API documentation.

" + "

Note that CURIEs are conflated with both GeneProtein and DrugChemical conflation, so that e.g. when searching for a protein, the identifier of the gene that encodes the protein will be returned itself. See Conflation documentation for more information.

", response_model=List[LookupResult], tags=["lookup"] ) @@ -346,8 +346,8 @@ async def lookup_curies_get( @app.post("/lookup", summary="Look up cliques for a fragment of a name or synonym.", description="

Returns cliques with a name or synonym that contains a specified string.

" - "

You can find out more about this endpoint in the API documentation.

" - "

Note that CURIEs are conflated with both GeneProtein and DrugChemical conflation, so that e.g. when searching for a protein, the identifier of the gene that encodes the protein will be returned itself. See Conflation documentation for more information.

", + "

You can find out more about this endpoint in the API documentation.

" + "

Note that CURIEs are conflated with both GeneProtein and DrugChemical conflation, so that e.g. when searching for a protein, the identifier of the gene that encodes the protein will be returned itself. See Conflation documentation for more information.

", response_model=List[LookupResult], tags=["lookup"] ) @@ -700,8 +700,8 @@ class NameResQuery(BaseModel): @app.post("/bulk-lookup", summary="Look up cliques for a fragment of multiple names or synonyms.", description="

Returns cliques for each query.

" - "

You can find out more about this endpoint in the API documentation.

" - "

Note that CURIEs are conflated with both GeneProtein and DrugChemical conflation, so that e.g. a protein that encodes a gene can be looked up with the gene's CURIE, not the protein's CURIE. See Conflation documentation for more information.

", + "

You can find out more about this endpoint in the API documentation.

" + "

Note that CURIEs are conflated with both GeneProtein and DrugChemical conflation, so that e.g. a protein that encodes a gene can be looked up with the gene's CURIE, not the protein's CURIE. See Conflation documentation for more information.

", response_model=Dict[str, List[LookupResult]], tags=["lookup"] ) diff --git a/documentation/API.md b/documentation/API.md index 834c839a..240545b0 100644 --- a/documentation/API.md +++ b/documentation/API.md @@ -338,14 +338,14 @@ Solr database. "status": "ok", "message": "Reporting results from primary core.", "babel_version": "2025sep1", - "babel_version_url": "https://github.com/ncatstranslator/Babel/blob/master/releases/2025sep1.md", + "babel_version_url": "https://github.com/ncatstranslator/Babel/blob/main/releases/2025sep1.md", "biolink_model": { "tag": "v4.2.6-rc5", "url": "https://github.com/biolink/biolink-model/tree/v4.2.6-rc5", "download_url": "https://raw.githubusercontent.com/biolink/biolink-model/v4.2.6-rc5/biolink-model.yaml" }, "conflations": ["GeneProtein", "DrugChemical"], - "conflation_url": "https://github.com/NCATSTranslator/Babel/blob/master/docs/Conflation.md", + "conflation_url": "https://github.com/NCATSTranslator/Babel/blob/main/docs/Conflation.md", "nameres_version": "v1.5.1", "startTime": "2025-12-19T11:53:09.638Z", "numDocs": 425583391, diff --git a/documentation/Babel.md b/documentation/Babel.md index c00bd4cf..cb4e3816 100644 --- a/documentation/Babel.md +++ b/documentation/Babel.md @@ -45,7 +45,7 @@ The practical consequence is that **there is no separate document to find for th member.** A protein is searchable, but the result you get back is identified by the gene that encodes it and carries the union of both sets of synonyms; likewise a drug is identified by its active ingredient. What each conflation merges, and in what order, is described in -[Babel's Conflation documentation](https://github.com/NCATSTranslator/Babel/blob/master/docs/Conflation.md). +[Babel's Conflation documentation](https://github.com/NCATSTranslator/Babel/blob/main/docs/Conflation.md). Once you have an identifier from NameRes you can use [NodeNorm](https://nodenormalization-sri.renci.org/) to see the clique with and without each @@ -61,12 +61,12 @@ type, falling back to `biolink:NamedThing` where UMLS asserts none. This is deliberate coverage, not a defect: it means a UMLS concept still resolves to *something* even when Babel could not merge it into a richer clique. See -[Babel's leftover-UMLS documentation](https://github.com/NCATSTranslator/Babel/blob/master/docs/sources/UMLS/Leftover.md). +[Babel's leftover-UMLS documentation](https://github.com/NCATSTranslator/Babel/blob/main/docs/sources/UMLS/Leftover.md). ## Gotchas in the returned fields Every field NameRes returns from `/lookup` and `/synonyms` comes from Babel's -[synonym file format](https://github.com/NCATSTranslator/Babel/blob/master/docs/DataFormats.md). +[synonym file format](https://github.com/NCATSTranslator/Babel/blob/main/docs/DataFormats.md). Four of them behave in ways that surprise people. ### `label` is not necessarily the label of the CURIE you looked up @@ -76,7 +76,7 @@ that is not the label of the clique leader — to disambiguate the concept, or t Translator UI something better to display. Looking up a CURIE and getting back a different-looking name is expected. How the preferred name is chosen (including the chemical-specific boost prefixes and the length demotion) is in -[Understanding Babel outputs](https://github.com/NCATSTranslator/Babel/blob/master/docs/Understanding.md). +[Understanding Babel outputs](https://github.com/NCATSTranslator/Babel/blob/main/docs/Understanding.md). ### `synonyms` is not ordered by quality @@ -120,14 +120,14 @@ A NameRes instance serves a fixed build, not live Babel output. `/status` report ```json { "babel_version": "2025sep1", - "babel_version_url": "https://github.com/ncatstranslator/Babel/blob/master/releases/2025sep1.md", + "babel_version_url": "https://github.com/ncatstranslator/Babel/blob/main/releases/2025sep1.md", "biolink_model": { "tag": "v4.2.6-rc5" } } ``` A concept that Babel has since fixed will still be wrong here until the instance is rebuilt and redeployed. Check `/status` before reporting a stale clique, and see -[Babel's releases](https://github.com/NCATSTranslator/Babel/blob/master/releases/README.md) for +[Babel's releases](https://github.com/NCATSTranslator/Babel/blob/main/releases/README.md) for what changed between builds. ## Reporting a problem @@ -142,11 +142,11 @@ Babel and NameRes have separate issue trackers, and the split is by *what is wro [NameRes issue tracker](https://github.com/NCATSTranslator/NameResolution/issues). If you are not sure, file it in Babel and it will be sorted out. -[Babel's guide to filing an issue](https://github.com/NCATSTranslator/Babel/blob/master/docs/NewIssue.md) +[Babel's guide to filing an issue](https://github.com/NCATSTranslator/Babel/blob/main/docs/NewIssue.md) describes what to include and how the priority/impact/size fields are used. ## Going deeper Everything above is what a NameRes user needs. If you want to know how the cliques are actually built — the per-source ingestion, the concord files, the union-find merge, running the pipeline -yourself — start at [Babel's documentation index](https://github.com/NCATSTranslator/Babel/blob/master/docs/). +yourself — start at [Babel's documentation index](https://github.com/NCATSTranslator/Babel/blob/main/docs/). diff --git a/documentation/NameResolution.ipynb b/documentation/NameResolution.ipynb index 1fd3b1ab..d60bba96 100644 --- a/documentation/NameResolution.ipynb +++ b/documentation/NameResolution.ipynb @@ -6,7 +6,7 @@ "source": [ "# Name Resolver (NameRes)\n", "\n", - "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NCATSTranslator/NameResolution/blob/master/documentation/NameResolution.ipynb)\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NCATSTranslator/NameResolution/blob/main/documentation/NameResolution.ipynb)\n", "\n", "The [Babel pipelines](https://github.com/NCATSTranslator/Babel) generates sets of equivalent identifiers to be used by the [Node Normalizer](https://github.com/NCATSTranslator/NodeNormalization) to harmonize identifiers from different sources, but it also collects all known synonyms for these identifiers. The Name Resolver can be used to search through those synonyms." ] From 814649fd0d4c28131ba21b34e0c485f0a86bdcdb Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Mon, 27 Jul 2026 03:42:40 -0400 Subject: [PATCH 43/46] Cover /status's conflation and environment reporting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The conflations field, its CONFLATIONS override and the environment variables that /status reports were all untested — /status is the only record of which Babel data an instance is serving, so a silent regression there is invisible. Also records in CLAUDE.md why cross-repo links must use /blob/main/, and why a local clone's origin/HEAD is the wrong place to check a default branch: it is cached at clone time and does not follow a remote rename, which is how the previous commit came to normalize these links in the wrong direction. Co-Authored-By: Claude Opus 5 --- CLAUDE.md | 14 +++++++++++++- tests/test_status.py | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index e5a2c39a..960b4201 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -99,4 +99,16 @@ Solr documents contain: `curie`, `preferred_name`, `names` (synonym list), and b new cross-repo links here so a reorganization there is a one-file fix. - `documentation/Deployment.md` - Docker/Kubernetes deployment guide - `documentation/Scoring.md` - Scoring algorithm details -- `documentation/NameResolution.ipynb` - Interactive usage examples \ No newline at end of file +- `documentation/NameResolution.ipynb` - Interactive usage examples +- `documentation/TranslatorGuide.md` - Translator-specific usage guidance + +### Linking to GitHub + +NameResolution, Babel and NodeNormalization all use `main` as their default branch. Babel's was +renamed from `master` fairly recently and NameResolution has no `master` branch at all, so +`/blob/master/` URLs resolve only through GitHub's post-rename redirect — they look fine until +that redirect goes away. Always write `/blob/main/`. + +Do not check this against a local clone's `origin/HEAD`: that ref is cached at clone time and does +not follow a remote rename, so it will still say `master` long after the rename. Use +`gh repo view / --json defaultBranchRef`. \ No newline at end of file diff --git a/tests/test_status.py b/tests/test_status.py index b48ddc07..7fd6ffbb 100644 --- a/tests/test_status.py +++ b/tests/test_status.py @@ -22,8 +22,40 @@ def test_status(): assert status['size'] != '' assert status['startTime'] + # The conflations baked into this index (documentation/Babel.md explains why they + # are baked in rather than applied per query). + assert status['conflations'] == ['GeneProtein', 'DrugChemical'] + assert status['conflation_url'].startswith('https://github.com/NCATSTranslator/Babel/') + # Count the specific number of test documents we load. assert status['numDocs'] == 89 assert status['maxDoc'] == 89 assert status['deletedDocs'] == 0 + +def test_status_reports_environment(monkeypatch): + """The environment variables documented in documentation/Deployment.md are the only + record of which data an instance is serving, so check they reach /status.""" + monkeypatch.setenv('BABEL_VERSION', '2026jul22') + monkeypatch.setenv('BABEL_VERSION_URL', 'https://example.org/releases/2026jul22.md') + monkeypatch.setenv('BIOLINK_MODEL_TAG', 'v4.2.6-rc5') + + status = TestClient(app).get("/status").json() + + assert status['babel_version'] == '2026jul22' + assert status['babel_version_url'] == 'https://example.org/releases/2026jul22.md' + assert status['biolink_model']['tag'] == 'v4.2.6-rc5' + # The other two Biolink URLs are derived from the tag rather than set separately. + assert status['biolink_model']['url'].endswith('/v4.2.6-rc5') + assert 'v4.2.6-rc5' in status['biolink_model']['download_url'] + + +def test_conflations_can_be_overridden(monkeypatch): + """CONFLATIONS is not detected from the data, so a differently-built index needs it + set. Whitespace and empty entries are tolerated so a trailing comma isn't a bug.""" + monkeypatch.setenv('CONFLATIONS', ' GeneProtein , ,DrugChemical,') + + status = TestClient(app).get("/status").json() + + assert status['conflations'] == ['GeneProtein', 'DrugChemical'] + From 914da54ccdf6640baaf5a4a5febb7625798c9981 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Mon, 27 Jul 2026 03:52:33 -0400 Subject: [PATCH 44/46] Fix the follow-ups this branch turned up - api/apidocs.py returned `app.openapi_schema()`, calling a dict. The branch is unreachable on the happy path (server.py builds the schema once at import), but any second call raises TypeError. Covered by tests/test_apidocs.py, which fails against the old line. - tests/test_docs_links.py checks, offline, that relative links resolve, that heading anchors exist, that the API.md anchors used in the endpoint descriptions exist, and that no link uses /blob/master/. Four of the six broken links this branch fixed by hand would have been caught mechanically. It runs in the existing pytest CI job rather than needing a new workflow, and it makes no network requests -- a link check that fails when GitHub is slow is one people learn to ignore. It immediately found nine more /blob/master/ links in releases/, which also used the pre-rename TranslatorSRI org; those are now NCATSTranslator/main. The remaining TranslatorSRI PR and issue links in old release notes are left alone: an org-rename redirect is durable, and rewriting historical release notes is churn. - data-loading/README.md's "Options considered" and "Issues addressed" were an argument for a merged PR, not documentation. Replaced with the three decisions a future reader could undo by accident; the rest is in the history. - documentation/TranslatorGuide.md now links to Babel.md where it talks about missing concepts and about caching across Babel releases. Co-Authored-By: Claude Opus 5 --- api/apidocs.py | 4 +- data-loading/README.md | 65 +++----------- documentation/TranslatorGuide.md | 9 +- releases/TranslatorFuguJuly2024.md | 2 +- releases/TranslatorGuppyAugust2024.md | 4 +- releases/TranslatorHammerheadNovember2024.md | 4 +- releases/v1.4.7.md | 4 +- releases/v1.5.1.md | 4 +- tests/test_apidocs.py | 17 ++++ tests/test_docs_links.py | 91 ++++++++++++++++++++ 10 files changed, 141 insertions(+), 63 deletions(-) create mode 100644 tests/test_apidocs.py create mode 100644 tests/test_docs_links.py diff --git a/api/apidocs.py b/api/apidocs.py index 9434b6b3..61d9c1c0 100644 --- a/api/apidocs.py +++ b/api/apidocs.py @@ -35,8 +35,10 @@ def construct_open_api_schema(app) -> Dict[str, str]: with open(Path(__file__).parent / 'resources' / 'openapi.yml', 'r') as apd_file: api_docs = load(apd_file, Loader=SafeLoader) + # Already built (FastAPI caches it on the app), so hand back the cached schema. + # This is a dict, not a callable -- calling it raises TypeError. if app.openapi_schema: - return app.openapi_schema() + return app.openapi_schema open_api_schema = get_openapi( title=api_docs['info']['title'], diff --git a/data-loading/README.md b/data-loading/README.md index 9b16710f..abe58589 100644 --- a/data-loading/README.md +++ b/data-loading/README.md @@ -146,54 +146,17 @@ release tarball; the servers (`docker-compose.yml`, CI, the Helm chart) track th `9.10` line, so they float forward onto patch releases and stay at or above the builder without anyone having to remember to bump them. -## Options considered - -We looked at three approaches before settling on the current one. - -### A. Status quo (cloud mode, replication backup, schema re-applied on restore) -Solr ran in cloud mode (`-DzkRun`) everywhere. The backup contained only the -Lucene index, so the restore code had to recreate the collection and re-declare -every field/type/copy-field over the Schema API. **Rejected:** ZooKeeper is -overhead for one node; the schema was defined twice (loader *and* restore) and -had already drifted; the backup was not self-contained; and the load committed -after every file with a fixed `sleep` between files. - -### B. PR [#249](https://github.com/NCATSTranslator/NameResolution/pull/249) (standalone + config in the backup) -Switched to standalone mode and started shipping `solrconfig.xml` / -`managed-schema.xml` in the backup. A real improvement, but it still created the -core through the API and did not tackle load speed. **Superseded** by C. - -### C. Chosen: standalone + checked-in configset + self-contained core backup + parallel load -The backup is the whole core (config + schema + index), so restore is just -"untar and start". The schema has a single source of truth (the checked-in -configset). The load is parallel with a single deferred commit, guarded by a -document count. This is the most maintainable *and* the fastest option. - -Sub-choices within C: - -- **Parallel load vs. cheap wins only.** We parallelize (the real speed lever) - and protect against dropped/corrupt data with the pre/post document-count - check and `curl --fail`, rather than only removing the per-file commit. -- **`tar` the stopped core vs. the replication backup API.** We `tar` the core - directory after a clean commit -- simpler, and the result is self-contained. -- **Optimize before export.** We run `optimize=true` before tarring: one segment - makes the index smaller and faster to serve (fewer files to keep in the OS - page cache), at the cost of a one-time forced merge during the build. - -## Issues addressed - -- Closes [#238](https://github.com/NCATSTranslator/NameResolution/issues/238) and - [#185](https://github.com/NCATSTranslator/NameResolution/issues/185): the backup - now includes the config and schema, so restore no longer recreates fields. -- Closes [#256](https://github.com/NCATSTranslator/NameResolution/issues/256): - the index is optimized before the snapshot is exported. -- Closes [#266](https://github.com/NCATSTranslator/NameResolution/issues/266): - `queryResultCache` is sized in the checked-in `solrconfig.xml`. -- Addresses [translator-devops#609](https://github.com/helxplatform/translator-devops/issues/609): - the download init container is idempotent, so a pod restart no longer wipes a - good core and leaves Solr empty. -- Makes progress on [#265](https://github.com/NCATSTranslator/NameResolution/issues/265): - the Helm chart's Solr heap is lowered to leave room for the OS page cache, and - optimizing shrinks the index. Final heap/memory sizing and the GC-flag rework - ([#272](https://github.com/NCATSTranslator/NameResolution/issues/272)) still - need to be validated under query load and are tracked separately. +## Why it is built this way + +Three decisions here are easy to undo by accident, so they are worth stating: + +- **Standalone, not cloud mode.** Solr used to run with `-DzkRun` everywhere. + ZooKeeper buys nothing for a single node, and cloud mode is what forced the + restore to recreate the collection and re-declare the schema over the Schema + API -- which is how the schema came to be defined in two places and drift. +- **`tar` the stopped core, not the replication backup API.** Tarring the core + directory after a clean commit is simpler, and the result is self-contained: + config, schema and index in one artifact, so restore is "untar and start". +- **Optimize before export.** `optimize=true` before tarring merges the index to + one segment, which makes it smaller and faster to serve (fewer files to hold + in the OS page cache), at the cost of a one-time forced merge during the build. diff --git a/documentation/TranslatorGuide.md b/documentation/TranslatorGuide.md index 763412cf..13a4eacc 100644 --- a/documentation/TranslatorGuide.md +++ b/documentation/TranslatorGuide.md @@ -82,7 +82,10 @@ Without `autocomplete`, `"diab"` will only match documents that literally contai ### 6. If the correct concept is consistently missing If your filtering is correct but the expected result never appears, the concept may be missing from the -Babel data that NameRes is built from. Consider filing an issue on: +Babel data that NameRes is built from — or it may be there under a conflated identifier you weren't +expecting. [Where NameRes data comes from](./Babel.md) covers which Babel files an instance is built +from and why, for example, there is no separate document for a protein. If the concept really is +absent, file an issue on: - [NameRes GitHub](https://github.com/NCATSTranslator/NameResolution/issues) — for search/ranking problems - [Babel GitHub](https://github.com/NCATSTranslator/Babel/issues) — for missing synonyms or identifiers @@ -182,4 +185,6 @@ application calls NameRes repeatedly for the same input strings, cache the resul GET /status ``` -The `babel_version` field in the response changes with each data release. +The `babel_version` field in the response changes with each data release. An instance serves one +fixed Babel build rather than live output — see +[Where NameRes data comes from](./Babel.md#the-index-is-a-snapshot-of-one-babel-release). diff --git a/releases/TranslatorFuguJuly2024.md b/releases/TranslatorFuguJuly2024.md index aaef527b..624ab161 100644 --- a/releases/TranslatorFuguJuly2024.md +++ b/releases/TranslatorFuguJuly2024.md @@ -1,6 +1,6 @@ # NameRes Translator "Fugu" July 2024 Release - Babel: [2024jul13](https://stars.renci.org/var/babel_outputs/2024jul13/) - ([Babel Translator July 2024 Release](https://github.com/TranslatorSRI/Babel/blob/master/releases/TranslatorFuguJuly2024.md)) + ([Babel Translator July 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorFuguJuly2024.md)) - NameRes: [v1.3.14](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.3.14) Next release: [Translator "Guppy" August 2024](./TranslatorGuppyAugust2024.md) diff --git a/releases/TranslatorGuppyAugust2024.md b/releases/TranslatorGuppyAugust2024.md index eee66d41..d4edc2f4 100644 --- a/releases/TranslatorGuppyAugust2024.md +++ b/releases/TranslatorGuppyAugust2024.md @@ -1,6 +1,6 @@ # NameRes Translator "Guppy" August 2024 Release - Babel: [2024aug18](https://stars.renci.org/var/babel_outputs/2024aug18/) - ([Babel Translator "Guppy" August 2024 Release](https://github.com/TranslatorSRI/Babel/blob/master/releases/TranslatorGuppyAugust2024.md)) + ([Babel Translator "Guppy" August 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorGuppyAugust2024.md)) - NameRes: [v1.4.3](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.4.3) Next release: [Translator "Hammerhead" November 2024](./TranslatorHammerheadNovember2024.md) Previous release: [Translator "Fugu" July 2024](./TranslatorFuguJuly2024.md) @@ -10,7 +10,7 @@ Previous release: [Translator "Fugu" July 2024](./TranslatorFuguJuly2024.md) * Added a Solr highlighter to report on matching labels and synonyms ([#156](https://github.com/TranslatorSRI/NameResolution/pull/156)) * Added support for multitype filtering ([#158](https://github.com/TranslatorSRI/NameResolution/pull/158)) -## Babel updates (from [Babel Translator "Guppy" August 2024 Release](https://github.com/TranslatorSRI/Babel/blob/master/releases/TranslatorGuppyAugust2024.md)) +## Babel updates (from [Babel Translator "Guppy" August 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorGuppyAugust2024.md)) * [Feature] Added support for generating DuckDB and Parquet files from the compendium and synonym files, allowing us to run queries such as looking for all the identically labeled cliques across all the compendia. Increased Babel Outputs file size to support DuckDB. diff --git a/releases/TranslatorHammerheadNovember2024.md b/releases/TranslatorHammerheadNovember2024.md index 0ab82b7b..a69a063c 100644 --- a/releases/TranslatorHammerheadNovember2024.md +++ b/releases/TranslatorHammerheadNovember2024.md @@ -1,6 +1,6 @@ # NameRes Translator "Hammerhead" November 2024 Release - Babel: [2024oct24](https://stars.renci.org/var/babel_outputs/2024oct24/) - ([Babel Translator November 2024 Release](https://github.com/TranslatorSRI/Babel/blob/master/releases/TranslatorHammerheadNovember2024.md)) + ([Babel Translator November 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorHammerheadNovember2024.md)) - NameRes: [v1.4.5](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.4.5) Next release: [NameRes v1.4.7](./v1.4.7.md) Previous release: [Translator "Guppy" August 2024](./TranslatorGuppyAugust2024.md) @@ -12,7 +12,7 @@ Previous release: [Translator "Guppy" August 2024](./TranslatorGuppyAugust2024.m * Add an MIT license for NameRes in [#169](https://github.com/TranslatorSRI/NameResolution/pull/169). * Delete tests/data/test-synonyms.jsonl, which does not appear to be used in [#170](https://github.com/TranslatorSRI/NameResolution/pull/170). -## Babel updates (from [Babel Translator "Hammerhead" November 2024 Release](https://github.com/TranslatorSRI/Babel/blob/master/releases/TranslatorHammerheadNovember2024.md)) +## Babel updates (from [Babel Translator "Hammerhead" November 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorHammerheadNovember2024.md)) - [New features] Added taxon information to proteins ([#349](https://github.com/TranslatorSRI/Babel/pull/349)) - [Updates] Upgraded RxNorm to 09032024. - [Updates] Changed NCBIGene download from FTP to HTTP. diff --git a/releases/v1.4.7.md b/releases/v1.4.7.md index fb93f257..5532c0f5 100644 --- a/releases/v1.4.7.md +++ b/releases/v1.4.7.md @@ -1,6 +1,6 @@ # NameRes v1.4.7 Release - Babel: [2025jan23](https://stars.renci.org/var/babel_outputs/2025jan23/) - ([Babel 2025jan23](https://github.com/TranslatorSRI/Babel/blob/master/releases/2025jan23.md)) + ([Babel 2025jan23](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025jan23.md)) - NameRes: [v1.4.7](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.4.7) Next release: [v1.5.1](./v1.5.1.md) Previous release: [Translator "Hammerhead" November 2024](./TranslatorHammerheadNovember2024.md) @@ -16,7 +16,7 @@ Previous release: [Translator "Hammerhead" November 2024](./TranslatorHammerhead * Fix reverse CURIE endpoint in [#172](https://github.com/TranslatorSRI/NameResolution/pull/172). * Queries ending with whitespace with autocomplete=true searched for everything by @gaurav in #173 -## Babel updates (from [Babel 2025jan23](https://github.com/TranslatorSRI/Babel/blob/master/releases/2025jan23.md)) +## Babel updates (from [Babel 2025jan23](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025jan23.md)) - [New feature] Added a check for duplicate CURIEs [#342](https://github.com/TranslatorSRI/Babel/pull/342). - [New feature] Added some additional manual concords for Disease/Phenotype cliques and DrugChemical conflation [#360](https://github.com/TranslatorSRI/Babel/pull/360). diff --git a/releases/v1.5.1.md b/releases/v1.5.1.md index f9924fb6..863afb22 100644 --- a/releases/v1.5.1.md +++ b/releases/v1.5.1.md @@ -1,6 +1,6 @@ # NameRes v1.5.1 Release - Babel: [2025sep1](https://stars.renci.org/var/babel_outputs/2025sep1/) - ([Babel 2025sep1](https://github.com/TranslatorSRI/Babel/blob/master/releases/2025sep1.md)) + ([Babel 2025sep1](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025sep1.md)) - NameRes: [v1.5.1](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.5.1) Next release: _None as yet_ @@ -26,7 +26,7 @@ Previous release: [NameRes v1.4.7](./v1.4.7.md) - Updated curl to use -T instead of -d to upload file in data-loading ([PR #183](https://github.com/TranslatorSRI/NameResolution/pull/197)). - Fix start-end quotes bug (i.e. "smart" quotes) ([PR #179](https://github.com/TranslatorSRI/NameResolution/pull/179)). -## Babel updates (from [Babel 2025sep1](https://github.com/TranslatorSRI/Babel/blob/master/releases/2025sep1.md)) +## Babel updates (from [Babel 2025sep1](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025sep1.md)) - [MAJOR] Information content values were incorrectly calculated as `100` for some cliques when they were in fact lower. - [MAJOR] UMLS proteins have now been removed from chemicals so that they only appear in proteins. - [MAJOR] Improved DrugChemical conflation order. diff --git a/tests/test_apidocs.py b/tests/test_apidocs.py new file mode 100644 index 00000000..5af9bcd6 --- /dev/null +++ b/tests/test_apidocs.py @@ -0,0 +1,17 @@ +from api.apidocs import construct_open_api_schema +from api.server import app + + +def test_construct_open_api_schema_returns_cached_schema(): + """FastAPI caches the schema on the app, and construct_open_api_schema() hands the + cached copy back on later calls. It used to call it -- `return app.openapi_schema()` + -- which raises TypeError on a dict.""" + first = construct_open_api_schema(app) + assert isinstance(first, dict) + + # Force the cached branch, whether or not importing the app already populated it. + app.openapi_schema = first + second = construct_open_api_schema(app) + + assert second is first + assert sorted(second["paths"]) == ["/bulk-lookup", "/lookup", "/reverse_lookup", "/status", "/synonyms"] diff --git a/tests/test_docs_links.py b/tests/test_docs_links.py new file mode 100644 index 00000000..79ae567a --- /dev/null +++ b/tests/test_docs_links.py @@ -0,0 +1,91 @@ +"""Checks on the links in our documentation and in the endpoint descriptions. + +Everything here is offline: it resolves relative paths and heading anchors on disk and +greps for a banned URL form. Nothing fetches a URL, because a test that fails when +GitHub is slow is a test people learn to ignore. +""" + +import re +from pathlib import Path + +REPO_ROOT = Path(__file__).parent.parent + +# Markdown we own. The notebook and anything vendored is deliberately excluded. +MARKDOWN_FILES = sorted( + p for p in REPO_ROOT.rglob("*.md") + if not any(part in {"venv", ".venv", "node_modules", ".git", ".pytest_cache"} for part in p.parts) +) + +# Files that carry links out to GitHub in code rather than in prose. +SOURCE_WITH_LINKS = [REPO_ROOT / "api" / "server.py", REPO_ROOT / "api" / "resources" / "openapi.yml"] + +INLINE_LINK = re.compile(r"\[[^\]]*\]\(([^)\s]+)\)") +HEADING = re.compile(r"^#+\s+(.*)$", re.MULTILINE) + + +def anchors_for(markdown_text): + """The anchor slugs GitHub generates for a document's headings.""" + slugs = set() + for heading in HEADING.findall(markdown_text): + slug = re.sub(r"[`*]", "", heading.strip().lower()) + slug = re.sub(r"[^a-z0-9 _-]", "", slug) + slugs.add(slug.replace(" ", "-")) + return slugs + + +def test_relative_links_resolve(): + """A relative link is written relative to the file it sits in, which is easy to get + wrong when a document moves into a subdirectory.""" + broken = [] + for path in MARKDOWN_FILES: + for target in INLINE_LINK.findall(path.read_text()): + if target.startswith(("http://", "https://", "#", "mailto:")): + continue + resolved = (path.parent / target.split("#")[0]).resolve() + if not resolved.exists(): + broken.append(f"{path.relative_to(REPO_ROOT)} -> {target}") + assert not broken, "Relative links that do not resolve:\n " + "\n ".join(broken) + + +def test_anchors_resolve(): + """Anchors are case-sensitive and GitHub lower-cases them, so `#Conflation` silently + lands at the top of the page rather than at the heading.""" + broken = [] + for path in MARKDOWN_FILES: + for target in INLINE_LINK.findall(path.read_text()): + if "#" not in target or target.startswith(("http://", "https://", "mailto:")): + continue + rel, _, anchor = target.partition("#") + if not anchor: + continue + target_path = path if rel == "" else (path.parent / rel) + if target_path.suffix != ".md" or not target_path.exists(): + continue + if anchor not in anchors_for(target_path.read_text()): + broken.append(f"{path.relative_to(REPO_ROOT)} -> {target}") + assert not broken, "Anchors with no matching heading:\n " + "\n ".join(broken) + + +def test_in_repo_anchors_in_source_resolve(): + """The endpoint descriptions link into documentation/API.md by absolute URL, so the + check above cannot see them.""" + api_md = REPO_ROOT / "documentation" / "API.md" + valid = anchors_for(api_md.read_text()) + broken = [] + for path in SOURCE_WITH_LINKS: + for anchor in re.findall(r"documentation/API\.md#([A-Za-z0-9_-]+)", path.read_text()): + if anchor not in valid: + broken.append(f"{path.relative_to(REPO_ROOT)} -> API.md#{anchor}") + assert not broken, "Anchors with no matching heading in API.md:\n " + "\n ".join(broken) + + +def test_no_master_branch_links(): + """NameResolution, Babel and NodeNormalization all default to `main`. A /blob/master/ + URL resolves only through GitHub's post-rename redirect, so it looks fine right up + until that redirect goes away. See CLAUDE.md.""" + offenders = [] + for path in MARKDOWN_FILES + SOURCE_WITH_LINKS: + for line_no, line in enumerate(path.read_text().splitlines(), start=1): + if re.search(r"github\.com/[^/\s]+/[^/\s]+/blob/master/", line, re.IGNORECASE): + offenders.append(f"{path.relative_to(REPO_ROOT)}:{line_no}") + assert not offenders, "Links using /blob/master/ instead of /blob/main/:\n " + "\n ".join(offenders) From b8aff0574b70749c21d85490320da57223b21416 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Mon, 27 Jul 2026 04:02:05 -0400 Subject: [PATCH 45/46] Refresh the documented /status example and keep it from drifting again The example in documentation/API.md still showed nameres_version v1.5.1 while the OpenAPI spec is at 1.7.0. The accompanying test treats the two kinds of drift differently. A stale patch or minor version is cosmetic and only warns. A stale major version fails, because a major release is when someone should re-run this example against a real instance and confirm the endpoint still behaves as documented. The field names are compared strictly in both cases: that is what catches a field added to /status and never written down, which is how `conflations` came to be undocumented in the first place. Co-Authored-By: Claude Opus 5 --- documentation/API.md | 2 +- tests/test_status.py | 39 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+), 1 deletion(-) diff --git a/documentation/API.md b/documentation/API.md index 240545b0..2179aa98 100644 --- a/documentation/API.md +++ b/documentation/API.md @@ -346,7 +346,7 @@ Solr database. }, "conflations": ["GeneProtein", "DrugChemical"], "conflation_url": "https://github.com/NCATSTranslator/Babel/blob/main/docs/Conflation.md", - "nameres_version": "v1.5.1", + "nameres_version": "v1.7.0", "startTime": "2025-12-19T11:53:09.638Z", "numDocs": 425583391, "maxDoc": 425586610, diff --git a/tests/test_status.py b/tests/test_status.py index 7fd6ffbb..e172a532 100644 --- a/tests/test_status.py +++ b/tests/test_status.py @@ -1,5 +1,10 @@ +import json import logging +import re +import warnings +from pathlib import Path +from api.apidocs import get_app_info from api.server import app from fastapi.testclient import TestClient @@ -59,3 +64,37 @@ def test_conflations_can_be_overridden(monkeypatch): assert status['conflations'] == ['GeneProtein', 'DrugChemical'] + +def test_documented_status_example_matches_the_real_response(): + """documentation/API.md shows an example /status payload, which had drifted to a + nameres_version three releases old. + + A stale patch or minor version in the example is cosmetic, so that only warns. A + stale *major* version fails: a major release is exactly when someone should re-run + this example and confirm the endpoint still behaves the way the docs describe. + + The field names are checked strictly either way -- that is what catches a field + added to /status and never documented, which is how `conflations` was missed.""" + api_md = (Path(__file__).parent.parent / "documentation" / "API.md").read_text() + # The first JSON block after the "### `/status`" heading is the example payload. + status_section = api_md.split("### `/status`", 1)[1] + example = json.loads(re.search(r"```json\n(.*?)```", status_section, re.DOTALL).group(1)) + + documented = example["nameres_version"].lstrip("v") + current = get_app_info()["version"] + if documented != current: + message = ( + f"documentation/API.md's /status example shows nameres_version " + f"{documented}, but this build is {current}." + ) + assert documented.split(".")[0] == current.split(".")[0], ( + message + " Refresh the example against a real instance for the new major version." + ) + warnings.warn(message) + + actual = TestClient(app).get("/status").json() + assert set(example) == set(actual), ( + "The documented /status example and the real response have different fields: " + f"only in docs={sorted(set(example) - set(actual))}, " + f"only in response={sorted(set(actual) - set(example))}" + ) From 6831d7ddc390aac7a1ed3ce8ad21446f34a60756 Mon Sep 17 00:00:00 2001 From: Gaurav Vaidya Date: Mon, 27 Jul 2026 05:49:49 -0400 Subject: [PATCH 46/46] Use the current org in release-note links, and ban the old one NameResolution, Babel and NodeNormalization all moved from TranslatorSRI to NCATSTranslator. 77 links across the eight release-note files still named the old org. They resolve through GitHub's org-rename redirect, which is more durable than the branch-rename redirect cleaned up earlier, but they still point at an org that no longer owns the code. The new test is scoped to those three repositories by name rather than banning the TranslatorSRI string. TranslatorSRI/RENCI-Python-image (used by data-loading/Dockerfile), TranslatorSRI/babel-validation and TranslatorSRI/r3 really do still live under that org -- NCATSTranslator equivalents are 404s -- so a blanket ban would push someone to "fix" three correct links into broken ones. The comment says so, since that is the trap here. Closes #292. Co-Authored-By: Claude Opus 5 --- releases/TranslatorDecember2023.md | 12 +++---- releases/TranslatorFuguJuly2024.md | 6 ++-- releases/TranslatorGuppyAugust2024.md | 36 ++++++++++---------- releases/TranslatorHammerheadNovember2024.md | 24 ++++++------- releases/TranslatorMay2024.md | 10 +++--- releases/TranslatorOctober2023.md | 4 +-- releases/v1.4.7.md | 26 +++++++------- releases/v1.5.1.md | 28 +++++++-------- tests/test_docs_links.py | 21 ++++++++++++ 9 files changed, 94 insertions(+), 73 deletions(-) diff --git a/releases/TranslatorDecember2023.md b/releases/TranslatorDecember2023.md index 60812ed6..84b8824f 100644 --- a/releases/TranslatorDecember2023.md +++ b/releases/TranslatorDecember2023.md @@ -1,7 +1,7 @@ # NameRes Translator December 2023 Release - Babel: [2023nov5](https://stars.renci.org/var/babel_outputs/2023nov5/) -- NameRes: [v1.3.11](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.3.11) +- NameRes: [v1.3.11](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.3.11) Next release: [Translator May 2024](TranslatorMay2024.md) @@ -24,9 +24,9 @@ Next release: [Translator May 2024](TranslatorMay2024.md) ## Releases since [Translator October 2023 release](TranslatorOctober2023.md) -* https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.3.7 +* https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.3.7 * fixes reverse lookup get endpoint. by @YaphetKG in #102 -* https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.3.8 +* https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.3.8 * Fix Terms of Service and service description by @gaurav in #114 * Return entire document in reverse lookups by @gaurav in #115 * Report scores for reverse lookup by @gaurav in #116 @@ -34,9 +34,9 @@ Next release: [Translator May 2024](TranslatorMay2024.md) * Add an autocomplete flag by @gaurav in #118 * Added a /status endpoint that tells us about this NameRes instance by @gaurav in #119 * Added instruction to make data/synonyms directory in target data/synonyms/done by @gaurav in #123 -* https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.3.9 +* https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.3.9 * Otel instrumentation by @YaphetKG in #121 -* https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.3.10 +* https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.3.10 * Update server.py by @YaphetKG in #130 -* https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.3.11 +* https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.3.11 * Incremented version that I forgot to do in Name Resolver v1.3.10. diff --git a/releases/TranslatorFuguJuly2024.md b/releases/TranslatorFuguJuly2024.md index 624ab161..cb2c0540 100644 --- a/releases/TranslatorFuguJuly2024.md +++ b/releases/TranslatorFuguJuly2024.md @@ -1,7 +1,7 @@ # NameRes Translator "Fugu" July 2024 Release - Babel: [2024jul13](https://stars.renci.org/var/babel_outputs/2024jul13/) ([Babel Translator July 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorFuguJuly2024.md)) -- NameRes: [v1.3.14](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.3.14) +- NameRes: [v1.3.14](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.3.14) Next release: [Translator "Guppy" August 2024](./TranslatorGuppyAugust2024.md) @@ -22,6 +22,6 @@ Next release: [Translator "Guppy" August 2024](./TranslatorGuppyAugust2024.md) * [Update] Updated PANTHER pathways from SequenceAssociationPathway3.6.7.txt to SequenceAssociationPathway3.6.8.txt. ## Releases since [Translator May 2024](./TranslatorMay2024.md) -* [1.3.14](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.3.14): Forgot to increment version number in +* [1.3.14](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.3.14): Forgot to increment version number in previous release. -* [1.3.13](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.3.13): Added release notes for Translator May 2024. \ No newline at end of file +* [1.3.13](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.3.13): Added release notes for Translator May 2024. \ No newline at end of file diff --git a/releases/TranslatorGuppyAugust2024.md b/releases/TranslatorGuppyAugust2024.md index d4edc2f4..0eb1f2f7 100644 --- a/releases/TranslatorGuppyAugust2024.md +++ b/releases/TranslatorGuppyAugust2024.md @@ -1,28 +1,28 @@ # NameRes Translator "Guppy" August 2024 Release - Babel: [2024aug18](https://stars.renci.org/var/babel_outputs/2024aug18/) ([Babel Translator "Guppy" August 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorGuppyAugust2024.md)) -- NameRes: [v1.4.3](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.4.3) +- NameRes: [v1.4.3](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.4.3) Next release: [Translator "Hammerhead" November 2024](./TranslatorHammerheadNovember2024.md) Previous release: [Translator "Fugu" July 2024](./TranslatorFuguJuly2024.md) ## New features -* Added exactish index for synonyms ([#150](https://github.com/TranslatorSRI/NameResolution/pull/150)) -* Added a Solr highlighter to report on matching labels and synonyms ([#156](https://github.com/TranslatorSRI/NameResolution/pull/156)) -* Added support for multitype filtering ([#158](https://github.com/TranslatorSRI/NameResolution/pull/158)) +* Added exactish index for synonyms ([#150](https://github.com/NCATSTranslator/NameResolution/pull/150)) +* Added a Solr highlighter to report on matching labels and synonyms ([#156](https://github.com/NCATSTranslator/NameResolution/pull/156)) +* Added support for multitype filtering ([#158](https://github.com/NCATSTranslator/NameResolution/pull/158)) ## Babel updates (from [Babel Translator "Guppy" August 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorGuppyAugust2024.md)) * [Feature] Added support for generating DuckDB and Parquet files from the compendium and synonym files, allowing us to run queries such as looking for all the identically labeled cliques across all the compendia. Increased Babel Outputs file size to support DuckDB. -* [Feature] Added labels from DrugBank (https://github.com/TranslatorSRI/Babel/pull/335). -* [Feature] Improved cell anatomy concords using Wikidata (https://github.com/TranslatorSRI/Babel/pull/329). -* [Feature] Added manual concords for the DrugChemical conflation (https://github.com/TranslatorSRI/Babel/pull/337). -* [Feature] Wrote a script for comparing between two summary files (https://github.com/TranslatorSRI/Babel/pull/320). +* [Feature] Added labels from DrugBank (https://github.com/NCATSTranslator/Babel/pull/335). +* [Feature] Improved cell anatomy concords using Wikidata (https://github.com/NCATSTranslator/Babel/pull/329). +* [Feature] Added manual concords for the DrugChemical conflation (https://github.com/NCATSTranslator/Babel/pull/337). +* [Feature] Wrote a script for comparing between two summary files (https://github.com/NCATSTranslator/Babel/pull/320). * [Feature] Added timestamping as an option to Wget. * [Feature] Reorganized primary label determination so that we can include it in compendia files as well. * This isn't currently used by the loader, but might be in the future. For now, this is only useful in helping track what labels are being chosen as the preferred label. -* [Bugfixes] Added additional ENSEMBL datasets to skip (https://github.com/TranslatorSRI/Babel/pull/297). +* [Bugfixes] Added additional ENSEMBL datasets to skip (https://github.com/NCATSTranslator/Babel/pull/297). * [Bugfixes] Fixed a bug in recognizing the end of file when reading the PubChem ID and SMILES files. * [Bugfixes] Fixed the lack of `clique_identifier_count` in leftover UMLS output. * [Bugfixes] Fixed unraised exception in Ensembl BioMart download. @@ -32,12 +32,12 @@ Previous release: [Translator "Fugu" July 2024](./TranslatorFuguJuly2024.md) * [Updates] Changed DrugBank ID types from 'ChemicalEntity' to 'Drug'. ## Releases since [Translator "Fugu" July 2024](./TranslatorFuguJuly2024.md) -* [1.4.3](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.4.3) - * Fix issue with empty Biolink type by @gaurav in [#159](https://github.com/TranslatorSRI/NameResolution/pull/159) -* [1.4.2](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.4.2) - * Added support for multitype filtering by @gaurav in [#158](https://github.com/TranslatorSRI/NameResolution/pull/158) -* [1.4.1](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.4.1) - * Added Translator Fugu release notes by @gaurav in [#155](https://github.com/TranslatorSRI/NameResolution/pull/155) - * Use a Solr highlighter to identify matching terms by @gaurav in [#156](https://github.com/TranslatorSRI/NameResolution/pull/156) -* [1.4.0](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.4.0) - * Add exactish index for synonyms by @gaurav in [#150](https://github.com/TranslatorSRI/NameResolution/pull/150) +* [1.4.3](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.4.3) + * Fix issue with empty Biolink type by @gaurav in [#159](https://github.com/NCATSTranslator/NameResolution/pull/159) +* [1.4.2](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.4.2) + * Added support for multitype filtering by @gaurav in [#158](https://github.com/NCATSTranslator/NameResolution/pull/158) +* [1.4.1](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.4.1) + * Added Translator Fugu release notes by @gaurav in [#155](https://github.com/NCATSTranslator/NameResolution/pull/155) + * Use a Solr highlighter to identify matching terms by @gaurav in [#156](https://github.com/NCATSTranslator/NameResolution/pull/156) +* [1.4.0](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.4.0) + * Add exactish index for synonyms by @gaurav in [#150](https://github.com/NCATSTranslator/NameResolution/pull/150) diff --git a/releases/TranslatorHammerheadNovember2024.md b/releases/TranslatorHammerheadNovember2024.md index a69a063c..2e2995af 100644 --- a/releases/TranslatorHammerheadNovember2024.md +++ b/releases/TranslatorHammerheadNovember2024.md @@ -1,27 +1,27 @@ # NameRes Translator "Hammerhead" November 2024 Release - Babel: [2024oct24](https://stars.renci.org/var/babel_outputs/2024oct24/) ([Babel Translator November 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorHammerheadNovember2024.md)) -- NameRes: [v1.4.5](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.4.5) +- NameRes: [v1.4.5](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.4.5) Next release: [NameRes v1.4.7](./v1.4.7.md) Previous release: [Translator "Guppy" August 2024](./TranslatorGuppyAugust2024.md) ## New features -* Searching for an empty string now returns an empty list in [#167](https://github.com/TranslatorSRI/NameResolution/pull/167). +* Searching for an empty string now returns an empty list in [#167](https://github.com/NCATSTranslator/NameResolution/pull/167). ## Cleanup and improvements -* Add an MIT license for NameRes in [#169](https://github.com/TranslatorSRI/NameResolution/pull/169). -* Delete tests/data/test-synonyms.jsonl, which does not appear to be used in [#170](https://github.com/TranslatorSRI/NameResolution/pull/170). +* Add an MIT license for NameRes in [#169](https://github.com/NCATSTranslator/NameResolution/pull/169). +* Delete tests/data/test-synonyms.jsonl, which does not appear to be used in [#170](https://github.com/NCATSTranslator/NameResolution/pull/170). ## Babel updates (from [Babel Translator "Hammerhead" November 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorHammerheadNovember2024.md)) -- [New features] Added taxon information to proteins ([#349](https://github.com/TranslatorSRI/Babel/pull/349)) +- [New features] Added taxon information to proteins ([#349](https://github.com/NCATSTranslator/Babel/pull/349)) - [Updates] Upgraded RxNorm to 09032024. - [Updates] Changed NCBIGene download from FTP to HTTP. -- [Updates] Increased DRUG_CHEMICAL_SMALLER_MAX_LABEL_LENGTH (introduced in [#330](https://github.com/TranslatorSRI/Babel/pull/330)) from 30 to 40. +- [Updates] Increased DRUG_CHEMICAL_SMALLER_MAX_LABEL_LENGTH (introduced in [#330](https://github.com/NCATSTranslator/Babel/pull/330)) from 30 to 40. ## Releases since [Translator "Guppy" August 2024](./TranslatorGuppyAugust2024.md) -* [1.4.5](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.4.5) - * Searching for an empty string now returns an empty list in [#167](https://github.com/TranslatorSRI/NameResolution/pull/167). - * Add an MIT license for NameRes in [#169](https://github.com/TranslatorSRI/NameResolution/pull/169). - * Delete tests/data/test-synonyms.jsonl, which does not appear to be used in [#170](https://github.com/TranslatorSRI/NameResolution/pull/170). -* [1.4.4](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.4.4) - * Added release notes for Translator Guppy in [#160](https://github.com/TranslatorSRI/NameResolution/pull/160). +* [1.4.5](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.4.5) + * Searching for an empty string now returns an empty list in [#167](https://github.com/NCATSTranslator/NameResolution/pull/167). + * Add an MIT license for NameRes in [#169](https://github.com/NCATSTranslator/NameResolution/pull/169). + * Delete tests/data/test-synonyms.jsonl, which does not appear to be used in [#170](https://github.com/NCATSTranslator/NameResolution/pull/170). +* [1.4.4](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.4.4) + * Added release notes for Translator Guppy in [#160](https://github.com/NCATSTranslator/NameResolution/pull/160). diff --git a/releases/TranslatorMay2024.md b/releases/TranslatorMay2024.md index e187e55b..14760818 100644 --- a/releases/TranslatorMay2024.md +++ b/releases/TranslatorMay2024.md @@ -1,23 +1,23 @@ # NameRes Translator May 2024 Release - Babel: [2024mar24](https://stars.renci.org/var/babel_outputs/2024mar24/) (Babel Translator May 2024 Release) -- NameRes: [v1.3.12](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.3.12) +- NameRes: [v1.3.12](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.3.12) Next release: [Translator "Fugu" July 2024](TranslatorFuguJuly2024.md) ## New features * Solr database now has two additional fields for each clique: * `taxa`: list of taxa that a clique is found in; in Babel 2024mar24, only NCBIGenes have taxa (as generated by - [TranslatorSRI/Babel#211](https://github.com/TranslatorSRI/Babel/pull/211)). + [NCATSTranslator/Babel#211](https://github.com/NCATSTranslator/Babel/pull/211)). * `clique_identifier_count`: the number of identifiers in this clique (as generated by - [TranslatorSRI/Babel#228](https://github.com/TranslatorSRI/Babel/pull/228)). + [NCATSTranslator/Babel#228](https://github.com/NCATSTranslator/Babel/pull/228)). * The `/lookup` endpoints now return `taxa` and `clique_identifier_count`. They also have an `only_taxa` filter that can be used to provide a pipe-delimited list of NCBITaxon entries to filter results with. ## Babel updates (from Babel Translator May 2024 Release) * [New identifiers] 36.9 million PubMed IDs (e.g. `PMID:25061375`) have been added as `biolink:JournalArticle`, as well as the mappings to PMC (e.g. `PMC:PMC4109484`) and DOIs (e.g. `doi:10.3897/zookeys.420.7089`) that are included in PubMed. - Details in [TranslatorSRI/Babel#227](https://github.com/TranslatorSRI/Babel/pull/227). + Details in [NCATSTranslator/Babel#227](https://github.com/NCATSTranslator/Babel/pull/227). * Fixed type determination for DrugChemical conflation. Details in - [TranslatorSRI/Babel#266](https://github.com/TranslatorSRI/Babel/pull/266). + [NCATSTranslator/Babel#266](https://github.com/NCATSTranslator/Babel/pull/266). * Synonym files now include the clique identifier count (the number of identifiers in each clique) in synonyms file. * Minor fixes. diff --git a/releases/TranslatorOctober2023.md b/releases/TranslatorOctober2023.md index d6140cd9..745bebe9 100644 --- a/releases/TranslatorOctober2023.md +++ b/releases/TranslatorOctober2023.md @@ -1,6 +1,6 @@ # NameRes Translator October 2023 Release -- Babel: [2023aug](https://stars.renci.org/var/babel_outputs/2023aug3-with-drugs/) (roughly Babel [v1.1.0](https://github.com/TranslatorSRI/Babel/releases/tag/v1.1.0)) -- NameRes: [v1.3.6](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.3.6) +- Babel: [2023aug](https://stars.renci.org/var/babel_outputs/2023aug3-with-drugs/) (roughly Babel [v1.1.0](https://github.com/NCATSTranslator/Babel/releases/tag/v1.1.0)) +- NameRes: [v1.3.6](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.3.6) Next release: [December 2023](TranslatorDecember2023.md) \ No newline at end of file diff --git a/releases/v1.4.7.md b/releases/v1.4.7.md index 5532c0f5..e6218fee 100644 --- a/releases/v1.4.7.md +++ b/releases/v1.4.7.md @@ -1,28 +1,28 @@ # NameRes v1.4.7 Release - Babel: [2025jan23](https://stars.renci.org/var/babel_outputs/2025jan23/) ([Babel 2025jan23](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025jan23.md)) -- NameRes: [v1.4.7](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.4.7) +- NameRes: [v1.4.7](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.4.7) Next release: [v1.5.1](./v1.5.1.md) Previous release: [Translator "Hammerhead" November 2024](./TranslatorHammerheadNovember2024.md) ## New Features -* Add a bulk endpoint for NameRes in [#165](https://github.com/TranslatorSRI/NameResolution/pull/165). +* Add a bulk endpoint for NameRes in [#165](https://github.com/NCATSTranslator/NameResolution/pull/165). ## Improvements -* Replaces /reverse_lookup with a more sensible name in [#168](https://github.com/TranslatorSRI/NameResolution/pull/168). -* Update OTEL to gRPC in [#171](https://github.com/TranslatorSRI/NameResolution/pull/171). +* Replaces /reverse_lookup with a more sensible name in [#168](https://github.com/NCATSTranslator/NameResolution/pull/168). +* Update OTEL to gRPC in [#171](https://github.com/NCATSTranslator/NameResolution/pull/171). ## Bugfixes -* Fix reverse CURIE endpoint in [#172](https://github.com/TranslatorSRI/NameResolution/pull/172). +* Fix reverse CURIE endpoint in [#172](https://github.com/NCATSTranslator/NameResolution/pull/172). * Queries ending with whitespace with autocomplete=true searched for everything by @gaurav in #173 ## Babel updates (from [Babel 2025jan23](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025jan23.md)) -- [New feature] Added a check for duplicate CURIEs [#342](https://github.com/TranslatorSRI/Babel/pull/342). +- [New feature] Added a check for duplicate CURIEs [#342](https://github.com/NCATSTranslator/Babel/pull/342). - [New feature] Added some additional manual concords for Disease/Phenotype cliques and DrugChemical - conflation [#360](https://github.com/TranslatorSRI/Babel/pull/360). -- [New feature] Replace use of `has_tradename` with `tradename_of` in RxNorm ([#377](https://github.com/TranslatorSRI/Babel/pull/377)). -- [New feature] Added processes from UMLS ([#395](https://github.com/TranslatorSRI/Babel/pull/395)). -- [New feature] Improved EFO relationships ([#396](https://github.com/TranslatorSRI/Babel/pull/396)). + conflation [#360](https://github.com/NCATSTranslator/Babel/pull/360). +- [New feature] Replace use of `has_tradename` with `tradename_of` in RxNorm ([#377](https://github.com/NCATSTranslator/Babel/pull/377)). +- [New feature] Added processes from UMLS ([#395](https://github.com/NCATSTranslator/Babel/pull/395)). +- [New feature] Improved EFO relationships ([#396](https://github.com/NCATSTranslator/Babel/pull/396)). - [Updates] Various updates - [Bugfix] Fixed a bug in choosing the best label shorter than a particular size in src/babel_utils.py:write_compendium() - [Bugfix] Cleaned up src/createcompendia/chemicals.py:parse_smifile() so that includes the ChEMBL ID and calculates the column index by name, with ValueErrors thrown if the column name is missing. @@ -30,10 +30,10 @@ Previous release: [Translator "Hammerhead" November 2024](./TranslatorHammerhead - [Bugfix] Other minor fixes. ## Releases since [Translator "Hammerhead" November 2024](./TranslatorHammerheadNovember2024.md) -* [1.4.7](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.4.7) +* [1.4.7](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.4.7) * Replaces /reverse_lookup with a more sensible name by @gaurav in #168 * Update OTEL to gRPC by @gaurav in #171 * Fix reverse CURIE endpoint by @gaurav in #172 * Add a bulk endpoint for NameRes by @gaurav in #165 -* [1.4.6](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.4.6) - * Bugfix: queries ending with whitespace with autocomplete=true searched for everything in [#173](https://github.com/TranslatorSRI/NameResolution/pull/173) +* [1.4.6](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.4.6) + * Bugfix: queries ending with whitespace with autocomplete=true searched for everything in [#173](https://github.com/NCATSTranslator/NameResolution/pull/173) diff --git a/releases/v1.5.1.md b/releases/v1.5.1.md index 863afb22..a193f959 100644 --- a/releases/v1.5.1.md +++ b/releases/v1.5.1.md @@ -1,7 +1,7 @@ # NameRes v1.5.1 Release - Babel: [2025sep1](https://stars.renci.org/var/babel_outputs/2025sep1/) ([Babel 2025sep1](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025sep1.md)) -- NameRes: [v1.5.1](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.5.1) +- NameRes: [v1.5.1](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.5.1) Next release: _None as yet_ Previous release: [NameRes v1.4.7](./v1.4.7.md) @@ -9,22 +9,22 @@ Previous release: [NameRes v1.4.7](./v1.4.7.md) ## Bugfixes - [MAJOR] Entries with clique_identifier_count=1 were being filtered out of search results because we were boosting with the log of clique_identifier_count, and log(1) = 0. This has now been fixed by using the log of - (clique_identifier_count + 1). ([PR #179](https://github.com/TranslatorSRI/NameResolution/pull/179)) + (clique_identifier_count + 1). ([PR #179](https://github.com/NCATSTranslator/NameResolution/pull/179)) - Fixed bug in NameRes Loading Docker image generation (d3f607af95e3fb254298d7f0a637a63ccb63fd6b). ## New Features - [MAJOR] This NameRes has GeneProtein conflation turned on. If you need to know the protein and gene identifiers for a particular NameRes result, you can normalize the identifier with the corresponding NodeNorm instance with `individual_types` set to true ([example](https://nodenormalization-sri.renci.org/1.5/get_normalized_nodes?curie=NCBIGene%3A1756&conflate=true&drug_chemical_conflate=false&description=false&individual_types=true)). -- Added Babel version and version URL to the `/status` endpoint ([PR #180](https://github.com/TranslatorSRI/NameResolution/pull/180)). -- Added instructions and script for running NameRes locally ([PR #186](https://github.com/TranslatorSRI/NameResolution/pull/186)). +- Added Babel version and version URL to the `/status` endpoint ([PR #180](https://github.com/NCATSTranslator/NameResolution/pull/180)). +- Added instructions and script for running NameRes locally ([PR #186](https://github.com/NCATSTranslator/NameResolution/pull/186)). ## Improvements -- [MAJOR] Update exact boosts ([PR #183](https://github.com/TranslatorSRI/NameResolution/pull/183)). +- [MAJOR] Update exact boosts ([PR #183](https://github.com/NCATSTranslator/NameResolution/pull/183)). - Exactish (case-insensitive exact) matches are now boosted by ~100x non-exactish matches. - Increased the basic boost by ~10x so that they override low clique_identifier_count matches. -- Updated curl to use -T instead of -d to upload file in data-loading ([PR #183](https://github.com/TranslatorSRI/NameResolution/pull/197)). -- Fix start-end quotes bug (i.e. "smart" quotes) ([PR #179](https://github.com/TranslatorSRI/NameResolution/pull/179)). +- Updated curl to use -T instead of -d to upload file in data-loading ([PR #183](https://github.com/NCATSTranslator/NameResolution/pull/197)). +- Fix start-end quotes bug (i.e. "smart" quotes) ([PR #179](https://github.com/NCATSTranslator/NameResolution/pull/179)). ## Babel updates (from [Babel 2025sep1](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025sep1.md)) - [MAJOR] Information content values were incorrectly calculated as `100` for some cliques when they were in fact lower. @@ -33,20 +33,20 @@ Previous release: [NameRes v1.4.7](./v1.4.7.md) - [MAJOR] Added secondary identifiers for ChEBI. ## Releases since [NameRes v1.4.7](./v1.4.7.md) -- [NameRes v1.4.8](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.4.8) +- [NameRes v1.4.8](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.4.8) - Added NameRes release information for November 2024 and January 2025 by @gaurav in #177 -- [NameRes v1.4.9](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.4.9) +- [NameRes v1.4.9](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.4.9) - Incremented version number (forgot to do that in NameRes v1.4.8). -- [NameRes v1.4.10](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.4.10) +- [NameRes v1.4.10](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.4.10) - Added Babel version and Babel version URL by @gaurav in #180 -- [NameRes v1.4.11](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.4.11) +- [NameRes v1.4.11](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.4.11) - Add instructions and script for running NameRes locally by @gaurav in #186 -- [NameRes v1.4.12](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.4.12) +- [NameRes v1.4.12](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.4.12) - Replace renci-python-image with GitHub Packages (d3f607a). -- [NameRes v1.5.0](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.5.0) +- [NameRes v1.5.0](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.5.0) - [MAJOR] Entries with clique_identifier_count=1 were being filtered out of search results because we were boosting with the log of clique_identifier_count, and log(1) = 0. This has now been fixed by using the log of (clique_identifier_count + 1) in #179. - [MAJOR] Tweaked exact boosts by @gaurav in #183 - Updated curl to use -T instead of -d to upload file in data-loading by @gaurav in #197 - Fix start-end quotes bug and stopped filtering out clique_identifier_count=1 records by @gaurav in #179 -- [NameRes v1.5.1](https://github.com/TranslatorSRI/NameResolution/releases/tag/v1.5.1) +- [NameRes v1.5.1](https://github.com/NCATSTranslator/NameResolution/releases/tag/v1.5.1) - Split GeneProteinConflated.txt instead of Gene.txt and Protein.txt by @gaurav in #203 diff --git a/tests/test_docs_links.py b/tests/test_docs_links.py index 79ae567a..b8a950bb 100644 --- a/tests/test_docs_links.py +++ b/tests/test_docs_links.py @@ -89,3 +89,24 @@ def test_no_master_branch_links(): if re.search(r"github\.com/[^/\s]+/[^/\s]+/blob/master/", line, re.IGNORECASE): offenders.append(f"{path.relative_to(REPO_ROOT)}:{line_no}") assert not offenders, "Links using /blob/master/ instead of /blob/main/:\n " + "\n ".join(offenders) + + +#: The three repositories that moved from the TranslatorSRI org to NCATSTranslator. Scoped to +#: these by name on purpose -- TranslatorSRI/RENCI-Python-image, TranslatorSRI/babel-validation +#: and TranslatorSRI/r3 really do still live under that org, so a blanket ban on the string +#: would push someone to "fix" three correct links into 404s. +MOVED_TO_NCATSTRANSLATOR = re.compile(r"TranslatorSRI/(Babel|NameResolution|NodeNormalization)\b") + + +def test_no_stale_org_links(): + """These three repositories are under NCATSTranslator now. The old URLs resolve through + GitHub's org-rename redirect, which is more durable than the branch-rename one, but they + still name an org that no longer owns the code.""" + offenders = [] + for path in MARKDOWN_FILES + SOURCE_WITH_LINKS: + for line_no, line in enumerate(path.read_text().splitlines(), start=1): + if MOVED_TO_NCATSTRANSLATOR.search(line): + offenders.append(f"{path.relative_to(REPO_ROOT)}:{line_no}") + assert not offenders, ( + "Links naming the pre-rename TranslatorSRI org:\n " + "\n ".join(offenders) + )