Speed up and simplify Solr data loading; fix /status for standalone

[gaurav]

What & why

Building a NameRes instance loads ~130 GB of Babel synonyms into Solr and ships a backup a worker restores. The old pipeline had three problems: a slow serial load (per-file hard commit + sleep 60), a Solr schema defined twice (loader and the Helm restore script) that had already drifted, and ZooKeeper running for what is a one-shard, one-node job.

This PR reworks the whole flow around a self-contained standalone core: the backup is a whole Solr core — config + schema + index — so restoring it is just "untar into the Solr home and start Solr". No ZooKeeper, no collection creation, no restore-time schema setup. This supersedes #249.

Key changes

• Single source of truth for the schema: new checked-in configset at data-loading/configsets/name_lookup/conf — two files, ~220 lines, written for NameRes rather than copied from Solr's _default. Field types are carried over verbatim so analysis behaviour is unchanged, the malformed types field is fixed, the index buffer is larger, autoSoftCommit is off and queryResultCache is grown. Everything in it is a decision we made, so there is nothing to reconcile against upstream on a Solr upgrade; the copy it replaced was 44 files, 70 field types and 69 dynamic fields, of which NameRes uses 6 field types and 13 fields (42 of the files were language stopword lists, which Babel synonyms can never use). The schema is now immutable (mutable=false), and schemaless mode is off in solrconfig.xml rather than in configoverlay.json, which Solr's Config API rewrites — so no file in the configset is one Solr can rewrite behind us. An unexpected Babel key is now a hard unknown field error instead of a silently invented field.
• Fast, guarded load: setup-and-load-solr.sh streams files to Solr in parallel with a single deferred commit (no per-file commit, no sleeps). Because a parallel load of a huge dataset is scary, it counts input documents before loading and compares the increase in Solr's count afterward, with curl --fail on every upload — a drop or a bad file aborts with a non-zero exit. The count is the load-bearing half: Solr answers 200 and indexes nothing for a file of plain text, so --fail alone would let it through.
• Standalone everywhere: loader Dockerfile, docker-compose.yml, and CI drop cloud mode; Solr bumped to 9.10. The Makefile creates the core from the configset, optimizes the index, and tars the core into snapshot.backup.tar.gz (with pigz when available — single-threaded gzip on an already-compressed 130 GB index was hours for a few percent).
• /status fixed for standalone: it looked up the SolrCloud core name name_lookup_shard1_replica_n1, so under standalone it reported Expected core not found. with no document counts — quietly, since the endpoint still answers 200 and the Helm probes only check that. The core now comes from SOLR_CORE (default name_lookup), with a fallback to the single core present so old cloud-built backups keep working.
• Version pinning by role: the builder names an exact release tarball (SOLR_VERSION=9.10.0) and is the floor for anything serving the backup; the servers (docker-compose, CI, Helm chart) track the 9.10 line, so they pick up patch releases and stay at or above the builder automatically.
• Deleted: setup_solr.sh, solr-restore/ and configoverlay.json (all obsolete).
• Load sized to the container, not the node: parallelism and pigz now come from the cgroup CPU quota rather than nproc, which inside a pod reports the node's cores — 8 CPUs of quota on a 64-core node used to mean 64 throttled uploads. The loading heap drops from 220G (solr -m commits both -Xms and -Xmx) to 31G, since indexing wants page cache rather than heap, and ramBufferSizeMB rises to 2G — the buffer is a budget shared across indexing threads, so at 32 parallel uploads that is ~64 MB per segment instead of ~16 MB, and four times fewer segments to merge.
• Kubernetes loading job reworked: the pod goes to 32 CPUs / 128Gi, and /var/solr is sized at 600Gi (the optimize writes the new segment before dropping the old ones, so it peaks at ~2x the finished index). Moving the index onto a node-local NVMe ephemeral volume is the change that would actually speed this up — it takes the whole write load of the indexing run and the optimize, and it is the one volume we could afford to lose — but it is blocked on Enable nvme-ephemeral volumes in the data-loading namespace #280 (the namespace cannot create nvme-ephemeral volumes), so both volumes are persistent PVCs today and both need deleting by hand. Makefile stamp files now live on the volume whose state they describe, so they disappear with the index if that volume is ever made ephemeral. New data-loading/kubernetes/README.md covers the volume split, sizing, the Grafana panels that answer “would more of this help”, and every knob.
• CI image builds cached: the data-loading image was rebuilt from scratch every run — apt upgrade, seven apt installs and a ~250MB Solr tarball — taking ~25 minutes. It now uses the GitHub Actions cache.
• Helm chart (in translator-devops, edited copy under data/name-lookup/): standalone Solr, idempotent download.sh, restore.sh gutted to blocklist-only, heap lowered for page cache.

Validation

CI now runs the whole thing end to end, against a Solr loaded by the new pipeline:

• pytest: 28/28 pass (including test_status, which the standalone switch initially broke).
• Configset verified against a real Solr: core creates from the two files, load succeeds (so the uuid processor and /query survived the trim), an unknown field is rejected with a 400, a Schema API write is refused with schema is not editable, and highlighting still returns spans.
• The loader must abort on a file that does not load — the guard is what stands between us and a silently half-loaded index, so it is tested rather than assumed.
• The loader must refuse a core that already has documents. Loading assigns fresh UUIDs, so a re-run over the same files doubles the index, and the count guard cannot see it — the delta is still exactly right. This is the one way the loader can return a corrupt index while reporting success, so it is a hard error (LOAD_APPEND=1 to override).
• Full backup roundtrip in CI: tar the core → drop it into a fresh Solr → 89 documents served with zero schema setup and no chown. The roundtrip deliberately chowns the source core to uid 1000 first, standing in for the loading image, so it proves the tarball's ownership stamping rather than hiding it behind a fixup.

Also checked by hand: an unexpected field is rejected with a 400 (schemaless really is off), filenames with spaces and blank lines are counted correctly, and make data/backup.done on a restarted container restarts Solr instead of failing — without re-running solr create on a core that already exists.

Also confirmed the failure the ownership fix prevents: a backup tarred without it restores into a core that will not load, with SolrException: /var/solr/data/name_lookup/data/index/write.lock — Solr takes a write lock on the core it serves, and the loading image's uid (1000) is not the serving image's (8983).

Issues

Closes #238, closes #185, closes #256, closes #266. Addresses helxplatform/translator-devops#609. Progresses #265 (heap knob + smaller index; final heap/GC tuning and #272 still need load testing). Out of scope: #267, running natively on SLURM/HPC.

Pre-merge cleanup — done

• Deleted the pull_request: trigger from .github/workflows/release-nameres-loading.yml (c563ded). It only existed to publish a pr-278 image so a load could run against this branch without merging.
• Reverted nameres-loading.k8s.yaml to image: …:latest (b6458cc). Caveat carried over: the publish workflow only tags latest on a published release, so latest becomes this pipeline only once a release is cut — run the next load after this is merged and released.

The load that already ran — done

A full Babel 2026jul22 load (~111Gi index, ~331M docs) was built with the pr-278
image and is now restored and serving on the cluster, so the workarounds this section
used to list are spent. That run predated the review fixes, so the backup's ownership
was stamped by hand — the merged Makefile now does it automatically with tar --owner.
The load ran to completion (no partial-load restart, no stray files to skip). Two
problems surfaced during it, both now fixed: the backup's uid ownership (in this
branch) and the serving chart's Solr image tag being read as 9.1 (see the reviewer
note).

Note for reviewers

The data/name-lookup/ Helm chart is gitignored here (it's a working copy); its edits live in translator-devops. One caveat found while bringing the load up: the Solr image tag must be quoted (tag: "9.10") — unquoted, YAML parses 9.10 as the float 9.1 and pulls solr:9.1, which cannot init a 9.10-built core (it fails with _version_ … not searchable). That fix still needs applying in translator-devops.

Point dataUrl at a backup built by the new loader — old backups won't restore under the new flow.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR reworks NameRes’s Solr data-loading and backup pipeline around a self-contained standalone Solr core backup (config + schema + index) to simplify restore, remove cloud/ZooKeeper dependencies, and substantially speed up bulk loading.

Changes:

• Replace serial per-file commit + sleep loading with a parallel streaming loader guarded by a pre/post document-count check.
• Switch local docker-compose and CI Solr to standalone mode and standardize on Solr 9.10.0, creating the core from a checked-in configset.
• Add a checked-in Solr configset (schema + config + language resources) and remove obsolete restore/schema-setup scripts.

Reviewed changes

Copilot reviewed 56 out of 57 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/create_test_synonyms.py	Write test synonym data in JSON-lines format to match the production loader’s expectations.
solr-restore/restore.sh	Removed obsolete restore script (standalone core restore no longer uses replication restore + schema setup).
solr-restore/README.md	Removed obsolete documentation for the deleted restore flow.
docker-compose.yml	Bump Solr image to 9.10.0 and document standalone core auto-discovery on startup.
data-loading/setup-and-load-solr.sh	Parallel streaming loader with deferred commit and doc-count verification guard.
data-loading/setup_solr.sh	Removed schema/field creation via Schema API (schema now lives in configset).
data-loading/README.md	Updated documentation for standalone core backups, parallel loading, and simplified restore.
data-loading/Makefile	End-to-end pipeline: download/split/load/optimize/tar a self-contained core backup in standalone mode.
data-loading/Dockerfile	Copy in configset and run Solr standalone (--user-managed) for loader container flow.
data-loading/configsets/name_lookup/conf/solrconfig.xml	Checked-in Solr core configuration tuned for bulk load + serving, used as schema/config source of truth.
data-loading/configsets/name_lookup/conf/managed-schema.xml	Checked-in managed schema for the name_lookup core (single source of truth).
data-loading/configsets/name_lookup/conf/configoverlay.json	Disable Solr autoCreateFields via config overlay (ensures schema consistency).
data-loading/configsets/name_lookup/conf/synonyms.txt	Add Solr synonym resource file referenced by analyzers in the configset.
data-loading/configsets/name_lookup/conf/stopwords.txt	Add Solr stopwords resource file referenced by analyzers in the configset.
data-loading/configsets/name_lookup/conf/protwords.txt	Add Solr protected-words resource file referenced by analyzers in the configset.
data-loading/configsets/name_lookup/conf/lang/userdict_ja.txt	Add Japanese tokenizer user dictionary resource used by default analyzers.
data-loading/configsets/name_lookup/conf/lang/stoptags_ja.txt	Add Japanese POS stop-tag list resource for analyzer chains.
data-loading/configsets/name_lookup/conf/lang/stopwords_ar.txt	Add Arabic stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_bg.txt	Add Bulgarian stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_ca.txt	Add Catalan stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_cz.txt	Add Czech stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_da.txt	Add Danish stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_de.txt	Add German stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_el.txt	Add Greek stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_en.txt	Add English stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_es.txt	Add Spanish stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_eu.txt	Add Basque stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_fa.txt	Add Persian stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_fi.txt	Add Finnish stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_fr.txt	Add French stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_ga.txt	Add Irish stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_gl.txt	Add Galician stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_hi.txt	Add Hindi stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_hu.txt	Add Hungarian stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_hy.txt	Add Armenian stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_id.txt	Add Indonesian stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_it.txt	Add Italian stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_ja.txt	Add Japanese stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_lv.txt	Add Latvian stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_nl.txt	Add Dutch stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_no.txt	Add Norwegian stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_pt.txt	Add Portuguese stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_ro.txt	Add Romanian stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_ru.txt	Add Russian stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_sv.txt	Add Swedish stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_th.txt	Add Thai stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stopwords_tr.txt	Add Turkish stopwords resource file.
data-loading/configsets/name_lookup/conf/lang/stemdict_nl.txt	Add Dutch stemmer override dictionary resource.
data-loading/configsets/name_lookup/conf/lang/hyphenations_ga.txt	Add Irish hyphenations resource.
data-loading/configsets/name_lookup/conf/lang/contractions_ca.txt	Add Catalan contraction list resource for elision filter.
data-loading/configsets/name_lookup/conf/lang/contractions_fr.txt	Add French contraction list resource for elision filter.
data-loading/configsets/name_lookup/conf/lang/contractions_ga.txt	Add Irish contraction list resource for elision filter.
data-loading/configsets/name_lookup/conf/lang/contractions_it.txt	Add Italian contraction list resource for elision filter.
CLAUDE.md	Update repository guidance to reflect standalone core creation and new loader script usage.
.github/workflows/tester.yml	Run Solr 9.10.0 standalone in CI and create the core from the checked-in configset before tests.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 21 out of 22 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 21 out of 22 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

data-loading/setup-and-load-solr.sh:42

• solr_count() parses Solr’s JSON response with a regex that assumes no whitespace after "numFound":. Since /query is configured with indent=true, the JSON writer may legally emit whitespace (e.g. "numFound": 123), which would make this return an empty count and abort an otherwise successful load. Make the match tolerate whitespace and also surface curl errors for easier debugging.

solr_count() {
  curl -sf "${SOLR_SERVER}/solr/${CORE}/query?q=*:*&rows=0" \
    | grep -oE '"numFound":[0-9]+' | head -1 | grep -oE '[0-9]+'
}