HuggingFace Hub integration: opt-in weight upload after training, default weight download for inference, tag dedup guards

README.md

@@ -113,6 +113,10 @@ Aetherscan reads secrets and path overrides from a `.env` file at the repo root.
 SLACK_BOT_TOKEN=your-slack-bot-token
 SLACK_CHANNEL=your-slack-channel
 
+# Only needed for uploading model weights to the HuggingFace Hub (train --hf-upload);
+# downloads (the inference default) hit a public repo and need no token
+HF_TOKEN=your-huggingface-write-token
+
 # If none specified, defaults to /datax/scratch/zachy/{data|models|outputs}/aetherscan
 # Note, CLI flags (--data-path, --model-path, --output-path) override these
 AETHERSCAN_DATA_PATH=/path/to/data
@@ -133,7 +137,7 @@ export SLACK_CHANNEL="your-slack-channel"
 ./utils/run_container.sh python -m aetherscan.main train ...
 ```
 
-The `AETHERSCAN_*` paths are bind-mounted 1:1 between host and container, so they must already exist on the host before the pipeline starts. The `utils/run_container.sh` wrapper forwards `SLACK_*` and `AETHERSCAN_*` into the container explicitly; if you need additional env vars on the container side, extend the wrapper's `--env` list.
+The `AETHERSCAN_*` paths are bind-mounted 1:1 between host and container, so they must already exist on the host before the pipeline starts. The `utils/run_container.sh` wrapper forwards `SLACK_*`, `AETHERSCAN_*`, and `HF_TOKEN` into the container explicitly; if you need additional env vars on the container side, extend the wrapper's `--env` list.
 
 **5. Run pipeline**
 
@@ -450,8 +454,10 @@ usage: train [-h] [--data-path DATA_PATH] [--model-path MODEL_PATH]
              [--patience-threshold PATIENCE_THRESHOLD]
              [--lr-reduction-factor LR_REDUCTION_FACTOR]
              [--max-retries MAX_RETRIES] [--retry-delay RETRY_DELAY]
+             [--hf-upload | --no-hf-upload] [--hf-repo-id HF_REPO_ID]
              [--load-dir LOAD_DIR] [--load-tag LOAD_TAG]
              [--start-round START_ROUND] [--save-tag SAVE_TAG]
+             [--force-tag | --no-force-tag]
 
 options:
   -h, --help            show this help message and exit
@@ -672,6 +678,15 @@ options:
   --retry-delay RETRY_DELAY
                         Delay in seconds between retry attempts after training
                         failure
+  --hf-upload, --no-hf-upload
+                        Upload the final model artifacts (encoder, decoder,
+                        random forest, config) plus a generated model card to
+                        the HuggingFace Hub after training completes, tagging
+                        the commit with --save-tag (default: disabled = local-
+                        only). Requires HF_TOKEN in the environment (via .env)
+  --hf-repo-id HF_REPO_ID
+                        HuggingFace model repo id (namespace/name) for weight
+                        upload/download (default: zachtheyek/aetherscan)
   --load-dir LOAD_DIR   Subdirectory for checkpoint loading (relative to
                         --model-path)
   --load-tag LOAD_TAG   Model tag for checkpoint loading. Accepted formats:
@@ -684,6 +699,12 @@ options:
   --save-tag SAVE_TAG   Tag for current pipeline run. Accepted formats:
                         final_vX, round_XX, test_vX. Current timestamp used
                         (YYYYMMDD_HHMMSS) if none specified
+  --force-tag, --no-force-tag
+                        Override the fail-early save-tag collision guard:
+                        proceed even when an explicitly-provided --save-tag
+                        matches existing artifacts, DB rows, or (with --hf-
+                        upload) an existing HuggingFace tag (default:
+                        disabled)
 ```
 
 ### Inference Command Help
@@ -721,7 +742,8 @@ usage: inference [-h] [--data-path DATA_PATH] [--model-path MODEL_PATH]
                  [--stamp-gallery-top-k STAMP_GALLERY_TOP_K]
                  [--max-candidate-plots MAX_CANDIDATE_PLOTS]
                  [--max-retries MAX_RETRIES] [--retry-delay RETRY_DELAY]
-                 [--save-tag SAVE_TAG]
+                 [--hf-repo-id HF_REPO_ID] [--hf-revision HF_REVISION]
+                 [--save-tag SAVE_TAG] [--force-tag | --no-force-tag]
 
 options:
   -h, --help            show this help message and exit
@@ -760,11 +782,19 @@ options:
                         provided, triggers the energy detection preprocessing
                         pipeline and takes precedence over --test-files
   --encoder-path ENCODER_PATH
-                        Path to trained VAE encoder model file (.keras)
-  --rf-path RF_PATH     Path to trained Random Forest model file (.joblib)
+                        Path to trained VAE encoder model file (.keras).
+                        Optional: when none of --encoder-path/--rf-
+                        path/--config-path are given, the artifacts are
+                        downloaded from the HuggingFace Hub (see --hf-repo-
+                        id/--hf-revision); provide either all three local
+                        paths or none
+  --rf-path RF_PATH     Path to trained Random Forest model file (.joblib).
+                        Optional: see --encoder-path for the all-three-or-none
+                        rule
   --config-path CONFIG_PATH
                         Path to config file from corresponding training run
-                        (.json)
+                        (.json). Optional: see --encoder-path for the all-
+                        three-or-none rule
   --per-replica-batch-size PER_REPLICA_BATCH_SIZE
                         Batch size per GPU/device replica during inference
   --classification-threshold CLASSIFICATION_THRESHOLD
@@ -863,8 +893,22 @@ options:
                         (including preprocessing) on failure
   --retry-delay RETRY_DELAY
                         Delay in seconds between inference retry attempts
+  --hf-repo-id HF_REPO_ID
+                        HuggingFace model repo id (namespace/name) for weight
+                        upload/download (default: zachtheyek/aetherscan)
+  --hf-revision HF_REVISION
+                        HuggingFace revision (tag, branch, or commit hash) to
+                        pin the model download to when no local artifact paths
+                        are given (default: the repo's latest release tag —
+                        highest semver vX.Y.Z tag, falling back to the highest
+                        final_vX training tag)
   --save-tag SAVE_TAG   Tag for current pipeline run. Current timestamp used
                         (YYYYMMDD_HHMMSS) if none specified
+  --force-tag, --no-force-tag
+                        Override the fail-early save-tag collision guard:
+                        proceed even when an explicitly-provided --save-tag
+                        matches a previous run's saved config or DB rows
+                        (default: disabled)
 ```
 
 ---

SECURITY.md

@@ -64,6 +64,14 @@ If you suspect a token has been compromised, rotate immediately:
 6. Update `SLACK_BOT_TOKEN` in all deployment environments
 7. Verify the new token works: `PYTHONPATH=src python utils/print_cli_help.py train` (should not show Slack errors)
 
+#### HuggingFace Hub Token
+
+1. Go to [HuggingFace access tokens](https://huggingface.co/settings/tokens)
+2. Locate the compromised token and invalidate (refresh) or delete it
+3. Create a replacement token — grant **write** access only if you upload artifacts (`train --hf-upload`); the default inference path downloads from a **public** repo and needs no token at all
+4. Update `HF_TOKEN` in all deployment environments (the gitignored `.env`, or exported in the shell)
+5. Verify the new token works by re-running an upload: `./utils/run_container.sh python -m aetherscan.main train --hf-upload ...` (the upload stage should authenticate without errors)
+
 ---
 
 ## Secrets Management
@@ -72,9 +80,12 @@ If you suspect a token has been compromised, rotate immediately:
 
 Aetherscan uses the following secrets that must be protected:
 
-| Secret          | Environment Variable | Purpose                        |
-| --------------- | -------------------- | ------------------------------ |
-| Slack Bot Token | `SLACK_BOT_TOKEN`    | Slack alerts and notifications |
+| Secret            | Environment Variable | Purpose                                       |
+| ----------------- | -------------------- | --------------------------------------------- |
+| Slack Bot Token   | `SLACK_BOT_TOKEN`    | Slack alerts and notifications                |
+| HuggingFace Token | `HF_TOKEN`           | Upload model artifacts to the HuggingFace Hub |
+
+`HF_TOKEN` is only needed to **upload** trained model artifacts to the HuggingFace Hub (opt-in via `train --hf-upload`); the default inference path downloads from a **public** repo and needs no token, so grant the token **write** scope only when uploading. Like `SLACK_BOT_TOKEN`, it is read from the gitignored `.env` (never committed) and forwarded into the NGC container through `utils/run_container.sh`'s explicit `--env` allowlist — never log it (INFO-level logs may reach Slack).
 
 ### Best Practices
 

docs/CONFIG_AND_CLI.md

@@ -76,7 +76,8 @@ subcommand uses it:
 | `DataConfig`         | Data shape, file lists, chunk sizes                                                                                                |
 | `TrainingConfig`     | Anything specific to the `train` command — sample counts, batch sizes, LR schedule, curriculum, latent-viz, retries                |
 | `InferenceConfig`    | Anything specific to the `inference` command — encoder/RF paths, classification threshold, energy-detection preprocessing, retries |
-| `CheckpointConfig`   | Load/save tags, start round                                                                                                        |
+| `HFConfig`           | HuggingFace Hub integration — target repo id, opt-in post-training upload, inference revision pin                                  |
+| `CheckpointConfig`   | Load/save tags, start round, tag-collision-guard override                                                                          |
 
 Two important consequences:
 
@@ -162,7 +163,9 @@ Current Pattern B flags:
 - `--gpu-memory-limit-mb` → `config.gpu.per_gpu_memory_limit_mb`
 - `--async-allocator` → `config.gpu.use_async_allocator`
 - `--num-replicas` → `config.gpu.num_replicas`
+- `--hf-repo-id` → `config.hf.repo_id`
 - `--save-tag` → `config.checkpoint.save_tag`
+- `--force-tag` → `config.checkpoint.force_tag`
 
 #### Pattern C — shared flag, divergent destination
 
@@ -364,5 +367,5 @@ mode && /^ *"--/{
 
 The second command lists every shared flag (those appearing in both subparsers); each
 should match either a single Pattern B `apply_args` block or a pair of Pattern C blocks
-with a `command` discriminator. As of this writing it yields 10 shared flags —
-7 Pattern B + 3 Pattern C — matching the tables above.
+with a `command` discriminator. As of this writing it yields 12 shared flags —
+9 Pattern B + 3 Pattern C — matching the tables above.

environment.yml

@@ -57,3 +57,6 @@ dependencies:
       - shap>=0.46.0,<0.47
       - slack-sdk>=3.31.0,<4
       - python-dotenv>=1.0,<2
+      # Model weight distribution via the HuggingFace Hub — pin range mirrors
+      # requirements-container.txt (see the rationale there).
+      - huggingface_hub>=1.21,<1.22

requirements-container.txt

@@ -20,6 +20,10 @@ umap-learn>=0.5.6,<0.6
 shap>=0.46.0,<0.47
 slack-sdk>=3.31.0,<4
 python-dotenv>=1.0,<2
+# Model weight distribution via the HuggingFace Hub (upload after training, download for
+# inference). 1.21 per SECURITY.md's version policy: two minors below the latest stable
+# (1.23) at pin time, newer than the newest >=6-month-old release.
+huggingface_hub>=1.21,<1.22
 # setuptools provides pkg_resources, which blimpy (transitive via setigen)
 # imports at module load. Recent pip / Python releases dropped setuptools
 # from the bootstrap, so we install it explicitly. Upper-bound at <81 because