Skip to content

[DevOps]: Simplify site-ingestion launcher configuration #336

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tomvothecoder wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[DevOps]: Simplify site-ingestion launcher configuration #336

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
235 changes: 112 additions & 123 deletions backend/app/scripts/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,93 +1,55 @@
# Operational Scripts

This directory contains administrative scripts for database management, account
provisioning, archive ingestion, and diagnostics discovery. These scripts are
internal operational entry points and are not part of the public API.

## Quick Start

### Requirements

Before running a script:

1. Set the required environment variables.
2. Confirm that the target database or API is accessible.
3. Activate the correct local, staging, or production environment.

Scripts are organized by domain:

```text
scripts/
├── ingestion/
│ ├── archive_client.py
│ ├── archive_discovery.py
│ ├── archive_ingestor_core.py
│ ├── archive_layout.py
│ ├── archive_workflow.py
│ ├── diagnostics_archives.py
│ ├── diagnostics_link_scanner.py
│ ├── hpc_upload_archive_ingestor.py
│ ├── nersc_archive_ingestor.py
│ ├── sites/
│ │ ├── lcrc-diagnostics-scanner.sh
│ │ ├── nersc-diagnostics-scanner.sh
│ │ ├── site_ingestion_launcher.sh
│ │ ├── chrysalis.config
│ │ └── nersc.config
│ └── v3_data/
│ ├── __init__.py
│ ├── lcrc-v3.env.example
│ ├── lcrc_v3.sh
│ ├── lcrc_v3_archive_ingestor.py
│ └── lcrc_v3_hpss_linker.py
├── db/
│ ├── seed.py
│ ├── rollback_seed.py
│ └── catalog.json
└── users/
├── create_admin_account.py
└── provision_service_account.py
```
This directory contains internal operational entry points for database
management, account provisioning, archive ingestion, and diagnostics discovery.
They are not part of the public API.

## In This Guide

Scripts may depend on:
- [Run a script](#run-a-script)
- [Choose an ingestion workflow](#ingestion-workflows)
- [Configure and schedule site collection](#site-collection-launcher)
- [Run the diagnostics scanner](#nersc-diagnostics-link-scanner)
- [Backfill E3SM v3 data](#one-time-chrysalis-e3sm-v3-archive-backfill)
- [Link E3SM v3 HPSS archives](#e3sm-v3-hpss-linker)

- Application configuration from `app.core.config`
- Database configuration from `app.core.database` or `database_async`
- SQLAlchemy models and application services
## Run a Script

### Run Scripts as Modules
Before running an operational script:

Run all scripts as modules from the project root. This ensures correct package
imports, configuration loading, and environment behavior.
1. Set its required environment variables.
2. Confirm the target database or API is reachable.
3. Select the intended local, staging, or production environment.

Run Python scripts as modules from the backend project root. This preserves
package imports, application configuration, and environment behavior.

```bash
python -m app.scripts.db.seed
python -m app.scripts.db.rollback_seed
python -m app.scripts.users.create_admin_account
python -m app.scripts.ingestion.hpc_upload_archive_ingestor
python -m app.scripts.ingestion.nersc_archive_ingestor
python -m app.scripts.ingestion.v3_data.lcrc_v3_archive_ingestor
python -m app.scripts.ingestion.v3_data.lcrc_v3_hpss_linker
uv run python -m app.scripts.db.seed
uv run python -m app.scripts.db.rollback_seed
uv run python -m app.scripts.users.create_admin_account
uv run python -m app.scripts.ingestion.hpc_upload_archive_ingestor
uv run python -m app.scripts.ingestion.nersc_archive_ingestor
uv run python -m app.scripts.ingestion.v3_data.lcrc_v3_archive_ingestor
uv run python -m app.scripts.ingestion.v3_data.lcrc_v3_hpss_linker
```

Do not execute scripts directly by file path:
> **Do not** execute a Python script directly by file path:

```bash
# Avoid
python app/scripts/db/seed.py
# Incorrect
uv run python app/scripts/ingestion/nersc_archive_ingestor.py
```

## Script Directory

Scripts are organized by domain:
### Script Areas

| Domain | Purpose |
| ------------ | ---------------------------------------------------------------- |
| `ingestion/` | Scheduled ingestion, archive backfill, and diagnostics workflows |
| `db/` | Database seeding and rollback utilities |
| `users/` | Administrative and service-account management |

The primary operational entry points are:
### Primary Entry Points

| Workflow | Entry point | Purpose |
| ------------------------ | ------------------------------------- | ------------------------------------------------- |
Expand All @@ -97,11 +59,19 @@ The primary operational entry points are:
| E3SM v3 archive backfill | `v3_data/lcrc_v3_archive_ingestor.py` | Backfill selected Chrysalis simulations |
| E3SM v3 HPSS linking | `v3_data/lcrc_v3_hpss_linker.py` | Add documented HPSS URLs to existing cases |

## Common Ingestion Behavior
## Ingestion Workflows

Choose the runner based on where the archive filesystem is available:

| Archive location | Runner | Submission method |
| --- | --- | --- |
| Mounted in the SimBoard backend environment | `nersc_archive_ingestor.py` | API path submission |
| Available only at a remote HPC site | `hpc_upload_archive_ingestor.py` | Per-case archive upload |

The NERSC path-based and HPC upload ingestors share archive discovery,
validation, deduplication, state tracking, dry-run, retry, and per-case
submission behavior.
Both runners share archive discovery, validation, deduplication, state tracking,
dry-run, retry, and per-case submission behavior.

## Common Ingestion Behavior

### Scan Modes

Expand All @@ -110,7 +80,7 @@ submission behavior.
| `staging` | `PERF_ARCHIVE_ROOT` | `/performance_archive` | Scan the current staging archive |
| `archive` | `OLD_PERF_ARCHIVE_ROOT` | `/OLD_PERF` | Scan historical archive snapshots |

Archive mode has these constraints:
In archive mode:

- Only top-level `YYYY-MM` directories are traversed. Other directories are
ignored.
Expand Down Expand Up @@ -141,24 +111,34 @@ Both automated ingestors persist immutable validation results before ingestion:
or conflicting stored outcome stops ingestion.
- Dry runs do not write discovery results or processed state.

### Common Environment Variables

| Variable | Required | Default | Purpose |
| ------------------------- | --------- | ---------------------- | ---------------------------------------- |
| `SIMBOARD_API_BASE_URL` | Live runs | None | SimBoard API endpoint |
| `SIMBOARD_API_TOKEN` | Live runs | None | Service-account token |
| `SCAN_MODE` | No | `staging` | Select staging or archive scanning |
| `PERF_ARCHIVE_ROOT` | No | `/performance_archive` | Staging archive root |
| `OLD_PERF_ARCHIVE_ROOT` | No | `/OLD_PERF` | Historical archive root |
| `MACHINE_NAME` | No | `perlmutter` | Source machine recorded during ingestion |
| `DRY_RUN` | No | `true` | Prevent ingestion and state changes |
| `MAX_CASES_PER_RUN` | No | Unlimited | Limit submissions per invocation |
| `MAX_ATTEMPTS` | No | Unlimited | Limit request attempts |
| `REQUEST_TIMEOUT_SECONDS` | No | `60` | Set the request timeout in seconds |
| `ARCHIVE_YEAR_START` | No | None | Earliest archive month to scan |
| `ARCHIVE_YEAR_END` | No | None | Latest archive month to scan |
### Environment Variables

## Ingestion Workflows
**API access**

| Variable | Required | Purpose |
| --- | --- | --- |
| `SIMBOARD_API_BASE_URL` | Live runs and remote-state dry runs | SimBoard API endpoint |
| `SIMBOARD_API_TOKEN` | Live runs and remote-state dry runs | Service-account token |

**Archive selection**

| Variable | Default | Purpose |
| --- | --- | --- |
| `SCAN_MODE` | `staging` | Select staging or archive scanning |
| `PERF_ARCHIVE_ROOT` | `/performance_archive` | Staging archive root |
| `OLD_PERF_ARCHIVE_ROOT` | `/OLD_PERF` | Historical archive root |
| `MACHINE_NAME` | `perlmutter` | Source machine recorded during ingestion |
| `ARCHIVE_YEAR_START` | None | Earliest archive month to scan |
| `ARCHIVE_YEAR_END` | None | Latest archive month to scan |

**Run controls**

| Variable | Default | Purpose |
| --- | --- | --- |
| `DRY_RUN` | `true` | Prevent ingestion and state changes |
| `MAX_CASES_PER_RUN` | Unlimited | Limit submissions per invocation |
| `MAX_ATTEMPTS` | Unlimited | Limit request attempts |
| `REQUEST_TIMEOUT_SECONDS` | `60` | Request timeout in seconds |

### NERSC Path-Based Archive Ingestion

Expand All @@ -176,14 +156,6 @@ MACHINE_NAME=perlmutter \
uv run python -m app.scripts.ingestion.nersc_archive_ingestor
```

#### NERSC Site Launcher

Use `sites/site_ingestion_launcher.sh nersc <staging|archive>` for host-side
NERSC collection. The launcher loads `sites/nersc.config` and runs
`python -m app.scripts.ingestion.nersc_archive_ingestor`. Override `SCAN_MODE`,
`DRY_RUN`, or another supported variable in the calling environment or cron
entry when a different behavior is required.

### HPC Upload Archive Ingestion

#### When to Use It
Expand Down Expand Up @@ -213,36 +185,51 @@ The common ingestion environment variables and archive rules apply.

`app/scripts/ingestion/sites/site_ingestion_launcher.sh` is the host-side
launcher for site collection. It loads `sites/<site>.config`, then selects the
configured Python ingestor. Use it as:
configured Python ingestor.

```bash
app/scripts/ingestion/sites/site_ingestion_launcher.sh nersc staging
app/scripts/ingestion/sites/site_ingestion_launcher.sh chrysalis archive
```

Each site config defines its machine name, archive roots, Python environment
file, token export file, API base URL, archive lower bound, and ingestor module.
Set `SIMBOARD_ROOT` to a shared operational directory containing
`repository/simboard` and `operations`; the launcher derives the backend and
working paths from it. A site config may instead set `SIMBOARD_MODULES` and
`SIMBOARD_WORKDIR` explicitly before using either variable. The launcher defaults to `DRY_RUN=true` with
`DRY_RUN_USE_REMOTE_STATE=true`, so it loads API credentials and performs
read-only state validation. Set `DRY_RUN_USE_REMOTE_STATE=false` for a
credential-free offline scan. Set `DRY_RUN=false` only after validating archive
access, token storage, network egress, and candidate counts. A capped
`MAX_CASES_PER_RUN` value limits real ingestion but still persists results.
#### Configuration Ownership

| Location | Configure | Notes |
| --- | --- | --- |
| Committed site config | Machine name, archive roots, API URL, archive lower bound, ingestor module | Never store credentials here. |
| Deployment configuration | `SIMBOARD_ROOT`, `SIMBOARD_ENV_FILE`, and protected token file | Required for the standard layout. |
| Nonstandard deployment only | `SIMBOARD_MODULES`, `SIMBOARD_WORKDIR`, `SIMBOARD_API_TOKEN_FILE` | Overrides the standard derived paths. |

For the standard layout, `SIMBOARD_ROOT` contains both
`repository/simboard/backend` and `operations`. The launcher derives its module
and working paths, then defaults the token file to
`$SIMBOARD_WORKDIR/.api_token_export`.

#### Run Safely

1. Start with the default `DRY_RUN=true`. It also defaults
`DRY_RUN_USE_REMOTE_STATE=true`, which validates remote state without writing.
2. Use `DRY_RUN_USE_REMOTE_STATE=false` only for a credential-free offline scan.
3. Confirm archive access, token storage, network egress, and candidate counts.
4. Set `DRY_RUN=false` only after that review. Use `MAX_CASES_PER_RUN` to cap a
live run; it limits submissions but still persists validation results.

Site configs are operational inputs. Keep credentials in their referenced,
protected files rather than committing them to a config file.

> **Upgrade note:** Remove any externally managed token-file default that expands
> `SIMBOARD_WORKDIR` while the site config is sourced. The launcher now derives
> that path after loading the config.

### Cron Setup

Copy `sites/crontab.example` outside the repository, set `SIMBOARD_ROOT` to the
shared operational directory, and install the adjusted file with `crontab`.
1. Copy `sites/crontab.example` outside the repository.
2. Set the deployment's `SIMBOARD_ROOT` and `SIMBOARD_ENV_FILE`.
3. Ensure `$SIMBOARD_ROOT/operations/.api_token_export` is readable only by the
cron user and exports `SIMBOARD_API_TOKEN`.
4. Install the adjusted copy with `crontab`.

The example schedules staging scans every 15 minutes and archive scans daily at
12:00 UTC. The token file referenced by the site config must be readable only by
the account that runs the scheduled job and export `SIMBOARD_API_TOKEN` when
sourced. Keep token values out of the repository and crontab.
12:00 UTC. Never place token values in the repository or crontab.

## NERSC Diagnostics Link Scanner

Expand Down Expand Up @@ -282,8 +269,10 @@ directory leaf names exactly, forces archive scanning from `2024-01`, and
reuses the HPC upload runner's discovery, validation, deduplication, packaging,
and `/api/v1/ingestions/from-hpc-upload` request logic.

For this one-time backfill, copy the committed template outside the repository,
secure it, replace its placeholders, then run a dry run:
### Configure

Copy the committed template outside the repository, secure it, and replace its
placeholders:

```bash
mkdir -p ~/.config/simboard
Expand All @@ -302,7 +291,7 @@ The environment file must define:
Set `OLD_PERF_ARCHIVE_ROOT` only when the Chrysalis archive is mounted somewhere
other than its documented default.

#### Run from the Repository Root
### Run from the Repository Root

Start with the Make dry run:

Expand Down Expand Up @@ -334,7 +323,7 @@ The Make targets override `DRY_RUN`; keep the external environment file focused
on the API credentials and optional archive-root override. They run Python with
unbuffered output so emitted structured events appear in the console immediately.

#### Fixed and Supported Settings
### Fixed and Supported Settings

The source site and scan scope are fixed. The runner ignores:

Expand All @@ -349,14 +338,14 @@ The following controls remain supported:
- `REQUEST_TIMEOUT_SECONDS`
- `ARCHIVE_YEAR_END`

#### State Behavior
### State Behavior

This targeted runner does not read or write database-backed archive snapshot
checkpoints. A filtered backfill cannot safely mark a mixed snapshot as complete
for the general archive runner. Processed-execution state and immutable
discovery results still make repeated runs idempotent.

### E3SM v3 HPSS Linker
## E3SM v3 HPSS Linker

#### Purpose

Expand Down
5 changes: 2 additions & 3 deletions backend/app/scripts/ingestion/sites/chrysalis.config
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
export SIMBOARD_ENV_FILE="${HOME}/envs/test_simboard/bin/activate"
export SIMBOARD_API_TOKEN_FILE="${SIMBOARD_API_TOKEN_FILE:-${SIMBOARD_WORKDIR:-}/.api_token_export}"
# Deployment configuration must set SIMBOARD_ENV_FILE. The launcher derives the
# default SIMBOARD_API_TOKEN_FILE from the resolved work directory.
export SIMBOARD_INGESTOR_MODULE="app.scripts.ingestion.hpc_upload_archive_ingestor"
export SIMBOARD_API_BASE_URL="${SIMBOARD_API_BASE_URL:-https://simboard-dev-api.e3sm.org}"
export SIMBOARD_DEFAULT_ARCHIVE_YEAR_START="${SIMBOARD_DEFAULT_ARCHIVE_YEAR_START:-2025-01}"
export DRY_RUN="${DRY_RUN:-true}"
# export MAX_CASES_PER_RUN=
export PERF_ARCHIVE_ROOT="${PERF_ARCHIVE_ROOT:-/lcrc/group/e3sm/PERF_Chrysalis/performance_archive}"
export OLD_PERF_ARCHIVE_ROOT="${OLD_PERF_ARCHIVE_ROOT:-/lcrc/group/e3sm/PERF_Chrysalis/OLD_PERF}"
export MACHINE_NAME="${MACHINE_NAME:-chrysalis}"
21 changes: 14 additions & 7 deletions backend/app/scripts/ingestion/sites/crontab.example
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,24 @@
# Example crontab entries for a generic site collection launcher.
# Install with something like:
# crontab backend/app/scripts/ingestion/sites/<site>.crontab.example
# Example crontab entries for a generic site collection launcher. Copy this file
# outside the repository, replace the placeholders, then install the copy with:
# crontab /path/to/<site>.crontab

SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
CRON_TZ=UTC

# Assume SIMBOARD_ROOT is a directory containing both
# "repository/simboard/backend/..." and
# "operations" subdirectories.
# Standard layout: SIMBOARD_ROOT contains both `repository/simboard/backend/`
# and `operations/`. The launcher derives its module and work paths from it.
SIMBOARD_ROOT=/path/to/simboard_root

# Staging scan: run every 15 minutes, offset 5 minutes into the hour.
# Deployment configuration: source the backend environment required to reach the
# API. Store the API token in ${SIMBOARD_ROOT}/operations/.api_token_export;
# it must be readable only by this cron user and export SIMBOARD_API_TOKEN.
SIMBOARD_ENV_FILE=/path/to/backend/environment.sh

# For a nonstandard layout only, set SIMBOARD_WORKDIR and SIMBOARD_MODULES here.
# Otherwise, leave them unset.

# Replace `chrysalis` with the site-config name. Staging scan: every 15 minutes.
5,20,35,50 * * * * cd "${SIMBOARD_ROOT}/operations" && "${SIMBOARD_ROOT}/repository/simboard/backend/app/scripts/ingestion/sites/site_ingestion_launcher.sh" chrysalis staging

# Archive scan: run daily at 12:00 UTC. The site config supplies the default
Expand Down
10 changes: 0 additions & 10 deletions backend/app/scripts/ingestion/sites/nersc.config
View file
Open in desktop

This file was deleted.

Loading
Loading