diff --git a/.gitignore b/.gitignore index e0371f5..0751f1c 100644 --- a/.gitignore +++ b/.gitignore @@ -46,3 +46,7 @@ studies/ # Default database location .data/ +# Personal study manifests and worker scripts +train_*.yaml +colab_worker.py + diff --git a/AGENTS.md b/AGENTS.md index e088e90..47c32b6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -72,6 +72,10 @@ When the user asks about study progress, trial results, or health: 1. Call `get_study_data(study_name)` to retrieve trial telemetry, health tier, fANOVA importances, best trials, and OOM patterns. 2. Summarize: current best score, health status, stagnation indicators, and any OOM warnings. 3. If health is `watch` or `intervene`, explain the reason and offer to explore the data further. +4. If `vram_telemetry.oom_count > 0`, inspect the `oom_trials` list: + - Cluster OOM trials by shared hyperparameters (e.g. "all OOM trials used resolution=1024 and batch_size >= 16") + - Compare peak VRAM values to `gpu_capacity_gb` — how close is the margin? + - Recommend narrowing the search space bounds for the offending parameter(s) ## IDE triggers & status polling (.hpo_status.json) @@ -87,5 +91,4 @@ When a completed trial is reported or the background daemon polls health, the sy - Never block the GPU worker on an LLM; the suggest path stays TPE. - Never auto-invoke the coordinator from the broker, dashboard, or a hook - only on user request. - Do not write/modify local JSON files for search space or configuration (e.g. `active_search_space.json`, `hpo_config.json`); state must reside in SQLite. -- Do not modify the root `colab_worker.py` unless the user owns the bridge-crack project. Cloners use `templates/`. -- Never modify `archive/` files — they are historical reference only. +- Cloners use `templates/`. Do not generate project-specific workers at the repo root. diff --git a/README.md b/README.md index 853a647..055b8be 100644 --- a/README.md +++ b/README.md @@ -3,12 +3,13 @@ [![Build Status](https://github.com/Ishaan1402/pathfinder/actions/workflows/integration.yml/badge.svg)](https://github.com/Ishaan1402/pathfinder/actions) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg?style=flat-square)](LICENSE) -Pathfinder is an MCP-integrated hyperparameter optimization dashboard that lets AI coding agents onboard your training script and inspect running experiments. It wraps Optuna's TPE sampler in a FastAPI broker with SQLite persistence and a vanilla JS dashboard, while exposing structured study data through Model Context Protocol tools so your IDE agent can meaningfully participate in the tuning loop. +Pathfinder is an MCP-integrated hyperparameter optimization dashboard that lets coding agents onboard your training script and inspect running experiments. It wraps Optuna's TPE sampler in a FastAPI broker with SQLite persistence and a dashboard UI. Study data is exposed through Model Context Protocol tools so your IDE agent can meaningfully participate and advise in the tuning loop. +
- Pathfinder Dashboard + Pathfinder Dashboard Hyperparameter Pathways Plot @@ -17,22 +18,24 @@ Pathfinder is an MCP-integrated hyperparameter optimization dashboard that lets
-**Designed for:** ML researchers tuning deep learning models on their own hardware (local GPU, Colab, cloud VMs). Connect your training loop in ~60 lines of code. ## Why Pathfinder? -ML practitioners waste GPU hours on poorly-bounded search spaces and have to manually inspect trial data by grepping logs or refreshing notebooks. Pathfinder gives you a live monitoring dashboard plus an MCP server so your IDE agent can read study state and help onboard new studies. It does not compete with W&B Sweeps or Ray Tune — it's a demonstration of agent-assisted HPO workflows. +ML practitioners spend varying amounts of time and compute on poorly-bounded search spaces and have to manually inspect trial data by reading logs or refreshing notebooks. Pathfinder offers a live monitoring dashboard plus an MCP server so your IDE agent can read study state and help onboard new studies. -Three independent layers: +2 layers: - **Broker (Optuna TPE)**: Fast, deterministic suggestion engine. Suggestions and pruning happen in <10ms. Workers hit the broker and continue training immediately. -- **Worker**: Trains autonomously in a loop. Reports metrics per epoch, handles pruning, OOM detection, and checkpointing. -- **Coordinator (you + optional LLM)**: Run episodic reviews when you decide. Inspect trial history, check search health, propose bounds changes. AI agents (Cursor, Claude Code) can assist via MCP tools. +- **Worker**: Trains your model autonomously in a loop. Reports metrics per epoch, handles pruning, OOM detection, and checkpointing. + +An MCP server lets coding agents inspect structured study data, validate manifests, and register new studies, only when you ask. The tuning path is never blocked by LLMs. -All state lives in **SQLite** — resumable, auditable, portable. +All state lives in SQLite. ## Quick Start + + ### Step 1: Start the Broker **Option A: Docker (zero-install)** @@ -52,6 +55,8 @@ python broker.py --daemon # Dashboard: http://127.0.0.1:8000 ``` + + ### Step 2: Connect Your Workers **Local worker (same machine)** @@ -91,57 +96,72 @@ Point your IDE at the MCP server for agent-driven onboarding and inspection. See ### Environment Variables Reference -| Variable | Default | Description | -|---|---|---| -| `HPO_DATABASE_URL` | `sqlite:///hpo_studies.db` | SQLite connection string. | -| `HPO_BROKER_URL` | `http://localhost:8000` | URL where the broker is running. Required by workers. | -| `HPO_STUDY_NAME` | *(none)* | Default study name when not passed explicitly. | -| `HPO_SECRET_TOKEN` | *(none)* | Bearer token for securing broker endpoints in remote deployments. | -| `HPO_DEBUG` | `0` | Set to `1` to enable verbose debug logging in the broker. | -| `HPO_SPARKLINES` | `0` | Set to `1` to print a Unicode performance curve on trial completion. | -| `HPO_BACKUP_ON_START` | `0` | Set to `1` to run a database backup when the broker starts. | -| `HPO_CAPTURE_FULL_ENV` | `0` | Set to `1` to capture full `pip freeze` output (default: ML whitelist only). | -| `HPO_TUNNEL_PROVIDER` | *(none)* | Tunnel provider for remote access: `ngrok` or `cloudflare`. | -| `HPO_TUNNEL_URL` | *(none)* | Static tunnel URL when using `cloudflare` provider. | -| `HPO_ALLOWED_ORIGINS` | *(none)* | Additional CORS origins (comma-separated) for the dashboard. | + +| Variable | Default | Description | +| ---------------------- | -------------------------- | ----------------------------------------------------------------------------------------------------------- | +| `HPO_DATABASE_URL` | `sqlite:///hpo_studies.db` | SQLite connection string | +| `HPO_BROKER_URL` | `http://localhost:8000` | URL where the broker is running (required) | +| `HPO_STUDY_NAME` | *(none)* | Default study name when not passed explicitly | +| `HPO_SECRET_TOKEN` | *(none)* | Bearer token for endpoints in remote deployments | +| `HPO_DEBUG` | `0` | Set to `1` to enable verbose debug logging | +| `HPO_SPARKLINES` | `0` | Set to `1` to print a neat performance curve on trial completion :) | +| `HPO_BACKUP_ON_START` | `0` | Set to `1` to run a database backup when the broker starts. | +| `HPO_CAPTURE_FULL_ENV` | `0` | Set to `1` to capture all installed packages; default captures only whitelisted core framework dependencies | +| `HPO_TUNNEL_PROVIDER` | *(none)* | Tunnel provider for remote access: `ngrok` or `cloudflare` | +| `HPO_TUNNEL_URL` | *(none)* | Static tunnel URL when using `cloudflare` provider | +| `HPO_ALLOWED_ORIGINS` | *(none)* | Additional CORS origins for the dashboard | + --- + + ## Core Features + + ### Optuna Engine -- **TPE Sampler**: Tree-structured Parzen Estimator — probability-based hyperparameter suggestions that beat grid and random search +- **Tree-structured Parzen Estimator Sampler**: Probability based hyperparameter suggestions that beat grid and random search - **Median Pruning**: Cuts underperforming trials early to save GPU time - **Single or Dual-Objective**: Optimize one target, or map a Pareto front between a maximize and a minimize metric (e.g., accuracy vs. loss) - **fANOVA Importances**: Identifies which hyperparameters actually matter + + ### Study Health Monitoring The dashboard and `.hpo_status.json` show a health tier: -| Tier | Meaning | -|------|---------| -| `healthy` | Trials are completing, metrics are improving | -| `watch` | Stagnation or early warning signs | + +| Tier | Meaning | +| ----------- | ------------------------------------------------------- | +| `healthy` | Trials are completing, metrics are improving | +| `watch` | Stagnation or early warning signs | | `intervene` | High OOM rate, prolonged stagnation, or 100% prune rate | -Health checks detect stagnation (best score flat-lining) and hardware failure patterns (CUDA OOM on specific batch sizes). + +Health checks detect stagnation (flatlining score, loss) and hardware failure patterns (CUDA OOM on specific batch sizes). ### Persistent SQLite State All configuration, trials, reviews, and metadata live in `hpo_studies.db`: + - Active search space and HPO config - Trial results with VRAM telemetry -- Coordinator review history +- Review history - Generated model cards + + ### MCP Server An MCP server (`hpo_mcp_server.py`) exposes structured study data through Model Context Protocol tools so your IDE agent can read study state, validate manifests, and register new studies. --- + + ## Agent Integration Pathfinder exposes MCP tools that let your IDE agent (Cursor, Claude Code, Antigravity) participate in two workflows: @@ -154,6 +174,8 @@ Pathfinder exposes MCP tools that let your IDE agent (Cursor, Claude Code, Antig 4. Agent calls `init_from_manifest` to register the study in Optuna and SQLite 5. Agent writes a minimal worker script from `templates/worker_minimal.py` + + ### Inspection Flow 1. Agent calls `get_study_data` to retrieve trial telemetry, health tier, fANOVA importances, and best trials @@ -168,9 +190,13 @@ See [AGENTS.md](AGENTS.md) for the full agent procedure. --- + + ## Onboarding Your Own Project -### 1. Write a manifest (`train.hpo.yaml`) + + +### 1. Write a manifest (`train.hpo.yaml`): Manually or have an agent do it for you ```yaml study_name: my_study @@ -204,6 +230,8 @@ python hpo_cli.py validate train.hpo.yaml python hpo_cli.py init train.hpo.yaml ``` + + ### 3. Update your training script ```python @@ -236,8 +264,12 @@ python train.py --- + + ## IDE Setup (Agent-Driven Onboarding & Inspection) + + ### Cursor **Settings → Features → MCP → + Add New MCP Server** @@ -260,10 +292,12 @@ Name: `pathfinder`, Type: `command`, Command: `source .venv/bin/activate && pyth } ``` -Pathfinder is compliant with the Model Context Protocol standard — it works with any MCP-compatible IDE. +Pathfinder is compliant with the Model Context Protocol standard, it works with any MCP-compatible IDE. --- + + ## Common Commands ```bash @@ -296,29 +330,39 @@ pytest tests/ -q --- + + ## Limitations -This is not a production HPO framework. It runs on a single machine with SQLite. It does not support distributed studies, Postgres backends, or advanced samplers like MOTPE or CMA-ES. Use Optuna's native dashboard or W&B Sweeps for production workloads. Pathfinder is a demonstration of MCP/agent integration for ML experiment workflows. +Pathfinder runs on a single machine with SQLite. It does not support Postgres backends or advanced samplers like MOTPE or CMA-ES. This is a demonstration of MCP/agent integration for ML experiment workflows. ## What I Learned -Building Pathfinder taught me the MCP architecture: how to expose structured tool surfaces so an IDE agent can participate in a tuning loop without blocking the hot path. I learned the lease/reap concurrency pattern for worker lifecycle management — detecting dead workers and reclaiming their trials without false positives. I also gained respect for SQLite as an application database; with WAL mode and careful connection pooling, it handled concurrent broker + dashboard + MCP reads without ever becoming the bottleneck. +- MCP tool design to inspect telemetry and modify training scripts, refactored the architecture to decouple agentic workflows from deterministic optimization path +- Implementing concurrency patterns for distributed workers, real-time detection of crashed processes +- Optimizing SQLite backend performance using Write-Ahead Logging; allowing concurrent broker writes, dashboard rendering, and MCP queries without read-write blocks --- + + ## Reference: crack-seg -Pathfinder was initially built to tune [crack-seg](https://github.com/Ishaan1402/crack-seg#crack-seg), a U-Net pixel-level segmentation model trained on UAV bridge imagery. The reference implementation (`colab_worker.py`) is preserved in [archive/](archive/). **Do not fork it** for new studies — use `templates/worker_minimal.py` instead. +Pathfinder was initially built to tune [crack-seg](https://github.com/Ishaan1402/crack-seg#crack-seg), a U-Net pixel-level segmentation model trained on UAV bridge imagery. --- + + ## Docs -- **[AGENTS.md](AGENTS.md)** — Guide for AI agents (Cursor, Claude Code, Antigravity) -- **[docs/INTEGRATION.md](docs/INTEGRATION.md)** — Worker integration contract details +- **[AGENTS.md](AGENTS.md)** — Guide for AI agents (Cursor, Claude Code, Antigravity, etc) +- **[docs/INTEGRATION.md](docs/INTEGRATION.md)** — Worker integration details --- + + ## License -MIT License - see the [LICENSE](LICENSE) file for details. +MIT License - see the [LICENSE](LICENSE) file for details. \ No newline at end of file diff --git a/archive/README.md b/archive/README.md deleted file mode 100644 index 1540e3f..0000000 --- a/archive/README.md +++ /dev/null @@ -1,11 +0,0 @@ -# Archive - -This directory contains code that was part of Pathfinder's development history but is not part of the current API surface. - -## colab_worker.py - -The original bridge-crack U-Net reference implementation. This was the study Pathfinder was initially built to tune — a pixel-level crack segmentation model trained on high-res UAV bridge imagery. - -It still works as a standalone Colab worker, but uses domain-specific naming (dice/bce) that predates Pathfinder's generic score/loss metric abstraction. It is kept here for historical context and as evidence of the project's origins. - -**Do not use this as a template for new studies.** Use `templates/worker_minimal.py` instead. diff --git a/archive/bridge_crack_500px_plots.png b/archive/bridge_crack_500px_plots.png deleted file mode 100644 index 3c3e667..0000000 Binary files a/archive/bridge_crack_500px_plots.png and /dev/null differ diff --git a/archive/bridge_crack_study_500px.csv b/archive/bridge_crack_study_500px.csv deleted file mode 100644 index 00cf235..0000000 --- a/archive/bridge_crack_study_500px.csv +++ /dev/null @@ -1,8 +0,0 @@ -trial_number,trial_id,state,score,loss,duration_seconds,epoch_reached,learning_rate,batch_size,resolution,encoder_name,loss_weight_ratio,model_capacity,gpu_model,max_vram_gb -0,10,COMPLETE,0.858018802569735,3.420776633773675,1666.775988,15,1.3000740917849293e-05,8,512,efficientnet-b0,0.4569677019352898,N/A,, -1,11,COMPLETE,0.8570024923984412,3.167852641157963,2674.331473,15,0.009016956432821428,4,1024,resnet34,0.8759492571891342,N/A,, -25,35,COMPLETE,0.857793011512137,3.20810041447732,339.773174,25,0.0016227567399957443,64,512,,0.9312941039111402,wide,, -26,36,COMPLETE,0.8579996422931605,3.450359302231028,140.290804,25,1.36045241634878e-05,16,512,,0.7556562618734615,narrow,, -27,93,COMPLETE,0.858148133915902,3.164196773923399,142.660719,25,0.009599869896398518,16,512,,0.6784632970852156,narrow,, -40,178,COMPLETE,0.8553575962463495,3.1704550543917884,564.41964,15,0.0011162195120225547,8,1024,,0.8482662431928073,narrow,NVIDIA A100-SXM4-40GB,39.4935302734375 -66,204,COMPLETE,0.8557610089292585,3.171118880123026,1338.960238,15,0.0034464570787237354,4,1024,,0.6326231660481053,narrow,NVIDIA L4,22.0343017578125 diff --git a/archive/bridge_crack_study_trials.json b/archive/bridge_crack_study_trials.json deleted file mode 100644 index bcc234a..0000000 --- a/archive/bridge_crack_study_trials.json +++ /dev/null @@ -1,2858 +0,0 @@ -[ - { - "trial_number": 0, - "trial_id": 10, - "state": "COMPLETE", - "params": { - "learning_rate": 1.3000740917849293e-05, - "batch_size": 8, - "resolution": 512, - "encoder_name": "efficientnet-b0", - "loss_weight_ratio": 0.4569677019352898 - }, - "datetime_start": "2026-06-03T13:43:13.054883", - "datetime_complete": "2026-06-03T14:10:59.830871", - "duration_seconds": 1666.775988, - "optuna_loss": 3.420776633773675, - "optuna_score": 0.858018802569735, - "epoch_reached": 15, - "primary_score": 0.858018802569735, - "primary_loss": 3.420776633773675, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 1, - "trial_id": 11, - "state": "COMPLETE", - "params": { - "learning_rate": 0.009016956432821428, - "batch_size": 4, - "resolution": 1024, - "encoder_name": "resnet34", - "loss_weight_ratio": 0.8759492571891342 - }, - "datetime_start": "2026-06-03T14:24:34.312874", - "datetime_complete": "2026-06-03T15:09:08.644347", - "duration_seconds": 2674.331473, - "optuna_loss": 3.167852641157963, - "optuna_score": 0.8570024923984412, - "epoch_reached": 15, - "primary_score": 0.8570024923984412, - "primary_loss": 3.167852641157963, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 2, - "trial_id": 12, - "state": "COMPLETE", - "params": { - "learning_rate": 0.0004925474076170948, - "batch_size": 2, - "resolution": 256, - "encoder_name": "resnet34", - "loss_weight_ratio": 0.31234114076019825 - }, - "datetime_start": "2026-06-03T15:13:22.895474", - "datetime_complete": "2026-06-03T15:16:10.457808", - "duration_seconds": 167.562334, - "optuna_loss": 0.05479208334007218, - "optuna_score": 0.9898519602789447, - "epoch_reached": 15, - "primary_score": 0.9898519602789447, - "primary_loss": 0.05479208334007218, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 3, - "trial_id": 13, - "state": "PRUNED", - "params": { - "learning_rate": 0.0014412064723689244, - "batch_size": 32, - "resolution": 512, - "encoder_name": "resnet50", - "loss_weight_ratio": 0.43849321487231385 - }, - "datetime_start": "2026-06-03T15:58:28.404445", - "datetime_complete": "2026-06-03T16:01:28.153573", - "duration_seconds": 179.749128, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 4, - "trial_id": 14, - "state": "PRUNED", - "params": { - "learning_rate": 0.000592418490813305, - "batch_size": 16, - "resolution": 512, - "encoder_name": "efficientnet-b0", - "loss_weight_ratio": 0.5883136720418263 - }, - "datetime_start": "2026-06-03T16:02:12.182605", - "datetime_complete": "2026-06-03T16:02:29.436115", - "duration_seconds": 17.25351, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 5, - "trial_id": 15, - "state": "PRUNED", - "params": { - "learning_rate": 0.00866044884543897, - "batch_size": 2, - "resolution": 512, - "encoder_name": "resnet34", - "loss_weight_ratio": 0.1362870798746597 - }, - "datetime_start": "2026-06-03T16:10:37.069107", - "datetime_complete": "2026-06-03T16:10:54.118335", - "duration_seconds": 17.049228, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 6, - "trial_id": 16, - "state": "COMPLETE", - "params": { - "learning_rate": 2.6272157099819104e-05, - "batch_size": 32, - "resolution": 256, - "encoder_name": "resnet50", - "loss_weight_ratio": 0.6982048083978067 - }, - "datetime_start": "2026-06-03T16:14:55.109066", - "datetime_complete": "2026-06-03T16:15:48.273632", - "duration_seconds": 53.164566, - "optuna_loss": 0.34094087169643195, - "optuna_score": 0.9900689937983594, - "epoch_reached": 15, - "primary_score": 0.9900689937983594, - "primary_loss": 0.34094087169643195, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 7, - "trial_id": 17, - "state": "PRUNED", - "params": { - "learning_rate": 1.5280813833109962e-05, - "batch_size": 32, - "resolution": 256, - "encoder_name": "efficientnet-b0", - "loss_weight_ratio": 0.5494909479212793 - }, - "datetime_start": "2026-06-03T16:20:02.454960", - "datetime_complete": "2026-06-03T16:20:10.660498", - "duration_seconds": 8.205538, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 8, - "trial_id": 18, - "state": "PRUNED", - "params": { - "learning_rate": 0.0015560408056926069, - "batch_size": 2, - "resolution": 512, - "model_capacity": "wide", - "loss_weight_ratio": 0.6541735359413245 - }, - "datetime_start": "2026-06-03T16:36:07.874740", - "datetime_complete": "2026-06-03T16:36:24.756243", - "duration_seconds": 16.881503, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 9, - "trial_id": 19, - "state": "PRUNED", - "params": { - "learning_rate": 1.3128228293775808e-05, - "batch_size": 8, - "resolution": 1024, - "model_capacity": "wide", - "loss_weight_ratio": 0.8645862853362126 - }, - "datetime_start": "2026-06-03T16:42:10.936967", - "datetime_complete": "2026-06-03T16:43:46.471404", - "duration_seconds": 95.534437, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 10, - "trial_id": 20, - "state": "PRUNED", - "params": { - "learning_rate": 0.004406605185153241, - "batch_size": 2 - }, - "datetime_start": "2026-06-03T16:44:51.217700", - "datetime_complete": "2026-06-03T16:45:22.544899", - "duration_seconds": 31.327199, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 11, - "trial_id": 21, - "state": "PRUNED", - "params": { - "learning_rate": 0.004606786973558291, - "batch_size": 64 - }, - "datetime_start": "2026-06-03T16:45:43.747381", - "datetime_complete": "2026-06-03T16:46:13.669698", - "duration_seconds": 29.922317, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 12, - "trial_id": 22, - "state": "FAIL", - "params": { - "learning_rate": 0.009238447012502312, - "batch_size": 8 - }, - "datetime_start": "2026-06-03T16:48:01.525344", - "datetime_complete": "2026-06-03T16:48:01.552523", - "duration_seconds": 0.027179, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 13, - "trial_id": 23, - "state": "FAIL", - "params": { - "learning_rate": 0.0006548097272413485, - "batch_size": 32 - }, - "datetime_start": "2026-06-03T16:48:21.372164", - "datetime_complete": "2026-06-03T16:48:21.385289", - "duration_seconds": 0.013125, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 14, - "trial_id": 24, - "state": "FAIL", - "params": { - "learning_rate": 2.4349414095370824e-05, - "batch_size": 8, - "model_capacity": "narrow", - "loss_weight_ratio": 0.9316343050565551 - }, - "datetime_start": "2026-06-03T16:49:13.338941", - "datetime_complete": "2026-06-03T16:49:28.821244", - "duration_seconds": 15.482303, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 15, - "trial_id": 25, - "state": "FAIL", - "params": { - "learning_rate": 0.00014020437791801606, - "batch_size": 4, - "model_capacity": "narrow", - "loss_weight_ratio": 0.1742679306216336 - }, - "datetime_start": "2026-06-03T16:49:28.830983", - "datetime_complete": "2026-06-03T16:49:36.991214", - "duration_seconds": 8.160231, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 16, - "trial_id": 26, - "state": "FAIL", - "params": { - "learning_rate": 0.0016347032047742205, - "batch_size": 4, - "model_capacity": "narrow", - "loss_weight_ratio": 0.03717144506513459 - }, - "datetime_start": "2026-06-03T16:49:37.001315", - "datetime_complete": "2026-06-03T16:51:21.026869", - "duration_seconds": 104.025554, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 17, - "trial_id": 27, - "state": "FAIL", - "params": { - "learning_rate": 0.001424331563772705, - "batch_size": 8, - "model_capacity": "narrow", - "loss_weight_ratio": 0.38679370235988164 - }, - "datetime_start": "2026-06-03T16:51:21.047576", - "datetime_complete": "2026-06-03T16:52:06.743811", - "duration_seconds": 45.696235, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 18, - "trial_id": 28, - "state": "PRUNED", - "params": { - "learning_rate": 0.0006710326147742782, - "batch_size": 16, - "model_capacity": "narrow", - "loss_weight_ratio": 0.002323245785979422 - }, - "datetime_start": "2026-06-03T16:52:06.749228", - "datetime_complete": "2026-06-03T16:52:06.767498", - "duration_seconds": 0.01827, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 19, - "trial_id": 29, - "state": "FAIL", - "params": {}, - "datetime_start": "2026-06-03T16:52:13.957698", - "datetime_complete": "2026-06-03T16:52:28.811225", - "duration_seconds": 14.853527, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 20, - "trial_id": 30, - "state": "PRUNED", - "params": { - "learning_rate": 0.005569662100056995, - "batch_size": 2, - "model_capacity": "wide", - "loss_weight_ratio": 0.9243719613528233 - }, - "datetime_start": "2026-06-03T16:52:28.818704", - "datetime_complete": "2026-06-03T16:52:28.836298", - "duration_seconds": 0.017594, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 21, - "trial_id": 31, - "state": "PRUNED", - "params": { - "learning_rate": 8.635129594504407e-05, - "batch_size": 8, - "model_capacity": "wide", - "loss_weight_ratio": 0.2159129532506059 - }, - "datetime_start": "2026-06-03T16:53:10.479054", - "datetime_complete": "2026-06-03T16:53:27.942296", - "duration_seconds": 17.463242, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 22, - "trial_id": 32, - "state": "PRUNED", - "params": { - "learning_rate": 1.2671531921681287e-05, - "batch_size": 32, - "model_capacity": "wide", - "loss_weight_ratio": 0.8084256426884586 - }, - "datetime_start": "2026-06-03T16:56:05.615491", - "datetime_complete": "2026-06-03T16:56:34.427798", - "duration_seconds": 28.812307, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 23, - "trial_id": 33, - "state": "PRUNED", - "params": { - "learning_rate": 2.5298748345870258e-05, - "batch_size": 32, - "model_capacity": "narrow", - "loss_weight_ratio": 0.8252409941205194 - }, - "datetime_start": "2026-06-03T17:00:35.578893", - "datetime_complete": "2026-06-03T17:01:04.174560", - "duration_seconds": 28.595667, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 24, - "trial_id": 34, - "state": "PRUNED", - "params": { - "learning_rate": 2.223369003885487e-05, - "batch_size": 2, - "model_capacity": "wide", - "loss_weight_ratio": 0.5713714827302104 - }, - "datetime_start": "2026-06-03T17:02:56.369030", - "datetime_complete": "2026-06-03T17:03:24.500350", - "duration_seconds": 28.13132, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 25, - "trial_id": 35, - "state": "COMPLETE", - "params": { - "learning_rate": 0.0016227567399957443, - "batch_size": 64, - "model_capacity": "wide", - "loss_weight_ratio": 0.9312941039111402, - "resolution": 512 - }, - "datetime_start": "2026-06-03T17:15:55.022375", - "datetime_complete": "2026-06-03T17:21:34.795549", - "duration_seconds": 339.773174, - "optuna_loss": 3.20810041447732, - "optuna_score": 0.857793011512137, - "epoch_reached": 25, - "primary_score": 0.857793011512137, - "primary_loss": 3.20810041447732, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 26, - "trial_id": 36, - "state": "COMPLETE", - "params": { - "learning_rate": 1.36045241634878e-05, - "batch_size": 16, - "model_capacity": "narrow", - "loss_weight_ratio": 0.7556562618734615, - "resolution": 512 - }, - "datetime_start": "2026-06-03T17:22:26.684040", - "datetime_complete": "2026-06-03T17:24:46.974844", - "duration_seconds": 140.290804, - "optuna_loss": 3.450359302231028, - "optuna_score": 0.8579996422931605, - "epoch_reached": 25, - "primary_score": 0.8579996422931605, - "primary_loss": 3.450359302231028, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 27, - "trial_id": 93, - "state": "COMPLETE", - "params": { - "learning_rate": 0.009599869896398518, - "batch_size": 16, - "model_capacity": "narrow", - "loss_weight_ratio": 0.6784632970852156, - "resolution": 512 - }, - "datetime_start": "2026-06-03T19:53:51.414244", - "datetime_complete": "2026-06-03T19:56:14.074963", - "duration_seconds": 142.660719, - "optuna_loss": 3.164196773923399, - "optuna_score": 0.858148133915902, - "epoch_reached": 25, - "primary_score": 0.858148133915902, - "primary_loss": 3.164196773923399, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 28, - "trial_id": 130, - "state": "PRUNED", - "params": { - "learning_rate": 3.126406973760604e-05, - "loss_weight_ratio": 0.11912673266270057, - "batch_size": 8, - "resolution": 512, - "model_capacity": "wide" - }, - "datetime_start": "2026-06-03T20:41:25.704340", - "datetime_complete": "2026-06-03T20:41:59.084033", - "duration_seconds": 33.379693, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 29, - "trial_id": 167, - "state": "FAIL", - "params": { - "learning_rate": 0.0003129825951856815, - "batch_size": 64, - "resolution": 1024, - "model_capacity": "wide", - "loss_weight_ratio": 0.6505910029778382 - }, - "datetime_start": "2026-06-05T13:58:19.470631", - "datetime_complete": "2026-06-05T13:59:43.751567", - "duration_seconds": 84.280936, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 30, - "trial_id": 168, - "state": "COMPLETE", - "params": { - "learning_rate": 1.2908087536051017e-05, - "batch_size": 32, - "resolution": 256, - "model_capacity": "narrow", - "loss_weight_ratio": 0.0336207768294422 - }, - "datetime_start": "2026-06-07T13:59:34.848713", - "datetime_complete": "2026-06-07T14:01:51.748743", - "duration_seconds": 136.90003, - "optuna_loss": 0.6156984318660784, - "optuna_score": 0.8673244251587711, - "epoch_reached": 15, - "primary_score": 0.8673244251587711, - "primary_loss": 0.6156984318660784, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA A100-SXM4-40GB", - "max_vram_gb": 39.4935302734375, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 31, - "trial_id": 169, - "state": "FAIL", - "params": { - "learning_rate": 0.005529602980992804, - "batch_size": 16, - "resolution": 1024, - "model_capacity": "narrow", - "loss_weight_ratio": 0.5764868199904598 - }, - "datetime_start": "2026-06-07T14:01:52.028617", - "datetime_complete": "2026-06-07T14:01:54.489417", - "duration_seconds": 2.4608, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 32, - "trial_id": 170, - "state": "PRUNED", - "params": { - "learning_rate": 5.619733318203318e-05, - "batch_size": 32, - "resolution": 256, - "model_capacity": "wide", - "loss_weight_ratio": 0.2684312186462182 - }, - "datetime_start": "2026-06-07T14:01:54.754181", - "datetime_complete": "2026-06-07T14:03:10.632234", - "duration_seconds": 75.878053, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 33, - "trial_id": 171, - "state": "PRUNED", - "params": { - "learning_rate": 0.0023088874045366766, - "batch_size": 2, - "resolution": 256, - "model_capacity": "narrow", - "loss_weight_ratio": 0.6236688985236228 - }, - "datetime_start": "2026-06-07T14:03:10.901884", - "datetime_complete": "2026-06-07T14:04:00.814544", - "duration_seconds": 49.91266, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 34, - "trial_id": 172, - "state": "FAIL", - "params": { - "learning_rate": 0.00965489857451385, - "batch_size": 8, - "resolution": 1024, - "model_capacity": "wide", - "loss_weight_ratio": 0.5779408885200207 - }, - "datetime_start": "2026-06-07T14:04:01.090942", - "datetime_complete": "2026-06-07T14:23:47.673944", - "duration_seconds": 1186.583002, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA A100-SXM4-40GB", - "max_vram_gb": 39.4935302734375, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 35, - "trial_id": 173, - "state": "FAIL", - "params": { - "learning_rate": 0.008667747925003964, - "batch_size": 8, - "resolution": 512, - "model_capacity": "wide", - "loss_weight_ratio": 0.0854227054576977 - }, - "datetime_start": "2026-06-07T14:23:48.011546", - "datetime_complete": "2026-06-07T14:26:55.783299", - "duration_seconds": 187.771753, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 36, - "trial_id": 174, - "state": "PRUNED", - "params": { - "learning_rate": 0.003964260072586964, - "batch_size": 32, - "resolution": 512, - "model_capacity": "narrow", - "loss_weight_ratio": 0.35151491618316477 - }, - "datetime_start": "2026-06-07T14:28:32.046807", - "datetime_complete": "2026-06-07T14:29:22.126494", - "duration_seconds": 50.079687, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.8560799390346749, - "primary_loss": 3.2290616176299407, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA A100-SXM4-40GB", - "max_vram_gb": 39.4935302734375, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 37, - "trial_id": 175, - "state": "PRUNED", - "params": { - "learning_rate": 6.190175730149249e-05, - "batch_size": 8, - "resolution": 256, - "model_capacity": "narrow", - "loss_weight_ratio": 0.8010454023210211 - }, - "datetime_start": "2026-06-07T14:29:23.171195", - "datetime_complete": "2026-06-07T14:29:50.114575", - "duration_seconds": 26.94338, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.9895626247940745, - "primary_loss": 0.4081219550426499, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA A100-SXM4-40GB", - "max_vram_gb": 39.4935302734375, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 38, - "trial_id": 176, - "state": "FAIL", - "params": { - "learning_rate": 6.88390653876011e-05, - "batch_size": 64, - "resolution": 1024, - "model_capacity": "narrow", - "loss_weight_ratio": 0.7564837061499687 - }, - "datetime_start": "2026-06-07T14:29:50.971427", - "datetime_complete": "2026-06-07T14:29:53.344969", - "duration_seconds": 2.373542, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA A100-SXM4-40GB", - "max_vram_gb": 39.4935302734375, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 39, - "trial_id": 177, - "state": "PRUNED", - "params": { - "learning_rate": 3.193828708214736e-05, - "batch_size": 8, - "resolution": 512, - "model_capacity": "wide", - "loss_weight_ratio": 0.8196726856529919 - }, - "datetime_start": "2026-06-07T14:29:53.663673", - "datetime_complete": "2026-06-07T14:31:27.262921", - "duration_seconds": 93.599248, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.855850139872812, - "primary_loss": 3.454779870399443, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA A100-SXM4-40GB", - "max_vram_gb": 39.4935302734375, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 40, - "trial_id": 178, - "state": "COMPLETE", - "params": { - "learning_rate": 0.0011162195120225547, - "batch_size": 8, - "resolution": 1024, - "model_capacity": "narrow", - "loss_weight_ratio": 0.8482662431928073 - }, - "datetime_start": "2026-06-07T14:31:28.087930", - "datetime_complete": "2026-06-07T14:40:52.507570", - "duration_seconds": 564.41964, - "optuna_loss": 3.1704550543917884, - "optuna_score": 0.8553575962463495, - "epoch_reached": 15, - "primary_score": 0.8553575962463495, - "primary_loss": 3.1704550543917884, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA A100-SXM4-40GB", - "max_vram_gb": 39.4935302734375, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 41, - "trial_id": 179, - "state": "PRUNED", - "params": { - "learning_rate": 8.376984597427127e-05, - "batch_size": 32, - "resolution": 256, - "model_capacity": "narrow", - "loss_weight_ratio": 0.08460947756974468 - }, - "datetime_start": "2026-06-07T14:40:52.830605", - "datetime_complete": "2026-06-07T14:41:18.493565", - "duration_seconds": 25.66296, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.97001750168377, - "primary_loss": 0.4865190936291771, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA A100-SXM4-40GB", - "max_vram_gb": 39.4935302734375, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 42, - "trial_id": 180, - "state": "PRUNED", - "params": { - "learning_rate": 0.006814745505395586, - "batch_size": 4, - "resolution": 256, - "model_capacity": "narrow", - "loss_weight_ratio": 0.08170498386801406 - }, - "datetime_start": "2026-06-07T14:41:19.613878", - "datetime_complete": "2026-06-07T14:41:47.336577", - "duration_seconds": 27.722699, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.984673943843873, - "primary_loss": 0.11464002246689445, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA A100-SXM4-40GB", - "max_vram_gb": 39.4935302734375, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 43, - "trial_id": 181, - "state": "FAIL", - "params": { - "learning_rate": 6.435284774708397e-05, - "batch_size": 32, - "resolution": 512, - "model_capacity": "wide", - "loss_weight_ratio": 0.5264320365100166 - }, - "datetime_start": "2026-06-07T14:41:48.511069", - "datetime_complete": "2026-06-07T14:41:50.863072", - "duration_seconds": 2.352003, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA A100-SXM4-40GB", - "max_vram_gb": 39.4935302734375, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 44, - "trial_id": 182, - "state": "PRUNED", - "params": { - "learning_rate": 1.1932677758682133e-05, - "batch_size": 16, - "resolution": 512, - "model_capacity": "wide", - "loss_weight_ratio": 0.7292200426298769 - }, - "datetime_start": "2026-06-07T14:41:51.173891", - "datetime_complete": "2026-06-07T14:43:22.767690", - "duration_seconds": 91.593799, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.7932973447561847, - "primary_loss": 3.5894378851234663, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA A100-SXM4-40GB", - "max_vram_gb": 39.4935302734375, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 45, - "trial_id": 183, - "state": "FAIL", - "params": { - "learning_rate": 0.0003957933664968164, - "batch_size": 32, - "resolution": 512, - "model_capacity": "wide", - "loss_weight_ratio": 0.3679429819753126 - }, - "datetime_start": "2026-06-07T14:43:23.404815", - "datetime_complete": "2026-06-07T14:43:25.751226", - "duration_seconds": 2.346411, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA A100-SXM4-40GB", - "max_vram_gb": 39.4935302734375, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 46, - "trial_id": 184, - "state": "PRUNED", - "params": { - "learning_rate": 0.00011093794054144854, - "batch_size": 32, - "resolution": 512, - "model_capacity": "narrow", - "loss_weight_ratio": 0.8695930976572441 - }, - "datetime_start": "2026-06-07T14:47:45.801056", - "datetime_complete": "2026-06-07T14:48:35.169769", - "duration_seconds": 49.368713, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.847776949003498, - "primary_loss": 3.545791560587501, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA A100-SXM4-40GB", - "max_vram_gb": 39.4935302734375, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 47, - "trial_id": 185, - "state": "FAIL", - "params": { - "learning_rate": 0.0016064112990469893, - "batch_size": 64, - "resolution": 1024, - "model_capacity": "narrow", - "loss_weight_ratio": 0.912719707686549 - }, - "datetime_start": "2026-06-07T14:48:35.876373", - "datetime_complete": "2026-06-07T14:48:38.240824", - "duration_seconds": 2.364451, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA A100-SXM4-40GB", - "max_vram_gb": 39.4935302734375, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 48, - "trial_id": 186, - "state": "PRUNED", - "params": { - "learning_rate": 1.866719221020291e-05, - "batch_size": 4, - "resolution": 512, - "model_capacity": "narrow", - "loss_weight_ratio": 0.4256559084249397 - }, - "datetime_start": "2026-06-07T14:48:38.561165", - "datetime_complete": "2026-06-07T14:49:30.149907", - "duration_seconds": 51.588742, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.8444927109183233, - "primary_loss": 3.504164194758934, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA A100-SXM4-40GB", - "max_vram_gb": 39.4935302734375, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 49, - "trial_id": 187, - "state": "FAIL", - "params": { - "learning_rate": 7.550066161936539e-05, - "batch_size": 32, - "resolution": 1024, - "model_capacity": "wide", - "loss_weight_ratio": 0.022409146295909954 - }, - "datetime_start": "2026-06-07T14:49:31.635068", - "datetime_complete": "2026-06-07T14:49:34.006746", - "duration_seconds": 2.371678, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA A100-SXM4-40GB", - "max_vram_gb": 39.4935302734375, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 50, - "trial_id": 188, - "state": "FAIL", - "params": { - "learning_rate": 0.0053350126131481235, - "batch_size": 4, - "resolution": 1024, - "model_capacity": "wide", - "loss_weight_ratio": 0.5049364450109906 - }, - "datetime_start": "2026-06-07T14:49:34.285916", - "datetime_complete": "2026-06-07T15:12:57.731695", - "duration_seconds": 1403.445779, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 51, - "trial_id": 189, - "state": "FAIL", - "params": { - "learning_rate": 0.000777600200657905, - "batch_size": 64, - "resolution": 512, - "model_capacity": "narrow", - "loss_weight_ratio": 0.5033823758278679 - }, - "datetime_start": "2026-06-09T12:27:32.048350", - "datetime_complete": "2026-06-09T12:27:42.606962", - "duration_seconds": 10.558612, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 52, - "trial_id": 190, - "state": "FAIL", - "params": { - "learning_rate": 0.0003510810257947738, - "batch_size": 16, - "resolution": 1024, - "model_capacity": "narrow", - "loss_weight_ratio": 0.3055155227639622 - }, - "datetime_start": "2026-06-09T12:27:43.506462", - "datetime_complete": "2026-06-09T12:27:46.401631", - "duration_seconds": 2.895169, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 53, - "trial_id": 191, - "state": "FAIL", - "params": { - "learning_rate": 0.00029149980386619276, - "batch_size": 8, - "resolution": 1024, - "model_capacity": "narrow", - "loss_weight_ratio": 0.9024384903900268 - }, - "datetime_start": "2026-06-09T12:27:47.394528", - "datetime_complete": "2026-06-09T12:27:52.254843", - "duration_seconds": 4.860315, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 54, - "trial_id": 192, - "state": "FAIL", - "params": { - "learning_rate": 0.006280369628664032, - "batch_size": 32, - "resolution": 1024, - "model_capacity": "narrow", - "loss_weight_ratio": 0.7468149969124053 - }, - "datetime_start": "2026-06-09T12:27:53.012884", - "datetime_complete": "2026-06-09T12:27:56.525848", - "duration_seconds": 3.512964, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 55, - "trial_id": 193, - "state": "FAIL", - "params": { - "learning_rate": 2.1608825057250263e-05, - "batch_size": 16, - "resolution": 512, - "model_capacity": "wide", - "loss_weight_ratio": 0.8276164452067589 - }, - "datetime_start": "2026-06-09T12:27:57.553135", - "datetime_complete": "2026-06-09T12:28:01.985592", - "duration_seconds": 4.432457, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 56, - "trial_id": 194, - "state": "PRUNED", - "params": { - "learning_rate": 0.006946225010911096, - "batch_size": 2, - "resolution": 512, - "model_capacity": "wide", - "loss_weight_ratio": 0.34343077115154985 - }, - "datetime_start": "2026-06-09T12:28:03.278519", - "datetime_complete": "2026-06-09T12:31:35.254234", - "duration_seconds": 211.975715, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.8567204705534222, - "primary_loss": 3.1828140226597523, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 57, - "trial_id": 195, - "state": "FAIL", - "params": { - "learning_rate": 0.0005722650461041644, - "batch_size": 32, - "resolution": 1024, - "model_capacity": "narrow", - "loss_weight_ratio": 0.5121602362719397 - }, - "datetime_start": "2026-06-09T12:31:37.404846", - "datetime_complete": "2026-06-09T12:31:41.226259", - "duration_seconds": 3.821413, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 58, - "trial_id": 196, - "state": "PRUNED", - "params": { - "learning_rate": 0.000421080095182812, - "batch_size": 2, - "resolution": 256, - "model_capacity": "wide", - "loss_weight_ratio": 0.3569257328985008 - }, - "datetime_start": "2026-06-09T12:31:41.992999", - "datetime_complete": "2026-06-09T12:33:28.062304", - "duration_seconds": 106.069305, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.9897473019936772, - "primary_loss": 0.05817848326096052, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 59, - "trial_id": 197, - "state": "FAIL", - "params": { - "learning_rate": 0.0006999779382349623, - "batch_size": 64, - "resolution": 1024, - "model_capacity": "wide", - "loss_weight_ratio": 0.32626756206545826 - }, - "datetime_start": "2026-06-09T12:33:29.930419", - "datetime_complete": "2026-06-09T12:33:32.766389", - "duration_seconds": 2.83597, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 60, - "trial_id": 198, - "state": "FAIL", - "params": { - "learning_rate": 1.660719226277921e-05, - "batch_size": 16, - "resolution": 512, - "model_capacity": "wide", - "loss_weight_ratio": 0.7440646190555177 - }, - "datetime_start": "2026-06-09T12:33:33.487153", - "datetime_complete": "2026-06-09T12:33:37.287858", - "duration_seconds": 3.800705, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 61, - "trial_id": 199, - "state": "FAIL", - "params": { - "learning_rate": 1.6136763657195547e-05, - "batch_size": 16, - "resolution": 1024, - "model_capacity": "narrow", - "loss_weight_ratio": 0.7194931228626711 - }, - "datetime_start": "2026-06-09T12:33:38.041676", - "datetime_complete": "2026-06-09T12:33:40.859672", - "duration_seconds": 2.817996, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 62, - "trial_id": 200, - "state": "FAIL", - "params": { - "learning_rate": 0.00644351011993277, - "batch_size": 64, - "resolution": 512, - "model_capacity": "narrow", - "loss_weight_ratio": 0.9831121443518326 - }, - "datetime_start": "2026-06-09T12:33:41.664483", - "datetime_complete": "2026-06-09T12:33:45.502201", - "duration_seconds": 3.837718, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 63, - "trial_id": 201, - "state": "FAIL", - "params": { - "learning_rate": 8.015086163668646e-05, - "batch_size": 16, - "resolution": 512, - "model_capacity": "narrow", - "loss_weight_ratio": 0.6247208153386133 - }, - "datetime_start": "2026-06-09T12:39:52.680426", - "datetime_complete": "2026-06-09T12:42:09.573070", - "duration_seconds": 136.892644, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 64, - "trial_id": 202, - "state": "FAIL", - "params": { - "learning_rate": 0.00010798109091729242, - "batch_size": 4, - "resolution": 1024, - "model_capacity": "narrow", - "loss_weight_ratio": 0.020768120149031843 - }, - "datetime_start": "2026-06-09T12:42:09.579066", - "datetime_complete": "2026-06-09T12:44:14.995999", - "duration_seconds": 125.416933, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 65, - "trial_id": 203, - "state": "PRUNED", - "params": { - "learning_rate": 1.0514744422959588e-05, - "batch_size": 2, - "resolution": 512, - "model_capacity": "narrow", - "loss_weight_ratio": 0.11626061471968552 - }, - "datetime_start": "2026-06-09T12:58:08.785871", - "datetime_complete": "2026-06-09T12:59:45.650142", - "duration_seconds": 96.864271, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.8570720076452774, - "primary_loss": 3.444088376524076, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 66, - "trial_id": 204, - "state": "COMPLETE", - "params": { - "learning_rate": 0.0034464570787237354, - "batch_size": 4, - "resolution": 1024, - "model_capacity": "narrow", - "loss_weight_ratio": 0.6326231660481053 - }, - "datetime_start": "2026-06-09T12:59:48.671857", - "datetime_complete": "2026-06-09T13:22:07.632095", - "duration_seconds": 1338.960238, - "optuna_loss": 3.171118880123026, - "optuna_score": 0.8557610089292585, - "epoch_reached": 15, - "primary_score": 0.8557610089292585, - "primary_loss": 3.171118880123026, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 67, - "trial_id": 205, - "state": "PRUNED", - "params": { - "learning_rate": 0.000655602631496065, - "batch_size": 2, - "resolution": 512, - "model_capacity": "narrow", - "loss_weight_ratio": 0.6165970770863961 - }, - "datetime_start": "2026-06-09T13:22:09.164919", - "datetime_complete": "2026-06-09T13:23:44.873486", - "duration_seconds": 95.708567, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.8502720038631986, - "primary_loss": 3.1791283390190026, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 68, - "trial_id": 206, - "state": "PRUNED", - "params": { - "learning_rate": 0.004488241212661547, - "batch_size": 2, - "resolution": 1024, - "model_capacity": "narrow", - "loss_weight_ratio": 0.6947912315068989 - }, - "datetime_start": "2026-06-09T13:23:47.124323", - "datetime_complete": "2026-06-09T13:30:50.154021", - "duration_seconds": 423.029698, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.856514060787977, - "primary_loss": 3.1795108579885105, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 69, - "trial_id": 207, - "state": "PRUNED", - "params": { - "learning_rate": 0.001526069664046176, - "batch_size": 4, - "resolution": 512, - "model_capacity": "wide", - "loss_weight_ratio": 0.33976780125584527 - }, - "datetime_start": "2026-06-09T13:30:57.204234", - "datetime_complete": "2026-06-09T13:34:34.567629", - "duration_seconds": 217.363395, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.8513724619433332, - "primary_loss": 3.180013144569558, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 70, - "trial_id": 208, - "state": "FAIL", - "params": { - "learning_rate": 8.036481706507076e-05, - "batch_size": 8, - "resolution": 1024, - "model_capacity": "wide", - "loss_weight_ratio": 0.3172719209651951 - }, - "datetime_start": "2026-06-09T13:34:37.873728", - "datetime_complete": "2026-06-09T13:34:42.152489", - "duration_seconds": 4.278761, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 71, - "trial_id": 209, - "state": "PRUNED", - "params": { - "learning_rate": 2.0588378463008213e-05, - "batch_size": 2, - "resolution": 256, - "model_capacity": "narrow", - "loss_weight_ratio": 0.5418362964684905 - }, - "datetime_start": "2026-06-09T13:34:43.516880", - "datetime_complete": "2026-06-09T13:35:45.401603", - "duration_seconds": 61.884723, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.9735566368410664, - "primary_loss": 0.41913800601717793, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 72, - "trial_id": 210, - "state": "PRUNED", - "params": { - "learning_rate": 2.5824788703582884e-05, - "batch_size": 32, - "resolution": 256, - "model_capacity": "wide", - "loss_weight_ratio": 0.1001732434526551 - }, - "datetime_start": "2026-06-09T13:35:48.655548", - "datetime_complete": "2026-06-09T13:37:33.419525", - "duration_seconds": 104.763977, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.9432637325874148, - "primary_loss": 0.5283325079372664, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 73, - "trial_id": 211, - "state": "FAIL", - "params": { - "learning_rate": 0.0012503262438584753, - "batch_size": 64, - "resolution": 256, - "model_capacity": "wide", - "loss_weight_ratio": 0.36453885264546326 - }, - "datetime_start": "2026-06-09T13:37:36.288939", - "datetime_complete": "2026-06-09T13:37:42.360303", - "duration_seconds": 6.071364, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 74, - "trial_id": 212, - "state": "FAIL", - "params": { - "learning_rate": 0.00015582382174458584, - "batch_size": 8, - "resolution": 1024, - "model_capacity": "wide", - "loss_weight_ratio": 0.5578577099740122 - }, - "datetime_start": "2026-06-09T13:37:43.645381", - "datetime_complete": "2026-06-09T13:37:47.480475", - "duration_seconds": 3.835094, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 75, - "trial_id": 213, - "state": "FAIL", - "params": { - "learning_rate": 0.00044334124591300603, - "batch_size": 32, - "resolution": 512, - "model_capacity": "wide", - "loss_weight_ratio": 0.5292311853371128 - }, - "datetime_start": "2026-06-09T13:37:49.004989", - "datetime_complete": "2026-06-09T13:37:52.819199", - "duration_seconds": 3.81421, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 76, - "trial_id": 214, - "state": "FAIL", - "params": { - "learning_rate": 1.223245529632777e-05, - "batch_size": 4, - "resolution": 1024, - "model_capacity": "wide", - "loss_weight_ratio": 0.3226911430366707 - }, - "datetime_start": "2026-06-09T13:37:53.719425", - "datetime_complete": "2026-06-09T13:37:57.511614", - "duration_seconds": 3.792189, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 77, - "trial_id": 215, - "state": "FAIL", - "params": { - "learning_rate": 0.004344956027911343, - "batch_size": 16, - "resolution": 256, - "model_capacity": "narrow", - "loss_weight_ratio": 0.1358767915932685 - }, - "datetime_start": "2026-06-11T13:55:37.316306", - "datetime_complete": "2026-06-11T13:56:00.102813", - "duration_seconds": 22.786507, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": false, - "failure_tag": "DIVERGED", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 78, - "trial_id": 216, - "state": "FAIL", - "params": { - "learning_rate": 3.2979458231192225e-05, - "batch_size": 64, - "resolution": 1024, - "model_capacity": "wide", - "loss_weight_ratio": 0.6172976418982675 - }, - "datetime_start": "2026-06-11T13:56:01.980573", - "datetime_complete": "2026-06-11T13:56:04.807001", - "duration_seconds": 2.826428, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 79, - "trial_id": 217, - "state": "FAIL", - "params": { - "learning_rate": 0.0011426478938092145, - "batch_size": 8, - "resolution": 512, - "model_capacity": "narrow", - "loss_weight_ratio": 0.28562228694664504 - }, - "datetime_start": "2026-06-11T13:56:05.919390", - "datetime_complete": "2026-06-11T13:56:29.007012", - "duration_seconds": 23.087622, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": false, - "failure_tag": "DIVERGED", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 80, - "trial_id": 218, - "state": "FAIL", - "params": { - "learning_rate": 0.007871157298782676, - "batch_size": 16, - "resolution": 256, - "model_capacity": "narrow", - "loss_weight_ratio": 0.8290396014049095 - }, - "datetime_start": "2026-06-11T13:56:29.728292", - "datetime_complete": "2026-06-11T13:56:43.485919", - "duration_seconds": 13.757627, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": false, - "failure_tag": "DIVERGED", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 81, - "trial_id": 219, - "state": "FAIL", - "params": { - "learning_rate": 0.0002372784275397014, - "batch_size": 2, - "resolution": 1024, - "model_capacity": "narrow", - "loss_weight_ratio": 0.23184013756127309 - }, - "datetime_start": "2026-06-11T13:56:44.586990", - "datetime_complete": "2026-06-11T14:11:41.460615", - "duration_seconds": 896.873625, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 82, - "trial_id": 220, - "state": "FAIL", - "params": { - "learning_rate": 2.922133857069148e-05, - "batch_size": 8, - "resolution": 1024, - "model_capacity": "narrow", - "loss_weight_ratio": 0.23999846195349261 - }, - "datetime_start": "2026-06-11T13:58:13.608825", - "datetime_complete": "2026-06-11T13:58:17.422688", - "duration_seconds": 3.813863, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 83, - "trial_id": 221, - "state": "FAIL", - "params": { - "learning_rate": 1.0827083945741795e-05, - "batch_size": 32, - "resolution": 256, - "model_capacity": "wide", - "loss_weight_ratio": 0.9421656316488691 - }, - "datetime_start": "2026-06-11T13:58:20.165570", - "datetime_complete": "2026-06-11T13:58:45.518922", - "duration_seconds": 25.353352, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": false, - "failure_tag": "DIVERGED", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 84, - "trial_id": 222, - "state": "FAIL", - "params": { - "learning_rate": 0.0013651455341181428, - "batch_size": 8, - "resolution": 1024, - "model_capacity": "narrow", - "loss_weight_ratio": 0.04946229416405579 - }, - "datetime_start": "2026-06-11T13:58:46.714834", - "datetime_complete": "2026-06-11T13:58:52.615385", - "duration_seconds": 5.900551, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 85, - "trial_id": 223, - "state": "FAIL", - "params": { - "learning_rate": 0.0014649725391093422, - "batch_size": 2, - "resolution": 256, - "model_capacity": "narrow", - "loss_weight_ratio": 0.37873331580397984 - }, - "datetime_start": "2026-06-11T13:58:53.699844", - "datetime_complete": "2026-06-11T13:59:07.562627", - "duration_seconds": 13.862783, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": false, - "failure_tag": "DIVERGED", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 86, - "trial_id": 224, - "state": "FAIL", - "params": { - "learning_rate": 0.003206537784664529, - "batch_size": 8, - "resolution": 256, - "model_capacity": "narrow", - "loss_weight_ratio": 0.3470651170360838 - }, - "datetime_start": "2026-06-11T13:59:10.192695", - "datetime_complete": "2026-06-11T13:59:27.999212", - "duration_seconds": 17.806517, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": false, - "failure_tag": "DIVERGED", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 87, - "trial_id": 225, - "state": "FAIL", - "params": { - "learning_rate": 5.02454787833256e-05, - "batch_size": 64, - "resolution": 1024, - "model_capacity": "narrow", - "loss_weight_ratio": 0.6253427203524324 - }, - "datetime_start": "2026-06-11T13:59:30.460971", - "datetime_complete": "2026-06-11T13:59:35.019583", - "duration_seconds": 4.558612, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 88, - "trial_id": 226, - "state": "FAIL", - "params": { - "learning_rate": 0.001075351250772297, - "batch_size": 32, - "resolution": 512, - "model_capacity": "wide", - "loss_weight_ratio": 0.5672267142760922 - }, - "datetime_start": "2026-06-11T13:59:35.874493", - "datetime_complete": "2026-06-11T13:59:39.953464", - "duration_seconds": 4.078971, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 89, - "trial_id": 227, - "state": "FAIL", - "params": { - "learning_rate": 0.001655021881946475, - "batch_size": 16, - "resolution": 1024, - "model_capacity": "narrow", - "loss_weight_ratio": 0.7220736783251582 - }, - "datetime_start": "2026-06-11T13:59:41.214776", - "datetime_complete": "2026-06-11T13:59:45.710518", - "duration_seconds": 4.495742, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 90, - "trial_id": 228, - "state": "FAIL", - "params": { - "learning_rate": 2.4095794895938903e-05, - "batch_size": 4, - "resolution": 1024, - "model_capacity": "narrow", - "loss_weight_ratio": 0.9722243737456475 - }, - "datetime_start": "2026-06-11T13:59:48.610190", - "datetime_complete": "2026-06-11T14:11:41.465213", - "duration_seconds": 712.855023, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": null, - "primary_score": null, - "primary_loss": null, - "oom_triggered": null, - "failure_tag": null, - "gpu_model": null, - "max_vram_gb": null, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 91, - "trial_id": 229, - "state": "FAIL", - "params": { - "learning_rate": 1.2631887352189931e-05, - "batch_size": 64, - "resolution": 1024, - "model_capacity": "wide", - "loss_weight_ratio": 0.6756570515061726 - }, - "datetime_start": "2026-06-11T14:11:17.852541", - "datetime_complete": "2026-06-11T14:11:19.258056", - "duration_seconds": 1.405515, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 92, - "trial_id": 230, - "state": "PRUNED", - "params": { - "learning_rate": 0.0005469877422305655, - "batch_size": 32, - "resolution": 256, - "model_capacity": "narrow", - "loss_weight_ratio": 0.04014802323930022 - }, - "datetime_start": "2026-06-11T14:11:20.269944", - "datetime_complete": "2026-06-11T14:12:23.208958", - "duration_seconds": 62.939014, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.9669699647951362, - "primary_loss": 0.6271721638959168, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 93, - "trial_id": 231, - "state": "PRUNED", - "params": { - "learning_rate": 2.456301968536136e-05, - "batch_size": 64, - "resolution": 256, - "model_capacity": "narrow", - "loss_weight_ratio": 0.518993915804978 - }, - "datetime_start": "2026-06-11T14:12:25.707012", - "datetime_complete": "2026-06-11T14:13:24.007531", - "duration_seconds": 58.300519, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.6370085282990985, - "primary_loss": 0.6926188302945487, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 94, - "trial_id": 232, - "state": "PRUNED", - "params": { - "learning_rate": 0.0016827718590978113, - "batch_size": 2, - "resolution": 512, - "model_capacity": "narrow", - "loss_weight_ratio": 0.3901213232480468 - }, - "datetime_start": "2026-06-11T14:13:27.009462", - "datetime_complete": "2026-06-11T14:15:15.158395", - "duration_seconds": 108.148933, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.8561622114054467, - "primary_loss": 3.172988403698563, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 95, - "trial_id": 233, - "state": "FAIL", - "params": { - "learning_rate": 0.00042245361689695283, - "batch_size": 64, - "resolution": 512, - "model_capacity": "narrow", - "loss_weight_ratio": 0.09698161427820617 - }, - "datetime_start": "2026-06-11T14:15:19.053044", - "datetime_complete": "2026-06-11T14:15:21.502024", - "duration_seconds": 2.44898, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 96, - "trial_id": 234, - "state": "PRUNED", - "params": { - "learning_rate": 0.0010085301314243155, - "batch_size": 4, - "resolution": 256, - "model_capacity": "wide", - "loss_weight_ratio": 0.6825227303515402 - }, - "datetime_start": "2026-06-11T14:15:22.979716", - "datetime_complete": "2026-06-11T14:17:09.701119", - "duration_seconds": 106.721403, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.9848687559506278, - "primary_loss": 0.07858677170699156, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 97, - "trial_id": 235, - "state": "FAIL", - "params": { - "learning_rate": 2.7283826334314898e-05, - "batch_size": 64, - "resolution": 256, - "model_capacity": "wide", - "loss_weight_ratio": 0.29036664563960757 - }, - "datetime_start": "2026-06-11T14:17:12.341718", - "datetime_complete": "2026-06-11T14:17:16.485723", - "duration_seconds": 4.144005, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 98, - "trial_id": 236, - "state": "PRUNED", - "params": { - "learning_rate": 0.0005765481949314836, - "batch_size": 8, - "resolution": 512, - "model_capacity": "narrow", - "loss_weight_ratio": 0.7465559395680328 - }, - "datetime_start": "2026-06-11T14:17:17.619303", - "datetime_complete": "2026-06-11T14:18:59.380148", - "duration_seconds": 101.760845, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.8532715976286795, - "primary_loss": 3.296048546642191, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 99, - "trial_id": 237, - "state": "PRUNED", - "params": { - "learning_rate": 0.0013883083869791772, - "batch_size": 64, - "resolution": 256, - "model_capacity": "narrow", - "loss_weight_ratio": 0.9210913786629441 - }, - "datetime_start": "2026-06-11T14:19:03.614194", - "datetime_complete": "2026-06-11T14:20:03.443717", - "duration_seconds": 59.829523, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.8453963661432513, - "primary_loss": 1.3442585114939807, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 100, - "trial_id": 238, - "state": "FAIL", - "params": { - "learning_rate": 0.00013018682232310546, - "batch_size": 32, - "resolution": 512, - "model_capacity": "narrow", - "loss_weight_ratio": 0.9347399927933793 - }, - "datetime_start": "2026-06-11T14:20:06.042537", - "datetime_complete": "2026-06-11T14:20:08.893357", - "duration_seconds": 2.85082, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 101, - "trial_id": 239, - "state": "PRUNED", - "params": { - "learning_rate": 1.0172045518922102e-05, - "batch_size": 8, - "resolution": 512, - "model_capacity": "wide", - "loss_weight_ratio": 0.005662841475867597 - }, - "datetime_start": "2026-06-11T14:20:09.560118", - "datetime_complete": "2026-06-11T14:23:51.931933", - "duration_seconds": 222.371815, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 5, - "primary_score": 0.8545924746829634, - "primary_loss": 3.441398318809799, - "oom_triggered": false, - "failure_tag": null, - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - }, - { - "trial_number": 102, - "trial_id": 240, - "state": "FAIL", - "params": { - "learning_rate": 0.00010302359944750134, - "batch_size": 4, - "resolution": 1024, - "model_capacity": "wide", - "loss_weight_ratio": 0.7977654326079116 - }, - "datetime_start": "2026-06-11T14:23:53.527265", - "datetime_complete": "2026-06-11T14:23:55.897358", - "duration_seconds": 2.370093, - "optuna_loss": null, - "optuna_score": null, - "epoch_reached": 0, - "primary_score": 0.0, - "primary_loss": 999.0, - "oom_triggered": true, - "failure_tag": "OOM", - "gpu_model": "NVIDIA L4", - "max_vram_gb": 22.0343017578125, - "health_tier": null, - "health_reason": null, - "git_commit": null, - "dataset_version": null - } -] \ No newline at end of file diff --git a/archive/colab_worker.py b/archive/colab_worker.py deleted file mode 100644 index 6c0dbe4..0000000 --- a/archive/colab_worker.py +++ /dev/null @@ -1,554 +0,0 @@ -import os -import glob -import time -import json -import numpy as np -import cv2 -import torch -import torch.nn as nn -import torch.optim as optim -from torch.utils.data import Dataset, DataLoader -import albumentations as A -from albumentations.pytorch import ToTensorV2 -from typing import Dict, Any, List, Optional, Tuple -import requests - -# Self-healing import block to fetch hpo_client.py dynamically from the broker if not present locally -try: - from src.hpo_client import TrialSession -except ImportError: - try: - from hpo_client import TrialSession - except ImportError: - print("hpo_client.py not found locally. Attempting to download from broker...") - # Resolve broker URL from environment - broker_url = os.getenv("HPO_BROKER_URL") - if not broker_url: - raise ValueError( - "Neither src.hpo_client nor hpo_client could be imported, " - "and HPO_BROKER_URL is not set to download hpo_client.py from the broker." - ) - try: - headers = {} - if "ngrok-free.app" in broker_url or "ngrok.io" in broker_url: - headers["ngrok-skip-browser-warning"] = "1" - token = os.getenv("HPO_SECRET_TOKEN") - if token: - headers["X-HPO-Token"] = token - r = requests.get( - f"{broker_url.rstrip('/')}/hpo_client.py", - headers=headers, - timeout=15, - ) - r.raise_for_status() - with open("hpo_client.py", "w") as f: - f.write(r.text) - print("Successfully downloaded hpo_client.py from broker.") - from hpo_client import TrialSession - except Exception as e: - print(f"Error downloading hpo_client.py from broker: {e}") - raise - -# --- 1. YOUR EXACT DATASET CLASS (EMBEDDED FOR EASY COLAB USE) --- -class CrackDataset(Dataset): - """ - PyTorch Dataset representing bridge defect imagery - and annotated segmentation binary masks. - """ - def __init__(self, img_paths: list, mask_paths: list, transform: A.Compose = None): - self.img_paths = img_paths - self.mask_paths = mask_paths - self.transform = transform - - def __len__(self) -> int: - return len(self.img_paths) - - def __getitem__(self, idx: int) -> tuple: - # Load imagery and convert to RGB channels - img = cv2.imread(self.img_paths[idx]) - img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) - - # Load annotation mask in grayscale - mask = cv2.imread(self.mask_paths[idx], cv2.IMREAD_GRAYSCALE) - - # Retrieve binary values from grayscale mask - _, mask = cv2.threshold(mask, 127, 1, cv2.THRESH_BINARY) - - # INVERSION: DeepCrack uses black (0) for cracks and white (1) for background. - # We invert it so cracks become 1 and background becomes 0 to match your model's expectations. - mask = 1 - mask - - mask = np.expand_dims(mask, axis=-1).astype(np.float32) - - # Match image and mask transformations - if self.transform: - augmented = self.transform(image=img, mask=mask) - img = augmented['image'] - mask = augmented['mask'] - - return img, mask - - -# Augmentation using your albumentations recipes, adapted for dynamic resolutions -def get_train_transform(resolution: int) -> A.Compose: - return A.Compose([ - A.Resize(resolution, resolution), # Injected dynamically for HPO tuning - A.HorizontalFlip(p=0.5), - A.VerticalFlip(p=0.5), - A.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2, hue=0.1, p=0.4), - A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)), - ToTensorV2() - ]) - - -def get_val_test_transform(resolution: int) -> A.Compose: - """ - Deterministic normalization transform for validation/test crack detection. - """ - return A.Compose([ - A.Resize(resolution, resolution), # Injected dynamically for HPO tuning - A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)), - ToTensorV2() - ]) - - -# --- 2. DATASET DOWNLOAD & EXTRACTION --- -# Automatically downloads and unzips DeepCrack if it is not present in /content/input/ -if not os.path.exists("/content/input/DeepCrack") and not os.path.exists("/content/DeepCrack"): - print("Downloading DeepCrack dataset from GitHub...") - os.makedirs("/content/input", exist_ok=True) - # Using raw download URL from yhlleo/DeepCrack - os.system("wget -q -O /content/input/DeepCrack.zip https://github.com/yhlleo/DeepCrack/raw/master/dataset/DeepCrack.zip") - print("Extracting DeepCrack.zip to /content/input/...") - os.system("unzip -o -q /content/input/DeepCrack.zip -d /content/input/") - print("DeepCrack dataset downloaded and extracted successfully!") - -# --- 3. MODEL FETCHING SETUPS --- -# Clones your repository if not already cloned in Google Colab -if not os.path.exists("/content/crack-seg"): - os.system("git clone https://github.com/Ishaan1402/crack-seg.git /content/crack-seg") - -import sys -sys.path.append("/content/crack-seg") - -from src.models.unet import UNet # Imports your real UNet model definition - -# --- 4. HPO BROKER CONFIGURATION --- -BROKER_URL = os.getenv("HPO_BROKER_URL") -if not BROKER_URL: - raise ValueError("HPO_BROKER_URL environment variable must be set to connect to Pathfinder.") - - - - - -def broker_get(path: str, timeout: int = 30): - base = BROKER_URL.rstrip("/") - for suffix in ("/api/suggest_trial", "/api/suggest_trials", "/api"): - if base.endswith(suffix): - base = base[: -len(suffix)] - if not path.startswith("/"): - path = "/" + path - - headers = {} - if "ngrok-free.app" in base or "ngrok.io" in base: - headers["ngrok-skip-browser-warning"] = "1" - token = os.getenv("HPO_SECRET_TOKEN") - if token: - headers["X-HPO-Token"] = token - - return requests.get( - base + path, - headers=headers, - timeout=timeout, - ) - -device = torch.device("cuda" if torch.cuda.is_available() else "cpu") -print(f"Using device: {device}") - - -# --- 6. YOUR SLIDING WINDOW NORMALIZATION CODE INTEGRATION --- -def run_sliding_window_inference(model: nn.Module, image: torch.Tensor, patch_size=448, overlap=0.5, t_val=0.5): - """ - Tiled sliding window inference combining overlapping predictions with a 2D Gaussian weight map. - """ - model.eval() - c, h_img, w_img = image.shape - - accum_p = np.zeros((h_img, w_img), dtype=np.float32) - accum_w = np.zeros((h_img, w_img), dtype=np.float32) - stride = int(patch_size * (1.0 - overlap)) - - gaussian_patch = np.outer(np.hamming(patch_size), np.hamming(patch_size)).astype(np.float32) - - with torch.no_grad(): - for y in range(0, h_img - patch_size + 1, stride): - for x in range(0, w_img - patch_size + 1, stride): - # Extract image patch - patch = image[:, y : y + patch_size, x : x + patch_size].unsqueeze(0).to(device) - logits = model(patch) - probs = torch.sigmoid(logits).squeeze(0).squeeze(0).cpu().numpy() - - accum_p[y : y + patch_size, x : x + patch_size] += probs * gaussian_patch - accum_w[y : y + patch_size, x : x + patch_size] += gaussian_patch - - # Element-wise division to normalize final blended probabilities - final_probs = accum_p / np.maximum(accum_w, 1e-5) - binary_mask = (final_probs > t_val).astype(np.uint8) - crack_area_ratio = float(np.sum(binary_mask) / (h_img * w_img)) - - return final_probs, binary_mask, crack_area_ratio - - -DEFAULT_HPO_CONFIG: Dict[str, Any] = { - "eval_protocol": { - "enabled": False, - "fixed_resolution": None, - "train_resolution_param": "resolution", - "fixed_dice_attr": "dice_eval_fixed", - "fixed_bce_attr": "bce_eval_fixed", - "use_fixed_metric_for_pruning": False, - "patch_size_below_512": 256, - "patch_size_at_512_plus": 448, - }, - "legacy_param_aliases": {"encoder_name": "model_capacity"}, - "legacy_capacity_values": { - "resnet34": "narrow", - "efficientnet-b0": "narrow", - "resnet50": "wide", - }, -} - - -def fetch_hpo_config() -> Dict[str, Any]: - if BROKER_URL: - try: - resp = broker_get("/api/hpo_config", timeout=15) - resp.raise_for_status() - return resp.json() - except Exception as exc: - print(f"Warning: could not fetch hpo_config from broker ({exc}); using defaults.") - return json.loads(json.dumps(DEFAULT_HPO_CONFIG)) - - -def normalize_trial_params(params: Dict[str, Any], config: Dict[str, Any]) -> Dict[str, Any]: - out = dict(params) - for old, new in config.get("legacy_param_aliases", {}).items(): - if old in out and new not in out: - val = out.pop(old) - mapped = config.get("legacy_capacity_values", {}).get(val, val) - out[new] = mapped - return out - - -def unet_features_from_params(params: Dict[str, Any], config: Dict[str, Any]) -> List[int]: - norm = normalize_trial_params(params, config) - capacity = norm.get("model_capacity", "narrow") - if capacity == "wide": - return [64, 128, 256, 512] - return [32, 64, 128, 256] - - -def patch_size_for_resolution(resolution: int, config: Dict[str, Any]) -> int: - ev = config.get("eval_protocol", {}) - if resolution < 512: - return int(ev.get("patch_size_below_512", 256)) - return int(ev.get("patch_size_at_512_plus", 448)) - - -def soft_dice_loss(logits: torch.Tensor, targets: torch.Tensor, eps: float = 1e-6) -> torch.Tensor: - probs = torch.sigmoid(logits) - intersection = (probs * targets).sum() - union = probs.sum() + targets.sum() - dice = (2.0 * intersection + eps) / (union + eps) - return 1.0 - dice - - -def run_validation_epoch( - model: nn.Module, - val_images: List[str], - val_masks: List[str], - resolution: int, - bce_loss_fn: nn.Module, - config: Dict[str, Any], -) -> Tuple[float, float]: - val_dataset = CrackDataset( - img_paths=val_images, - mask_paths=val_masks, - transform=get_val_test_transform(resolution), - ) - model.eval() - val_dice_list = [] - val_bce_list = [] - patch_size = patch_size_for_resolution(resolution, config) - - for val_img, val_mask in val_dataset: - final_probs, binary_mask, _ = run_sliding_window_inference( - model=model, - image=val_img, - patch_size=patch_size, - overlap=0.5, - t_val=0.5, - ) - target_np = val_mask.numpy().astype(np.uint8) - if target_np.ndim == 3: - target_np = np.squeeze(target_np) - val_dice_list.append(calculate_dice_score(binary_mask, target_np)) - - probs_t = torch.tensor(final_probs).unsqueeze(0).unsqueeze(0) - if val_mask.dim() == 3 and val_mask.shape[2] == 1: - val_mask = val_mask.permute(2, 0, 1) - target_for_loss = val_mask.unsqueeze(0) - val_loss = bce_loss_fn( - torch.logit(torch.clamp(probs_t, 1e-6, 1 - 1e-6)), target_for_loss - ) - val_bce_list.append(val_loss.item()) - - return float(np.mean(val_dice_list)), float(np.mean(val_bce_list)) - - -def calculate_dice_score(pred_mask: np.ndarray, target_mask: np.ndarray) -> float: - intersection = np.sum(pred_mask * target_mask) - union = np.sum(pred_mask) + np.sum(target_mask) - if union == 0: - return 1.0 - return float((2. * intersection) / union) - - -# --- 7. MAIN HPO RUNNER FOR COLAB --- -COLAB_WORKER_REV = "2025-06-11-1" # bump after broker-side edits; Colab should re-fetch /colab_worker.py - -# -# Public entrypoints (bridge-crack reference only): -# train_colab_trial — run one trial (suggest → train → complete / prune / fail) -# train_colab_trial_loop — call train_colab_trial N times; survives guardrail skips, -# caught OOM, and transient suggest errors; clears CUDA cache -# between iterations. - - -def train_colab_trial(study_name: str, epochs=15): - """Run a single HPO trial on Colab (one suggest → complete cycle).""" - # Ensure folder for saved checkpoints exists - os.makedirs("checkpoints", exist_ok=True) - - # Detect GPU hardware telemetry if torch is available - gpu_model = "CPU" - max_vram_gb = 0.0 - try: - import torch - if torch.cuda.is_available(): - gpu_model = torch.cuda.get_device_name(0) - max_vram_gb = torch.cuda.get_device_properties(0).total_memory / (1024 ** 3) - except ImportError: - pass - - session = TrialSession(broker_url=BROKER_URL, study_name=study_name) - print(f"colab_worker rev {COLAB_WORKER_REV}") - print(session.health()) - - try: - trial = session.suggest() - except Exception as exc: - print(f"ERROR: could not get suggestion from HPO broker: {exc}") - return - - trial_id = trial["trial_id"] - trial_number = trial.get("trial_number", trial_id) - params = trial["params"] - trial_label = f"#{trial_number}" - - print(f"\n--- Starting HPO Trial {trial_label} on Colab GPU (broker id={trial_id}) ---") - - def _fail_trial(epoch: int = 0, oom: bool = False, reason: str = ""): - if reason: - print(reason) - try: - session.complete( - epoch, - 0.0, - 999.0, - state="FAIL", - gpu_model=gpu_model, - max_vram_gb=max_vram_gb, - oom_triggered=oom, - ) - print(f" >> Trial {trial_label} reported as FAILED to broker.") - except Exception as report_err: - print(f" >> Could not report failure to broker: {report_err}") - - try: - hpo_config = fetch_hpo_config() - print(f"Parameters: {params}") - - required = ["learning_rate", "batch_size", "resolution", "model_capacity", "loss_weight_ratio"] - missing = [k for k in required if k not in params] - if missing: - _fail_trial(reason=f"ERROR: Trial {trial_id} has incomplete hyperparameters (missing: {missing}).") - return - - # Pre-flight guardrail (predictable VRAM blow-up; not a caught CUDA OOM) - if params.get("resolution", 512) == 1024 and params.get("batch_size", 8) >= 16: - _fail_trial(oom=True, reason="Guardrail: resolution 1024 with batch_size >= 16 — skipping training.") - return - - lr = params.get("learning_rate", 1e-3) - batch_size = int(params.get("batch_size", 8)) - resolution = int(params.get("resolution", 512)) - loss_weight_ratio = params.get("loss_weight_ratio", 0.5) - - unet_features = unet_features_from_params(params, hpo_config) - ev = hpo_config.get("eval_protocol", {}) - - DEEPCRACK_DIR = "/content/input" - if os.path.exists(f"{DEEPCRACK_DIR}/DeepCrack/train_img"): - DEEPCRACK_DIR = f"{DEEPCRACK_DIR}/DeepCrack" - - print(f"Loading DeepCrack images from: {DEEPCRACK_DIR}") - - train_images = sorted(glob.glob(f"{DEEPCRACK_DIR}/train_img/*.jpg") + - glob.glob(f"{DEEPCRACK_DIR}/train_img/*.png")) - train_masks = sorted(glob.glob(f"{DEEPCRACK_DIR}/train_lab/*.jpg") + - glob.glob(f"{DEEPCRACK_DIR}/train_lab/*.png")) - - val_images = sorted(glob.glob(f"{DEEPCRACK_DIR}/test_img/*.jpg") + - glob.glob(f"{DEEPCRACK_DIR}/test_img/*.png")) - val_masks = sorted(glob.glob(f"{DEEPCRACK_DIR}/test_lab/*.jpg") + - glob.glob(f"{DEEPCRACK_DIR}/test_lab/*.png")) - - if not train_images or not train_masks: - raise FileNotFoundError( - f"Could not find training images or masks in {DEEPCRACK_DIR}! " - f"(Checked subfolders train_img/ and train_lab/)" - ) - - train_dataset = CrackDataset( - img_paths=train_images, - mask_paths=train_masks, - transform=get_train_transform(resolution) - ) - dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, drop_last=True) - - model = UNet(in_channels=3, out_channels=1, features=unet_features).to(device) - optimizer = optim.Adam(model.parameters(), lr=lr) - bce_loss_fn = nn.BCEWithLogitsLoss() - - pruned = False - - for epoch in range(1, epochs + 1): - t_start = time.time() - model.train() - train_bce = 0.0 - - for images, targets in dataloader: - images, targets = images.to(device), targets.to(device) - if targets.dim() == 4 and targets.shape[3] == 1: - targets = targets.permute(0, 3, 1, 2) - optimizer.zero_grad() - logits = model(images) - bce = bce_loss_fn(logits, targets) - dice_component = soft_dice_loss(logits, targets) - lw = float(loss_weight_ratio) - loss = lw * bce + (1.0 - lw) * dice_component - loss.backward() - optimizer.step() - train_bce += loss.item() * images.size(0) - - train_bce /= len(dataloader.dataset) - - mean_dice, mean_bce = run_validation_epoch( - model, val_images, val_masks, resolution, bce_loss_fn, hpo_config - ) - - dice_eval_fixed = None - bce_eval_fixed = None - fixed_res = ev.get("fixed_resolution") - if ev.get("enabled") and fixed_res is not None: - fixed_res = int(fixed_res) - if fixed_res != resolution: - dice_eval_fixed, bce_eval_fixed = run_validation_epoch( - model, val_images, val_masks, fixed_res, bce_loss_fn, hpo_config - ) - print( - f" Fixed eval @{fixed_res}px | BCE: {bce_eval_fixed:.4f} | Dice: {dice_eval_fixed:.4f}" - ) - else: - dice_eval_fixed, bce_eval_fixed = mean_dice, mean_bce - - t_elapsed = time.time() - t_start - total_images_processed = len(dataloader.dataset) + len(val_images) - speed_ips = total_images_processed / t_elapsed if t_elapsed > 0 else 0.0 - - gpu_memory = 0.0 - if torch.cuda.is_available(): - gpu_memory = torch.cuda.memory_allocated(device) / (1024 ** 2) - - print(f" Epoch {epoch:02d} | Train BCE: {train_bce:.4f} | Val BCE: {mean_bce:.4f} | Val Dice: {mean_dice:.4f} | GPU Mem: {gpu_memory:.1f}MB | Speed: {speed_ips:.1f} img/s") - - should_prune = session.report_epoch( - epoch, - mean_dice, - mean_bce, - score_eval_fixed=dice_eval_fixed, - loss_eval_fixed=bce_eval_fixed, - gpu_memory=gpu_memory, - speed_ips=speed_ips - ) - - if should_prune: - print(f" >> Trial {trial_label} performing poorly. PRUNING at epoch {epoch}!") - session.complete( - epoch, - mean_dice, - mean_bce, - score_eval_fixed=dice_eval_fixed, - loss_eval_fixed=bce_eval_fixed, - state="PRUNED", - gpu_model=gpu_model, - max_vram_gb=max_vram_gb, - oom_triggered=False - ) - pruned = True - break - - if not pruned: - weights_path = f"checkpoints/trial_{trial_id}_unet_res_{resolution}.pt" - torch.save(model.state_dict(), weights_path) - - session.complete( - epoch, - mean_dice, - mean_bce, - weights_path=weights_path, - score_eval_fixed=dice_eval_fixed, - loss_eval_fixed=bce_eval_fixed, - state="COMPLETE", - gpu_model=gpu_model, - max_vram_gb=max_vram_gb, - oom_triggered=False - ) - print(f" >> Trial {trial_label} marked as COMPLETED successfully!") - - except Exception as exc: - oom = type(exc).__name__ == "OutOfMemoryError" or "out of memory" in str(exc).lower() - if oom: - _fail_trial(oom=True, reason=f"CUDA OOM during trial {trial_label}: {exc}") - else: - _fail_trial(reason=f"Trial {trial_label} crashed: {exc}") - finally: - try: - import torch - if torch.cuda.is_available(): - torch.cuda.empty_cache() - except Exception: - pass - - -def train_colab_trial_loop(study_name: str, n_trials: int = 12, epochs: int = 15): - """Run ``train_colab_trial`` repeatedly — the usual Colab entrypoint for a full study session.""" - for i in range(n_trials): - print(f"\n========== Colab HPO iteration {i + 1}/{n_trials} ==========") - train_colab_trial(study_name, epochs=epochs) - - -if __name__ == "__main__": - train_colab_trial_loop("bridge_crack_study", n_trials=12, epochs=15) diff --git a/archive/extract_bridge_crack_study.py b/archive/extract_bridge_crack_study.py deleted file mode 100644 index 251f23f..0000000 --- a/archive/extract_bridge_crack_study.py +++ /dev/null @@ -1,117 +0,0 @@ -import optuna -import json -import os -import datetime -from sqlalchemy.orm import Session -from src.db_manager import engine, SessionLocal -from src.schema import TrialResult - -def extract_study_data(): - study_name = "bridge_crack_study" - print(f"Loading Optuna study '{study_name}'...") - - # 1. Load the Optuna study - db_url = os.getenv("HPO_DATABASE_URL", "sqlite:///hpo_studies.db") - try: - study = optuna.load_study(study_name=study_name, storage=db_url) - except KeyError: - print(f"Error: Study '{study_name}' not found in database.") - return - - print(f"Loading Pathfinder-specific trial results for '{study_name}'...") - # 2. Query Pathfinder TrialResult metadata from the DB - pathfinder_results = {} - with SessionLocal() as session: - results = session.query(TrialResult).filter_by(study_name=study_name).all() - for r in results: - pathfinder_results[r.trial_id] = r.to_dict() - - print(f"Aggregating {len(study.trials)} trials...") - # 3. Combine Optuna trials with Pathfinder trial results - all_trials_data = [] - - for t in study.trials: - # Get basic Optuna trial info - trial_number = t.number - trial_id = t._trial_id - state = t.state.name - params = t.params - datetime_start = t.datetime_start.isoformat() if t.datetime_start else None - datetime_complete = t.datetime_complete.isoformat() if t.datetime_complete else None - duration = t.duration.total_seconds() if t.duration else None - values = t.values # This is a list/tuple of objective values - - # Primary score and loss from values or parameters - # In multi-objective: direction minimize (loss/obj 0), maximize (score/obj 1) - optuna_loss = values[0] if values and len(values) > 0 else None - optuna_score = values[1] if values and len(values) > 1 else None - - # Get Pathfinder metadata if available using _trial_id - pf_info = pathfinder_results.get(trial_id, {}) - - # Merge data - trial_record = { - "trial_number": trial_number, - "trial_id": trial_id, - "state": state, - "params": params, - "datetime_start": datetime_start, - "datetime_complete": datetime_complete, - "duration_seconds": duration, - "optuna_loss": optuna_loss, - "optuna_score": optuna_score, - "epoch_reached": pf_info.get("epoch_reached"), - "primary_score": pf_info.get("primary_score"), - "primary_loss": pf_info.get("primary_loss"), - "oom_triggered": pf_info.get("oom_triggered"), - "failure_tag": pf_info.get("failure_tag"), - "gpu_model": pf_info.get("gpu_model"), - "max_vram_gb": pf_info.get("max_vram_gb"), - "health_tier": pf_info.get("health_tier"), - "health_reason": pf_info.get("health_reason"), - "git_commit": pf_info.get("git_commit"), - "dataset_version": pf_info.get("dataset_version"), - } - all_trials_data.append(trial_record) - - # Sort by trial_number - all_trials_data.sort(key=lambda x: x["trial_number"]) - - # Save to JSON - output_path = "bridge_crack_study_trials.json" - with open(output_path, "w") as f: - json.dump(all_trials_data, f, indent=2) - print(f"Successfully saved all trials data to '{output_path}'.") - - # Filter completed trials to show progress - completed_trials = [t for t in all_trials_data if t["state"] == "COMPLETE"] - print(f"Total trials: {len(all_trials_data)}") - print(f"Completed trials: {len(completed_trials)}") - - if not completed_trials: - print("No completed trials found to evaluate improvement.") - return - - # Find start trials vs best trials - first_completed = completed_trials[:3] - best_completed = sorted(completed_trials, key=lambda x: x["primary_score"] or 0, reverse=True)[:3] - - print("\n--- FIRST COMPLETED TRIALS ---") - for t in first_completed: - print(f"Trial #{t['trial_number']} (ID={t['trial_id']}): Score={t['primary_score']}, Loss={t['primary_loss']}, Params={t['params']}") - - print("\n--- BEST COMPLETED TRIALS ---") - for t in best_completed: - print(f"Trial #{t['trial_number']} (ID={t['trial_id']}): Score={t['primary_score']}, Loss={t['primary_loss']}, Params={t['params']}") - - initial_score = first_completed[0]['primary_score'] if first_completed else None - best_score = best_completed[0]['primary_score'] if best_completed else None - - if initial_score is not None and best_score is not None: - diff = best_score - initial_score - print(f"\nImprovement in Best Score: {initial_score} -> {best_score} (Gain: +{diff:.6f})") - else: - print("\nCould not calculate improvement due to missing scores.") - -if __name__ == "__main__": - extract_study_data() diff --git a/archive/extract_to_csv.py b/archive/extract_to_csv.py deleted file mode 100644 index 47c194f..0000000 --- a/archive/extract_to_csv.py +++ /dev/null @@ -1,99 +0,0 @@ -import optuna -import csv -import os -from sqlalchemy.orm import Session -from src.db_manager import SessionLocal -from src.schema import TrialResult - -def extract_filtered_csv(): - study_name = "bridge_crack_study" - print(f"Loading study '{study_name}'...") - - db_url = os.getenv("HPO_DATABASE_URL", "sqlite:///hpo_studies.db") - try: - study = optuna.load_study(study_name=study_name, storage=db_url) - except KeyError: - print(f"Error: Study '{study_name}' not found.") - return - - # Load Pathfinder metadata - pathfinder_results = {} - with SessionLocal() as session: - results = session.query(TrialResult).filter_by(study_name=study_name).all() - for r in results: - pathfinder_results[r.trial_id] = r.to_dict() - - # Filter and flatten completed trials with resolution >= 500 - rows = [] - headers = [ - "trial_number", "trial_id", "state", "score", "loss", - "duration_seconds", "epoch_reached", "learning_rate", - "batch_size", "resolution", "encoder_name", "loss_weight_ratio", - "model_capacity", "gpu_model", "max_vram_gb" - ] - - for t in study.trials: - if t.state.name != "COMPLETE": - continue - - params = t.params - res = params.get("resolution") - if res is None or res < 500: - continue - - trial_id = t._trial_id - pf_info = pathfinder_results.get(trial_id, {}) - - # Primary score and loss from Optuna values or Pathfinder - values = t.values - optuna_loss = values[0] if values and len(values) > 0 else None - optuna_score = values[1] if values and len(values) > 1 else None - - score = pf_info.get("primary_score") or optuna_score - loss = pf_info.get("primary_loss") or optuna_loss - - row = { - "trial_number": t.number, - "trial_id": trial_id, - "state": t.state.name, - "score": score, - "loss": loss, - "duration_seconds": t.duration.total_seconds() if t.duration else None, - "epoch_reached": pf_info.get("epoch_reached"), - "learning_rate": params.get("learning_rate"), - "batch_size": params.get("batch_size"), - "resolution": res, - "encoder_name": params.get("encoder_name"), - "loss_weight_ratio": params.get("loss_weight_ratio"), - "model_capacity": params.get("model_capacity", "N/A"), - "gpu_model": pf_info.get("gpu_model"), - "max_vram_gb": pf_info.get("max_vram_gb") - } - rows.append(row) - - # Sort by trial number - rows.sort(key=lambda x: x["trial_number"]) - - # Save to CSV - csv_file = "bridge_crack_study_500px.csv" - with open(csv_file, mode="w", newline="") as f: - writer = csv.DictWriter(f, fieldnames=headers) - writer.writeheader() - for r in rows: - writer.writerow(r) - - print(f"Successfully extracted {len(rows)} trials to '{csv_file}'.") - - if len(rows) > 0: - first = rows[0] - best = max(rows, key=lambda x: x["score"] if x["score"] is not None else 0) - print("\n--- RESULTS OVERVIEW (Resolution >= 500px) ---") - print(f"First Trial #{first['trial_number']}: Score={first['score']}, Loss={first['loss']}, Params={first['learning_rate'], first['batch_size'], first['resolution']}") - print(f"Best Trial #{best['trial_number']}: Score={best['score']}, Loss={best['loss']}, Params={best['learning_rate'], best['batch_size'], best['resolution']}") - if first['score'] is not None and best['score'] is not None: - print(f"Improvement: {first['score']} -> {best['score']} (Gain: +{best['score'] - first['score']:.6f})") - else: - print("No completed trials matching criteria found.") - -if __name__ == "__main__": - extract_filtered_csv() diff --git a/archive/plot_results.py b/archive/plot_results.py deleted file mode 100644 index efa3168..0000000 --- a/archive/plot_results.py +++ /dev/null @@ -1,93 +0,0 @@ -import os -import sys - -# Ensure pandas and matplotlib are installed -try: - import pandas as pd - import matplotlib.pyplot as plt -except ImportError: - print("Required packages (pandas, matplotlib) are missing.") - print("Installing packages...") - import subprocess - subprocess.check_call([sys.executable, "-m", "pip", "install", "pandas", "matplotlib"]) - import pandas as pd - import matplotlib.pyplot as plt - -def plot_study_results(): - csv_file = "bridge_crack_study_500px.csv" - if not os.path.exists(csv_file): - print(f"Error: {csv_file} not found. Run extract_to_csv.py first.") - return - - # Load data - df = pd.read_csv(csv_file) - print("Loaded data:") - print(df[["trial_number", "score", "loss", "learning_rate", "batch_size", "resolution"]]) - - if len(df) == 0: - print("No completed trials to plot.") - return - - # Modern styling - plt.style.use("seaborn-v0_8-whitegrid" if "seaborn-v0_8-whitegrid" in plt.style.available else "default") - fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 6)) - - # Color definitions - accent_color = "#3B82F6" # Premium Indigo/Blue - highlight_color = "#10B981" # Emerald Green for best - neutral_dark = "#1F2937" - - # Plot 1: Score Progression over completed trials - ax1.plot(df["trial_number"], df["score"], marker="o", color=accent_color, linewidth=2, markersize=8, label="Dice Score") - - # Highlight best trial - best_idx = df["score"].idxmax() - best_trial = df.loc[best_idx] - ax1.scatter(best_trial["trial_number"], best_trial["score"], color=highlight_color, s=200, zorder=5, label=f"Best (Trial #{int(best_trial['trial_number'])}: {best_trial['score']:.4f})") - - # Labels and Titles - ax1.set_title("Dice Score Progression (Resolution >= 500px)", fontsize=14, fontweight="bold", pad=15, color=neutral_dark) - ax1.set_xlabel("Optuna Trial Number", fontsize=12, labelpad=10) - ax1.set_ylabel("Dice Score (Higher is Better)", fontsize=12, labelpad=10) - ax1.legend(loc="lower right", frameon=True, facecolor="white", edgecolor="#E5E7EB") - ax1.set_ylim(df["score"].min() - 0.005, df["score"].max() + 0.005) - - # Plot 2: Learning Rate vs Score colored by Batch Size - scatter = ax2.scatter( - df["learning_rate"], - df["score"], - c=df["batch_size"], - cmap="viridis", - s=120, - edgecolors="none", - alpha=0.85 - ) - # Highlight best trial in scatter - ax2.scatter( - best_trial["learning_rate"], - best_trial["score"], - color=highlight_color, - edgecolors="black", - s=250, - zorder=5, - label="Best Model" - ) - - ax2.set_xscale("log") - ax2.set_title("Learning Rate vs. Score (Size: Batch Size)", fontsize=14, fontweight="bold", pad=15, color=neutral_dark) - ax2.set_xlabel("Learning Rate (Log Scale)", fontsize=12, labelpad=10) - ax2.set_ylabel("Dice Score", fontsize=12, labelpad=10) - - # Colorbar for Batch Size - cbar = plt.colorbar(scatter, ax=ax2) - cbar.set_label("Batch Size", fontsize=11, rotation=270, labelpad=15) - ax2.legend(loc="lower left", frameon=True, facecolor="white", edgecolor="#E5E7EB") - - plt.tight_layout() - plot_path = "bridge_crack_500px_plots.png" - plt.savefig(plot_path, dpi=300, facecolor="white") - print(f"Successfully generated and saved plots to '{plot_path}'.") - plt.close() - -if __name__ == "__main__": - plot_study_results() diff --git a/docs/INTEGRATION.md b/docs/INTEGRATION.md index 9bc2171..397e4d4 100644 --- a/docs/INTEGRATION.md +++ b/docs/INTEGRATION.md @@ -3,8 +3,8 @@ A human quickstart for wiring your own model to the broker. For the agent-driven version of this (have your IDE assistant do it), see [`AGENTS.md`](../AGENTS.md). -The reference crack-seg implementation (`colab_worker.py`) lives in [`archive/`](../archive/). You -do **not** need to fork it. Cloners start from `templates/worker_minimal.py` and the 3-call +The reference crack-seg implementation is not included in this repository. +You do **not** need to fork it. Cloners start from `templates/worker_minimal.py` and the 3-call client in `src/hpo_client.py`. ## 1. Install @@ -146,12 +146,6 @@ That is the entire contract: > - Pass your lower-is-better metric (e.g. Cross-Entropy, Perplexity, MAE) as `loss`. > - You can customize their display names on the UI dashboard under **Settings > Eval protocol** by setting "Loss metric display name" and "Score metric display name". -### Google Colab (reference only) - -The historical bridge-crack U-Net reference implementation is preserved in -[`archive/colab_worker.py`](../archive/colab_worker.py). **Do not use it for new studies.** -Cloners should use `templates/worker_minimal.py`. - ## 5. Create the study and validate Call the MCP `init_from_manifest` tool (or CLI `init`) to create the Optuna study and seed diff --git a/docs/images/pathfinder_dashboard_example.png b/docs/images/pathfinder_dashboard_example.png new file mode 100644 index 0000000..a4ec1ac Binary files /dev/null and b/docs/images/pathfinder_dashboard_example.png differ diff --git a/examples/onboarding/README.md b/examples/onboarding/README.md deleted file mode 100644 index 009df7d..0000000 --- a/examples/onboarding/README.md +++ /dev/null @@ -1,42 +0,0 @@ -# Pathfinder Onboarding Walkthrough - -This directory contains a complete, self-contained walkthrough for onboarding a new ML project to Pathfinder using a **manifest file** (`train.hpo.yaml`). - ---- - -## The Happy Path (3 Steps) - -### Step 1: Draft the Manifest -Create a file named `train.hpo.yaml` defining your search space, objective metrics, environment variables, and entrypoint command. You can start from [train.hpo.yaml](train.hpo.yaml). - -> [!NOTE] -> **Worker Metrics Slot Mapping** -> Even if you name your objectives `"accuracy"` (maximize) and `"loss"` (minimize) in the manifest, the worker client script will report them through the generic API slots: `report_epoch(..., score=..., loss=...)`. Pathfinder maps them dynamically in the backend using your objective directions. - -### Step 2: Validate the Manifest -Validate your manifest file before registering: -- **CLI**: - ```bash - python hpo_cli.py validate examples/onboarding/train.hpo.yaml - ``` -- **Dashboard**: Open the dashboard, click **+ New Study** in the top right, and drag-and-drop your `train.hpo.yaml` file into the modal. -- **MCP**: Call the `validate_manifest` tool with your YAML content. - -### Step 3: Initialize the Study -Register your study and write configurations to the SQLite database: -- **CLI**: - ```bash - python hpo_cli.py init examples/onboarding/train.hpo.yaml - ``` -- **Dashboard**: Click **Initialize Study** in the New Study modal once validation succeeds. -- **MCP**: Call the `init_from_manifest` tool. - ---- - -## Running the Worker - -Once the study is initialized: -1. Open the dashboard. -2. Select your study (`mnist_tuning`) in the dropdown. -3. Click the **Worker Setup** tab in the sidebar. -4. Copy the fully customized setup commands and run them on your training GPU machine or in a Google Colab notebook! diff --git a/examples/onboarding/train.hpo.yaml b/examples/onboarding/train.hpo.yaml deleted file mode 100644 index a823dcf..0000000 --- a/examples/onboarding/train.hpo.yaml +++ /dev/null @@ -1,31 +0,0 @@ -# examples/onboarding/train.hpo.yaml -# Sample manifest for onboarding a custom PyTorch model to Pathfinder. -study_name: "mnist_tuning" - -metrics: - primary_score: "accuracy" - objectives: - - name: "accuracy" - direction: "maximize" - label: "Validation Accuracy" - - name: "loss" - direction: "minimize" - label: "Cross-Entropy Loss" - -params: - - name: "learning_rate" - type: "float_log" - min: 1e-4 - max: 1e-1 - - name: "batch_size" - type: "categorical" - options: [32, 64, 128] - - name: "dropout" - type: "float" - min: 0.1 - max: 0.5 - -worker: - entrypoint: "python train.py --lr {learning_rate} --batch_size {batch_size} --dropout {dropout}" - env: - CUDA_VISIBLE_DEVICES: "0" diff --git a/hpo_cli.py b/hpo_cli.py index 5a0104f..f9d2fca 100644 --- a/hpo_cli.py +++ b/hpo_cli.py @@ -681,10 +681,14 @@ def cmd_modelcard(args): lines.append(f"- GPU Model: {vram.get('gpu_model', 'Unknown')}") lines.append(f"- GPU Capacity: {vram.get('gpu_capacity_gb', 'N/A')} GB") lines.append(f"- OOM Count: {vram.get('oom_count', 0)}") - oom_risk = vram.get("bounds_oom_risk") - if oom_risk: - lines.append(f"- OOM Risk Level: {oom_risk.get('risk_level', 'N/A')}") - lines.append(f"- Predicted Max VRAM: {oom_risk.get('predicted_max_vram_gb', 'N/A'):.2f} GB") + oom_trials = vram.get("oom_trials", []) + if oom_trials: + lines.append("- OOM Trials:") + for ot in oom_trials: + tid = ot.get("trial_id", "?") + peak = ot.get("peak_vram_gb", "N/A") + params = ot.get("params", {}) + lines.append(f" - Trial {tid}: peak VRAM {peak} GB, params={params}") lines.append("") health = packet.get("health", {}) diff --git a/hpo_mcp_server.py b/hpo_mcp_server.py index bbe7f27..dc282d4 100644 --- a/hpo_mcp_server.py +++ b/hpo_mcp_server.py @@ -1,4 +1,4 @@ -from typing import Optional, Dict, Any, List +from typing import Optional, Dict, Any from mcp.server.fastmcp import FastMCP mcp = FastMCP("Pathfinder") @@ -15,10 +15,20 @@ def get_study_data(study_name: str) -> Dict[str, Any]: @mcp.tool() -def get_study_cards(study_name: Optional[str] = None) -> List[Dict[str, Any]]: +def get_study_cards(study_name: Optional[str] = None) -> Dict[str, Any]: """Retrieves generated study cards (model cards, recaps) from the database to enable cross-study queries.""" from src.analytics import load_study_cards - return load_study_cards(study_name) + + if study_name is not None: + import optuna + from src.db_manager import DATABASE_URL + try: + optuna.load_study(study_name=study_name, storage=DATABASE_URL) + except KeyError: + return {"success": False, "error": f"Study '{study_name}' not found."} + + cards = load_study_cards(study_name) + return {"success": True, "cards": cards} @mcp.tool() @@ -61,10 +71,14 @@ def init_from_manifest(yaml_str: str, force: bool = False) -> Dict[str, Any]: @mcp.tool() -def export_manifest(study_name: str) -> str: +def export_manifest(study_name: str) -> Dict[str, Any]: """Export the active search space, HPO config, and context of an existing study as a valid manifest YAML string.""" from src.manifest import export_manifest_yaml - return export_manifest_yaml(study_name) + try: + result = export_manifest_yaml(study_name) + return {"success": True, "yaml_str": result} + except Exception as e: + return {"success": False, "error": str(e)} # --- MCP PROMPT RESOURCES --- diff --git a/simulators/training_worker.py b/simulators/training_worker.py index 4b72681..e577f11 100644 --- a/simulators/training_worker.py +++ b/simulators/training_worker.py @@ -33,7 +33,7 @@ def simulate_training_epoch( res_perf = 0.06 # 3. Model capacity (wide vs narrow U-Net channel widths) - cap = params.get("model_capacity") or params.get("encoder_name", "narrow") + cap = params.get("model_capacity", "narrow") if cap in ("wide", "resnet50"): enc_perf = 0.04 elif cap in ("narrow", "resnet34", "efficientnet-b0"): diff --git a/src/analytics.py b/src/analytics.py index ce1b85a..d73ef4d 100644 --- a/src/analytics.py +++ b/src/analytics.py @@ -59,8 +59,8 @@ def bin_trials(study, db_metrics: Dict[int, Any], search_space: Dict[str, Any]) if t.state == TrialState.COMPLETE: s = get_score(t, study) score = s if s is not None else 0.0 - l = get_loss(t, study) - loss = l if l is not None else 0.0 + loss_val = get_loss(t, study) + loss = loss_val if loss_val is not None else 0.0 metric = db_metrics.get(t._trial_id, {}) score = metric.get("primary_score") if metric.get("primary_score") is not None else score @@ -185,11 +185,9 @@ def get_fanova_importances(study, config: Dict[str, Any]) -> Dict[str, float]: except Exception: return {} - aliases = config.get("legacy_param_aliases", {}) display: Dict[str, float] = {} for param, value in importances.items(): - canonical = aliases.get(param, param) - label = param_display_name(canonical, config) + label = param_display_name(param, config) display[label] = max(display.get(label, 0.0), float(value)) return display @@ -336,59 +334,12 @@ def compute_fidelity_durations(study, config: Dict[str, Any]) -> Dict[str, Any]: # --- VRAM telemetry --- -def fit_vram_model(trials: List, db_metrics: Dict[int, Any], train_param: str) -> Optional[Dict[str, Any]]: - points: List[tuple] = [] - for t in trials: - if t.state != TrialState.COMPLETE: - continue - metric = db_metrics.get(t._trial_id, {}) - oom = metric.get("oom_triggered") or t.user_attrs.get("oom_triggered", False) - if oom: - continue - vram = metric.get("max_vram_gb") or t.user_attrs.get("max_vram_gb") - if vram is None: - continue - bs = t.params.get("batch_size") - res = t.params.get(train_param) - if bs is not None and res is not None: - try: - points.append((float(bs), float(res), float(vram))) - except (ValueError, TypeError): - continue - - if len(points) < 6: - return None - - X = [p[0] * (p[1] ** 2) for p in points] - Y = [p[2] for p in points] - if len(set(X)) < 2: - return None - - n = len(points) - sum_x = sum(X) - sum_y = sum(Y) - sum_xx = sum(x * x for x in X) - sum_xy = sum(X[i] * Y[i] for i in range(n)) - denom = n * sum_xx - sum_x * sum_x - if abs(denom) < 1e-12: - return None - - slope = (n * sum_xy - sum_x * sum_y) / denom - intercept = (sum_y - slope * sum_x) / n - ssr = sum((Y[i] - (slope * X[i] + intercept)) ** 2 for i in range(n)) - rse = (ssr / (n - 2)) ** 0.5 if n > 2 else 0.0 - return {"slope": slope, "intercept": intercept, "n_points": n, "rse": rse} - - -def compute_vram_telemetry(study, db_metrics: Dict[int, Any], search_space: Dict[str, Any], config: Dict[str, Any]) -> Dict[str, Any]: - ev = config.get("eval_protocol", {}) - train_param = ev.get("train_resolution_param", "resolution") +def compute_vram_telemetry(study, db_metrics: Dict[int, Any]) -> Dict[str, Any]: trials = list(study.trials) - model = fit_vram_model(trials, db_metrics, train_param) - gpu_capacity_gb = 0.0 gpu_models: List[str] = [] oom_count = 0 + oom_trials: List[Dict[str, Any]] = [] for t in trials: metric = db_metrics.get(t._trial_id, {}) @@ -401,50 +352,19 @@ def compute_vram_telemetry(study, db_metrics: Dict[int, Any], search_space: Dict gpu_models.append(gpu) if oom: oom_count += 1 + oom_trials.append({ + "trial_id": t.number, + "params": dict(t.params), + "peak_vram_gb": float(vram) if vram else 0.0, + }) gpu_model = max(set(gpu_models), key=gpu_models.count) if gpu_models else "Unknown" - oom_risk = None - - if model and gpu_capacity_gb > 0: - max_bs = None - bs_info = search_space.get("batch_size", {}) - if bs_info.get("type") == "categorical": - active_bs = bs_info.get("active", []) - if active_bs: - max_bs = max(active_bs) - else: - max_bs = bs_info.get("max") - - max_res = None - res_info = search_space.get(train_param, {}) - if res_info.get("type") == "categorical": - active_res = res_info.get("active", []) - if active_res: - max_res = max(active_res) - else: - max_res = res_info.get("max") - - if max_bs is not None and max_res is not None: - predicted_mean_vram = model["slope"] * (float(max_bs) * (float(max_res) ** 2)) + model["intercept"] - margin = max(1.0, 1.96 * model["rse"]) - predicted_max_vram = predicted_mean_vram + margin - if predicted_max_vram > 0.9 * gpu_capacity_gb: - oom_risk = { - "max_batch_size": max_bs, - "max_resolution": max_res, - "predicted_mean_vram_gb": predicted_mean_vram, - "margin_gb": margin, - "predicted_max_vram_gb": predicted_max_vram, - "gpu_capacity_gb": gpu_capacity_gb, - "risk_level": "high" if predicted_max_vram > gpu_capacity_gb else "medium", - } return { "gpu_model": gpu_model, "gpu_capacity_gb": gpu_capacity_gb, "oom_count": oom_count, - "vram_model": model, - "bounds_oom_risk": oom_risk, + "oom_trials": oom_trials, } @@ -600,7 +520,7 @@ def build_compacted_packet( pareto_numbers = pareto_trial_numbers_deploy_aware(study, config) if len(study.directions) > 1 else [] boundary_hits = check_boundary_hits(study, pareto_numbers, search_space) fidelity_durations = compute_fidelity_durations(study, config) - vram_telemetry = compute_vram_telemetry(study, db_metrics, search_space, config) + vram_telemetry = compute_vram_telemetry(study, db_metrics) return { "study_name": study_name, diff --git a/src/db_manager.py b/src/db_manager.py index 8458234..a51e993 100644 --- a/src/db_manager.py +++ b/src/db_manager.py @@ -2,18 +2,15 @@ import contextlib import logging import os -from sqlalchemy import create_engine +from sqlalchemy import create_engine, event from sqlalchemy.orm import sessionmaker, Session from .schema import Base - from .settings import settings logger = logging.getLogger(__name__) DATABASE_URL = settings.database_url -from sqlalchemy import event - connect_args = {} if DATABASE_URL.startswith("sqlite"): connect_args["check_same_thread"] = False @@ -67,10 +64,6 @@ def init_db(): # Run additive migrations for altered tables _apply_additive_migrations() - # Migrate data from the legacy segmentation_metrics table to trial_results if present. - if "segmentation_metrics" in existing_tables: - _migrate_segmentation_metrics_to_trial_results() - # Additive, idempotent column migrations for tables that predate a new field. # Keeps an already-populated SQLite DB from 500ing when the ORM adds a nullable column. @@ -118,63 +111,6 @@ def _apply_additive_migrations(): print(f"Error migrating column {col_name} in {table}: {e}") -def _migrate_segmentation_metrics_to_trial_results(): - from sqlalchemy import inspect, text - try: - with engine.begin() as conn: - # Check if trial_results table exists and is empty - res = conn.execute(text("SELECT COUNT(*) FROM trial_results")).fetchone() - if res and res[0] > 0: - return - - study_name = settings.study_name - try: - s_res = conn.execute(text("SELECT study_name FROM study_status LIMIT 1")).fetchone() - if s_res: - study_name = s_res[0] - else: - s_res = conn.execute(text("SELECT study_name FROM study_reviews LIMIT 1")).fetchone() - if s_res: - study_name = s_res[0] - except Exception as e: - logger.warning(f"Failed to extract study_name during DB migration: {e}") - - inspector = inspect(engine) - seg_cols = {c["name"] for c in inspector.get_columns("segmentation_metrics")} - - cols_to_select = [] - cols_to_insert = [] - - mapping = { - "trial_id": "trial_id", - "epoch_reached": "epoch_reached", - "final_dice_score": "primary_score", - "final_bce_loss": "primary_loss", - "val_loss_history": "score_history_json", - "weights_path": "weights_path", - "gpu_model": "gpu_model", - "max_vram_gb": "max_vram_gb", - "oom_triggered": "oom_triggered", - "created_at": "created_at" - } - - for old_col, new_col in mapping.items(): - if old_col in seg_cols: - cols_to_select.append(old_col) - cols_to_insert.append(new_col) - - if cols_to_select: - select_clause = ", ".join(cols_to_select) - insert_clause = ", ".join(cols_to_insert) - - stmt = f""" - INSERT INTO trial_results (study_name, {insert_clause}) - SELECT :study_name, {select_clause} FROM segmentation_metrics - """ - conn.execute(text(stmt), {"study_name": study_name}) - print("Successfully migrated data from segmentation_metrics to trial_results.") - except Exception as e: - print(f"Error migrating segmentation_metrics data to trial_results: {e}") @contextlib.contextmanager diff --git a/src/hpo_client.py b/src/hpo_client.py index e777b31..5a3f313 100644 --- a/src/hpo_client.py +++ b/src/hpo_client.py @@ -397,14 +397,20 @@ def complete( # Auto-detect defaults env = dict(self.detected_env) if hasattr(self, "detected_env") else {} - # Override with explicit values if passed - if git_commit is not None: env["git_commit"] = git_commit - if python_version is not None: env["python_version"] = python_version - if cuda_version is not None: env["cuda_version"] = cuda_version - if pip_freeze is not None: env["pip_freeze"] = pip_freeze - if dataset_version is not None: env["dataset_version"] = dataset_version - if hostname is not None: env["hostname"] = hostname - if platform is not None: env["platform"] = platform + if git_commit is not None: + env["git_commit"] = git_commit + if python_version is not None: + env["python_version"] = python_version + if cuda_version is not None: + env["cuda_version"] = cuda_version + if pip_freeze is not None: + env["pip_freeze"] = pip_freeze + if dataset_version is not None: + env["dataset_version"] = dataset_version + if hostname is not None: + env["hostname"] = hostname + if platform is not None: + env["platform"] = platform # Merge into payload for k, v in env.items(): diff --git a/src/hpo_config.py b/src/hpo_config.py index 06babfd..b825fb6 100644 --- a/src/hpo_config.py +++ b/src/hpo_config.py @@ -40,42 +40,6 @@ "param_labels": {}, } -# Legacy defaults for U-Net crack segmentation studies (config_version = 1). -LEGACY_DEFAULT_HPO_CONFIG: Dict[str, Any] = { - "metric_loss_label": "BCE", - "metric_score_label": "Dice", - "validation_rules": { - "score_min": 0.0, - "loss_min": 0.0, - "max_epoch_jump": 0.5, - "enabled": True, - }, - "eval_protocol": { - "enabled": False, - "fixed_resolution": None, - "train_resolution_param": "resolution", - "fixed_dice_attr": "dice_eval_fixed", - "fixed_bce_attr": "bce_eval_fixed", - "dice_train_label": "Dice (train)", - "dice_fixed_label": "Dice (eval)", - "use_fixed_metric_for_pruning": True, - "prune_min_epoch": 5, - "prune_compare_same_resolution_only": True, - "prune_exclude_low_res_from_baseline": True, - "pareto_deploy_resolution_only": True, - "low_train_res_warning": 384, - "patch_size_below_512": 256, - "patch_size_at_512_plus": 448, - }, - "param_labels": {}, - "legacy_param_aliases": {"encoder_name": "model_capacity"}, - "legacy_capacity_values": { - "resnet34": "narrow", - "efficientnet-b0": "narrow", - "resnet50": "wide", - }, -} - def load_hpo_config(study_name: Optional[str] = None) -> Dict[str, Any]: from .settings import settings @@ -93,21 +57,8 @@ def load_hpo_config(study_name: Optional[str] = None) -> Dict[str, Any]: row = session.query(SystemConfiguration).filter_by( study_name=study_name, config_key="hpo_config" ).first() - if not row and study_name != "_global": - # Fallback to _global config in DB - row = session.query(SystemConfiguration).filter_by( - study_name="_global", config_key="hpo_config" - ).first() if row: - loaded_data = json.loads(row.config_value) - # If we fell back to _global and the current study is NOT a legacy U-Net study, - # check if the global config is legacy (version 1). If it is, ignore it to - # prevent legacy poisoning of generic studies. - if row.study_name == "_global": - if loaded_data.get("config_version", 1) == 1: - loaded_data = None - if loaded_data: - data = loaded_data + data = json.loads(row.config_value) except Exception as e: print(f"Error loading hpo_config from DB: {e}") @@ -120,25 +71,14 @@ def load_hpo_config(study_name: Optional[str] = None) -> Dict[str, Any]: except Exception as e: logger.warning(f"Failed to seed hpo_config in DB: {e}") - config_version = data.get("config_version", 2) - defaults = LEGACY_DEFAULT_HPO_CONFIG if config_version == 1 else DEFAULT_HPO_CONFIG + defaults = DEFAULT_HPO_CONFIG try: merged = copy.deepcopy(defaults) - merged.update({k: v for k, v in data.items() if k not in ("eval_protocol", "param_labels", "legacy_param_aliases", "legacy_capacity_values", "validation_rules")}) + merged.update({k: v for k, v in data.items() if k not in ("eval_protocol", "param_labels", "validation_rules")}) merged["eval_protocol"] = {**defaults.get("eval_protocol", {}), **data.get("eval_protocol", {})} merged["validation_rules"] = {**defaults.get("validation_rules", {}), **data.get("validation_rules", {})} merged["param_labels"] = {**defaults.get("param_labels", {}), **data.get("param_labels", {})} - if "legacy_param_aliases" in defaults or "legacy_param_aliases" in data: - merged["legacy_param_aliases"] = { - **defaults.get("legacy_param_aliases", {}), - **data.get("legacy_param_aliases", {}), - } - if "legacy_capacity_values" in defaults or "legacy_capacity_values" in data: - merged["legacy_capacity_values"] = { - **defaults.get("legacy_capacity_values", {}), - **data.get("legacy_capacity_values", {}), - } return merged except Exception: return copy.deepcopy(defaults) @@ -151,9 +91,7 @@ def save_hpo_config(config: Dict[str, Any], study_name: Optional[str] = None) -> if not study_name or not study_name.strip(): raise ValueError("study_name cannot be empty.") - # Enforce config_version on save to prevent client downgrades config = dict(config) - config["config_version"] = config.get("config_version", 2) try: from .db_manager import get_db_session @@ -191,13 +129,5 @@ def param_display_name(param: str, config: Optional[Dict[str, Any]] = None, stud def normalize_trial_params(params: Dict[str, Any], config: Optional[Dict[str, Any]] = None, study_name: Optional[str] = None) -> Dict[str, Any]: - """Map legacy param names/values for display and workers.""" - config = config or load_hpo_config(study_name) - out = dict(params) - for old, new in config.get("legacy_param_aliases", {}).items(): - if old in out and new not in out: - val = out.pop(old) - mapped = config.get("legacy_capacity_values", {}).get(val, val) - out[new] = mapped - return out + return dict(params) diff --git a/src/manifest.py b/src/manifest.py index 1d284e5..23be9cd 100644 --- a/src/manifest.py +++ b/src/manifest.py @@ -438,7 +438,7 @@ def _manifest_to_hpo_config(data: Dict[str, Any]) -> Dict[str, Any]: obj_label = obj.get("label", obj_name) if obj_name == primary_score: score_label = obj_label - elif "loss" in obj_name.lower() or "bce" in obj_name.lower(): + elif "loss" in obj_name.lower(): loss_label = obj_label eval_proto = data.get("eval_protocol", {}) @@ -456,7 +456,6 @@ def _manifest_to_hpo_config(data: Dict[str, Any]) -> Dict[str, Any]: max_epoch_jump = rules.get("max_epoch_jump") hpo_config = { - "config_version": 2, "metric_loss_label": loss_label, "metric_score_label": score_label, "eval_protocol": { diff --git a/src/metrics.py b/src/metrics.py index 5c64781..63d66ce 100644 --- a/src/metrics.py +++ b/src/metrics.py @@ -186,8 +186,8 @@ def _trial_metric_snapshot( def get_eval_attr_names(ev: dict) -> tuple[str, str]: """Return the score and loss user-attribute names for fixed-eval tracking.""" - score_fixed_key = ev.get("fixed_score_attr", ev.get("fixed_dice_attr", "score_eval_fixed")) - loss_fixed_key = ev.get("fixed_loss_attr", ev.get("fixed_bce_attr", "loss_eval_fixed")) + score_fixed_key = ev.get("fixed_score_attr", "score_eval_fixed") + loss_fixed_key = ev.get("fixed_loss_attr", "loss_eval_fixed") return score_fixed_key, loss_fixed_key def get_completed_trials(study) -> List[FrozenTrial]: diff --git a/src/onboarding.py b/src/onboarding.py index 9c50e74..3e617f2 100644 --- a/src/onboarding.py +++ b/src/onboarding.py @@ -4,9 +4,9 @@ from typing import Dict, Any, List, Optional from src.db_manager import get_db_session, DATABASE_URL +from src.schema import SystemConfiguration, StudyStatus logger = logging.getLogger(__name__) -from src.schema import SystemConfiguration, StudyStatus def initialize_study( study_name: str, @@ -18,10 +18,6 @@ def initialize_study( directions: Optional[List[str]] = None ) -> str: """Initializes a new study: creates Optuna study and stores search space, config, context, and source files in DB.""" - if "config_version" not in hpo_config: - hpo_config = dict(hpo_config) - hpo_config["config_version"] = 2 - try: # Create Optuna study if multi_objective: diff --git a/src/pruning.py b/src/pruning.py index dfaa088..0ea61ff 100644 --- a/src/pruning.py +++ b/src/pruning.py @@ -59,13 +59,13 @@ def _epoch_composite_score(study, trial, epoch: int, ev: Dict[str, Any]) -> Opti if entry.get("epoch") == epoch: if use_fixed and entry.get("score_eval_fixed") is not None: s = entry.get("score_eval_fixed") - l = entry.get("loss_eval_fixed", entry.get("loss", 0.0)) + loss_val = entry.get("loss_eval_fixed", entry.get("loss", 0.0)) else: s = entry.get("score") - l = entry.get("loss") - if s is not None and l is not None: + loss_val = entry.get("loss") + if s is not None and loss_val is not None: scores.append(float(s)) - losses.append(float(l)) + losses.append(float(loss_val)) break # 3. Z-score normalize if we have enough history (>= 10 values) diff --git a/src/search_space.py b/src/search_space.py index 4f80448..a2cb9e2 100644 --- a/src/search_space.py +++ b/src/search_space.py @@ -3,13 +3,13 @@ import logging from typing import Optional, Dict, Any, List -logger = logging.getLogger(__name__) from fastapi import HTTPException from optuna.trial import TrialState from src.db_manager import get_db_session from src.schema import SystemConfiguration, CompactedPacket -from src.hpo_config import load_hpo_config + +logger = logging.getLogger(__name__) # Default search space definition DEFAULT_SEARCH_SPACE = { @@ -18,18 +18,6 @@ } -def _migrate_search_space(space: Dict[str, Any], study_name: Optional[str] = None) -> Dict[str, Any]: - """Rename legacy encoder_name → model_capacity for older studies/files.""" - config = load_hpo_config(study_name) - if "encoder_name" in space and "model_capacity" not in space: - enc = space.pop("encoder_name") - mapping = config.get("legacy_capacity_values", {}) - enc["active"] = [mapping.get(v, v) for v in enc.get("active", enc.get("options", []))] - enc["options"] = [mapping.get(v, v) for v in enc.get("options", enc.get("active", []))] - space["model_capacity"] = enc - return space - - def load_search_space(study_name: Optional[str] = None) -> Dict[str, Any]: from .settings import settings if not study_name: @@ -44,7 +32,7 @@ def load_search_space(study_name: Optional[str] = None) -> Dict[str, Any]: ).first() if row: space = json.loads(row.config_value) - return _migrate_search_space(space, study_name) + return space except Exception as e: print(f"Error loading search space from DB: {e}") diff --git a/src/suggest.py b/src/suggest.py index 77cedea..4958b6e 100644 --- a/src/suggest.py +++ b/src/suggest.py @@ -1,5 +1,7 @@ import json +import logging +import random from typing import Optional from pydantic import BaseModel from fastapi import HTTPException @@ -133,8 +135,6 @@ def handle_api_suggest_trial(req: SuggestRequest): if repaired: print(f"Repaired {repaired} corrupt categorical param row(s) in study '{req.study_name}'.") except Exception as e: - # Fallback: if there is a database issue during repair, do not crash suggestion - import logging logging.getLogger(__name__).warning(f"Failed to repair categorical param indices: {e}") space = load_search_space(req.study_name) hpo_config = load_hpo_config(req.study_name) @@ -174,10 +174,14 @@ def handle_api_suggest_trial(req: SuggestRequest): # Optuna does not support narrowing categorical distributions after the first # trial. TPE always samples from the full historical choice set. When a user # deactivates a choice (dashboard Settings), we must handle TPE sampling an - # inactive value. We retry up to 20 times, failing each attempt in Optuna. - # This tells TPE that the deactivated region is unproductive — a best-effort - # heuristic given Optuna's static-distribution design. On the final attempt, - # we substitute a random active choice instead of 500-ing the worker. + # inactive value. We FAIL the attempt and enqueue a trial with a random active + # value for the violated categorical(s) so the next study.ask() returns a valid + # trial immediately instead of retrying up to 20 times. TPE does not learn from + # FAIL trials, so narrowed categoricals revert to random selection for the + # remainder of the study — a bounded cost in practice (small active sets are + # exhaustively covered in a few trials, and TPE guidance on discrete dimensions + # is inherently weak). The final-attempt random fallback is kept as a + # belt-and-suspenders safety net. MAX_RESAMPLE_ATTEMPTS = 20 for attempt in range(MAX_RESAMPLE_ATTEMPTS): trial = study.ask() @@ -191,9 +195,7 @@ def handle_api_suggest_trial(req: SuggestRequest): params = suggest_params_from_space(study, trial, space) break except ValueError: - # TPE sampled an inactive categorical if attempt < MAX_RESAMPLE_ATTEMPTS - 1: - # Fail the trial and retry with a fresh one try: study.tell(trial.number, state=TrialState.FAIL) except Exception: @@ -201,9 +203,25 @@ def handle_api_suggest_trial(req: SuggestRequest): with get_db_session() as session: delete_lease_by_trial_id(session, trial_id) session.commit() + # Enqueue a trial with random active values for violated + # categoricals so the next study.ask() returns a valid trial. + fixed = _finalize_trial_params(dict(trial.params), space) + violated: dict = {} + for param, cfg in space.items(): + if isinstance(cfg, dict) and cfg.get("type") == "categorical": + active = list(cfg.get("active") or cfg.get("options") or []) + if active and fixed.get(param) not in active: + violated[param] = random.choice(active) + if violated: + try: + study.enqueue_trial(violated) + except Exception: + logging.getLogger(__name__).warning( + "enqueue_trial failed for violated categoricals %s", + list(violated.keys()), + ) continue # Final attempt — fallback: pick random active values for violated categoricals - import random fixed = _finalize_trial_params(dict(trial.params), space) for param, cfg in space.items(): if isinstance(cfg, dict) and cfg.get("type") == "categorical": diff --git a/studies/bridge_crack_study_model_card.md b/studies/bridge_crack_study_model_card.md deleted file mode 100644 index 8e0291d..0000000 --- a/studies/bridge_crack_study_model_card.md +++ /dev/null @@ -1,32 +0,0 @@ -# Study Model Card: bridge_crack_study - -## Executive Summary -This model card synthesizes results for study `bridge_crack_study`. - -- **Best Achieved Score (Dice):** 0.9901 -- **Optimal Hyperparameters:** - - `learning_rate`: 2.6272157099819104e-05 - - `batch_size`: 32 - - `resolution`: 256 - - `encoder_name`: resnet50 - - `loss_weight_ratio`: 0.6982048083978067 - -## Search Space Performance -- **Total Trials Evaluated:** 77 -- **Successful Runs:** 10 -- **Pruned Runs:** 33 -- **Failed/OOM Runs:** 34 - -### Key Parameter Importances (fANOVA) -- `train_resolution`: 0.5317 -- `batch_size`: 0.3536 -- `loss_weight_ratio`: 0.0833 -- `learning_rate`: 0.0314 - -## Telemetry Profile -- **GPU Device:** NVIDIA L4 -- **Peak VRAM Recorded:** 94.97 GB -- **OOM Failures:** 23 - ---- -*Generated by Pathfinder on 2026-06-09T21:32:59.308077* diff --git a/studies/unet_crack_segmentation_test_model_card.md b/studies/unet_crack_segmentation_test_model_card.md deleted file mode 100644 index f8d837c..0000000 --- a/studies/unet_crack_segmentation_test_model_card.md +++ /dev/null @@ -1,33 +0,0 @@ -# Study Model Card: unet_crack_segmentation_test - -## Executive Summary -This model card synthesizes results for study `unet_crack_segmentation_test`. - -- **Best Achieved Score (Dice):** 0.5000 -- **Optimal Hyperparameters:** - - `learning_rate`: 1.66144714685308e-05 - - `batch_size`: 32 - - `resolution`: 1024 - - `model_capacity`: narrow - - `loss_weight_ratio`: 0.5534297134135798 - -## Search Space Performance -- **Total Trials Evaluated:** 8 -- **Successful Runs:** 7 -- **Pruned Runs:** 0 -- **Failed/OOM Runs:** 1 - -### Key Parameter Importances (fANOVA) -- `learning_rate`: 0.4944 -- `batch_size`: 0.2712 -- `resolution`: 0.1175 -- `loss_weight_ratio`: 0.0701 -- `model_capacity`: 0.0468 - -## Telemetry Profile -- **GPU Device:** Unknown -- **Peak VRAM Recorded:** 0.00 GB -- **OOM Failures:** 1 - ---- -*Generated by Pathfinder on 2026-06-19T13:48:12.043832* diff --git a/tests/__init__.py b/tests/__init__.py index 625516d..1dd5e96 100644 --- a/tests/__init__.py +++ b/tests/__init__.py @@ -1,4 +1,5 @@ # Test package init — sets up shared test database and sys.path +# ruff: noqa: E402 import os import sys import tempfile diff --git a/tests/conftest.py b/tests/conftest.py index 3ecbf8e..183e991 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -4,6 +4,7 @@ every test file: it points HPO_DATABASE_URL at a throwaway SQLite file BEFORE any ``src.*`` import binds the SQLAlchemy engine, and exposes reusable fixtures. """ +# ruff: noqa: E402 import os import sys import tempfile diff --git a/tests/test_concurrency.py b/tests/test_concurrency.py index 337ac09..5edcfdf 100644 --- a/tests/test_concurrency.py +++ b/tests/test_concurrency.py @@ -4,7 +4,7 @@ Verifies that multiple concurrent workers each receive unique leased trials, and that no trial duplication occurs across simultaneous suggest requests. """ - +# ruff: noqa: E402 import os import sys @@ -31,12 +31,11 @@ def _cleanup(): import threading import uuid import optuna -import json from datetime import datetime, timedelta from optuna.trial import TrialState from src.db_manager import init_db, get_db_session -from src.schema import TrialLease, SystemConfiguration +from src.schema import TrialLease class TestConcurrencyLeases(unittest.TestCase): @@ -79,7 +78,7 @@ def worker_suggest(worker_id): trial_id=trial_id, study_name=study_name, leased_to=worker_id, - lease_expires_at=datetime.utcnow() + timedelta(seconds=LEASE_TTL) + lease_expires_at=datetime.now() + timedelta(seconds=LEASE_TTL) ) session.add(lease) session.commit() @@ -132,7 +131,7 @@ def test_expired_lease_is_not_double_assigned(self): trial_id=trial_a._trial_id, study_name=study_name, leased_to="worker-A", - lease_expires_at=datetime.utcnow() - timedelta(seconds=10) + lease_expires_at=datetime.now() - timedelta(seconds=10) ) session.add(lease) session.commit() @@ -145,7 +144,7 @@ def test_expired_lease_is_not_double_assigned(self): if trial_b._trial_id == trial_a._trial_id: # Same trial recycled — only one lease row should exist - matching = [l for l in all_leases if l.trial_id == trial_a._trial_id] + matching = [lease for lease in all_leases if lease.trial_id == trial_a._trial_id] self.assertEqual( len(matching), 1, "Recycled trial should have exactly one lease row" @@ -162,7 +161,7 @@ def test_heartbeat_refreshes_lease(self): LEASE_TTL = 300 # Create a lease about to expire (30 seconds remaining) - initial_expiry = datetime.utcnow() + timedelta(seconds=30) + initial_expiry = datetime.now() + timedelta(seconds=30) with get_db_session() as session: lease = TrialLease( trial_id=trial._trial_id, @@ -181,7 +180,7 @@ def test_heartbeat_refreshes_lease(self): leased_to=worker_id ).first() self.assertIsNotNone(lease, "Lease should exist before heartbeat") - lease.lease_expires_at = datetime.utcnow() + timedelta(seconds=LEASE_TTL) + lease.lease_expires_at = datetime.now() + timedelta(seconds=LEASE_TTL) session.commit() # Verify lease was extended @@ -203,7 +202,7 @@ def test_wrong_worker_cannot_refresh_lease(self): intruder_id = f"worker-intruder-{uuid.uuid4()}" trial = study.ask() - initial_expiry = datetime.utcnow() + timedelta(seconds=60) + initial_expiry = datetime.now() + timedelta(seconds=60) with get_db_session() as session: lease = TrialLease( trial_id=trial._trial_id, @@ -231,6 +230,56 @@ def test_wrong_worker_cannot_refresh_lease(self): self.assertIsNotNone(lease) self.assertEqual(lease.leased_to, owner_id) + def test_enqueue_trial_overrides_tpe_categorical(self): + """enqueue_trial with partial params overrides suggest_categorical for that param.""" + study, study_name = self._make_study() + + # Seed first trial to lock the categorical distribution in Optuna + trial0 = study.ask() + trial0.suggest_categorical("resolution", [256, 512, 1024]) + trial0.suggest_float("lr", 1e-4, 1e-2, log=True) + study.tell(trial0.number, state=TrialState.COMPLETE, values=[0.5, 0.5]) + + # Enqueue a trial with resolution=256 (partial fix) + study.enqueue_trial({"resolution": 256}) + + # The enqueued trial should return 256 for resolution, TPE-sampled for lr + trial1 = study.ask() + res = trial1.suggest_categorical("resolution", [256, 512, 1024]) + self.assertEqual(res, 256, + "Enqueued categorical value should override TPE sampling") + + lr = trial1.suggest_float("lr", 1e-4, 1e-2, log=True) + self.assertGreaterEqual(lr, 1e-4) + self.assertLessEqual(lr, 1e-2) + + def test_multiple_enqueued_trials_consumed_in_order(self): + """Multiple enqueued trials are consumed sequentially by ask() calls.""" + study, study_name = self._make_study() + + # Seed first trial to lock the categorical distribution + trial0 = study.ask() + trial0.suggest_categorical("resolution", [256, 512, 1024]) + trial0.suggest_float("lr", 1e-4, 1e-2, log=True) + study.tell(trial0.number, state=TrialState.COMPLETE, values=[0.5, 0.5]) + + study.enqueue_trial({"resolution": 256}) + study.enqueue_trial({"resolution": 512}) + + # FIFO + trial1 = study.ask() + res1 = trial1.suggest_categorical("resolution", [256, 512, 1024]) + self.assertEqual(res1, 256, "First enqueued trial should be consumed first") + study.tell(trial1.number, state=TrialState.COMPLETE, values=[0.6, 0.5]) + + trial2 = study.ask() + res2 = trial2.suggest_categorical("resolution", [256, 512, 1024]) + self.assertEqual(res2, 512, "Second enqueued trial should be consumed second") + study.tell(trial2.number, state=TrialState.COMPLETE, values=[0.6, 0.5]) + + self.assertNotEqual(trial1.number, trial2.number, + "Sequential enqueued trials must have distinct trial numbers") + if __name__ == "__main__": unittest.main() diff --git a/tests/test_health_tier.py b/tests/test_health_tier.py index 8b9072f..e998b36 100644 --- a/tests/test_health_tier.py +++ b/tests/test_health_tier.py @@ -1,3 +1,4 @@ +# ruff: noqa: E402 import os import sys @@ -25,7 +26,7 @@ def _cleanup(): from optuna.trial import TrialState from src.db_manager import init_db, get_db_session -from src.schema import TrialResult, StudyStatus +from src.schema import TrialResult from src.health import compute_health_tier diff --git a/tests/test_http_auth.py b/tests/test_http_auth.py index 1cbcf48..4da551f 100644 --- a/tests/test_http_auth.py +++ b/tests/test_http_auth.py @@ -49,18 +49,3 @@ def test_complete_requires_lease_for_inflight_trial(client, initialized_study): "history": [{"epoch": 1, "score": 0.6, "loss": 0.4}], "state": "COMPLETE"}, ) assert bad.status_code == 403 - - -def test_api_requires_token_when_configured(client, initialized_study, monkeypatch): - monkeypatch.setenv("HPO_SECRET_TOKEN", "sekret") - assert client.get(f"/api/study_details?study_name={initialized_study}").status_code == 401 - ok = client.get(f"/api/study_details?study_name={initialized_study}", headers={"X-HPO-Token": "sekret"}) - assert ok.status_code == 200 - - -def test_login_cookie_then_authorizes(client, initialized_study, monkeypatch): - monkeypatch.setenv("HPO_SECRET_TOKEN", "sekret") - assert client.post("/api/login", json={"token": "nope"}).status_code == 401 - assert client.post("/api/login", json={"token": "sekret"}).status_code == 200 - # The httpOnly cookie now authorizes /api requests without any header. - assert client.get(f"/api/study_details?study_name={initialized_study}").status_code == 200 diff --git a/tests/test_http_concurrency.py b/tests/test_http_concurrency.py index 3106a18..ec2bd55 100644 --- a/tests/test_http_concurrency.py +++ b/tests/test_http_concurrency.py @@ -50,5 +50,5 @@ def test_expired_lease_can_be_reclaimed_by_new_worker(initialized_study): from src.schema import TrialLease with get_db_session() as session: lease = session.query(TrialLease).filter_by(trial_id=trial_id).first() - lease.lease_expires_at = datetime.utcnow() - timedelta(seconds=30) + lease.lease_expires_at = datetime.now() - timedelta(seconds=30) assert _claim_in_new_session(initialized_study, trial_id, "owner-B") is True diff --git a/tests/test_manifest.py b/tests/test_manifest.py index 9f991b2..5e91bfc 100644 --- a/tests/test_manifest.py +++ b/tests/test_manifest.py @@ -2,16 +2,13 @@ import os import sys -import json import yaml import pytest import subprocess from src.manifest import ( validate_manifest, _manifest_params_to_search_space, - _manifest_to_hpo_config, - ParamType, - ObjectiveDirection + _manifest_to_hpo_config ) from src.db_manager import get_db_session from src.schema import SystemConfiguration @@ -263,36 +260,11 @@ def test_mappings(base_manifest_data): assert space["num_epochs"]["options"] == [15] config = _manifest_to_hpo_config(base_manifest_data) - assert config["config_version"] == 2 assert config["metric_score_label"] == "Score" assert config["metric_loss_label"] == "BCE Loss" assert config["eval_protocol"]["enabled"] is True assert config["eval_protocol"]["fixed_resolution"] == 512 -def test_hpo_config_versioning(): - from src.hpo_config import load_hpo_config, save_hpo_config - # Test fallback / default config yields version 2 - cfg_new = load_hpo_config("nonexistent_test_study_v2") - assert cfg_new["config_version"] == 2 - assert cfg_new["metric_score_label"] == "Score" - assert "legacy_param_aliases" not in cfg_new - - # Save a version 1 legacy configuration and verify it merges with legacy defaults - legacy_cfg = { - "config_version": 1, - "metric_score_label": "Dice", - "eval_protocol": { - "enabled": True, - "fixed_resolution": 256 - } - } - save_hpo_config(legacy_cfg, "legacy_test_study_v1") - cfg_legacy = load_hpo_config("legacy_test_study_v1") - assert cfg_legacy.get("config_version", 1) == 1 - assert cfg_legacy["metric_score_label"] == "Dice" - assert cfg_legacy["metric_loss_label"] == "BCE" - assert cfg_legacy["legacy_param_aliases"] == {"encoder_name": "model_capacity"} - def test_cli_validate_success(tmp_path, base_manifest_data): yaml_file = tmp_path / "manifest.yaml" with open(yaml_file, "w") as f: @@ -382,8 +354,6 @@ def test_api_endpoints(base_manifest_data): def test_manifest_metric_ordering(client, base_manifest_data): import optuna from src.onboarding import init_study_from_manifest_dict - from src.db_manager import get_db_session - from src.schema import TrialResult # 1. Order [maximize, minimize] -> (Dice, Loss) study_name_1 = "test_order_max_min" @@ -490,8 +460,7 @@ def test_single_objective_minimize(client, base_manifest_data): def test_deep_cleanup_on_force_overwrite(client, base_manifest_data): from src.onboarding import init_study_from_manifest_dict - from src.db_manager import get_db_session - from src.schema import SystemConfiguration, TrialResult + from src.schema import TrialResult study_name = "test_deep_cleanup_study" data = base_manifest_data.copy() diff --git a/tests/test_mcp_server.py b/tests/test_mcp_server.py new file mode 100644 index 0000000..20dfae6 --- /dev/null +++ b/tests/test_mcp_server.py @@ -0,0 +1,264 @@ +"""Tests for the MCP server tools — verify unified error handling, return shapes, and edge cases.""" + +import yaml + +from hpo_mcp_server import ( + get_study_data, + get_study_cards, + validate_manifest, + init_from_manifest, + export_manifest, + resource_grill, +) + +# --------------------------------------------------------------------------- +# Manifest YAML helpers +# --------------------------------------------------------------------------- + +VALID_MANIFEST_YAML = """study_name: mcp_test_study +metrics: + primary_score: score + objectives: + - name: score + label: Score + direction: maximize + - name: loss + label: Loss + direction: minimize +params: + - name: learning_rate + type: float_log + min: 0.00001 + max: 0.1 + - name: batch_size + type: categorical + options: [16, 32, 64] +worker: + entrypoint: python train.py +""" + + +# --------------------------------------------------------------------------- +# get_study_data +# --------------------------------------------------------------------------- + +def test_get_study_data_valid_study_with_completed_trial(client, initialized_study): + """Return a valid packet with counts and health when study has completed trials.""" + study_name = initialized_study + + resp = client.post( + "/api/suggest_trial", + json={"study_name": study_name, "worker_id": "w1"}, + ) + assert resp.status_code == 200 + trial_id = resp.json()["trial_id"] + + resp = client.post( + "/api/complete_trial", + json={ + "study_name": study_name, + "trial_id": trial_id, + "worker_id": "w1", + "epoch": 1, + "score": 0.80, + "loss": 0.20, + "weights_path": "model.pt", + "history": [{"epoch": 1, "score": 0.80, "loss": 0.20}], + "state": "COMPLETE", + }, + ) + assert resp.status_code == 200 + + result = get_study_data(study_name) + assert isinstance(result, dict) + assert result.get("study_name") == study_name + assert "counts" in result + assert result["counts"].get("complete", 0) >= 1 + assert "trial_bins" in result + assert "health" in result + + +def test_get_study_data_nonexistent_study(): + """Returns success=False, error when study doesn't exist.""" + result = get_study_data("nonexistent_study_xyz") + assert isinstance(result, dict) + assert result.get("success") is False + assert "error" in result + + +def test_get_study_data_empty_study(client, initialized_study): + """Returns a valid packet with zero completed trials for a fresh study.""" + result = get_study_data(initialized_study) + assert isinstance(result, dict) + assert result.get("study_name") == initialized_study + assert result.get("counts", {}).get("complete") == 0 + + +# --------------------------------------------------------------------------- +# get_study_cards +# --------------------------------------------------------------------------- + +def test_get_study_cards_valid_study_no_cards(initialized_study): + """Returns success=True, cards=[] when study exists but has no cards.""" + result = get_study_cards(initialized_study) + assert isinstance(result, dict) + assert result.get("success") is True + assert isinstance(result.get("cards"), list) + assert result.get("cards") == [] + + +def test_get_study_cards_nonexistent_study(): + """Returns success=False, error when study doesn't exist.""" + result = get_study_cards("nonexistent_study_xyz") + assert isinstance(result, dict) + assert result.get("success") is False + assert "error" in result + assert "not found" in result["error"] + + +def test_get_study_cards_no_argument(): + """Returns success=True with a list when no study_name is passed.""" + result = get_study_cards() + assert isinstance(result, dict) + assert result.get("success") is True + assert isinstance(result.get("cards"), list) + + +# --------------------------------------------------------------------------- +# validate_manifest +# --------------------------------------------------------------------------- + +def test_validate_manifest_valid_yaml(): + """Valid YAML with valid schema returns success=True, no errors.""" + result = validate_manifest(VALID_MANIFEST_YAML) + assert isinstance(result, dict) + assert result.get("success") is True + assert result.get("errors") == [] + + +def test_validate_manifest_garbage_string(): + """Non-YAML string returns success=False with YAML parse error.""" + result = validate_manifest("<<: *does_not_exist") + assert result.get("success") is False + assert "Invalid YAML structure" in result["errors"][0] + + +def test_validate_manifest_non_dict_yaml(): + """Valid YAML that is not a dict returns success=False.""" + result = validate_manifest('"just a string"') + assert result.get("success") is False + assert "Manifest root must be a dictionary" in result["errors"][0] + + +def test_validate_manifest_missing_study_name(): + """Valid YAML but missing required field returns errors.""" + manifest = yaml.safe_load(VALID_MANIFEST_YAML) + del manifest["study_name"] + yaml_str = yaml.dump(manifest) + result = validate_manifest(yaml_str) + assert result.get("success") is False + assert any("study_name" in e.lower() for e in result["errors"]) + + +# --------------------------------------------------------------------------- +# init_from_manifest +# --------------------------------------------------------------------------- + +def test_init_from_manifest_fresh_study(): + """Fresh manifest with force=False succeeds.""" + data = yaml.safe_load(VALID_MANIFEST_YAML) + data["study_name"] = "mcp_test_init_fresh" + yaml_str = yaml.dump(data) + result = init_from_manifest(yaml_str, force=True) + assert isinstance(result, dict) + assert result.get("success") is True + assert "initialized" in result.get("message", "").lower() + + +def test_init_from_manifest_duplicate_no_force(): + """Same study name without force returns error.""" + data = yaml.safe_load(VALID_MANIFEST_YAML) + data["study_name"] = "mcp_test_init_dup" + yaml_str = yaml.dump(data) + + result1 = init_from_manifest(yaml_str, force=True) + assert result1.get("success") is True + + result2 = init_from_manifest(yaml_str, force=False) + assert result2.get("success") is False + assert "already exists" in result2.get("error", "") + + +def test_init_from_manifest_duplicate_with_force(): + """Same study name with force=True overwrites successfully.""" + data = yaml.safe_load(VALID_MANIFEST_YAML) + data["study_name"] = "mcp_test_init_force" + yaml_str = yaml.dump(data) + + init_from_manifest(yaml_str, force=True) + result = init_from_manifest(yaml_str, force=True) + assert result.get("success") is True + assert "initialized" in result.get("message", "").lower() + + +def test_init_from_manifest_garbage_yaml(): + """Garbage YAML string returns success=False.""" + result = init_from_manifest("<<: *does_not_exist", force=False) + assert result.get("success") is False + assert "Invalid YAML structure" in result.get("error", "") + + +def test_init_from_manifest_schema_errors(): + """Valid YAML with missing required fields returns success=False.""" + data = yaml.safe_load(VALID_MANIFEST_YAML) + del data["params"] + yaml_str = yaml.dump(data) + result = init_from_manifest(yaml_str, force=True) + assert result.get("success") is False + assert "Cannot initialize study" in result.get("error", "") + + +# --------------------------------------------------------------------------- +# export_manifest +# --------------------------------------------------------------------------- + +def test_export_manifest_valid_study(): + """Returns success=True with parseable YAML string for an existing study.""" + data = yaml.safe_load(VALID_MANIFEST_YAML) + study_name = "mcp_test_export_valid" + data["study_name"] = study_name + yaml_str = yaml.dump(data) + init_from_manifest(yaml_str, force=True) + + result = export_manifest(study_name) + assert isinstance(result, dict) + assert result.get("success") is True + assert "yaml_str" in result + yaml_str = result["yaml_str"] + assert isinstance(yaml_str, str) + assert len(yaml_str) > 0 + + reparsed = yaml.safe_load(yaml_str) + assert reparsed.get("study_name") == study_name + + +def test_export_manifest_nonexistent_study(): + """Returns success=False with error for a non-existent study (no longer raises).""" + result = export_manifest("nonexistent_study_xyz") + assert isinstance(result, dict) + assert result.get("success") is False + assert "error" in result + assert "not found" in result["error"].lower() + + +# --------------------------------------------------------------------------- +# resource_grill +# --------------------------------------------------------------------------- + +def test_resource_grill_static_content(): + """Resource returns a non-empty string containing expected onboarding keywords.""" + content = resource_grill() + assert isinstance(content, str) + assert len(content) > 0 + assert "validate_manifest" in content + assert "AGENTS.md" in content diff --git a/tests/test_metrics.py b/tests/test_metrics.py index 41a4f0a..c43b168 100644 --- a/tests/test_metrics.py +++ b/tests/test_metrics.py @@ -1,7 +1,7 @@ +# ruff: noqa: E402 import os import sys import unittest -from unittest.mock import patch _project_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) if _project_root not in sys.path: @@ -17,10 +17,7 @@ get_loss_from_dirs, get_score, get_score_from_dirs, - loss_objective_index, - score_objective_index, ) -from src.analytics import get_fanova_importances def _complete_trial(study, values, params=None): @@ -34,16 +31,6 @@ def _complete_trial(study, values, params=None): class TestMetrics(unittest.TestCase): - def test_legacy_minimize_maximize_indices(self): - study = optuna.create_study(directions=["minimize", "maximize"]) - self.assertEqual(loss_objective_index(study), 0) - self.assertEqual(score_objective_index(study), 1) - - def test_reversed_maximize_minimize_indices(self): - study = optuna.create_study(directions=["maximize", "minimize"]) - self.assertEqual(score_objective_index(study), 0) - self.assertEqual(loss_objective_index(study), 1) - def test_single_objective_maximize(self): study = optuna.create_study(direction="maximize") trial = _complete_trial(study, 0.75) @@ -58,7 +45,6 @@ def test_single_objective_minimize(self): def test_get_best_trial_and_score(self): study = optuna.create_study(directions=["minimize", "maximize"]) - t0 = _complete_trial(study, [0.5, 0.6]) t1 = _complete_trial(study, [0.3, 0.9]) completed = [t for t in study.trials if t.state == TrialState.COMPLETE] best = get_best_trial(completed, study) @@ -71,26 +57,6 @@ def test_get_score_from_dirs(self): self.assertEqual(get_score_from_dirs(trial, study.directions), 0.8) self.assertEqual(get_loss_from_dirs(trial, study.directions), 0.2) - def test_fanova_importances_no_score_objective(self): - study = optuna.create_study( - study_name="test_fanova_no_score_" + self._testMethodName, - storage="sqlite:///:memory:", - directions=["minimize", "minimize"], - load_if_exists=True, - ) - _complete_trial(study, [0.5, 0.3], {"x": 0.1}) - _complete_trial(study, [0.4, 0.2], {"x": 0.9}) - result = get_fanova_importances(study, {}) - self.assertEqual(result, {}) - - def test_fanova_importances_mocked_score_index_none(self): - study = optuna.create_study(directions=["minimize", "maximize"]) - _complete_trial(study, [0.5, 0.6]) - _complete_trial(study, [0.4, 0.7]) - with patch("src.analytics.score_objective_index", return_value=None): - result = get_fanova_importances(study, {}) - self.assertEqual(result, {}) - if __name__ == "__main__": unittest.main() diff --git a/tests/test_onboarding_pipeline.py b/tests/test_onboarding_pipeline.py index 36d4627..a815876 100644 --- a/tests/test_onboarding_pipeline.py +++ b/tests/test_onboarding_pipeline.py @@ -1,46 +1,7 @@ -import subprocess import pytest from pydantic import ValidationError from src.reporting import ReportEpochRequest, CompleteTrialRequest -def test_hpo_cli_validate_standalone(): - """Verify that hpo_cli validate works independently.""" - # Create a temporary manifest - import tempfile - import os - - valid_manifest = """study_name: test_validate_study -metrics: - primary_score: loss - objectives: - - name: loss - direction: minimize - label: Loss -params: - - name: x - type: float - min: 0.0 - max: 1.0 -worker: - entrypoint: python test.py -""" - with tempfile.NamedTemporaryFile("w", delete=False, suffix=".yaml") as f: - f.write(valid_manifest) - manifest_path = f.name - - import sys - try: - # Run CLI validate - result = subprocess.run( - [sys.executable, "hpo_cli.py", "validate", manifest_path], - capture_output=True, - text=True - ) - assert result.returncode == 0 - assert "Manifest is valid" in result.stdout - finally: - os.remove(manifest_path) - def test_report_epoch_single_metric(): """Verify that ReportEpochRequest accepts single metrics and rejects empty metrics.""" # Only score diff --git a/tests/test_pruning.py b/tests/test_pruning.py index a27fb14..7d88538 100644 --- a/tests/test_pruning.py +++ b/tests/test_pruning.py @@ -1,3 +1,4 @@ +# ruff: noqa: E402 import os import sys @@ -22,7 +23,6 @@ def _cleanup(): import unittest import optuna -from optuna.trial import TrialState from src.db_manager import init_db from src.pruning import _epoch_composite_score @@ -65,23 +65,6 @@ def test_composite_score_thin_data(self): score = _epoch_composite_score(self.study, frozen, 1, {"enabled": False}) self.assertEqual(score, 0.0) # 0.5 - 0.5 = 0.0 - def test_composite_score_zscore_and_zero_variance_clamp(self): - """11 trials with identical scores (zero variance) -> Z-score with epsilon clamp returns 0.0.""" - last_number = None - for i in range(11): - self.study.enqueue_trial({"learning_rate": 1e-3, "batch_size": 16}) - t = self.study.ask() - self.study._storage.set_trial_user_attr(t._trial_id, "history", [{"epoch": 1, "score": 0.8, "loss": 0.2}]) - if i < 10: - self.study.tell(t.number, [0.2, 0.8]) - else: - last_number = t.number - - frozen = self._get_frozen_trial(last_number) - score = _epoch_composite_score(self.study, frozen, 1, {"enabled": False}) - # z_score = (0.8 - 0.8) / 1.0 = 0.0, z_loss = -(0.2 - 0.2) / 1.0 = 0.0 - self.assertEqual(score, 0.0) - def test_composite_score_zscore_normal_variance(self): """11 trials with varying scores -> Z-score normalization produces meaningful non-zero result.""" score_vals = [0.3, 0.4, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85] diff --git a/tests/test_robustness_features.py b/tests/test_robustness_features.py index 47fc2ae..05338de 100644 --- a/tests/test_robustness_features.py +++ b/tests/test_robustness_features.py @@ -2,13 +2,8 @@ import tempfile import json import pytest -import uuid -import math -import sqlite3 -from fastapi.testclient import TestClient -from optuna.trial import TrialState -from src.db_manager import get_db_session, DATABASE_URL -from src.schema import TrialResult, SystemConfiguration +from src.db_manager import get_db_session +from src.schema import TrialResult import hpo_cli def test_zero_metric_rejection(client, initialized_study): @@ -151,32 +146,6 @@ class ImportArgs: except Exception: pass -def test_cli_backup_command(): - temp_dir = tempfile.mkdtemp() - backup_path = os.path.join(temp_dir, "test_backup.db") - - class BackupArgs: - output = backup_path - - try: - hpo_cli.cmd_backup(BackupArgs()) - except SystemExit as e: - assert e.code == 0 - - assert os.path.exists(backup_path) - conn = sqlite3.connect(backup_path) - cursor = conn.cursor() - cursor.execute("SELECT name FROM sqlite_master WHERE type='table'") - tables = [t[0] for t in cursor.fetchall()] - assert "trial_results" in tables - conn.close() - - try: - os.remove(backup_path) - os.rmdir(temp_dir) - except Exception: - pass - def test_flat_env_fields_api(client, initialized_study): resp = client.post("/api/suggest_trial", json={"study_name": initialized_study, "worker_id": "env_worker"}) @@ -422,3 +391,42 @@ class ImportArgs: os.rmdir(temp_dir) except Exception: pass + + +def test_search_space_update(client, initialized_study): + """Narrowing numeric bounds, toggling categorical options, and rejection of invalid inputs.""" + study = initialized_study + + # 1. Narrow learning_rate bounds + resp = client.post( + "/api/update_search_space", + json={"study_name": study, "learning_rate": {"min": 1e-4, "max": 1e-3}}, + ) + assert resp.status_code == 200 + space = resp.json()["space"] + assert space["learning_rate"]["min"] == 1e-4 + assert space["learning_rate"]["max"] == 1e-3 + + # 2. Restrict categorical options + resp = client.post( + "/api/update_search_space", + json={"study_name": study, "batch_size": {"active": [4, 8]}}, + ) + assert resp.status_code == 200 + assert set(resp.json()["space"]["batch_size"]["active"]) == {4, 8} + + # 3. Reject unknown parameter + resp = client.post( + "/api/update_search_space", + json={"study_name": study, "nonexistent_param": {"min": 0}}, + ) + assert resp.status_code == 400 + assert "not recognized" in resp.json()["detail"] + + # 4. Reject min >= max + resp = client.post( + "/api/update_search_space", + json={"study_name": study, "learning_rate": {"min": 0.01, "max": 0.001}}, + ) + assert resp.status_code == 400 + assert "strictly less" in resp.json()["detail"] diff --git a/tests/test_vram_telemetry.py b/tests/test_vram_telemetry.py index 94e1886..d3c3fb6 100644 --- a/tests/test_vram_telemetry.py +++ b/tests/test_vram_telemetry.py @@ -1,9 +1,7 @@ -import pytest -from optuna.trial import TrialState from src.analytics import build_study_packet -def test_vram_telemetry_flows_to_review_packet(client, initialized_study): +def test_vram_telemetry_flows_to_study_packet(client, initialized_study): """ Submits a trial with VRAM data and verifies the review packet contains populated VRAM telemetry rather than empty defaults. @@ -48,3 +46,10 @@ def test_vram_telemetry_flows_to_review_packet(client, initialized_study): assert vram.get("oom_count", -1) >= 0, ( f"Expected oom_count >= 0, got {vram.get('oom_count')}" ) + + # Regression model fields removed — simplify verified + assert "vram_model" not in vram, "vram_model should not be in simplified telemetry" + assert "bounds_oom_risk" not in vram, "bounds_oom_risk should not be in simplified telemetry" + + # Flat OOM trials list present (empty for non-OOM trial) + assert isinstance(vram.get("oom_trials"), list), "oom_trials should be a list" diff --git a/web/index.html b/web/index.html index 6a2c48e..2442a19 100644 --- a/web/index.html +++ b/web/index.html @@ -19,12 +19,12 @@ @@ -42,8 +42,7 @@

Dashboard

STUDY: +
/
WORKERS:0
@@ -107,21 +106,12 @@

-
Study HealthHEALTHY
+
Study HealthHEALTHY
-
Current State
— — — — —
Healthy

No anomalies detected. Search space is healthy.

+
Current State
Healthy

No anomalies detected.

-
-
Latest Coordinator Action
No coordinator action recorded
-
-
-
- Show audit history
-
-
-
@@ -133,14 +123,6 @@

Active Search Space
Loading space bounds...
-
- Remote Worker Setup -
-

export HPO_BROKER_URL=http://your-broker:8000

-

export HPO_STUDY_NAME=your_study_name

-

See docs/INTEGRATION.md for full worker setup instructions.

-
-
@@ -210,6 +192,8 @@

Score train column label

Score fixed column label
+
+ Advanced
@@ -237,6 +221,7 @@

Min loss threshold

Max epoch score jump (fraction)
+
@@ -253,26 +238,6 @@

Hyperparameters

-
diff --git a/web/js/api.js b/web/js/api.js index a818ba7..9e2e88e 100644 --- a/web/js/api.js +++ b/web/js/api.js @@ -25,6 +25,17 @@ async function populateStudyList() { if (study === activeStudy) opt.selected = true; select.appendChild(opt); }); + if (!activeStudy && studiesToShow.length > 0) { + select.value = studiesToShow[0]; + window.HPOState.session.studyName = studiesToShow[0]; + const url = new URL(window.location); + url.searchParams.set('study', studiesToShow[0]); + window.history.pushState({}, '', url); + window.fetchStudyDetails(); + window.fetchHpoConfig(); + window.fetchSearchSpace(); + window.fetchFanova(); + } } } } catch (err) { @@ -33,6 +44,7 @@ async function populateStudyList() { } async function fetchStudyDetails(throwOnError = false) { + if (!window.HPOState.session.studyName) return; try { const res = await fetch(`/api/study_details?study_name=${window.HPOState.session.studyName}`); if (!res.ok) { @@ -42,11 +54,7 @@ async function fetchStudyDetails(throwOnError = false) { const data = await res.json(); window.HPOState.data.latestStudyData = data; - const completedCount = data.trials.filter(t => t.state === "COMPLETE" || t.state === "FAIL" || t.state === "PRUNED").length; - if (window.HPOState.telemetry.lastCompletedCount !== null && completedCount !== window.HPOState.telemetry.lastCompletedCount) { - window.HPOState.ui.reviewPillDismissed = false; - } - window.HPOState.telemetry.lastCompletedCount = completedCount; + window.HPOState.telemetry.lastCompletedCount = data.trials.filter(t => t.state === "COMPLETE" || t.state === "FAIL" || t.state === "PRUNED").length; const select = document.getElementById("study-select"); if (select) select.value = data.study_name; @@ -58,8 +66,6 @@ async function fetchStudyDetails(throwOnError = false) { window.HPOState.data.review = data.review || null; const nComplete = data.completed_count ?? data.trials.filter((t) => t.state === "COMPLETE").length; updateStatConfidenceBanner(data.statistical_confidence || "low", nComplete); - renderPastReviews(data.past_reviews || []); - applyReviewUi(); updateAnalysisStatusTicker(); syncTelemetryFromTrials([...data.trials].sort((a, b) => b.number - a.number)); @@ -76,6 +82,7 @@ async function fetchStudyDetails(throwOnError = false) { } async function fetchHpoConfig() { + if (!window.HPOState.session.studyName) return; const studyName = window.HPOState.session.studyName || ''; try { const res = await fetch(`/api/hpo_config?study_name=${encodeURIComponent(studyName)}`); @@ -89,6 +96,7 @@ async function fetchHpoConfig() { } async function fetchSearchSpace() { + if (!window.HPOState.session.studyName) return; const studyName = window.HPOState.session.studyName || ''; try { const res = await fetch(`/api/search_space?study_name=${encodeURIComponent(studyName)}`); @@ -96,7 +104,6 @@ async function fetchSearchSpace() { window.HPOState.data.activeSearchSpace = await res.json(); renderSearchSpace(); renderDashboardSearchSpaceSummary(); - fetchPendingChanges(); if (window.HPOState.data.latestStudyData) { renderStudyDetails(window.HPOState.data.latestStudyData); } @@ -106,6 +113,7 @@ async function fetchSearchSpace() { } async function fetchFanova() { + if (!window.HPOState.session.studyName) return; try { const res = await fetch(`/api/fanova?study_name=${window.HPOState.session.studyName}`); if (!res.ok) return; diff --git a/web/js/charts_modal.js b/web/js/charts_modal.js index c3d0112..ea4a630 100644 --- a/web/js/charts_modal.js +++ b/web/js/charts_modal.js @@ -1,3 +1,72 @@ +function openTrialDetails(trialNumber) { + trialNumber = Number(trialNumber); + const trials = window.HPOState.data.trials || []; + const trial = trials.find(t => t.number === trialNumber); + if (!trial) return; + + document.getElementById("modal-trial-title").textContent = `Trial #${trial.number}`; + + const paramsEl = document.getElementById("modal-params"); + if (paramsEl) { + const params = trial.params_display || trial.params || {}; + paramsEl.innerHTML = Object.entries(params) + .map(([k, v]) => `
${k}
${v}
`) + .join(""); + } + + const oomSection = document.getElementById("modal-oom-section"); + if (oomSection) { + oomSection.style.display = trial.oom_triggered ? "block" : "none"; + } + + const state = trial.state || ""; + const titleEl = document.getElementById("modal-rationale-title"); + if (titleEl) { + titleEl.textContent = state === "COMPLETE" ? "Result" : "Status"; + } + + const reasoningEl = document.getElementById("modal-reasoning"); + if (reasoningEl) { + reasoningEl.textContent = state === "COMPLETE" + ? `Completed with ${trial.loss != null ? `loss ${Number(trial.loss).toFixed(4)}` : "unknown loss"} and ${trial.score != null ? `score ${Number(trial.score).toFixed(4)}` : "unknown score"}.` + : state === "PRUNED" ? "Pruned by median-rule early stopping." : + state === "FAIL" ? `Failed${trial.oom_triggered ? " (Out of Memory)" : ""}.` : + "Awaiting results."; + } + + const history = trial.history || []; + updateModalChart(history); + + const modal = document.getElementById("detail-modal"); + if (modal) modal.classList.add("active"); + window.HPOState.ui.isModalOpen = true; +} + +function closeModal(event) { + if (event && event.target !== event.currentTarget) return; + closeModalDirect(); +} + +function closeModalDirect() { + const modal = document.getElementById("detail-modal"); + if (modal) modal.classList.remove("active"); + window.HPOState.ui.isModalOpen = false; + if (window.HPOState.charts.modalHistory.instance) { + window.HPOState.charts.modalHistory.instance.destroy(); + window.HPOState.charts.modalHistory.instance = null; + } + if (window.HPOState.render.pendingRender) { + window.HPOState.render.pendingRender = false; + if (window.renderStudyDetails && window.HPOState.data.latestStudyData) { + window.renderStudyDetails(window.HPOState.data.latestStudyData); + } + } +} + +window.openTrialDetails = openTrialDetails; +window.closeModal = closeModal; +window.closeModalDirect = closeModalDirect; + function updateModalChart(history) { const ctx = document.getElementById("modal-history-chart")?.getContext("2d"); if (!ctx) return; diff --git a/web/js/charts_pareto.js b/web/js/charts_pareto.js index 85a24c0..10dc68b 100644 --- a/web/js/charts_pareto.js +++ b/web/js/charts_pareto.js @@ -45,9 +45,13 @@ function updateChart(trials, paretoSet, directions) { const trialNum = parseInt(match[1], 10); const trial = trials.find(t => t.number === trialNum); if (trial) { - const paramsList = Object.entries(trial.params) - .map(([k, v]) => `${k.replace(/_/g, " ")}: ${typeof v === 'number' && v % 1 !== 0 ? v.toFixed(4) : v}`) + const paramKeys = Object.keys(trial.params); + const first3 = paramKeys.slice(0, 3) + .map(k => `${k.replace(/_/g, " ")}: ${typeof trial.params[k] === 'number' && trial.params[k] % 1 !== 0 ? trial.params[k].toFixed(4) : trial.params[k]}`) .join(", "); + const paramsList = paramKeys.length > 3 + ? `${first3}, +${paramKeys.length - 3} more` + : first3; let labelText = `${rawPoint.label} | `; if (isSingleObj) { diff --git a/web/js/health.js b/web/js/health.js index b14ed29..698651f 100644 --- a/web/js/health.js +++ b/web/js/health.js @@ -1,66 +1,15 @@ -function renderHealthLatestAction() { - const body = document.getElementById("health-action-body"); - if (!body) return; - body.replaceChildren(); - const action = window.getLatestCoordinatorAction(window.HPOState.data.pastReviews, window.HPOState.data.pendingChanges); - if (!action) { - body.appendChild(Object.assign(document.createElement("div"), { className: "health-action-line health-action-muted", textContent: "No coordinator action recorded" })); - return; - } - body.appendChild(Object.assign(document.createElement("div"), { className: "health-action-line", textContent: action.label })); - body.appendChild(Object.assign(document.createElement("div"), { className: "health-action-target", textContent: action.detail })); - if (action.timestamp) { - body.appendChild(Object.assign(document.createElement("div"), { className: "health-action-time", textContent: window.formatReviewTimestamp(action.timestamp) })); - } - if (action.kind === "pending") { - body.appendChild(Object.assign(document.createElement("a"), { className: "health-action-link", href: "#search-space", textContent: "Review in Search Space →" })); - } -} - function renderHealthPanelExtras() { - const tier = window.HPOState.data.healthTier || "healthy", latestReview = (window.HPOState.data.pastReviews || [])[0], rating = latestReview?.health_rating; - const tierLine = document.getElementById("health-tier-line"), blocksEl = document.getElementById("health-rating-blocks"); + const tier = window.HPOState.data.healthTier || "healthy"; + const tierLine = document.getElementById("health-tier-line"); if (tierLine) tierLine.textContent = tier.charAt(0).toUpperCase() + tier.slice(1); - if (blocksEl) { - const blocks = window.buildHealthBlocks(rating); - blocksEl.textContent = blocks ? (blocks + (rating ? ` ${rating}/5` : "")) : "— — — — —"; - blocksEl.classList.toggle("health-blocks-empty", !blocks); - } - renderHealthLatestAction(); -} - -function renderPastReviews(reviews) { - const details = document.getElementById("audit-history-details"), summary = document.getElementById("audit-history-summary"), list = document.getElementById("past-reviews-list"); - if (!list) return; - window.HPOState.data.pastReviews = reviews || []; - list.replaceChildren(); - const count = (reviews || []).length; - if (summary) summary.textContent = count ? `Show audit history (${count} logs)` : "No audit history"; - if (details) details.style.display = count ? "" : "none"; - if (!count) { renderHealthPanelExtras(); return; } - reviews.forEach((r) => { - const row = Object.assign(document.createElement("div"), { className: "audit-history-row" + (r.policy_action === "no_change" ? " no-change" : "") }); - const parts = [`#${r.id}`, window.policyActionLabel(r.policy_action), window.formatReviewDate(r.created_at), (r.summary || "").slice(0, 80)].filter(Boolean); - row.appendChild(Object.assign(document.createElement("div"), { textContent: parts.join(" · ") })); - if (r.policy_action && r.policy_action !== "no_change") { - const btn = Object.assign(document.createElement("button"), { type: "button", className: "audit-flag-btn", textContent: r.quality_flagged ? "Unflag" : "Flag" }); - btn.addEventListener("click", () => toggleReviewFlag(r.id, !r.quality_flagged)); - row.appendChild(btn); - } - list.appendChild(row); - }); - renderHealthPanelExtras(); } function updateAnalysisStatusTicker() { const ticker = document.getElementById("analysis-status-ticker"), track = document.getElementById("analysis-ticker-track"); if (!ticker || !track) return; - const tier = window.HPOState.data.healthTier || "healthy", reason = window.HPOState.data.healthReason || window.HPOState.data.studyHealthReason || "", review = window.HPOState.data.review, confidence = window.HPOState.data.statisticalConfidence || "low", nComplete = window.HPOState.data.completedCount || 0; + const tier = window.HPOState.data.healthTier || "healthy", reason = window.HPOState.data.healthReason || window.HPOState.data.studyHealthReason || "", confidence = window.HPOState.data.statisticalConfidence || "low", nComplete = window.HPOState.data.completedCount || 0; const messages = []; if (tier === "watch" || tier === "intervene") messages.push(`${tier.charAt(0).toUpperCase() + tier.slice(1)}: ${reason}`); - if (review?.review_recommended && (review.reasons || []).length && !window.HPOState.ui.reviewPillDismissed) { - messages.push(`Coordinator review recommended — ` + review.reasons.map((r) => r.message || r.code).join(" · ")); - } if (confidence === "low") messages.push(`${nComplete} complete trial${nComplete === 1 ? "" : "s"}. Early signal; treat rankings as indicative.`); track.replaceChildren(); if (!messages.length) { @@ -68,7 +17,7 @@ function updateAnalysisStatusTicker() { ticker.classList.remove("tier-healthy", "tier-watch", "tier-intervene"); return; } - let displayTier = tier === "intervene" ? "intervene" : (tier === "watch" || (review?.review_recommended && (review.reasons || []).length && !window.HPOState.ui.reviewPillDismissed) || confidence === "low" ? "watch" : "healthy"); + let displayTier = tier === "intervene" ? "intervene" : (tier === "watch" || confidence === "low" ? "watch" : "healthy"); let separator = " ▲ "; if (displayTier === "watch") { separator = " ◆ "; @@ -82,121 +31,32 @@ function updateAnalysisStatusTicker() { track.style.setProperty("--ticker-duration", `${Math.max(20, Math.min(45, text.length * 0.28))}s`); } -async function toggleReviewFlag(reviewId, flagged) { - try { - await fetch(`/api/flag_review?review_id=${reviewId}&flagged=${flagged}`, { method: "POST" }); - fetchStudyDetails(); - } catch (err) { console.error("Failed to flag review:", err); } -} - -function applyReviewUi() { - const recommended = window.HPOState.data.review?.review_recommended, reasons = window.HPOState.data.review?.reasons || []; - const show = !!(recommended && reasons.length) && !window.HPOState.ui.reviewPillDismissed; - const pill = document.getElementById("dashboard-review-pill"), pillReasons = document.getElementById("dashboard-review-reasons"); - if (pill) { - pill.classList.toggle("hidden", !show); - if (pillReasons) pillReasons.textContent = show ? reasons.map((r) => r.message || r.code).join(" • ") : ""; - } - updateAnalysisStatusTicker(); -} - -async function copyReviewPrompt() { - const pillAction = document.querySelector("#dashboard-review-pill .review-pill-action"); - try { - const res = await fetch(`/api/review_packet?study_name=${window.HPOState.session.studyName}`); - const data = await res.json(); - await navigator.clipboard.writeText(data.review_prompt || ""); - if (pillAction) { - const prev = pillAction.textContent; pillAction.textContent = "Copied!"; - setTimeout(() => { pillAction.textContent = prev; }, 1600); - } - fetch(`/api/dismiss_coordinator_nudge?study_name=${window.HPOState.session.studyName}`, { method: "POST" }).catch(e => {}); - const mark = document.querySelector('.brand-mark'); if (mark) mark.classList.remove("ping-twice"); - if (window.HPOState.ui.reviewPillDismissTimeout) clearTimeout(window.HPOState.ui.reviewPillDismissTimeout); - window.HPOState.ui.reviewPillDismissTimeout = setTimeout(() => { dismissReviewPill(); }, 120000); - } catch (err) { console.error("Could not copy review prompt:", err); } -} - async function checkStudyHealth(throwOnError = false) { + if (!window.HPOState.session.studyName) return; try { const res = await fetch(`/api/study_health?study_name=${window.HPOState.session.studyName}`); if (!res.ok) { if (throwOnError) throw new Error("HTTP error " + res.status); return; } const data = await res.json(), card = document.getElementById("study-health-card"), badge = document.getElementById("health-badge"); - const msg = document.getElementById("health-message"), actionBtn = document.getElementById("health-action-btn"), dismissBtn = document.getElementById("health-dismiss-btn"), mark = document.querySelector('.brand-mark'); + const msg = document.getElementById("health-message"), mark = document.querySelector('.brand-mark'); if (!card) return; window.HPOState.data.studyHealthReason = window.HPOState.data.healthReason = data.health_reason || ""; window.HPOState.data.healthTier = data.health_tier || "healthy"; - card.className = "card health-card " + (data.health_tier || "healthy"); + const tier = data.health_tier || "healthy"; + card.className = "card health-card"; + const orb = document.getElementById("health-orb"); + if (orb) orb.className = "status-orb " + tier; if (mark) mark.classList.remove("ping-watch", "ping-intervene", "ping-twice"); if (badge) { badge.className = `badge ${data.health_tier || 'healthy'}`; badge.textContent = (data.health_tier || 'healthy').toUpperCase(); } - const isDismissed = !!data.is_dismissed; if (data.health_tier === "healthy") { msg.textContent = "No anomalies detected. Search space is healthy."; - if (actionBtn) actionBtn.style.display = "none"; - if (dismissBtn) dismissBtn.style.display = "none"; } else { msg.textContent = data.health_reason || (data.health_tier === "watch" ? "Watch condition detected." : "Intervention recommended."); - if (actionBtn) actionBtn.style.display = data.health_tier === "intervene" ? "inline-block" : "none"; - if (dismissBtn) dismissBtn.style.display = isDismissed ? "none" : "inline-block"; - if (mark && !isDismissed) mark.classList.add(data.health_tier === "watch" ? "ping-watch" : "ping-intervene"); + if (mark) mark.classList.add(data.health_tier === "watch" ? "ping-watch" : "ping-intervene"); } renderHealthPanelExtras(); updateAnalysisStatusTicker(); } catch (err) { console.error("Error checking study health:", err); if (throwOnError) throw err; } } -async function dismissHealthAlert() { - try { - const res = await fetch(`/api/dismiss_coordinator_nudge?study_name=${window.HPOState.session.studyName}`, { method: "POST" }); - if (res.ok) checkStudyHealth(); - } catch (err) { console.error("Error dismissing health alert:", err); } -} - -function copyDiagnosticPrompt() { - const diagnosticPayload = { - anomaly: window.HPOState.data.studyHealthReason || "unknown anomaly", - active_search_space: window.HPOState.data.activeSearchSpace || {}, - recent_trials: window.HPOState.data.trials.slice(0, 3).map(t => ({ - trial_id: t.number, state: t.state, params: t.params, score: t.score, loss: t.loss - })) - }; - const promptText = `I need help debugging my Pathfinder study because it is failing health checks: -ANOMALY: ${diagnosticPayload.anomaly} - -ACTIVE SEARCH SPACE: -${JSON.stringify(diagnosticPayload.active_search_space, null, 2)} - -RECENT TRIALS: -${JSON.stringify(diagnosticPayload.recent_trials, null, 2)} - -Please analyze the failure and suggest one policy action (update_search_space or enqueue_one_manual_trial).`; - - navigator.clipboard.writeText(promptText).then(() => { - const btn = document.getElementById("health-action-btn"); - if (btn) { - const orig = btn.textContent; btn.textContent = "Copied!"; - setTimeout(() => btn.textContent = orig, 2000); - } - }).catch(err => console.error("Could not copy diagnostic prompt:", err)); -} - -function dismissReviewPill() { - window.HPOState.ui.reviewPillDismissed = true; - const pill = document.getElementById("dashboard-review-pill"); - if (pill) pill.classList.add("hidden"); - - // Clear double ping on dismiss via backend DB - fetch(`/api/dismiss_coordinator_nudge?study_name=${window.HPOState.session.studyName}`, { method: "POST" }) - .catch(err => console.error("Error dismissing nudge:", err)); - - const brandMark = document.querySelector('.brand-mark'); - if (brandMark) brandMark.classList.remove("ping-twice"); - - if (window.HPOState.ui.reviewPillDismissTimeout) { - clearTimeout(window.HPOState.ui.reviewPillDismissTimeout); - window.HPOState.ui.reviewPillDismissTimeout = null; - } -} - function updateStatConfidenceBanner(confidence, nComplete) { window.HPOState.data.statisticalConfidence = confidence || "low"; window.HPOState.data.completedCount = Number.isFinite(nComplete) ? nComplete : 0; @@ -227,12 +87,10 @@ function syncTelemetryFromTrials(trials) { return; } + window.HPOState.telemetry.trialNumber = running.number; const epochSuffix = running.latest_epoch != null ? ` · E${running.latest_epoch}` : ""; const label = `Trial #${running.number}${epochSuffix}`; - if (window.HPOState.telemetry.trialNumber !== running.number) { - window.HPOState.telemetry.trialNumber = running.number; - } setWorkerRunIndicator(true, label); } @@ -247,17 +105,9 @@ function setWorkerRunIndicator(isLive, label) { if (wrap) wrap.title = label; } -window.renderHealthLatestAction = renderHealthLatestAction; window.renderHealthPanelExtras = renderHealthPanelExtras; -window.renderPastReviews = renderPastReviews; window.updateAnalysisStatusTicker = updateAnalysisStatusTicker; -window.toggleReviewFlag = toggleReviewFlag; -window.applyReviewUi = applyReviewUi; -window.copyReviewPrompt = copyReviewPrompt; window.checkStudyHealth = checkStudyHealth; -window.dismissHealthAlert = dismissHealthAlert; -window.copyDiagnosticPrompt = copyDiagnosticPrompt; -window.dismissReviewPill = dismissReviewPill; window.updateStatConfidenceBanner = updateStatConfidenceBanner; window.syncTelemetryFromTrials = syncTelemetryFromTrials; window.setWorkerRunIndicator = setWorkerRunIndicator; diff --git a/web/js/main.js b/web/js/main.js index 8735d1c..21382f3 100644 --- a/web/js/main.js +++ b/web/js/main.js @@ -1,5 +1,5 @@ document.addEventListener("DOMContentLoaded", () => { - window.initHpoMarks(); + window.populateStudyList(); window.syncTopPerformersFilterCheckboxes(); const savedAccentName = localStorage.getItem("hpo_accent_name") || "cyan"; diff --git a/web/js/settings.js b/web/js/settings.js index 1e6a519..1ee1077 100644 --- a/web/js/settings.js +++ b/web/js/settings.js @@ -70,15 +70,7 @@ async function saveEvalProtocol() { } function togglePill(param, option, btn) { - const list = window.HPOState.data.activeSearchSpace[param].active; - const val = isNaN(option) ? option : Number(option); - const index = list.indexOf(val); - if (index > -1) { - if (list.length > 1) { list.splice(index, 1); btn.classList.remove("active"); } - } else { - list.push(val); - btn.classList.add("active"); - } + btn.classList.toggle("active"); markParamModified(param); } @@ -87,6 +79,10 @@ async function applySingleParamConstraints(param) { if (val.type === "float" || val.type === "float_log" || val.type === "int") { const minEl = document.getElementById(`${param}-min`), maxEl = document.getElementById(`${param}-max`); if (minEl && maxEl) { val.min = Number(minEl.value); val.max = Number(maxEl.value); } + } else if (val.type === "categorical") { + const pills = [...document.querySelectorAll(`#group-${param} .pill-btn.active`)] + .map(b => { const v = b.textContent; return isNaN(v) ? v : Number(v); }); + val.active = pills; } try { const res = await fetch(`/api/update_search_space?study_name=${window.HPOState.session.studyName}`, { @@ -193,6 +189,37 @@ function renderDashboardSearchSpaceSummary() { } function markParamModified(param) { + const currentSpace = window.HPOState.data.activeSearchSpace || {}; + const cfg = currentSpace[param]; + if (!cfg) return clearParamModified(param); + + if (cfg.type === "categorical") { + const activePills = [...document.querySelectorAll(`#group-${param} .pill-btn.active`)] + .map(b => { const v = b.textContent; return isNaN(v) ? v : Number(v); }); + const original = (cfg.active || []).slice().sort(); + const current = activePills.slice().sort(); + if (original.length === current.length && original.every((v, i) => v === current[i])) { + clearParamModified(param); + return; + } + window.HPOState.ui.modifiedParams.add(param); + const indicator = document.getElementById(`indicator-${param}`); + const btn = document.getElementById(`apply-btn-${param}`); + if (indicator) indicator.style.opacity = 1; + if (btn) btn.style.display = "inline-block"; + return; + } + + const minInput = document.getElementById(`${param}-min`); + const maxInput = document.getElementById(`${param}-max`); + const minChanged = minInput && parseFloat(minInput.value) !== parseFloat(cfg.min); + const maxChanged = maxInput && parseFloat(maxInput.value) !== parseFloat(cfg.max); + + if (!minChanged && !maxChanged) { + clearParamModified(param); + return; + } + window.HPOState.ui.modifiedParams.add(param); const indicator = document.getElementById(`indicator-${param}`); const btn = document.getElementById(`apply-btn-${param}`); diff --git a/web/js/state.js b/web/js/state.js index ba7273d..caa9a06 100644 --- a/web/js/state.js +++ b/web/js/state.js @@ -2,7 +2,7 @@ window.HPOState = { session: { studyName: new URLSearchParams(window.location.search).get("study") || new URLSearchParams(window.location.search).get("study_name") - || "bridge_crack_study", + || null, }, data: { trials: [], @@ -15,8 +15,6 @@ window.HPOState = { studyHealthReason: "", healthTier: "healthy", healthReason: "", - pastReviews: [], - pendingChanges: null, statisticalConfidence: "low", completedCount: 0, }, @@ -24,8 +22,6 @@ window.HPOState = { accentColorHex: "#06b6d4", filterTopPerformersOnly: false, isModalOpen: false, - reviewPillDismissed: false, - reviewPillDismissTimeout: null, toastTimeout: null, columnWidths: {}, modifiedParams: new Set(), @@ -59,24 +55,6 @@ window.HPOState = { constants: { HPO_FILTER_TOP_KEY: "hpo_filter_top", DASHBOARD_TABLE_COLS: 9, - HPO_MARK_SVG: ` - `, }, }; window.HPOState.ui.filterTopPerformersOnly = diff --git a/web/js/table_render.js b/web/js/table_render.js index 0725fb6..311e9b7 100644 --- a/web/js/table_render.js +++ b/web/js/table_render.js @@ -49,11 +49,7 @@ function renderDashboardTableBody(displayTrials, paretoSet, data) { const tbody = document.getElementById("trials-table-body"); if (!tbody) return; const ev = data.hpo_config?.eval_protocol || {}; - let paramKeys = []; - data.trials.forEach((t) => Object.keys(t.params).forEach((k) => { - if (!paramKeys.includes(k)) paramKeys.push(k); - })); - paramKeys.sort(); + const paramKeys = getParamKeys(); const piped = applyTablePipeline(displayTrials, "dashboard"); if (!piped.length) { @@ -101,7 +97,7 @@ function renderDashboardTableBody(displayTrials, paretoSet, data) { syncTrialColumnWidth(document.getElementById("dashboard-trial-table"), piped); } -function getAnalysisParamKeys() { +function getParamKeys() { const activeSpace = window.HPOState.data.activeSearchSpace || {}; let keys = Object.keys(activeSpace).filter(k => !k.startsWith("_") && typeof activeSpace[k] === "object"); if (keys.length === 0 && window.HPOState.data.trials && window.HPOState.data.trials.length > 0) { @@ -122,7 +118,7 @@ function renderAnalysisTableBody(displayTrials) { if (!tbody) return; applyAnalysisTableHeaders(); const ev = window.HPOState.data.hpoConfig?.eval_protocol || {}; - const paramKeys = getAnalysisParamKeys(); + const paramKeys = getParamKeys(); const piped = applyTablePipeline(displayTrials, "analysis"); if (!piped.length) { const totalCols = 4 + (ev.enabled ? 1 : 0) + paramKeys.length + 1; @@ -172,7 +168,7 @@ function renderAnalysisTrialsTable(trials) { function applyAnalysisTableHeaders() { const ev = window.HPOState.data.hpoConfig?.eval_protocol || {}; const paramLabels = window.HPOState.data.hpoConfig?.param_labels || {}; - const paramKeys = getAnalysisParamKeys(); + const paramKeys = getParamKeys(); const lossLabel = window.HPOState.data.hpoConfig?.metric_loss_label || "Loss"; const scoreLabel = window.HPOState.data.hpoConfig?.metric_score_label || "Score"; @@ -197,7 +193,7 @@ function applyAnalysisTableHeaders() { const label = paramLabels[k] || k.replace(/_/g, " "); thHtml += buildSortableTh(label, `param:${k}`, "analysis"); }); - thHtml += `Actions`; + thHtml += `Actions`; tableHeader.innerHTML = thHtml; const table = tableHeader.closest("table"); const baseTrials = window.HPOState.data.trials || []; @@ -212,25 +208,9 @@ function renderStudyDetails(data) { if (data.hpo_config) applyAnalysisTableHeaders(); applyEvalInsightsUi(); - applyReviewUi(); - // Find all parameters defined in trials or search space dynamically - let paramKeys = []; const paramLabels = data.hpo_config?.param_labels || {}; - - if (data.trials && data.trials.length > 0) { - data.trials.forEach(t => { - Object.keys(t.params).forEach(k => { - if (!paramKeys.includes(k)) paramKeys.push(k); - }); - }); - } else if (window.HPOState.data.activeSearchSpace) { - Object.keys(window.HPOState.data.activeSearchSpace).forEach(k => { - if (!k.startsWith("_") && !paramKeys.includes(k)) paramKeys.push(k); - }); - } - paramKeys.sort(); - + const paramKeys = getParamKeys(); const ev = data.hpo_config?.eval_protocol || {}; // Render live monitor table headers dynamically @@ -256,7 +236,7 @@ function renderStudyDetails(data) { thHtml += buildSortableTh(label, `param:${k}`, "dashboard"); }); thHtml += `★`; - thHtml += `Actions`; + thHtml += `Actions`; tableHeader.innerHTML = thHtml; const table = tableHeader.closest("table"); syncTrialColumnWidth(table, data.trials || []); diff --git a/web/js/utils.js b/web/js/utils.js index 1f896f9..be8d0ab 100644 --- a/web/js/utils.js +++ b/web/js/utils.js @@ -4,8 +4,7 @@ function paramLabel(key) { } function fanovaParamLabel(param) { - const legacy = { encoder_name: "model_capacity" }; - return paramLabel(legacy[param] || param); + return paramLabel(param); } function formatTrainResolution(trial) { @@ -70,85 +69,6 @@ function copyTrialsToJson() { }); } -function policyActionLabel(action) { - const labels = { - no_change: "No change", - update_search_space: "Search space adjusted", - enqueue_one_manual_trial: "Manual trial queued", - }; - return labels[action] || action || "Unknown action"; -} - -function formatReviewDate(createdAt) { - if (!createdAt) return ""; - const d = new Date(createdAt); - if (Number.isNaN(d.getTime())) return String(createdAt); - return d.toLocaleDateString(undefined, { month: "short", day: "numeric", year: "numeric" }); -} - -function formatOutcomeStatus(status) { - const labels = { - measured: "Outcome measured", - pending: "Outcome pending", - not_applicable: "No change applied", - inconclusive: "Outcome inconclusive", - }; - return labels[status] || status || ""; -} - -function buildHealthBlocks(rating) { - if (rating == null || rating <= 0) return null; - const n = Math.max(0, Math.min(5, Math.round(rating))); - const filled = "■ ".repeat(n).trim(); - const empty = "□ ".repeat(5 - n).trim(); - return (filled + (filled && empty ? " " : "") + empty).trim(); -} - -function formatReviewTimestamp(createdAt) { - if (!createdAt) return ""; - const d = new Date(createdAt); - if (Number.isNaN(d.getTime())) return String(createdAt); - return d.toISOString().replace("T", " ").slice(0, 16) + " UTC"; -} - -function getLatestCoordinatorAction(pastReviews, pendingChanges) { - if (pendingChanges && Object.keys(pendingChanges).length > 0) { - const keys = Object.keys(pendingChanges).map((k) => window.paramLabel(k)).join(", "); - return { - kind: "pending", - label: "Pending approval", - detail: `Search space adjustment (${keys})`, - timestamp: null, - }; - } - const actionable = (pastReviews || []).find((r) => r.policy_action && r.policy_action !== "no_change"); - if (actionable) { - return { - kind: "review", - label: policyActionLabel(actionable.policy_action), - detail: (actionable.summary || "").slice(0, 100), - timestamp: actionable.created_at, - reviewId: actionable.id, - }; - } - return null; -} - -window.paramLabel = paramLabel; -window.fanovaParamLabel = fanovaParamLabel; -window.formatTrainResolution = formatTrainResolution; -window.formatMetric = formatMetric; -window.formatLR = formatLR; -window.formatTime = formatTime; -window.copyTrialConfigToClipboard = copyTrialConfigToClipboard; -window.copyTrialsToJson = copyTrialsToJson; -window.policyActionLabel = policyActionLabel; -window.formatReviewDate = formatReviewDate; -window.formatOutcomeStatus = formatOutcomeStatus; -window.buildHealthBlocks = buildHealthBlocks; -window.formatReviewTimestamp = formatReviewTimestamp; -window.getLatestCoordinatorAction = getLatestCoordinatorAction; - function showEmptyState(containerId, message) { const container = document.getElementById(containerId); if (!container) return; @@ -244,15 +164,6 @@ function hideEmptyState(containerId, recreateCanvas = false) { } } -function initHpoMarks() { - const svg = window.HPOState?.constants?.HPO_MARK_SVG; - if (!svg) return; - document.querySelectorAll(".brand-mark").forEach((el) => { - el.innerHTML = svg; - }); -} - window.showEmptyState = showEmptyState; window.hideEmptyState = hideEmptyState; -window.initHpoMarks = initHpoMarks; diff --git a/web/styles.css b/web/styles.css index 325edd8..d7904d7 100644 --- a/web/styles.css +++ b/web/styles.css @@ -155,6 +155,7 @@ margin-bottom: 18px; overflow: hidden; width: 100%; + padding-left: 12px; } .brand-logo-group { @@ -165,12 +166,11 @@ .brand-mark { flex-shrink: 0; - width: 3px; - height: 20px; + width: 10px; + height: 10px; + border-radius: 50%; background-color: var(--accent-color); - border-radius: 0; - transition: background-color 0.3s ease, transform 0.2s cubic-bezier(0.25, 1, 0.5, 1), box-shadow 0.2s ease; - transform-origin: center; + transition: background-color 0.3s ease, opacity 0.3s ease; } .brand-mark.collision-flash { @@ -484,7 +484,6 @@ .sidebar-footer { margin-top: auto; border-top: 1px solid var(--border-color); - padding-top: 12px; display: flex; flex-direction: column; gap: 0; @@ -495,9 +494,10 @@ align-items: center; gap: 10px; width: 100%; - padding: 8px 4px; + padding: 12px 12px; background: none; border: none; + border-bottom: 1px solid var(--border-color); color: var(--text-muted); font-family: var(--font-main); font-size: 0.8rem; @@ -506,6 +506,7 @@ letter-spacing: 0.05em; cursor: pointer; transition: color 0.2s ease; + line-height: 1; } .accent-toggle:hover { @@ -514,7 +515,7 @@ .accent-preview { width: 14px; - height: 10px; + height: 12px; border-radius: 0; background: var(--accent-color); border: 2px solid rgba(255, 255, 255, 0.25); @@ -575,6 +576,7 @@ cursor: pointer; padding: 5px 2px; text-align: center; + line-height: 1; transition: all 0.15s ease; } @@ -593,8 +595,6 @@ align-items: center; gap: 10px; padding: 12px 4px 4px; - border-top: 1px solid var(--border-color); - margin-top: 4px; } .worker-run-dot { @@ -736,7 +736,6 @@ #dashboard-view > .dashboard-sidebar { height: 100%; min-height: 0; - overflow: hidden; } /* Left column: trial table + pareto (original layout) */ @@ -919,11 +918,11 @@ } .section-label { - font-size: 0.68rem; + font-size: 0.75rem; font-weight: 500; text-transform: uppercase; letter-spacing: 0.06em; - color: var(--text-muted); + color: #94a3b8; margin: 0 0 10px 0; } @@ -1005,6 +1004,7 @@ /* Live Monitor Table Styling */ .trial-table { width: 100%; + min-width: 100%; border-collapse: collapse; font-family: var(--font-mono); font-size: 0.8rem; @@ -1026,6 +1026,7 @@ .table-scroll-wrapper { overflow: auto; + box-shadow: inset 0 -1px 0 var(--border-color), inset -1px 0 0 var(--border-color); } .card-content--table { @@ -1153,7 +1154,7 @@ position: absolute; top: 0; right: 0; - width: 4px; + width: 8px; cursor: col-resize; user-select: none; height: 100%; @@ -1276,7 +1277,7 @@ } .trial-table .col-trial { width: 60px; min-width: 60px; } - .trial-table .col-state { width: 100px; } + .trial-table .col-state { width: 120px; } .trial-table .col-metric { width: 72px; } .trial-table .col-score-fixed { width: 72px; } .trial-table .col-lr { width: 64px; } @@ -1395,6 +1396,7 @@ min-height: 0; flex: 1; background-color: transparent; + height: 100%; } .dashboard-sidebar > .card { @@ -1840,13 +1842,13 @@ cursor: pointer; transition: all 0.2s ease; font-family: var(--font-mono); + line-height: 1; } .pill-btn.active { background-color: rgba(6, 182, 212, 0.12); border-color: var(--accent-color); color: var(--accent-color); - box-shadow: 0 0 10px var(--accent-glow); } .apply-btn { @@ -2198,67 +2200,21 @@ } .brand-mark.ping-watch { - animation: brand-mark-ping-watch-anim 3s infinite ease-in-out; + animation: brand-mark-ping-watch-anim 2s infinite ease-in-out; } @keyframes brand-mark-ping-watch-anim { - 0%, 100% { - transform: scale(1); - box-shadow: none; - background-color: var(--accent-color); - } - 10% { - transform: scale(1.2); - box-shadow: 0 0 10px #ffffff; - background-color: #ffffff; - } - 20% { - transform: scale(1); - box-shadow: none; - background-color: var(--accent-color); - } - 30% { - transform: scale(1.2); - box-shadow: 0 0 10px #ffffff; - background-color: #ffffff; - } - 40%, 90% { - transform: scale(1); - box-shadow: none; - background-color: var(--accent-color); - } + 0%, 100% { opacity: 1; } + 50% { opacity: 0.35; } } .brand-mark.ping-intervene { - animation: brand-mark-ping-intervene-anim 3s infinite ease-in-out; + animation: brand-mark-ping-intervene-anim 2s infinite ease-in-out; } @keyframes brand-mark-ping-intervene-anim { - 0%, 100% { - transform: scale(1); - box-shadow: none; - background-color: var(--accent-color); - } - 10% { - transform: scale(1.2); - box-shadow: 0 0 10px #ff3b30; - background-color: #ff3b30; - } - 20% { - transform: scale(1); - box-shadow: none; - background-color: var(--accent-color); - } - 30% { - transform: scale(1.2); - box-shadow: 0 0 10px #ff3b30; - background-color: #ff3b30; - } - 40%, 90% { - transform: scale(1); - box-shadow: none; - background-color: var(--accent-color); - } + 0%, 100% { opacity: 1; background-color: #ff3b30; } + 50% { opacity: 0.2; background-color: #ff3b30; } } .pending-banner { @@ -2368,21 +2324,11 @@ .health-card { flex: 0 0 auto; transition: all 0.2s ease; - border-left: 3px solid #10B981 !important; - } - .health-card.healthy { - border-left-color: #10B981 !important; - } - .health-card.watch { - border-left-color: #f59e0b !important; - } - .health-card.intervene { - border-left-color: #ef4444 !important; } .status-orb { - width: 16px; - height: 6px; - border-radius: 0; + width: 8px; + height: 8px; + border-radius: 50%; display: inline-block; transition: all 0.2s ease; flex-shrink: 0; @@ -2688,7 +2634,6 @@ .control-group { display: flex; align-items: center; - justify-content: space-between; gap: 12px; padding: 10px 12px; border-bottom: 1px solid var(--border-color); @@ -2700,21 +2645,23 @@ } .control-label { - flex: 0 0 200px; - margin-right: 24px; + flex: 0 0 180px; + margin-right: 12px; font-weight: 500; font-size: 0.85rem; color: var(--text-color); + line-height: 1.2; } .eval-text-input { width: 220px; - padding: 4px 8px; + padding: 6px 8px; font-family: var(--font-mono); - font-size: 0.82rem; + font-size: 0.78rem; background: rgba(255, 255, 255, 0.03); border: 1px solid var(--border-color); border-radius: 0; + color: var(--text-color); color: #fff; } @@ -2726,9 +2673,16 @@ .number-input--compact { flex: 0 0 auto; width: 110px; - padding: 4px 8px; + padding: 6px 8px; border-radius: 0; - font-size: 0.82rem; + font-size: 0.78rem; + -moz-appearance: textfield; + } + + .number-input--compact::-webkit-inner-spin-button, + .number-input--compact::-webkit-outer-spin-button { + -webkit-appearance: none; + margin: 0; } #settings-tab-eval .hpo-toggle--eval .hpo-toggle-box { @@ -2749,7 +2703,7 @@ .eval-form-section .section-label { padding: 0 12px; - margin-bottom: 0; + margin-bottom: 8px; } .eval-form-section .control-group:last-child { @@ -2762,6 +2716,11 @@ margin-top: 14px; } + .eval-form-actions .toolbar-btn { + font-size: 0.8rem; + padding: 8px 16px; + } + .page-header { border-bottom: 1px solid var(--border-color); padding: 14px 24px; @@ -2830,3 +2789,16 @@ color: #000000; } + /* Chart.js tooltip constraints */ + #chartjs-tooltip { + max-width: 280px; + white-space: normal; + word-break: break-word; + border-radius: 6px; + pointer-events: none; + } + + #chartjs-tooltip table { + margin: 0; + } +