Pre-flight training controls: metric, model families, trial budget, wall-clock/cost caps

backend/agents/trainer.yaml

@@ -242,6 +242,11 @@ system: |
   - Start with a quick scan: train 2-3 model types on a sample (max 10k rows) to identify the best approach
   - Then do a full run with thorough hyperparameter tuning on the complete dataset
   - Budget tuning appropriately: 30-50 optuna trials for the final model
+  - If a `## User training constraints` block appears below, it OVERRIDES
+    these defaults: restrict the quick scan to the allowed model families,
+    optimize the user's metric, and never exceed the user's trial budget
+    or wall-clock/cost cap. Declare `optimization_metric` and `max_trials`
+    on your start-training call.
 
   ## Mandatory Experiment Lifecycle (NON-NEGOTIABLE)
   Ownership note: chat / orchestrator typically open the experiment

backend/db.py

@@ -86,6 +86,11 @@ def _run_migrations(connection):
                 text("ALTER TABLE projects ADD COLUMN sandbox_config JSON")
             )
             logger.info("[DB] Added sandbox_config column to projects table")
+        if "training_config" not in columns:
+            connection.execute(
+                text("ALTER TABLE projects ADD COLUMN training_config JSON")
+            )
+            logger.info("[DB] Added training_config column to projects table")
 
     # ------------------------------------------------------------------
     # Phase A — projects foundation

backend/models.py

@@ -75,6 +75,10 @@ class Project(Base):
     description = Column(Text, default="")
     created_at = Column(String, default=lambda: utcnow().isoformat())
     sandbox_config = Column(JSON, default=dict)
+    # Pre-flight training controls (metric, model families, trial budget,
+    # wall-clock/cost cap) — see schemas.TrainingConfig. Empty dict = the
+    # trainer agent keeps full autonomy.
+    training_config = Column(JSON, default=dict)
     updated_at = Column(String, default=lambda: utcnow().isoformat())
 
     experiments = relationship(
@@ -114,6 +118,7 @@ def to_dict(
             "name": self.name,
             "description": self.description or "",
             "sandbox_config": self.sandbox_config or {},
+            "training_config": self.training_config or {},
             "created_at": self.created_at,
             "updated_at": self.updated_at,
             "experiment_count": experiment_count,

backend/routers/projects.py

@@ -59,6 +59,11 @@ async def create_project(body: ProjectCreate, db: AsyncSession = Depends(get_db)
         name=body.name or "New project",
         description=body.description or "",
         sandbox_config=body.sandbox_config.model_dump() if body.sandbox_config else {},
+        training_config=(
+            body.training_config.model_dump(exclude_none=True)
+            if body.training_config
+            else {}
+        ),
         created_at=now,
         updated_at=now,
     )
@@ -146,6 +151,9 @@ async def update_project(
         project.description = body.description
     if body.sandbox_config is not None:
         project.sandbox_config = body.sandbox_config.model_dump()
+    if body.training_config is not None:
+        # exclude_none so cleared fields drop out — {} means "no constraints".
+        project.training_config = body.training_config.model_dump(exclude_none=True)
     project.updated_at = _now()
 
     await db.commit()

backend/schemas.py

@@ -2,9 +2,10 @@
 
 from __future__ import annotations
 
+import re
 from typing import Literal, Optional
 
-from pydantic import BaseModel, Field
+from pydantic import BaseModel, Field, field_validator
 
 # Generous-but-not-infinite caps to stop runaway inputs from swamping the
 # database or bloating an agent's context window. Calibrated so legitimate
@@ -30,6 +31,78 @@ class SandboxConfig(BaseModel):
     training: Optional[SandboxProfile] = None
 
 
+# Model families a user can restrict the trainer to. Mirrors the `framework`
+# vocabulary of the start-training skill (see skills/start-training/schema.yaml).
+KNOWN_MODEL_FAMILIES = (
+    "xgboost",
+    "lightgbm",
+    "sklearn",
+    "pytorch",
+    "tensorflow",
+    "huggingface",
+    "other",
+)
+
+# Metric names are rendered verbatim into agent system prompts (inside a
+# backtick fence) — restrict to a plain-identifier charset so a crafted value
+# can't break out of the fence or smuggle prompt directives.
+_METRIC_RE = re.compile(r"^[A-Za-z0-9 _\-./@:()]+$")
+
+
+class TrainingConfig(BaseModel):
+    """Pre-flight training controls (issue #104).
+
+    Everything is optional — an empty config means the trainer agent keeps
+    full autonomy (today's behavior). Any field the user sets becomes a
+    constraint the orchestrator/trainer must honor: it is injected into the
+    agent system prompt and enforced at the start-training skill boundary.
+    """
+
+    # Metric the tuning loop must optimize (e.g. "roc_auc", "pr_auc", "f1", "rmse").
+    optimization_metric: Optional[str] = Field(default=None, max_length=64)
+    # Allowed model families. Empty/None = agent's choice.
+    model_families: Optional[list[str]] = Field(default=None, max_length=16)
+    # Hard cap on hyperparameter-search trials (Optuna or equivalent).
+    max_trials: Optional[int] = Field(default=None, ge=1, le=1000)
+    # Wall-clock budget for training work, in minutes. Also clamps the
+    # training sandbox profile's per-call timeout.
+    max_wallclock_minutes: Optional[int] = Field(default=None, ge=1, le=1440)
+    # Advisory spend cap for the training run, in USD.
+    max_cost_usd: Optional[float] = Field(default=None, gt=0, le=100_000)
+
+    @field_validator("optimization_metric")
+    @classmethod
+    def _clean_metric(cls, v: Optional[str]) -> Optional[str]:
+        v = (v or "").strip()
+        if not v:
+            return None
+        if not _METRIC_RE.fullmatch(v):
+            raise ValueError(
+                "optimization_metric may only contain letters, digits, spaces "
+                "and _-./@:() characters"
+            )
+        return v
+
+    @field_validator("model_families")
+    @classmethod
+    def _clean_families(cls, v: Optional[list[str]]) -> Optional[list[str]]:
+        if v is None:
+            return None
+        cleaned: list[str] = []
+        for fam in v:
+            fam = (fam or "").strip().lower()
+            if not fam:
+                continue
+            if fam not in KNOWN_MODEL_FAMILIES:
+                raise ValueError(
+                    f"Unknown model family '{fam}'. "
+                    f"Valid: {', '.join(KNOWN_MODEL_FAMILIES)}"
+                )
+            if fam not in cleaned:
+                cleaned.append(fam)
+        return cleaned or None
+
+
 class ExperimentCreate(BaseModel):
     name: str = Field(..., min_length=1, max_length=_NAME_MAX)
     description: str = Field(default="", max_length=_DESC_MAX)
@@ -81,12 +154,14 @@ class ProjectCreate(BaseModel):
     name: Optional[str] = Field(default=None, max_length=_NAME_MAX)
     description: Optional[str] = Field(default=None, max_length=_DESC_MAX)
     sandbox_config: Optional[SandboxConfig] = None
+    training_config: Optional[TrainingConfig] = None
 
 
 class ProjectUpdate(BaseModel):
     name: Optional[str] = Field(default=None, min_length=1, max_length=_NAME_MAX)
     description: Optional[str] = Field(default=None, max_length=_DESC_MAX)
     sandbox_config: Optional[SandboxConfig] = None
+    training_config: Optional[TrainingConfig] = None
 
 
 class ExperimentUpdate(BaseModel):

backend/services/agent/runner.py

@@ -164,19 +164,27 @@ async def _load_conversation_history(session_id: str) -> list[dict]:
     return messages
 
 
-async def _load_project_context(experiment_id: str) -> tuple[str, str, str, dict]:
-    """Return (project_id, project_name, project_files_listing, sandbox_config).
+async def _load_project_context(
+    experiment_id: str,
+) -> tuple[str, str, str, dict, dict]:
+    """Return (project_id, project_name, project_files_listing, sandbox_config,
+    training_config).
 
     project_files_listing is a multi-line string describing all files currently
     present under /projects/{project_id}/datasets/. If the project has no data,
     returns the placeholder "(no data uploaded yet)".
 
     sandbox_config is the project's per-profile compute settings (default and
     training profiles, each with optional gpu + timeout). Empty dict if unset.
+
+    training_config is the project's pre-flight training controls (optimization
+    metric, model families, trial budget, wall-clock/cost cap — see
+    schemas.TrainingConfig). Empty dict if unset.
     """
     project_id = ""
     project_name = ""
     sandbox_config: dict = {}
+    training_config: dict = {}
     try:
         async with async_session() as db:
             result = await db.execute(
@@ -192,6 +200,7 @@ async def _load_project_context(experiment_id: str) -> tuple[str, str, str, dict
                 if project:
                     project_name = project.name
                     sandbox_config = project.sandbox_config or {}
+                    training_config = project.training_config or {}
     except Exception as e:
         logger.warning("Failed to load project for experiment %s: %s", experiment_id, e)
 
@@ -241,7 +250,94 @@ async def _load_project_context(experiment_id: str) -> tuple[str, str, str, dict
                 + "/datasets/` directly)"
             )
 
-    return project_id, project_name, files_listing, sandbox_config
+    return project_id, project_name, files_listing, sandbox_config, training_config
+
+
+def _apply_training_wallclock_cap(sandbox_config: dict, training_config: dict) -> dict:
+    """Clamp the training sandbox profile's per-call timeout to the user's
+    wall-clock budget (training_config.max_wallclock_minutes).
+
+    This is the hard-enforcement half of the pre-flight controls: even if the
+    agent ignores the prompt-level constraint, a heavy execute-code call cannot
+    run past the cap. Returns a new dict; the input is not mutated.
+    """
+    cap_minutes = (training_config or {}).get("max_wallclock_minutes")
+    if not cap_minutes:
+        return sandbox_config
+    cap_seconds = int(cap_minutes) * 60
+    config = dict(sandbox_config or {})
+    training_profile = dict(config.get("training") or {})
+    current = training_profile.get("timeout") or settings.sandbox_timeout
+    training_profile["timeout"] = min(int(current), cap_seconds)
+    config["training"] = training_profile
+    return config
+
+
+def _format_training_constraints(training_config: dict) -> str:
+    """Render the user's pre-flight training controls as a prompt block.
+
+    Injected into the system prompt of any agent that can call start-training
+    (orchestrator, trainer, chat). Empty string when no constraint is set so
+    unconfigured projects behave exactly as before.
+    """
+    cfg = training_config or {}
+    metric = cfg.get("optimization_metric")
+    families = cfg.get("model_families") or []
+    max_trials = cfg.get("max_trials")
+    max_wallclock = cfg.get("max_wallclock_minutes")
+    max_cost = cfg.get("max_cost_usd")
+
+    constraints: list[str] = []
+    if metric:
+        constraints.append(
+            f"- **Optimization metric**: `{metric}`. Every model-selection and "
+            f"hyperparameter-tuning decision (including the Optuna objective) "
+            f"MUST optimize this metric. Report other metrics too, but select on this one."
+        )
+    if families:
+        fam_list = ", ".join(f"`{f}`" for f in families)
+        constraints.append(
+            f"- **Allowed model families**: {fam_list}. Do NOT train or tune "
+            f"models outside these families — not even for the quick scan. "
+            f"start-training rejects other frameworks."
+        )
+    if max_trials:
+        constraints.append(
+            f"- **Trial budget**: at most {max_trials} hyperparameter-search "
+            f"trials TOTAL across the whole run. This overrides any default "
+            f"trial count in your instructions (e.g. '30-50 optuna trials')."
+        )
+    if max_wallclock:
+        constraints.append(
+            f"- **Wall-clock cap**: {max_wallclock} minutes of training compute. "
+            f"The training sandbox profile's per-call timeout is clamped to this "
+            f"cap; plan fits/sweeps to finish within it."
+        )
+    if max_cost:
+        constraints.append(
+            f"- **Cost cap**: ${max_cost:g} for this training effort. Prefer "
+            f"cheaper models/fewer trials as you approach it."
+        )
+
+    if not constraints:
+        return ""
+
+    lines = [
+        "## User training constraints (MANDATORY)",
+        "",
+        "The user configured pre-flight training controls in Project Settings.",
+        "These are hard requirements, not suggestions — they OVERRIDE any",
+        "conflicting default strategy in your instructions:",
+        "",
+        *constraints,
+        "",
+        "When delegating training work to another agent, restate these",
+        "constraints verbatim in the delegation instructions so they are not",
+        "lost. The start-training skill validates its arguments against them,",
+        "and REQUIRES you to declare `optimization_metric` and `max_trials`",
+        "explicitly whenever the corresponding constraint is set above.",
+    ]
+    return "\n".join(lines)
 
 
 def _format_compute_env(sandbox_config: dict) -> str:
@@ -888,8 +984,14 @@ async def _publish(
             project_name,
             project_files,
             sandbox_config,
+            training_config,
         ) = await _load_project_context(experiment_id)
 
+        # Hard enforcement of the user's wall-clock budget: clamp the training
+        # sandbox profile's per-call timeout before the config flows into
+        # execute-code / delegate-task handlers.
+        sandbox_config = _apply_training_wallclock_cap(sandbox_config, training_config)
+
         system_prompt = render_agent_system_prompt(
             agent_type,
             experiment_id=experiment_id,
@@ -923,6 +1025,14 @@ async def _publish(
         if "execute-code" in get_agent_skills(agent_type):
             system_prompt += "\n\n" + _format_compute_env(sandbox_config)
 
+        # Pre-flight training controls (issue #104) — only meaningful for
+        # agents that can open a training window. Empty config renders to ""
+        # so unconfigured projects get a byte-identical prompt.
+        if "start-training" in get_agent_skills(agent_type):
+            constraints_block = _format_training_constraints(training_config)
+            if constraints_block:
+                system_prompt += "\n\n" + constraints_block
+
         if user_prompt:
             prompt = _apply_mentions(user_prompt, mentions)
         else:

backend/skills/start-training/SKILL.md

@@ -23,6 +23,26 @@ sidebar — which is the signal that something went wrong.
 - `experiment_id` (required): from `create-experiment`.
 - `framework` (required): one of `xgboost | lightgbm | sklearn | pytorch | tensorflow | huggingface | other`.
 - `hyperparams` (optional but encouraged): the dict you'll pass to `.fit()`. Saved on the experiment row so the lineage view can show "this model was trained with these hyperparams" without parsing the snapshot manifest.
+- `optimization_metric` (optional): the metric your tuning loop optimizes (e.g. `roc_auc`, `pr_auc`, `f1`, `rmse`).
+- `max_trials` (optional): the number of hyperparameter-search trials you plan to run.
+
+## User training constraints
+
+Projects can carry pre-flight training controls set by the user in Project
+Settings (optimization metric, allowed model families, trial budget,
+wall-clock/cost cap). When set, this skill enforces them:
+
+- `framework` outside the allowed model families → **rejected**.
+- `optimization_metric` that conflicts with the user's metric → **rejected**.
+  When the user configured a metric, `optimization_metric` becomes
+  **required** — omitting it is rejected too.
+- `max_trials` above the user's trial budget → **rejected**. When the user
+  configured a budget, `max_trials` becomes **required** — omitting it is
+  rejected too.
+
+A successful call echoes the active constraints back in `user_constraints` —
+honor them for the whole run. If no constraints are configured, the call
+behaves exactly as before.
 
 ## Returns