fix(profile): find the steady-state trace the injected profiler dir redirected

[BaoYunkai]

Problem

Magpie's launcher emits its own --profiler-config.torch_profiler_dir <workspace>/torch_trace before EXTRA_VLLM_ARGS, so the placeholder
_workload_envs appends -- the task root -- is the value vLLM resolves under
its dotted-flag last-wins merge. Observed argv order on a real round:

14  --profiler-config.profiler torch                       <- Magpie
16  --profiler-config.torch_profiler_dir <ws>/torch_trace  <- Magpie
...
34  --profiler-config.torch_profiler_dir <task root>       <- EXTRA_VLLM_ARGS, wins

and the server confirms it: ProfilerConfig(torch_profiler_dir='<task root>').

The server therefore writes its steady-state traces straight into the task root
and its graph-capture sidecars into that root's capture_traces/, while
Magpie's own torch_trace/ stays empty and says so
([PROFILE] No trace found for relay under: .../torch_trace).

_candidate_trace_dirs probed the workspace and the root's capture_traces,
so it saw only sidecars and fell back to capture-only. Roofline rejects a
capture-only trace, so every attempt failed with profile_capture_only_failed
while the traces it wanted sat one directory up.

Session 20260917T191938Z-1ad81699 lost both roofline tasks that way, 3 of 3
attempts each, against 24 complete annotated steady-state traces (8 ranks x 3
attempts, ~250MB each) already on disk. The knock-on is the whole kernel phase:
GEMM tuning had no shapes to derive (no candidates to E2E validate) and
forge-fusion was skipped outright (no decode trace yet), leaving KB replay as
the session's only source of gain.

Fix

Fix the launch contract instead of teaching the reader to hunt the task root.

• Write side (_workload_envs.py): when materializing a vLLM profile run,
  keep profiler=torch and the delay/max-iteration bounds in EXTRA_VLLM_ARGS,
  but do not append --profiler-config.torch_profiler_dir (the old task-root
  placeholder). With only Magpie's earlier torch_profiler_dir on the real
  vllm serve argv, last-wins no longer overrides Magpie; steady-state traces
  land under <workspace>/torch_trace as Magpie intended.
• Read side (profile.py): discovery stays on workspace/torch_trace (and
  capture sidecars under the workspace). No task-root candidate, no mtime
  watermark — those were dropped in favour of fixing where the server writes.

CI / test hygiene: test_phase_state_machine.py stubs GEAK runner and
controller subprocess boundaries in its session_dir fixture (same pattern as
the optimize-loop walkthrough) and checks phase-history idempotency while parked
in FRAMEWORK_AGENT after prelude_done, instead of disabling the framework
phase and racing KERNEL→SWEEP closeout on the second tick.

Test plan

• test_profile_and_kernel_handlers.py: materialize tests assert
  torch_profiler_dir is not injected into EXTRA_VLLM_ARGS; executor
  tests prefer workspace/torch_trace over capture-only sidecars.
• test_phase_state_machine.py: kernel subprocess stubs + idempotency in
  FRAMEWORK_AGENT (46 passed in file).
• src/hyperloom/orchestrator/ and inference_optimizer profile suite as
  run locally before push.
• ruff check + ruff format --check on changed files.
• Replayed against session 20260917T191938Z-1ad81699: roofline admits
  steady-state traces under workspace torch_trace once launch no longer
  redirects writes to the task root.

[ZhengGong-amd]

Reconsider this as a write-side fix. The task-root search should not exist.

Magpie emits --profiler-config.torch_profiler_dir <workspace>/torch_trace first. _workload_envs then appends the same flag pointed at the task root, and vLLM last-wins keeps the task root. That directory is shared by every attempt. Probing workspace.parent and filtering trace_files by mtime does not scope the attempt: on the roofline path _preferred_main_trace_path returns the directory as main_trace_path, and discover_trace_inputs rglobs it with no watermark. From attempt 2 the analysis includes earlier traces.

Drop the placeholder from the launched EXTRA_VLLM_ARGS unless the operator already set one. baseline.py already injects <output_dir>/torch_trace on the probe argv only; bypass_engine already puts the real dir after extra args. Leave discovery on <workspace>/torch_trace. Delete workspace.parent, the scoped watermark, and the two new tests. Leave the _workload_envs comment — it is true once the launched flag stops overriding Magpie.

[BaoYunkai]

Adopted the write-side approach from your comment.

• _workload_envs no longer appends a task-root torch_profiler_dir placeholder to EXTRA_VLLM_ARGS (operator-set values unchanged). Magpie’s <workspace>/torch_trace flag now wins last-wins merge at launch.
• Reverted task-root probing in _candidate_trace_dirs, the scoped mtime watermark on workspace.parent, and the two executor tests that encoded the read-side fix.
• Updated the vLLM profile materialization test to expect profiler=torch without a launched torch_profiler_dir; argv preflight still uses the probe-only injection in baseline.py / bypass_engine.py.
• Kept the expanded _workload_envs comment, updated to describe the new behavior.

Pushed on bugfix/yunkai/profile-steady-state-trace (commit message: keep torch traces under workspace/torch_trace).

[ZhengGong-amd]

Rebase onto main; this head conflicts, so Lint and Tests with Coverage never started and must pass on the rebased commit.

Rewrite the PR body: it still describes probing the task root and the mtime watermark, but the change now stops EXTRA_VLLM_ARGS from overriding Magpie so traces stay in <workspace>/torch_trace.

Correct the new comment and test note: Magpie emits torch_profiler_dir before EXTRA_VLLM_ARGS, and the fix works because that flag is no longer written into the launched args, not because Magpie's copy comes last.

[ZhengGong-amd]

Approved