fix: restore hourly execution for Intent Signal Discovery (workflow c10f1d63)

[princepspolycap]

Problem

Intent Signal Discovery is not actually running hourly in Poly Operations.

Evidence

• Workflow: c10f1d63-0e63-4c03-bfea-aa16c31d2a6a::1.0
  • execution_status=RUNNING
  • is_scheduled=false
• Hourly task: 8c929111-2380-49bb-b07d-e6c2429927c3::1.0
  • is_recurring=true
  • recurrence_pattern=hourly
  • execution_status=not_started
  • stale timestamps (updated_at + next_recurrence_date stuck on 2026-03-24)
• Known blocker in escalation record: UserConcurrencyLimitError and reset behavior.

This creates orchestration drift: workflow marked running while no tasks execute.

Goal

Make Stage 1 Intent Signal Discovery reliably execute every hour with observable proof.

Scope

1. Reconcile state drift between workflow-level RUNNING and task-level not_started.
2. Clear/handle orphaned active executions so concurrency checks stop blocking valid runs.
3. Enforce a single scheduler source of truth (pick one):
   • workflow-level scheduler (is_scheduled=true + hourly config), OR
   • stage-level cadence (auto_trigger=true, cadence_hours=1).
4. Add guardrails so stale next_recurrence_date < now is auto-recovered.
5. Add monitoring signal for missed cadence (last_cycle_completed_at > 90m).

Acceptance Criteria

• A 6-hour observation window shows >=5 successful hourly cycles.
• Task updated_at advances roughly hourly.
• next_recurrence_date always rolls forward to future (~+1h).
• No UserConcurrencyLimitError blocks on normal hourly runs.
• RUNNING state reflects actual active task execution (no false RUNNING idle state).
• Alert fires when cadence misses 90 minutes.

Verification Plan

• Run staged test in non-prod or controlled environment.
• Capture execution timeline for 6 hours.
• Include logs for scheduler decisions, lock acquisition/release, and recurrence computation.

References

• Monitoring playbook:
  https://polyalphablobstorage.blob.core.windows.net/workspace-a39b2977-0668-48e6-8578-0cbb6849f091-files/markdown/intent_signal_discovery__hourly_monitoring_playbook.md

Priority

High (pipeline health + demand capture reliability).

[princepspolycap]

Update after engineering review of PR #11:

• Reviewed PR diff and validated in compute sandbox.
• Result: not mergeable for incident resolution yet.

Why this does not resolve the production blocker

1. The PR introduces a new in-memory scheduler (/api/scheduler/*) rather than patching the existing persisted Task/TaskExecution concurrency accounting path.
2. Incident is caused by persisted/orphaned executions inflating active counts; in-memory state cannot reconcile real orphan rows.
3. Branch currently fails npm run build due to TS unused-symbol errors in new tests.
4. No check runs were present on PR head at review time.

Required implementation direction

• Patch the real limiter query/cache logic used by execution gating.
• Add persisted cleanup rule:
  • if task is not running at both task status and execution_status levels, close/cancel active TaskExecution rows for that task.
• Enforce idempotent enqueue:
  • no enqueue when same task already has queued/running execution.
• Add integration tests against persistence-backed execution flow.

I submitted REQUEST_CHANGES on PR #11 with detailed blocker notes and acceptance path.

[princepspolycap]

Status update from GitHub Engineer execution:

• Re-validated linked PR fix: restore hourly execution for Intent Signal Discovery (workflow c10f1d63) #11 against issue requirements.
• Posted updated technical validation notes on PR fix: restore hourly execution for Intent Signal Discovery (workflow c10f1d63) #11.
• Submitted formal REQUEST_CHANGES review on PR fix: restore hourly execution for Intent Signal Discovery (workflow c10f1d63) #11.

Reason for hold:

• Fix currently targets in-memory scheduler logic, not persistence-backed execution/concurrency path used by production.
• No check runs currently attached to PR head SHA.

Issue #10 remains open pending a persistence-backed fix and integration-level evidence.

[princepspolycap]

Execution update (2026-03-31 UTC):

• Re-reviewed PR fix: restore hourly execution for Intent Signal Discovery (workflow c10f1d63) #11 against issue fix: restore hourly execution for Intent Signal Discovery (workflow c10f1d63) #10 acceptance criteria.
• Submitted formal review: REQUEST_CHANGES on PR fix: restore hourly execution for Intent Signal Discovery (workflow c10f1d63) #11.

Why issue remains open:

• PR fix: restore hourly execution for Intent Signal Discovery (workflow c10f1d63) #11 currently implements an in-memory scheduler layer, not the persistence-backed execution/concurrency path that is actually broken in production.
• Critical implementation items in PR body remain unchecked (scheduler daemon loop, completion callback endpoint, lifecycle wiring, full ::1.0 ID handling).
• CI on PR head is not yet green (check currently in progress at review time).

Additional validation evidence:

• Compute sandbox attempt to validate PR branch:
  • clone + checkout succeeded
  • npm ci terminated with exit code 137 (process killed), so local full-build verification did not complete in compute.

Status:

• Issue fix: restore hourly execution for Intent Signal Discovery (workflow c10f1d63) #10 should remain OPEN until a persistence-backed fix is submitted with passing CI and evidence for hourly recurrence + concurrency recovery in real state.

[princepspolycap]

Execution update (2026-04-03 UTC):

• Re-reviewed PR fix: restore hourly execution for Intent Signal Discovery (workflow c10f1d63) #11 at latest head d66d48f1fd078e83b8ebeea39ed46f2391cd094f.
• Submitted formal review on PR fix: restore hourly execution for Intent Signal Discovery (workflow c10f1d63) #11: REQUEST_CHANGES.

Current blocker summary:

1. CI is not currently green for the PR branch (CI workflow runs on this branch show action_required).
2. No completed successful check run is attached to the current head SHA.
3. Incident acceptance criteria still require runtime proof over time window (hourly cycles + cadence/recurrence evidence), not only test claims.

Issue #10 should remain open until CI is green on current head and runtime verification evidence is posted.

[princepspolycap]

Engineering status update (2026-04-03 UTC):

• Re-reviewed linked PR fix: restore hourly execution for Intent Signal Discovery (workflow c10f1d63) #11 and submitted REQUEST_CHANGES review: fix: restore hourly execution for Intent Signal Discovery (workflow c10f1d63) #11 (review)

Current blockers to close this incident:

1. No check runs currently attached to PR fix: restore hourly execution for Intent Signal Discovery (workflow c10f1d63) #11 head SHA bf11b27613733b22ed34bef1b49e2fab86faad43.
2. Runtime evidence for hourly cadence is still missing from deployed environment (test-only evidence is strong but not sufficient for incident closure).
3. Persistence path must be explicitly mapped to production execution state source to confirm the fix addresses real drift/concurrency records.

Issue remains OPEN pending those three closure conditions.