Skip to content

test: 22 real-worker unit tests exceed deadlines on local macOS dev machine (CI macOS passes) #598

Open

Description

@teleworksai

Summary

On the maintainer's local Mac, 22 unit tests that launch real sandboxed workers / child processes fail by hitting internal deadlines. The same tests pass on GitHub's macos-latest runner (ci/unit, e.g. PR #597 run 36367036870) and fail identically on unmodified main locally, so this is environmental, not a regression.

Evidence

  • Full -m unit run under load (2026-09-27): 29 failures in the first ~57%; one (test_freeze_registry) was a real hash-pin regression in a PR branch and was fixed.
  • Rerun of the remaining 28 on an idle machine: 6 passed, 22 still fail (1h15m for 28 tests).
  • Direct run_validation probe on an idle machine: ok: true in 13.7 s, so typed validation itself works; its time budget is the problem.

Failure signatures (all deadline-shaped):

  • TimeoutError: tool controller response deadline expired (matched_tool_service._request)
  • precheck infrastructure_exclusion: child command timed out after 15.86 seconds
  • aisle.harness.cli monolith check ... timed out after 20 seconds (test_cli_check_selects_bound_worker)
  • downstream: result["process"] is None (TypeError: 'NoneType' object is not subscriptable), DID NOT RAISE KeyboardInterrupt, infrastructure_exclusion == tool_result

Still failing on an idle machine (22)

  • test_dynamic_monolithic_journal.py::test_dynamic_monolithic_child_retains_auditable_authored_failure
  • test_dynamic_typed_journal.py::test_ordinary_typed_child_retains_preparation_and_gate_refusal
  • test_matched_preparation_service.py::test_frontend_run_request_uses_private_current_source_preparation[monolithic]
  • test_matched_preparation_session.py::test_session_runner_provisions_real_arm_precheck[monolithic]
  • test_matched_run_config.py::test_invalid_run_binding_refuses_before_rollout[hash|purpose|runtime|stage]
  • test_matched_run_launch.py::test_actual_run_child_retains_json_refusal_and_exact_launch[typed|monolithic]
  • test_matched_run_launch.py::test_constructed_tool_run_reaches_real_child_preflight[typed|monolithic]
  • test_matched_run_launch.py::test_controller_bootstrap_imports_explicit_bound_dependency
  • test_matched_run_launch.py::test_run_controller_reaps_child_on_wait_failure[False|True]
  • test_matched_run_launch.py::test_run_controller_wait_failure_stops_detached_descendant[False-configured|True-configured]
  • test_matched_worker_journal.py::test_actual_worker_precheck_journal_rejects_unindexed_collection_file[extra|missing_index]
  • test_monolith_worker_config.py::test_cli_check_selects_bound_worker
  • test_monolithic_run_prepare.py::test_monolithic_preparation_uses_current_source_and_real_worker[0|31]

Load-sensitive only (passed when idle): the 6 in test_matched_dynamic_run.py and test_matched_run_config.py::test_bound_typed_dispatch_passes_verified_stage_factory_to_rollout.

Leading hypothesis (unconfirmed)

Microsoft Defender for Endpoint (Endpoint Security system extension active; wdavdaemon ~17% CPU at idle) scans the freshly copied worker packages (numpy/pyarrow trees copied into per-test temp dirs) and new process executions, pushing child startup past the 15-30 s budgets. CI runners have no EDR. A UTM/QEMU VM was also running. First import of numpy+pyarrow from a fresh copy took 2.2 s vs 1.2 s from the venv.

Next steps

  • With IT: Defender exclusion for the pytest temp root ($TMPDIR/pytest-of-$USER) and .venv, then rerun the 22; confirms or refutes the hypothesis.
  • Measure per-phase child startup time (worker bundle copy, sandbox exec, first import) to see which budget is exceeded.
  • Decide whether local deadlines should scale (e.g. an env-configurable test multiplier) without loosening the measured-record budgets (CON-5 determinism, frozen budgets unaffected).
  • The full local -m unit run once hung for 4.5 h; still unexplained. Consider pytest-timeout in the dev extras.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions