Repository navigation
test: 22 real-worker unit tests exceed deadlines on local macOS dev machine (CI macOS passes) #598
Description
Summary
On the maintainer's local Mac, 22 unit tests that launch real sandboxed workers / child processes fail by hitting internal deadlines. The same tests pass on GitHub's macos-latest runner (ci/unit, e.g. PR #597 run 36367036870) and fail identically on unmodified main locally, so this is environmental, not a regression.
Evidence
- Full
-m unitrun under load (2026-09-27): 29 failures in the first ~57%; one (test_freeze_registry) was a real hash-pin regression in a PR branch and was fixed. - Rerun of the remaining 28 on an idle machine: 6 passed, 22 still fail (1h15m for 28 tests).
- Direct
run_validationprobe on an idle machine:ok: truein 13.7 s, so typed validation itself works; its time budget is the problem.
Failure signatures (all deadline-shaped):
TimeoutError: tool controller response deadline expired(matched_tool_service._request)- precheck
infrastructure_exclusion: child commandtimed out after 15.86 seconds aisle.harness.cli monolith check ... timed out after 20 seconds(test_cli_check_selects_bound_worker)- downstream:
result["process"]isNone(TypeError: 'NoneType' object is not subscriptable),DID NOT RAISE KeyboardInterrupt,infrastructure_exclusion == tool_result
Still failing on an idle machine (22)
test_dynamic_monolithic_journal.py::test_dynamic_monolithic_child_retains_auditable_authored_failuretest_dynamic_typed_journal.py::test_ordinary_typed_child_retains_preparation_and_gate_refusaltest_matched_preparation_service.py::test_frontend_run_request_uses_private_current_source_preparation[monolithic]test_matched_preparation_session.py::test_session_runner_provisions_real_arm_precheck[monolithic]test_matched_run_config.py::test_invalid_run_binding_refuses_before_rollout[hash|purpose|runtime|stage]test_matched_run_launch.py::test_actual_run_child_retains_json_refusal_and_exact_launch[typed|monolithic]test_matched_run_launch.py::test_constructed_tool_run_reaches_real_child_preflight[typed|monolithic]test_matched_run_launch.py::test_controller_bootstrap_imports_explicit_bound_dependencytest_matched_run_launch.py::test_run_controller_reaps_child_on_wait_failure[False|True]test_matched_run_launch.py::test_run_controller_wait_failure_stops_detached_descendant[False-configured|True-configured]test_matched_worker_journal.py::test_actual_worker_precheck_journal_rejects_unindexed_collection_file[extra|missing_index]test_monolith_worker_config.py::test_cli_check_selects_bound_workertest_monolithic_run_prepare.py::test_monolithic_preparation_uses_current_source_and_real_worker[0|31]
Load-sensitive only (passed when idle): the 6 in test_matched_dynamic_run.py and test_matched_run_config.py::test_bound_typed_dispatch_passes_verified_stage_factory_to_rollout.
Leading hypothesis (unconfirmed)
Microsoft Defender for Endpoint (Endpoint Security system extension active; wdavdaemon ~17% CPU at idle) scans the freshly copied worker packages (numpy/pyarrow trees copied into per-test temp dirs) and new process executions, pushing child startup past the 15-30 s budgets. CI runners have no EDR. A UTM/QEMU VM was also running. First import of numpy+pyarrow from a fresh copy took 2.2 s vs 1.2 s from the venv.
Next steps
- With IT: Defender exclusion for the pytest temp root (
$TMPDIR/pytest-of-$USER) and.venv, then rerun the 22; confirms or refutes the hypothesis. - Measure per-phase child startup time (worker bundle copy, sandbox exec, first import) to see which budget is exceeded.
- Decide whether local deadlines should scale (e.g. an env-configurable test multiplier) without loosening the measured-record budgets (CON-5 determinism, frozen budgets unaffected).
- The full local
-m unitrun once hung for 4.5 h; still unexplained. Considerpytest-timeoutin the dev extras.
Activity