Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion TIME_WASTERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Signature β†’ cause β†’ what to do. One lesson per line, no incident history.
- **Confirm a cause by REMOVING it and re-running, before writing the fix.** Binary-search by cutting half the code.
- **A/B two revisions on ONE machine, back to back.** Byte-identical output (same SHA-256, empty `ImageChops.difference` bbox) ends the question. Two machines β€” or one machine hours apart β€” compares host state as much as code.
- **"Fails on clean main too" β‰  environmental.** A shared harness mistake fails on clean main as well.
- **Host flake vs regression, from the log alone:** `wait_for_idle: timed out after 1 iterations` means the process was never scheduled (a real non-convergence burns thousands of iterations in the same wall clock); sum the gaps between `Running robot_<name>...` lines excluding the suspect for a load index (the suite reproduces within a second β€” 843/843/844s β€” so a 970s run is unmistakable); then `gh workflow run heavy-selfhosted.yml --ref <branch>` to re-run the identical commit. Do that dispatch first, not last.
- **Host flake vs regression, from the log alone:** read the diagnostic fields, not the iteration count. `wait_for_idle: timed out after 1 iterations` means the process was never scheduled, but the converse does not hold: a huge count rules starvation *out* and proves nothing else. `needs_update=false, has_animations=false, waiting_for_present=true` is a slow compositor no matter how many iterations ran β€” `robot_idle_fps_after_tab_walk` burned 1,280,577 of them with composition already settled, and passed in isolation and in a full 121-test suite on the same commit. Only fields that accuse the application justify blaming it. Then sum the gaps between `Running robot_<name>...` lines excluding the suspect for a load index (the suite reproduces within a second β€” 843/843/844s β€” so a 970s run is unmistakable); then `gh workflow run heavy-selfhosted.yml --ref <branch>` to re-run the identical commit. Do that dispatch first, not last.
- **A failure one hair over its bound with a timing metric that moved with it is contention.** A real leak grows every cycle and clears the tolerance by a mile. Never widen the bound β€” for `robot_text_handle_cycle_stability` the accumulation ratio *is* the guard.
- CI preserves full stdout on the runner (`Preserving robot result artifacts after status 1: <dir>`) β€” complete, unlike the truncated Actions log.
- Before acting on a proxy metric, read how the tool decides. `scripts/public_api_test_coverage.py` matched names as *substrings* (`with_timeout` "covered" by `exit_with_timeout`) and scanned `crates/` only, so the robot suite β€” the sole exercise most of the driver API gets β€” counted as untested.
Expand Down Expand Up @@ -175,3 +175,10 @@ Signature β†’ cause β†’ what to do. One lesson per line, no incident history.
- **`Span::mixed_site()` does not shield generated `let` patterns from call-site `const` items** β€” pattern resolution treats a visible const as a const pattern, and item lookup for mixed-site tokens happens at the call site. `macro_rules!` has the identical hole (probe: a macro emitting `let value = 1u32` under a call-site `const value: u8` fails E0308 "interpreted as a constant"); only nightly `def_site` hygiene closes it. Mixed-site still isolates generated locals from user *locals* β€” worth doing β€” but a module const named exactly like a crate-internal generated binding is unfixable on stable.
- Host-dependent dependency-budget trap: `just dep-budget` (and so `just budgets` / `just ci`) can be red on a Mac while the same tree is green in CI. The budget walks `cargo tree --duplicates`, which resolves for the host platform only, and the CI budgets job runs on Linux (`.github/workflows/rust.yml`, "architecture budgets (linux)"). Any Apple-only duplicate is therefore invisible to CI and unmissable locally β€” on 2026-08-27 it was `objc2`/`objc2-app-kit`/`objc2-foundation` (0.5.2 via `accesskit_macos`, plus 0.6.4). Before blaming your change, re-run the budget on a stashed tree; a duplicate that reproduces on a clean baseline is platform-scoped, not yours.
- Doubled event-delivery diagnosis: when a composition-scoped collector reports every platform event exactly twice, count the *service registry's observers* before suspecting a double publish β€” two live registrations feeding one collector look identical to one publisher firing twice, and only the observer count separates them. `two_event_streams_in_one_composable_each_deliver_exactly_once` in `crates/cranpose-core/tests/effects_and_frames.rs` is the harness shape: a fake service that counts registrations, plus per-publish delivery assertions.
- **A robot `wait_for_idle` timeout with `waiting_for_present=true` is a host-load symptom, not a composition bug.** The budget is an iteration count, not wall clock, so on a machine under load (an Android emulator and Android Studio alone put this one at `load_1m=24/10`) the app can still be legitimately mid-present when the count runs out β€” `needs_update=false, has_animations=false` says composition already settled. Check `uptime` and re-run the one example (`just robot-one <name>`) before believing it: the same test passed in isolation and the full 121-test suite passed on the second run, same commit.
- **Targeted `-p <crate>` test runs miss the source-hygiene gates, and they are the ones that take main red.** `apps/desktop-demo/tests/source_hygiene_aliases.rs` scans *every* crate's sources β€” `workspace_tests_do_not_default_to_tmpfs_paths` rejects a `/tmp/` literal anywhere, even in a pure in-memory fixture that never opens the path, and `no_cargo_config_enables_unstable_web_sys_bindings` guards the rustflags. A per-crate gate set can be entirely green while these fail, and a sandboxed reviewer's `just test` can die at the socket-bound `cranpose-services::peer` tests before ever reaching them: both happened at once, so a `/tmp/` literal reached main and four sessions then fixed the same line in parallel (#503, #504, #506, #507). Run `cargo test --profile ci -p desktop-app --test source_hygiene_aliases` alongside targeted runs, or the full `just test`, before pushing. And when a gate does go red, `gh pr list --search "<failing test name>"` before writing a patch: a break that is *in* main is not cleared by rebasing, and someone else's fix is usually already open.
- **A full disk reports itself as a compile or test failure, never as a disk error.** The signatures are `failed to build archive ...: failed to open object file (os error 2)`, `extern location for <crate> does not exist: .../deps/lib<crate>-*.rmeta`, and a test binary that `could not execute process ... (never executed)` β€” all of which read as a dependency or code problem. They are truncated artifacts from a build that was killed mid-write. Check `df -h /` first. Two follow-on traps: `cargo clean --profile <p>` can itself die partway on the corrupted tree and print `No such file or directory` while leaving it broken, so confirm the directory is actually gone or `mv` it aside and let the build recreate it; and at genuinely zero bytes every tool that writes a temp file fails, including the ones you would use to diagnose, so free space before anything else. On this workspace `target/` reaches ~200G across profiles and a review worktree adds ~60G more.
- **A parser that reads `cargo tree` output must strip ANSI, not just the box-drawing characters.** `.github/workflows/rust.yml` sets `CARGO_TERM_COLOR: always`, and cargo honours it even through a pipe, so a line that reads `β”‚ └── name` locally arrives as `ESC[2mβ”‚ESC[0m ESC[2m└──ESC[0m name` in CI β€” a `starts_with('β”‚')` guard never fires and the name comes out carrying the drawing. It is invisible on every dev machine because a pipe auto-disables colour: `cargo tree | od -c` shows no `033` bytes, `CARGO_TERM_COLOR=always cargo tree | od -c` shows `033 [ 2 m` ahead of every prefix. Reproduce any CI-only text-parsing difference with `CARGO_TERM_COLOR=always <command>`, and fix it in both layers β€” strip ANSI in the parser so any colour source is handled, and pass `--color never` so it is never generated.
- **A CI waiter that greps for "pending" fires on an output gap.** `until ! gh pr checks <n> 2>/dev/null | grep -q pending` reads "the word is absent" as "the run finished", so one transient gh failure ends the wait exactly like completion β€” and the `2>/dev/null` hides the error that would have shown it. `gh pr checks` also exits 8 while checks are still queued, so a waiter keyed on its status reports failure while its own banner says complete. Require all three: gh exited 0, output non-empty, and every check terminal via `[.statusCheckRollup[] | (.status // .state // "UNKNOWN")]` rejecting `QUEUED|IN_PROGRESS|PENDING|WAITING|REQUESTED|UNKNOWN`. That still only sees *registered* checks, so a partial list reads as complete β€” assert the expected set by name; a cranpose PR carries seven (Android release build, architecture budgets, fmt + tests + clippy, iOS build, robot e2e, robot external captures, wasm build). Require *pass*, not "nothing failed" β€” a skipped or cancelled check is not green. Then check what the merge actually did rather than printing success after calling it: a `gh pr merge` refused for a conflict still returns to the next line, and its exit status must be read directly, never through a pipe (same trap as the `| tail` entry above). Finish by confirming the squash commit is an ancestor of `origin/main` with the expected subject β€” `state == MERGED` can be set by auto-merge, a merge queue or an admin action, so it never asserts that *your* merge is what landed.
- **Verify the verifier: a guard you have not watched reject a bad input is not a guard.** Every hardening step is itself a predicate nobody tested, and the test is the easy thing to get silently wrong. `timeout` does not exist on macOS, so `timeout 8 ./wait.sh && echo FIRED || echo held` prints "held" from `command not found` β€” a green result for a script that never ran, and AGENTS.md sends people to `macm3` over ssh where a `timeout`-based test copied from Linux fails exactly this way. Feed the guard the states it exists to catch and watch each one: for a CI gate that is empty rollup, partial-but-all-green, one QUEUED, one FAILURE, one CANCELLED, one SKIPPED, an unexpected extra check, and only then all-green. The partial case is not hypothetical β€” a PR was observed going "no checks reported" to 4 registered to 7, so a gate that trusts the registered list can pass before the slowest job is created.
- **An app's own background work is measured as framework frame cost.** Scroll on a 2018-class Android device read as 15-17 fps with `present` p50 29-33 ms; the app was in fact still running on-device inference, settled at 442% CPU before a finger touched the screen. Re-measured on the same scene with the app quiet at 0-3% CPU: 22-28 fps, `present` p50 13.5-17 ms β€” the headline numbers were inflated about 1.5x. Note what did *not* move: the ranked causes filed alongside them β€” render passes, isolated layers, and a 35% layer-cache hit rate re-measured at 34.6% with 1.78 MP of blurred offscreen per frame β€” reproduced on the quiet device, two of them worse than filed. Contamination inflates the headline figures you quote in the title; it does not necessarily touch the per-frame counters underneath, so re-verify rather than retracting wholesale. Before trusting any on-device frame number, poll `adb shell top` for the package until its CPU settles and print what it settled at; a run that does not report its settled CPU is not a measurement, and the same applies to any app doing background indexing or sync. Never A/B across two package names either β€” separate installs carry separate app data and settings, so the arms differ in more than the build (theirs disagreed on `vsync_period_ms`, 15.841 vs 16.632, which is how the contamination surfaced). Install both arms over the same package, or flip them inside one binary with a `debug.cranpose.*` property override as `segment_surface` and `pipeline_disk_cache` already do. `idle_iters` tells you whether a telemetry window was a real scroll window or mostly idle.