Handover safety, dusk-anchored cadence, and slots that can actually run - #3
Merged
Conversation
… not happen
Three failure modes of the handover bookkeeping, all seen live, share one
root: `finished` doubles as the "handed over" marker and the chimera
program id is a bare sqlite rowid.
* The queue clean deleted the row of the program the scheduler is
EXECUTING. The scheduler-program thread then dies on ObjectDeletedError
as soon as it logs the abort it just handled - every operator lock
during a program does this (opd-40 2026-07-28, the abort landed 6 min
after the delete) - and the freed rowid goes to the next handover, so a
completion is applied to whichever program inherited the id (2026-07-26:
eta Car recorded as observed out of a hand-loaded bias run). The clean
now asks the scheduler which program is running and leaves that row (and
its in-memory handover record) alone.
* A completion could credit an observation to a program that never
started. Attribution now requires the program's own `program_begin`:
a SUCCESS for a program that never began is either the reused-rowid
mis-delivery above (the program is still queued, so the event is
ignored) or a program the scheduler finished without running it
("Program not valid anymore"), in which case the handover is released
and the program goes back on offer.
* A handed-but-unrun program whose chimera row vanished was lost for good:
the recovery sweep only recognised a `chimera_id` still in the queue, and
the dangling branch cleared the link but left `finished = True`. BRUCH's
3.4 h block and four focus occurrences died that way on 2026-07-26.
`Program.handed_at` now records the handover itself and is cleared when
a completion is applied, so "handed but unrun" survives the death of the
chimera row. A program whose row is finished still counts as run.
An ABORTED completion is an operator decision, not a program failure: it no
longer counts towards `max_consecutive_errors`, and the program keeps its
open handover so the next clean re-offers it.
`times: [0, 3, 6, 10]` is documented as hours after the evening twilight, but parse_time_entry anchored on `obs_start` — which `--tonight` sets to NOW. A 04:00 UT replan on 2026-07-28 therefore produced 04:00/07:00/10:00/ 14:00 instead of the night's 21:58/00:58/03:58/07:58: two occurrences past dawn, one expired unrun, and no further focus for the rest of the night. Nothing warned; the rows look perfectly plausible in the database. Numeric entries now count from the dusk that opened the night (the site knows it; the window start no longer defines the cadence), and an occurrence resolving outside the observing window is dropped at generation time — the treatment absolute UT entries already got — instead of sitting in `timeddb` unrunnable and, with nothing later to supersede it, unexpirable.
The allocator lays slots on a fixed `slot_len` cadence and stops when the slot START passes the window end; whether the BLOCK fits in what is left of the night was only checked later, per poll, by check_conditions. With slot_len (1500 s) longer than the block (1260 s), EXOPL's nineteenth slot on 2026-07-27 carried a block ending 08:22:03 against a night end of 08:18:54. It was rejected on every poll for the rest of the night, so the project's queue never legitimately emptied, night_status showed an unfinished program that is unrunnable by construction, and a `process-queue` preview reported 0.00 h open shutter for a night that was in fact fully booked. Block length is known at allocation time (`obsblock.length`, stored at ingest), so it joins the per-slot eligibility mask: a block too long for the remaining night is simply not a candidate, and a shorter one can still take the slot.
`obsblock.length` was estimated with two global constants — 12 s of readout per frame and 600 s per autofocus sweep. Measured on opd-40 (QHY600, 1x1, fits_rice) the real numbers are ~2.8 s/frame and ~270 s, so stored lengths run about 4x long, and the same wrong number lands in three places: the night-end fit check rejects blocks that would have finished (EXOPL's last visit: 1260 s estimated, 985 s real, 3 min inside the night), the derived slot length gives ~24 visits where ~37 fit, and `process-queue` reports open-shutter and idle times off by the same margin. The overhead is a property of the camera, its binning and its compression, so no constant can be right for every deployment. `add-observing-block` and `add-project-inputs` now take --readout-overhead / --autofocus-overhead / --autoflat-frame-overhead, defaulting to today's values so nothing changes silently, and the estimate is printed per block at ingest.
astroufsc/chimera#271 made `ChimeraObject.get_site()` the only way an object reaches the site and dropped the Scheduler's `site:` option. The harness built its Manager without a site and configured one on the scheduler, so the scheduler-program thread raised ObjectNotFoundException and the script hung short of its exit - reported as a 300 s subprocess timeout, which reads as "slow" rather than "broken". The Manager now gets the Site injected (as production does, and as the core's own test fixture does), the scheduler's dead `site:` key is gone, and the script arms a hard internal deadline that dumps every thread stack before exiting non-zero, so the next hang diagnoses itself.
Takes main's stricter plot-log fallback (resolve an orphaned log entry by name only when exactly one target carries it); this branch's own cli/robobs.py changes are unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Six unfixed scheduler/robobs items from
lna40/docs/PENDING_ISSUES.md, allseen live on opd-40. Each commit is independent and carries a discriminating
test (every new test was verified to fail without its fix).
Handover bookkeeping (issues #37, #38, and the robobs half of #41)
finisheddoubles as the "handed over" marker and the chimera program id isa bare sqlite rowid. Three failure modes follow:
scheduler-program thread then dies on
ObjectDeletedErroras it logs theabort it just handled — every operator lock during a program does this
(2026-07-28) — and the freed rowid goes to the next handover, so a
completion is applied to whichever program inherited the id (2026-07-26:
eta Car recorded as observed out of a hand-loaded bias run). The clean now
asks the scheduler which program is running and leaves that row alone.
requires the program's own
program_begin. A SUCCESS for a program thatnever began is either the mis-delivery above (still queued → the event is
ignored) or a program the scheduler finished without running ("Program not
valid anymore") → the handover is released and the program goes back on
offer.
Program.handed_at(new column, migrated inopen_database) records thehandover itself and is cleared when a completion is applied, so
"handed but unrun" survives the death of the chimera row. BRUCH's 3.4 h
block and four focus occurrences died that way on 2026-07-26.
An ABORTED completion is an operator decision, not a program failure: it no
longer counts towards
max_consecutive_errors, and the program keeps itsopen handover so the next clean re-offers it.
times:cadence anchored on dusk (issue #42)times: [0, 3, 6, 10]means hours after the evening twilight, but the anchorwas
obs_start— which--tonightsets to NOW. A 04:00 UT replan produced04:00/07:00/10:00/14:00 instead of 21:58/00:58/03:58/07:58: two occurrences
past dawn and no further focus for the night, with nothing warning. Numeric
entries now count from the night's dusk, and an occurrence resolving outside
the window is dropped at generation time (as absolute UT entries already
were) instead of sitting in
timeddbunrunnable and unexpirable.Slots that cannot finish before dawn (issue #40)
The allocator bounded the slot START by the window, not the block END: with
slot_len(1500 s) longer than the block (1260 s), EXOPL's 19th slot on2026-07-27 ended 3 min after the night did. It was rejected on every poll for
the rest of the night, the queue never legitimately emptied, and a
process-queuepreview reported 0.00 h open shutter for a fully bookednight. Block length is known at allocation time, so it joins the per-slot
eligibility mask — a shorter block can still take the last slot.
Per-site block-length overheads (issue #43)
obsblock.lengthused two global constants (12 s readout/frame, 600 s focussweep) where opd-40 measures ~2.8 s and ~270 s — lengths ~4x long, and the
same wrong number lands in the night-end fit check, the derived slot length
and the
process-queuesimulation. The overhead belongs to the camera, itsbinning and its compression, so
add-observing-block/add-project-inputsnow take--readout-overhead/--autofocus-overhead/--autoflat-frame-overhead, defaulting to today's values so nothing changessilently.
Integration harness (issue #46)
astroufsc/chimera#271 made
get_site()the only accessor and dropped theScheduler's
site:option, sotests/integration/fast_night.pyhung and thetest reported a bare 300 s subprocess timeout. The Manager now gets the Site
injected, and the script arms its own deadline that dumps thread stacks, so
the next hang diagnoses itself. The compressed-night run passes again,
which is also an end-to-end check of the handover changes above against a
real bus and scheduler.
Companion core-side fixes (issues #41 and #20) are pushed to
astroufsc/chimera#255.
Unit tests: 174 passed. Integration:
test_compressed_nightALL PASS.