cronstable is a fork of yacron, continuing from yacron 0.19. The 1.0.x entries below document the fork; the entries from 0.19.0 onward document the history of the original yacron project, on which cronstable is based.
Defect fixes from a full-application review.
- Windows: cancelling a job (executionTimeout, Replace, shutdown, or the cancel API) now kills the process tree while its root is still alive. Before, the real workload of a shell-form command could survive as an orphan, and a Forbid job could double-run after a timeout.
- The Forbid/Replace launch gate is serialised per job. A manual start racing the scheduled fire (or a double-click on Run) could previously launch two instances of a Forbid job.
- DAG catch-up after a suspend or a forward clock jump honours
onMissedand the same catch-up limit plain jobs use, instead of replaying every missed occurrence. - Duplicate job names are refused at config load; previously all but the last definition silently never ran.
- A web listener whose every bind fails (port still held by a draining predecessor, say) is retried on the next housekeeping pass instead of staying down for the life of the process.
- SSE log tails end at web teardown instead of waiting out aiohttp's 60-second shutdown timeout, so a web, TLS, or MCP config change no longer stalls scheduling while open dashboard tails drain.
- CORS preflights reach the
/mcpOPTIONS route with authentication on, so browser MCP clients onmcp.allowedOriginscan connect to a token-protected daemon. - The MCP
cron_decide_gatetool requires theapprovetoken scope, like the REST decision route it fronts. - Bearer-token redaction covers the full RFC 6750 charset: tokens carrying
+,/or~no longer escape redaction, whole or in part. The Basic pattern learned the base64url alphabet too. - The durable store keeps record names monotonic across a backward clock step. Pruning could otherwise delete a just-written record while keeping stale future-dated ones, and the missed-run/retry watermarks could skip new history.
- Stream names carrying lone surrogates (from non-UTF-8 crontab filenames) round-trip the store audit exactly; the orphan-blob sweep no longer deletes artifact payloads such streams still reference.
- Removing
expand:from a task while a run is mid-flight no longer wedges that run: the recorded fan-out keeps dispatching and the run reaches a terminal state. - Boot reconciliation of DAG runs isolates a stalled or unreadable run document. One bad document used to abort state rehydration entirely, which also skipped starting the job API.
- Custom
web.metrics.durationBucketsno longer discard the persisted duration histogram on every restart. - Dashboard: the drawer Logs pane re-attaches when a new run starts producing output; a poll that never settles can no longer freeze all data loading behind a still-green "live" indicator; the live CPU/RSS chip and the command cell update when their rendered values change.
- TUI: a run finishing in the local future (clock skew against a remote daemon) no longer crashes the heatmap, and 8-bit C1 escape sequences from job output are stripped like their 7-bit forms.
- The MCP stdio bridge pins its stdio to UTF-8, so non-ASCII tool results no longer kill it on Windows consoles.
- CI: a fork pull request from a branch named
maincan no longer cancel an in-flight release, and the pyinstaller Docker build works from the repo-root context again.
- Code cleanup - removal of dead and unused code
- Author email update
- Logo changes: simplification
- Address GitHub security findings
A performance-audit release, the follow-up to 1.2.32's engine pass. That release took the hot frames a profiler shows; this one takes the distributed costs only end-to-end measurement shows, across the durable state store, the run loop's between-fires bookkeeping, the web and metrics endpoints, config parsing, the cluster paths, and both UIs. It adds no configuration surface. Where a rewrite touches observable output, the old and new paths were pinned against each other during review: the cron-expression work by a 3,000-case differential against 1.2.32's engine (the golden vectors are untouched), redaction byte-for-byte over a hostile corpus, and the Prometheus value formatter over 9,000 fuzz cases. The benchmark gate that will hold these wins grows from 8 to 24 armed metrics, and the audit surfaced and fixed a long-standing Windows locking race (see Fixes).
- Blocking store calls dispatch to pooled daemon threads instead of spawning a thread per operation, which cost about 130 microseconds of pure dispatch per call, several times the store work itself on a read from a warm page cache. Abandonability is unchanged: a worker rejoins the pool only once its operation returns, so one wedged in a dead-mount syscall is never handed more work and the next call spawns a replacement.
- Lease and coordination traffic gets its own lane: a dedicated slot pool, and an exemption from the write rate limiter, so a burst of bulk record writes (or bulk workers wedged on a hung mount) can never queue a lease renew past its TTL and expire a live holder's fence.
- Re-reads of unchanged record files are served from a validated in-memory cache, capped by entry count and total bytes. Entries are admitted only after full schema validation, every hit hands the caller a private copy, and strict reads bypass the cache in both directions.
- Documents a caller has already validated can be dumped as trusted,
skipping the portability pre-walk; the dump's own failure path re-walks
and raises the same
UnsupportedValueit always did. - Per-operation store statistics (count, errors, seconds, measured on the worker thread around the store work itself) accumulate per label and surface through the backend's stats.
- The due-fire heap tolerates stale entries and compacts in one pass only when they outnumber live jobs, instead of rebuilding eagerly.
- Pause refresh and cross-node retry-claim scans run on their own intervals, re-anchored when the state backend generation changes, rather than riding every housekeeping tick; the claim scan enumerates the store's retry streams once instead of probing one stream per configured job.
- Reload reparse runs with the garbage collector held: the parse's
temporary object storm no longer triggers collections over the live job
set mid-reload, and the config signature that decides whether anything
changed is computed off the event loop. Two new gated metrics
(
config.reload_gc_100k,mem.gc_pause_100k) watch exactly this. ResourceMonitor.stop()reuses the ticker's last process-tree snapshot for the final sample instead of a fresh table walk; per-member readings are taken live, and members spawned inside the final interval are absent from that sample, the same blind spot the periodic samples already carry.
GET /andGET /jobsanswer with a strong validator and gzip: a repeat dashboard load revalidates into an empty 304 instead of resending the document, and clients that accept gzip get the compressed body.GET /metricsrenders in two phases, snapshotting the metric families and building the exposition text on the executor; the per-label block memo introduced in 1.2.32 now persists across scrapes (label values are part of its key, so it can never serve a stale block).- JSON responses are written from pre-encoded bytes: the Content-Type
is bare
application/json(bodies are UTF-8) and non-ASCII values are sent raw rather than\uXXXX-escaped. - SSE log-tail replay is one write of the retained buffer instead of a write per retained line, and per-line frames are encoded through the same trusted-dump fast path.
- statsd endpoints are opened once and reused (refreshed after 60 seconds, never cached on a failed open, queued bytes bounded) instead of an open-send-close per emit.
- The jobs table diffs per-row signatures and rebuilds exactly the rows whose rendered inputs moved; countdown cells keep ticking client-side. The pendulum logo's LQR gain seed is computed once per parameter set and cached.
- The TUI's line clipping and padding take an ASCII fast path, SGR color runs dispatch on a single token scan, heat-strip and ANSI-scrub results are cached with pruning, wrapped-log windows are sliced by arithmetic instead of wrapping the whole buffer, and the log search carries its match state incrementally across appends instead of rescanning the tail.
- Included files ride the content-hash parse cache that already served config directories, so an unchanged include is not reparsed; cycle and overlap detection are unchanged.
- A crontab parsed once is carried through the defaults merge instead
of being reparsed per job, schedule lint results are cached per parse,
and a job whose
onLateblock is the untouched default settles that validation with one identity test. - Job digests share a per-reload memo for the config blocks jobs share (the digest bytes are unchanged, so persisted fingerprints and the golden are unaffected), and SLA payload views are precomputed at parse time.
- The cron-expression engine got a further pass over its search internals and its equality check (resolved-text comparison first), pinned by the differential above; parse error messages are unchanged.
- Rendezvous-hash ownership streams its candidate bytes and memoizes
answers against the peer-state mutation generation, the gossiped fleet
summary body is cached as bytes and replayed for 304 responses, and
@rebootadvertising reuses the same gates as the membership test. - The etcd campaign tries a plain linearizable range read first and falls to the create-if-absent transaction only when the key is empty; the Kubernetes backend gets its own client pool. Lease fencing semantics are unchanged in both.
- Redaction pre-screens each line with a casefolded gate for the secret keywords before running the full pattern pass; import-time checks pin the gate letters to the keyword set, and output is byte-identical to 1.2.32.
- The daemon no longer imports PyNaCl to learn it exists: the startup
probe asks importlib for the module spec (the real import cost about 10
ms and 1 MB of RSS on every start, for a reporter most installs never
configure). The import happens at the first sealed-box use; a PyNaCl
installation that is findable but broken now surfaces there, as a
per-send
PushError, instead of at config load. --validate-configand--job-set-idanswer without building a scheduler, and thejobcli/mcp/tuientry stubs register their flags without importing asyncio or the daemon; a parity suite asserts the stubs match the real registrations flag for flag.- The push log-tail trim bisects to the drop count instead of removing one line at a time (verified equivalent over 400 randomized payloads).
- 16 more metrics are release-gated (state mutation, reload GC, idle
wake rate, SSE fan-out, resource-monitor stop, DAG advance, cluster
ownership, web UI render and append, push sealing among them), bringing
the expected-gated ledger to 24;
push.seal_500arms now that the baseline release ships the module. - Five new absolute ceilings wait in a
proposedblock (not read by the comparer) until one release publishes a CI-observed value to size them against; the local measurements recorded there (for examplestate.mutate_document_1kat 1.161 s,webui.render_term_5kat 69 ms) exist to make that later sizing auditable. - About 790 lines of new tests pin the rewrites: the worker pool's wedge and spawn-failure paths, record-cache isolation, the Prometheus two-phase seam, resource-monitor sampling, CLI stub parity, and fast path versus slow path agreement in the portable JSON codec.
- Two schedulers sharing a Windows state directory could crash a claim
pass on first contact with a fresh lock file. The lock bootstrap
(msvcrt needs a byte present before it can lock one) raced: both sides
could observe the empty file, and the loser's own bootstrap write then
landed on the winner's just-locked byte range, which Windows rejects, so
a
PermissionErrorescaped the lease acquire instead of the loser waiting its turn. Present since the durable store's lock files were introduced; this release's pooled lease dispatch made rivals arrive close enough together to hit it. The loser now falls through and contends on the lock; two regression tests cover the simulated loss and the real byte-range collision.
An alerting-reach release, with a round of hot-path tuning alongside it. A
fifth reporter delivers end-to-end encrypted push alerts to paired devices
through a hosted relay that never sees plaintext; a device-pairing API and a
dashboard QR panel manage those pairings; a new GET /whoami tells any client
what its bearer token may do; and an opt-in Bonjour/mDNS advert lets a
companion app on the same network find the daemon without a typed URL. Both
capabilities are optional extras (push, discovery) and both fail closed at
config load: a config that asks for either on an install without the library
refuses to start instead of silently not alerting or not advertising.
A full CPU-efficiency review then took the remaining cost out of the cron engine, the Prometheus scrape and the run loop's per-tick bookkeeping. Every figure in those sections is from an A/B benchmark of this release against the previous one on the same machine, and every change is output-identical: the engine's answers are pinned by the golden vectors plus a 104,244-case differential against the previous implementation, and a scrape renders byte-for-byte the same exposition text.
- A fifth reporter,
push, seals every alert to each paired device's public key (a libsodium sealed box: X25519 + XSalsa20-Poly1305) and hands the ciphertext to a hosted relay that forwards it to the platform push service (APNs). The relay sees only a device token, a ciphertext, an opaque collapse-id hash, a priority, and an event flag, never job names, hostnames, or log lines, so a self-hosted daemon can use a shared relay without trusting it. See the Push Notifications wiki page; the versioned wire contract isdocs/relay-protocol.md. report: push:rides the existing report schema ononFailure/onPermanentFailure/onSuccess/onLateand undernotify.report, so DAG failures, approval gates, and leader/quorum events can push too. Its keys:enabled(defaultfalse),priority(time-sensitiveorpassive, relayed as the APNs interruption level), andincludeLogTail(defaulttrue).- A daemon-global
push:section says where alerts go and where pairings live:push.relay.url(required, http(s)) plus an optionalpush.relay.timeout, and registry storage that rides the durablestate:store when one is configured (one document per device, cluster-visible, never swept by state GC) or a localpush.devicesFileotherwise. - The sealed payload always fits the platform cap: the ciphertext is sized so the relay's final APNs JSON stays under APNs' 4096-byte limit, trimming the log tail oldest-first (the newest lines carry the failure) and keeping the alert's identity (name, kind, host) intact.
- Push is a new optional extra (
push, PyNaCl) and fails closed: apush:section without PyNaCl installed,report.push.enabledanywhere without apush:section, apush:section with neither astate:section nordevicesFile, and apush:section on a daemon whose web API listens on a routable address with noweb.authToken/web.authTokens(the/push/devicespairing endpoints would answer anyone who can reach the listener;push.allowUnauthenticated: trueis the override) are allConfigErrors at parse time. The release binaries bundle PyNaCl per architecture behind a sealed-box verification step (a broken build is dropped); a lane that cannot build it ships without the extra, and the config error then says so.
- Four new endpoints manage the paired-device registry:
GET /push/deviceslists pairings (push tokens redacted to their trailing characters),POST /push/devicespairs a device ({name, platform, publicKey, pushToken}; the same public key pairing again updates its record in place and keeps its id, so revocation references stay stable),DELETE /push/devices/{id}revokes one, andPOST /push/devices/{id}/testround-trips a test alert through the relay so a silent phone is debuggable from the dashboard. All ride the existing listeners, bearer tokens, and scopes (viewto list,controlto mutate) and answer404until apush:section is configured, so a reload that adds the section needs no web-app restart. - The dashboard gains a "Pair a device" panel (command palette and
settings): a QR code of
{v: 1, name, url, token}plus the same JSON as a copyable string, with a warning when the stored token is the all-scopes one, pointing at a scopedweb.authTokensentry for phones instead.
- A new introspection endpoint describes the presented bearer token:
{authenticated, label, scopes, allScopes}, so a companion app can show what it is allowed to do and the dashboard can warn before its pairing QR hands a phone the all-scopes token. With no token configured there is no auth middleware to match against:authenticatedisfalse,allScopesistrue, and every scope is effectively granted.
web.bonjour: trueadvertises the web API as a_cronstable._tcpmDNS service, so a companion app (ordns-sd -B _cronstable._tcp) finds the daemon without a typed URL. The advert names one LAN-reachable listener (the first bound https listen entry another machine can dial, else the first such http one; loopback and unix listeners are never advertised) and carries the instance name (the hostname, or the map form'sname:override), that listener's actually bound TCP port (correct even for an ephemeral:0listen) and scheme, and a TXT recordv(the daemon version); its SRV target is a dedicated<name>-cronstable.local.hostname, never the machine's own.localname. No secrets are advertised; a discovered client still needs a bearer token to read anything. See the LAN Discovery wiki page.- It requires the new
discoveryextra (python-zeroconf) and a TCP listener, both enforced at parse time:web.bonjourwithout the library, or with everyweb.listenentry a unix socket, is aConfigError. Unlike push, a runtime mDNS failure only logs and leaves the advert off until the next config apply: discovery is a convenience and must never take down a scheduler.
- The day walk stops calling a method per candidate day. Both day columns
are plain value sets in all but the L / W /
#schedules, and the unrestricted spelling of either column has already expanded to that column's full range, so the both-restricted AND rule needs no special case: such a schedule now tests a day with two set lookups inline, carrying the weekday forward instead of building a freshdatetime.datefor it. The_day_matches/_dom_matchescall pair it replaces was about 28% of anext()on the profile, because the loop runs once per candidate day and a sparse schedule (0 0 29 2 *) walks most of a month to find its answer. Schedules that do use the L / W /#families take the old path unchanged. - The month length is computed, not looked up.
calendar.monthrangevalidates its argument through anenum.Monthlookup and computes the month's first weekday, the half of its result pair this engine never reads; the arithmetic that replaces it measured about 12x cheaper, and it runs once per candidate month and once pertest(). - The time-of-day columns are binary-searched. Locating the first
matching hour, minute and second used to scan the sorted columns one value
at a time, so a plain
* * * * *seeded at 10:30 stepped through roughly 40 values;bisectgets there in a couple of probes. - A per-call copy is gone from every search entry point.
next,prevandoccurrenceseach opened by stripping a tzinfo to get a naive civil label, which for an already-naive argument only bought a copy of it; the aware paths ofnextandoccurrencesthen overwrote that label with a properly seeded one without ever reading it. - Together:
next()over 20k plain tabs is 37% faster,test()over 200k matches 38%, occurrence enumeration 36%,next()over complex tabs 18%, and zonednext()across real DST transitions 15%. The scheduler inherits it: reseeding a 100k-job index is 39% faster, a cold 100k-job build 12%, and a due-fire pass 16%. The iCal feed (17%) and the schedule preview endpoints (21%) ride the same searches.
- A sample's
{k="v",...}block is memoized for the pass. One job's label set recurs across about nine families in a scrape (each counter, each gauge), and the block was reassembled from a generator expression, astr.joinand aformatper label every single time; only the escaping was cached. Keying the memo on the label items collapses that to one assembly per distinct label set, and the one- and two-label shapes that are nearly every sample skip the generator machinery entirely on a miss. The memo lives and dies with the one render pass, so nothing is served stale across scrapes. - The per-sample line is built inline in the renderer instead of through
a helper whose call frame and
(name, block)tuple cost more than the two lines they saved. Rendering a 500-job scrape is 29% faster; on a 3,000-job fleet the exposition step measured 85ms before and 71ms now, for the same 72,006 samples and byte-identical output.
- The sub-minute check is memoized.
run()asks whether any enabled job fires at second granularity on every loop iteration, which a second-level job makes once a second, and the underlying scan is O(all jobs). It was worst-cased exactly where it hurts most, since the scan only short-circuits early when a second-level job happens to sit near the front of the job set. The answer can only change on a reload, so it now rides the same cache lifecycle as the job-set fingerprint and is cleared wherever the job set is swapped. /jobs,/status,/summaryand the gossiped fleet summaries read the clock once per request instead of once per job. Besides the redundant work at fleet scale, this makes each snapshot internally consistent: every countdown is now measured from one instant rather than from instants drifting apart across the loop. The/jobspayload for 500 jobs is 8% faster, and the MCP tool dispatch that shares these builders 24%.
A reporting, orchestration-visibility, and API-contract release. Report
payloads now carry the run's host, schedule, start time and ledger id; a global
defaults: block finally reaches DAG tasks; a new notify: block reports on
DAG failures, approval gates, and leadership changes (not just job runs); the
REST API gains a single-job and a batched-summary endpoint; the web API can now
issue scoped, per-device bearer tokens so a phone or wallboard need not
carry an all-powerful token; and the whole HTTP control API is now described by
a CI-checked OpenAPI spec. Every change is useful to existing webhook/ntfy/CLI
users, not just to API clients. (Rename this Unreleased heading to the cut
version at release time; see Contributing.)
template_varsgainshost,schedule,started_atandrun_id, so a webhook, ntfy, mail or Sentry report can identify the run (which node ran it, its crontab line (object schedules rendered), the ISO-8601 start instant, and its durable-ledger id) without the template digging throughenvironment.started_at/run_idareNonebefore a run starts and on anonLatebreach (which describes a run that did not happen);hostandscheduleare always populated. Existing templates are unaffected.- The shell reporter exports
CRONSTABLE_HOST,CRONSTABLE_RUN_IDandCRONSTABLE_STARTED_ATalongside the existingCRONSTABLE_*variables, so a notify script sees the same run context.
- A DAG task inherits the file's
defaults:block just as a job does: globalshell,environment,env_file, capture,monitorResources, run-scopedsecrets, and reporter (onFailure/onSuccess) config now reach DAG tasks, with the task's own value winning on any key it sets. - DAG task runs now fire their
onFailure/onSuccessreporters, set per-task (a new report-only task key; there is noonFailure.retryon a task, attempts stay graph-driven) or inherited fromdefaults:. Every failed attempt reports viaonFailure; cancelled and replaced instances do not report, matching job semantics. Reports are spawned off the reaper, so a slow reporter never stalls completion handling or the graph advance. Previously a task template was built over the built-in defaults only, so a global reporter or environment silently skipped DAG tasks. Graph-shape fields (dependsOn,triggerRule,retries,expand,onReject, the poke settings) are never touched bydefaults:, and the DAG's synthetic schedule-trigger job stays on the built-in defaults so a global reporter fires per DAG run, not on every tick.
- A new top-level
notify:block reports on daemon and orchestration events, over the same four reporters (sentry / mail / shell / webhook) a job uses. It fires ondag_failure(a DAG run ended failed),approval_waiting(an approval gate began awaiting a decision, once per gate),leader_change(this node acquired or lost scheduled-job leadership), andquorum_loss(this node left quorum). An optionaleventsallow-list filters which fire; the default is all. Its default templates key on the event (event/subject/message) rather than the completed/failed job wording, and it has its own Sentry grouping. Until now only job runs could report; DAG failures, gate waits, and cluster events never fired a notification. Notifications are fire-and-forget and never block the scheduler, cluster loop, or a DAG advance.
GET /jobs/{name}returns one job's detail in the identical shape as an entry ofGET /jobs, so a client can refresh a single job without pulling and filtering the whole fleet. (The same detail was already an MCP tool,cron_get_job; this puts it on REST too.)GET /summaryreturns one batched fleet overview for an at-a-glance client (a widget, a status tile): fleet job counts (total / enabled / disabled / running / paused / failing / never-fires), the soonest upcoming fire, this node's identity, and its compact cluster role: one small poll instead of folding the whole/jobsarray.
- The HTTP control API is now described by an OpenAPI 3.0 spec at
docs/openapi.yaml, kept honest by two CI checks:.github/scripts/check_openapi.py(tox -e openapi) validates the document itself (schema, refs, duplicated keys), andtests/test_openapi.pydiffs the spec's paths and methods against the served route table (cronstable.cron.WEB_ROUTES, now the single source the aiohttp app builds its routes from) in both directions, so a route added or renamed without a spec edit fails the suite. It is the contract a generated client is built from; the HTTP-API wiki page remains the field-by-field reference.
- A new
web.authTokenslist issues per-device bearer tokens, each with its own scopes (view/control/approve), so a phone, a wallboard, or a CI trigger need not carry the all-powerfulweb.authToken.viewcovers every read-onlyGET;controlcovers the mutatingPOSTs (start / cancel / pause / resume, DAG trigger / backfill) andPOST /mcp;approvecovers only the DAG approval-gate decision.controlandapproveeach implyview. A recognised token that lacks a route's scope is now403 Forbidden(naming the token and the missing scope), distinct from the401for an unknown token. New routes get a safe default (aGETneedsview, any other method needscontrol), so nothing is ever unguarded by omission. - The scalar
web.authTokenis unchanged: it remains an all-scopes token, every configured token is accepted, and both keys compose. Each scoped entry resolves its secret from the samevalue/fromFile/fromEnvVarsources and fails closed the same way (a configured-but-empty entry refuses to start the web API); two tokens resolving to the same secret are refused at startup, since matching is by secret and only one entry's scopes could apply.mcp.enabled's fail-closed gate is satisfied by either key. - Revoke a device by dropping its entry and reloading; its optional
labelidentifies it in logs and 403 bodies. These transport scopes are unrelated to the loopback job-state API's key-valuescope. See the HTTP-API wiki page.
A configuration release. A YAML config can now pull its own values from the
environment. ${VAR} and ${VAR:-default} in any string value are expanded
from cronstable's process environment when the file loads, so one config file
serves many environments without a wrapper entrypoint templating the YAML from
env first. A listen address, a state path, a timezone, an include path or a
webhook URL can come from an environment variable while the config itself stays
declarative. It is opt-in by syntax: a config that never writes ${...}
behaves exactly as before, and expansion introduces no new keys.
${VAR}expands to an environment variable and${VAR:-default}supplies a fallback when the variable is unset or set-but-empty (the shell's:-).$$is a literal$, so$${VAR}stays${VAR}. Only these braced forms are recognised: a lone$, a bare$VAR, or a malformed${...}is left exactly as written, so a value that never used the syntax passes through untouched.- An unset variable with no default is a hard
ConfigErrorthat names the variable, the config value it appeared in (e.g.web.listen[0]), and the file. A missing deploy-time variable therefore fails at load, andcronstable --validate-configcatches it in CI or a deploy check before the scheduler ever starts. - Expansion runs after the document is validated, over the parsed values,
so it reaches every string-typed field. A consequence is that a numeric key
cannot itself be a bare
${VAR}(its value fails schema validation before expansion is reached); put the variable inside a string, as inlisten: ["0.0.0.0:${PORT}"]. The section builders then validate the expanded value, so an interpolatedstate.paththat resolves to empty is rejected the usual way. - Each file is expanded against the environment as it is parsed. An
includepath is an ordinary string value, so it too may be built from a variable (- ${ENVIRONMENT:-prod}.yaml), and an included file resolves its own references. An environment value is never rescanned for further${...}, so a variable cannot expand into another expansion. - Job, DAG-task and shell-reporter
command/shellare left untouched: their${VAR}belongs to the runtime shell, which expands it against the job's own environment (env_file, per-jobenvironment, staged secrets) at execution time, not the daemon's. Theloggingsection is likewise left for Python'slogging.config, whose$-style formatters legitimately write${asctime}in aformatstring; interpolating it would fail an otherwise valid logging config to load. - The job-set id is taken over the expanded config, so a job set that interpolates per-environment values gets a different id per environment by design (the deployments really are running different job sets). Keep interpolated variables out of fingerprinted job fields if one id must compare across environments.
A performance and hardening release. The cron engine, the config loader, the web and metrics endpoints, the cluster mesh and the durable store each do the same work with fewer CPU cycles, fewer allocations and fewer redundant reads, and two guards bound memory on the durable write and mapped read paths. It introduces no new configuration.
- The fire walk stops recomputing the month length for every candidate
day.
CronTab._day_matchesnow receives the month's last day from the caller that already holds it (test, and the forward and backward civil walks), rather than callingcalendar.monthrangeagain per day. On a schedule-pressure scan over a thousand per-minute jobs that removed on the order of a million redundant calls per request. The result is byte for byte the same, checked by a differential fuzz against the previous engine across 221k cases spanning next, prev, test and occurrences in naive and DST-aware frames. - The DST linter scans each zone once instead of once per job. The schedule linter used to walk 366 days of UTC offsets for every zoned job on every config load. The transition days in a year are now computed once per (zone, year) and cached, so a fleet sharing a zone pays the scan a single time and every later reload reuses it. The findings are unchanged.
- The collision heatmap walks each distinct schedule once. A fleet
duplicates schedules heavily, and the fire enumeration depends only on the
schedule and the zone, never on the job name, so
schedule_pressurenow walks each distinct (schedule, zone) once and replays the resulting cells for the jobs that share it.
- A one-file edit no longer reparses the whole config directory. When a
watched file changes, the reload used to rebuild every job in the directory
through strictyaml. A per-file cache now returns the unchanged files'
already-parsed configs and reruns the parser only for the file that changed.
An entry is validated by hashing each input's bytes (the file, its
transitive
includes and its jobs'env_files), so a size-preserving edit whose modification time is pinned back (a coarse-granularity network filesystem, or tooling such asrsync -a,cp --preserve=timestampsor a backup restore) is still picked up. Reading the bytes is work the reload already did; only the parse is skipped. - A job's schedule is compiled once per load. The linter reads the
CronTabthe scheduler already built instead of parsing the expression a second time. - A shared
env_fileis read once per document rather than once per job that names it.
- JSON responses are encoded with orjson. A shared response helper
serializes the data endpoints with orjson and compact separators, falling
back to the standard library for any value orjson rejects, and every web
route, the cluster
/peerbody and the server-sent-events line encoder go through it. GET /jobsanswers conditional requests. The response carries a contentETagand honorsIf-None-Matchwith a304, so a poll that finds nothing changed does not re-encode or re-send the body. The tag is computed over the payload with each job's relative countdown swapped for its absolute next-fire instant, so it holds steady while the countdown ticks and moves the moment a fire lands or any other field changes. For a large fleet the encode runs off the event loop.- The SLA trends drawer stops rescanning the ledger on every poll.
GET /jobs/{name}/trendsserves its built payload for a few seconds and drops that cache the instant a run for the job finishes, so a drawer several clients are watching reads up toTREND_SCAN_LIMITledger records at most once per window instead of once per poll.
- Label escaping is skipped when there is nothing to escape, and each distinct label value is escaped once per scrape rather than once per sample, so a job name that appears across dozens of series is processed a single time.
- The MCP metrics query filters structured samples directly.
cron_query_metricsreads the metric families in place rather than rendering the full exposition text and parsing it back with a regular expression.
- Ownership is derived once per pass, not once per job. The spread-owner
and available-member sets are memoized and the member names are pre-encoded,
so
job_ownerandavailable_job_ownerno longer rebuild the member list for every job; the per-job, per-peer encoding collapses to one derivation a pass.
- The newest-record lookup stops at the first match.
artifact_get_recordscans the stream once and stops early instead of paging the whole listing twice, cutting a representative newest-record read by close to a third. _derive_maxno longer sorts the whole listing; it anchors on the maximum and sorts only the small set of newer records the fold needs for its tie-break.- Durable writes are bounded in time and in number. Every run-record,
inflight, counter and archive write runs under the state-operation timeout,
and the pending-write set is capped: past the cap a write is shed and counted
as
cronstable_state_dropped_writes{kind="overflow"}rather than queued without limit. - Mapped-XCom reads are bounded in size. A mapped fan-in checks a record's declared size and refuses an oversized blob before fetching it, and a list past the item cap is returned unwalked, so one runaway upstream cannot exhaust a downstream task's memory.
A transport release. Every HTTP surface cronstable serves could previously
be encrypted only by putting a reverse proxy in front of it; now the daemon
can do it itself. web.listen accepts https:// addresses, the job-facing
state API accepts one too, the certificates behind both are declared in
config rather than in a sidecar, a web certificate replaced in place is
picked up while the daemon runs, and every client that dials a web listener
gained the same four options for saying which CA to trust and what
certificate to present. The listener can also be told to require a client
certificate, so it authenticates its callers at the transport instead of
only encrypting them. Each block below is optional and empty by default.
The machinery is the cluster mesh's, which has spoken mutual TLS since
1.1.x: its context builders, its on-disk certificate fingerprint and its
"does the new material load yet?" dry run moved into a shared
cronstable/tlsutil.py, a standard-library-only leaf module, and the
cluster's own functions delegate to it under their existing names.
cluster.tls is unchanged.
web.listenacceptshttps://entries, served from a newweb.tlsblock whosecertandkeyare required together. The context is built once per app start and applied per listener, not per runner, so a listen list can mixhttp://andhttps://entries serving the same app on one process, each with its own transport.unix://listeners stay plaintext: they are confined to the host's filesystem, where the socket's own permissions (web.socketMode) are the access control.web.tls.clientCarequires a client certificate signed by that CA (mutual TLS). It is required, never merely requested: a client that presents nothing is refused at the handshake rather than completing it and being sorted out later. The CA file is consequently the caller allowlist, because a server does no hostname verification and accepts any certificate that CA ever signed, so it wants to be a CA minted for this purpose rather than a shared organisational one.mcp.enabledis now allowed on a routable listener with noweb.authTokenwhenweb.tls.clientCais set, since mTLS authenticates the caller. Plainhttps://does not qualify and still raises the original configuration error: encryption is not authentication.- Misconfigurations fail at parse time, so
--validate-configcatches them: a cert without its key or a key without its cert, aclientCawith no certificate of the listener's own, TLS material with nohttps://listener to use it (it would be silently ignored), and anhttps://listener with no material to serve. Whether the files exist or load is deliberately not checked there; config parsing touches no filesystem,--validate-configmay run somewhere that is not the deployment target, and a mounted Kubernetes secret need not exist at first boot. That check happens at the listener, which logs and declines to start rather than falling back to cleartext on a port an operator asked to encrypt.
state.jobApi.listenaccepts anhttps://URL served fromstate.jobApi.tls.certand.key, with the same paired validation. The endpoint hands every run a bearer token and stages that job's secrets, so it is the surface where cleartext cost the most.- Jobs are handed a trust anchor.
state.jobApi.tls.cais injected into every run asCRONSTABLE_STATE_CACERTand read by the in-job CLI, socronstable state|cursor|lock|artifact|idempotent|secretcan verify a certificate no public root signed, which is the normal case for an internally-issued one. Note the asymmetry withweb.tls.clientCa: thiscais the client-side anchor the daemon gives its jobs, not a CA that authenticates callers. The in-job CLI has no TLS flags at all and no way to switch verification off, by design: nothing running inside a job should be able to downgrade the channel carrying that job's own secrets. - An in-place rotation of the endpoint's certificate is picked up while
the daemon runs, the same as a web certificate. The
cert/keyfiles are fingerprinted as the listener loaded them; a change (same paths, new bytes, which is how cert-manager, Vault and Kubernetes secret refreshes renew) rebuilds just this listener on the next housekeeping pass, without disturbing the store backend or its leases, and gated on the new material loading first so a half-written refresh keeps the old certificate up.state.jobApi.tls.cais exempt, because jobs read it fresh by path. - A wildcard host over
https://is a configuration error. Jobs dial the address they are handed and no certificate carries a SAN for every interface, sohttps://0.0.0.0:9000could only ever fail verification. Name the interface explicitly. - The advertised URL now prefers the configured host over the bound one,
which also fixes a latent bug: a plaintext
listen: 0.0.0.0:9000used to advertiseCRONSTABLE_STATE_URL=http://0.0.0.0:9000, which happened to work on Linux and failed on Windows. The bound address is still used when nothing was configured, where the ephemeral loopback default makes it the right answer. - Mutual TLS is deliberately not offered on this endpoint. The per-run bearer token already authenticates the caller, and requiring client certificates would mean injecting key material into every job's environment.
allowNonLoopbackBindchanged its wording, and gained a warning. Its configuration error no longer asserts that the endpoint is plaintext or tells the reader to add a reverse proxy, because TLS is now available in process. Enabling it alongside a plaintext off-host listen logs a warning naming the exposure at every boot. It is a warning and not an error: the documented pairing was a reverse proxy, and that remains a valid answer.
- An in-place rotation restarts the web listener. The SSL context is
built once and never reloaded, so new bytes at the same paths, which is
exactly how cert-manager, Vault and a Kubernetes secret refresh renew,
would otherwise be invisible until the old certificate expired. An
(mtime, size)fingerprint of the configured files is compared on the ordinary housekeeping reload and a change restarts the listener. - The restart is gated on the new material loading. Make before break is impossible here, because the new runner binds the port the old one still holds, so a half-written rotation (none of those refreshes is atomic across the files) would otherwise tear down a working listener and then fail to rebuild it. When the new material does not load, the running listener is kept, a warning says why, and the next reload retries.
- The restart drops connections open at that moment, including the SSE log streams the dashboard and the terminal dashboard hold open. They reconnect on their own; a live log tail blips.
- The job state API listener does not do any of this. It builds its context once at startup, so rotating its certificate needs a daemon restart.
--cacert,--client-cert,--client-keyand--insecure, with the identical names and meanings incronstable tuiand thecronstable mcpstdio bridge (and in the thin__main__stubs that advertise them without importing either). Each falls back to an environment variable (CRONSTABLE_WEB_CACERT,CRONSTABLE_WEB_CLIENT_CERT,CRONSTABLE_WEB_CLIENT_KEY,CRONSTABLE_WEB_INSECURE), the same flag then env precedence the bearer token already used, so one exported set of variables serves every client. With none of them set the clients build no context at all and keep exactly the transport they had.--insecurewarns on stderr every time. Verification is off but theAuthorizationheader is still sent, so the bearer token goes to whoever answers the connection.--client-keywithout--client-certis refused. A key alone cannot present an identity, so the only alternative to failing is accepting the flag and ignoring it, which would leave the caller believing it had authenticated to a listener that had in fact refused it.- A failed handshake reads as a failed handshake.
urllibdelivers one wrapped as a generic connection error, which would have sent an operator hunting a firewall or a wrong port while the socket connected fine; both the MCP bridge and the in-job CLI now tell the two apart and name the CA variable or flag involved. - Client hostname verification is on, so a certificate must cover the
name actually dialled:
https://127.0.0.1:8443needs an IP SAN for127.0.0.1andhttps://localhost:8443needs a DNS SAN. This is the most likely first-run failure. - The webhook reporter has no CA option yet.
report.webhookstill verifies against the system trust store only, so an endpoint holding an internally-issued or self-signed certificate cannot be reported to.
See Listener TLS in the wiki for the full configuration, certificate requirements and the trust models.
Two operator features land this release: a runtime job pause and per-job SLA
(late-run) monitoring, each wired through the HTTP API, the web and terminal
dashboards, Prometheus, and MCP. Both are control-and-alerting layers only:
neither changes what a job runs or when, and both stay off the scheduling hot
path. A pause is runtime-only state, and the sla/onLate config keys are
excluded from the job-set-id fingerprint, so replicas never read either as
drift. The rest of the branch hardens their edge cases; one internal change
stands apart from both.
POST /jobs/{name}/pauseholds a job's scheduled fires;/resumeends the hold. The body is optional:durationSeconds(default3600, range1to2592000) or, exclusively, an absoluteuntil(future, at most 30 days out, naive timestamps read as UTC), plus anote(at most 500 chars) andby(at most 100 chars). An unknown job is404; both time keys at once, an out-of-range or past deadline, a wrong type, or an oversized field is400. Re-pausing overwrites the window, which is how a pause is extended. Both routes sit behindweb.authTokenand the cross-site request defense, likestartandcancel.- A pause is always bounded. Every pause carries an
until; there is no indefinite pause (editenabled: falsefor that). Expiry takes no timer: an elapsed window reads as absent everywhere at once, and the once-a-minute housekeeping pass sweeps the record and logs the auto-resume, with nothing to reconcile if the daemon restarts across the deadline. - Skipped slots are recorded, not silent. Each due slot inside the window
writes a synthetic run-ledger row with
outcome: "skipped"andskip_reason: "paused"(nostarted_at, noexit_code), so history shows a deliberate skip rather than a gap, and stamps neither success nor failure. - Interactions are conservative. Catch-up owes nothing for a paused
window, including slots the daemon slept through. Pending retries defer
rather than cancel. Manual
startstill launches a paused job (a disabled one is refused409), andcanceland running instances are untouched. A paused@rebootjob defers its once-per-boot run instead of forfeiting it, firing when the pause lifts (still exactly once per OS boot). SLA checks are suppressed while paused, and any active OVERDUE clears on the pause itself. - Durable and fleet-wide with a state store. Without a
state:store a pause is in-memory and forgotten on restart. With one, each pause and resume appends to a durablepaused/<job>stream (newest wins): boot rehydrates active windows before the first fire, every node sharing the store honours the pause, and a resume revokes it fleet-wide even on a node that had not yet seen it. Propagation rides the housekeeping pass (up to about a minute; the accepting node applies it immediately), and the fire-time check is memory-only, so an unreadable store never blocks firing. - Every surface shows it.
pausedis always present onGET /jobs({since, until, note, by, channel}ornull);/schedule/whyadds apausednote naming expiry, actor, and note; the dashboards add a Paused status, a⏸chip, a summary pill, and aptoggle; Prometheus addscronstable_job_paused{job_name}and counts skips undercronstable_job_runs_total{status="skipped"}; MCP gainscron_pause_jobandcron_resume_jobin theacttoolset.channelrecords the acting surface (apiormcp).
- A new
sla:block declares three independent thresholds (seconds).maxTimeSinceSuccessSeconds(no successful finish in the window: the dead-man check for a wedged or silently dead job),lateAfterSeconds(a due slot has not started within the window), andmaxRuntimeSeconds(a run has been going longer than the window; observes only, never kills, useexecutionTimeoutto enforce). Each is off (null) until set and must be> 0. - A new
onLatereporting hook, the fourth alongside onFailure / onPermanentFailure / onSuccess. It fires once per breach through the same mail, sentry, shell, and webhook reporters, with defaults reworded for an overdue condition (an "is overdue" mail subject and body naming check, threshold, observed value, and last success, a Slack-compatible webhook body, and the sentry fingerprint["cronstable", "sla", "{{ name }}"]so breaches group apart from run failures). Configuring anonLatereporter with noslathreshold set is a load-timeConfigError(onLate requires sla). - Evaluated in-process, once per minute, memory-only. A state store is not required (it does improve the staleness reference across restarts). With no success on record the check references when the monitor first saw the job, so a fresh boot ages into the breach rather than paging instantly. Disabled and paused jobs are skipped, and time a job spends paused or disabled is credited to the staleness check, so a resumed or re-enabled job gets a fresh window instead of paging the instant it comes back. Under leader election only the owning node evaluates, so one breach pages once and an ownership handoff does not false-page.
- Breaches latch per
(job, check). Entry firesonLateonce, setscronstable_job_late{job_name, check}to1, incrementscronstable_job_sla_breaches_total, and logs a warning; nothing re-fires while the breach holds; recovery clears the gauge and logs, with no recovery report. The latch is in-memory, so a still-breached check reports once more after a restart. Reports dispatch off the scheduler loop, ordered after the same job's in-flight completion reports. - Breach variables.
sla_check,threshold_seconds,observed_seconds, andlast_success_atjoin the standard template set (run-shaped fields empty), and the shell reporter also receives them asCRONSTABLE_SLA_CHECK,CRONSTABLE_SLA_THRESHOLD_SECONDS,CRONSTABLE_SLA_OBSERVED_SECONDS, andCRONSTABLE_LAST_SUCCESS_AT. - Every surface shows it.
GET /jobscarries anslaobject for configured jobs ({thresholds, state, breaches},observed_secondsre-measured at payload time); the dashboards add an OVERDUE badge (row, drawer, wallboard) independent of run status; Prometheus addscronstable_job_lateandcronstable_job_sla_breaches_total; MCP observe toolscron_list_jobsandcron_get_jobreturn the same object. - The monitor cannot report its own death. An in-process check dies with
the daemon; pair it with the external Prometheus staleness alert on
cronstable_job_last_success_timestamp_secondsas the outside backstop.
- The classic-crontab control-character guard is rebuilt from explicit code
points. The refused set (C0 controls minus TAB and LF, DEL and the C1
range, and the Unicode line and paragraph separators) is unchanged, so every
crontab that parsed before parses the same; enumerating the points instead of
a literal range keeps the TAB/LF carve-out visible and clears a
static-analysis false positive (CodeQL
py/overly-large-range).
A bounding and hardening release. The previous two passes made the hot paths cheap; this one puts ceilings on the paths that had none, so a fleet's worst minute is bounded rather than merely fast on a good one. The batching work lands where a fan-out finishes all at once, the ceilings land where a queue or a stream could previously grow with the workload, and the benchmark harness learns to tell a real regression from runner noise. All figures below are from an A/B benchmark of this release against 1.2.24 on the same machine, interleaved rounds, with the scenario sizes noted per figure.
The hardening half comes from a property-based fuzzing campaign that drove the parsers, encoders and state machines with several million generated inputs across 38 surfaces, then re-argued every candidate against the module's own documented contract. The four critical and seventeen high-severity defects it confirmed are fixed in the sections below ("A fuzzing campaign..." onward), each with a regression test pinning the fuzzer's repro; the medium- and low-severity findings are catalogued for a follow-up pass.
- A mapped fan-in used to pay a full run-document rewrite per finished task: each completion took its own locked read-modify-write and fsync of the whole document and its own graph advance, so a thousand instances finishing together cost a thousand of each. Completions are now buffered while the reaper drains a batch and applied through one batched transform per run, followed by a single advance for the whole batch. Every mark keeps its own proc-token, attempt and poke fences and applies against only its own task entry, so a superseded or duplicate completion is still dropped on its own while the rest of the batch lands. Finishing a 1,000-instance fan-in dropped from 13.3s to 12.7ms.
- A completion is now durable at the next reaper flush rather than
the instant the task is reaped. A crash inside that window leaves
the task's entry reading
RUNNING; boot reconciliation recovers it by pid liveness exactly as it recovers a task the daemon died while running, so no outcome is lost, but the recovery path is reconciliation rather than the record itself. - One job's failure to finish no longer strands the rest of its batch. Buffering made the reaper's per-job work shared: a job that raised while being finished (a state backend answering 503 during its finish step is enough) abandoned the remaining jobs in that batch and skipped the flush, leaving completions that earlier jobs in the same batch had already buffered sitting in memory until some unrelated job happened to complete. Finishing a job is now guarded per job, and the flush runs whether or not one of them raised.
/dagsstops re-parsing finished runs. The rollup listed and fully parsed every retained run document of every dag on every call, which a three-second dashboard poll repeated forever. Run terminality is monotonic, so each terminal run's summary is now cached and the rollup reads only keys plus the documents that are new or still running, falling back to a single bulk sweep pastDAG_ROLLUP_BULK_THRESHOLD(8) unread documents or on a backend with no keys-only listing. A warm rollup dropped from 11.1ms to 1.7ms. Switching state backends drops that cache along with the rest of the per-store bookkeeping: run keys are derived from the dag name and logical date, so the new store's live run reuses the key of whatever the old store had finished under it, and nothing rebuilds this cache on a timer to correct it later.
- A live-log subscriber can no longer pin a run's whole output. The
1,000-line ring buffer bounded what the dashboard showed but not what
a subscriber's delivery queue held, so one stalled SSE viewer (a
backgrounded tab, a full TCP window) accumulated every line the job
ever wrote. Subscriber queues are now bounded at
LIVE_LOG_SUBSCRIBER_QUEUE_LIMIT(8,192 lines, ample headroom over the ring) and overflow drops the oldest queued line, so a viewer that falls behind keeps receiving current output instead of a growing backlog. A viewer more than 8,192 lines behind now misses lines in its live tail; captured output, archived runs and failure reports are unaffected, and a reconnect re-snapshots the ring. The number of lines dropped this way is counted per stream. - Artifact and XCom streams bound to their distinct names. A put appended a record and superseded versions accumulated until the job was garbage collected, so a job republishing one name every minute grew its stream without limit. Appends now carry a name-keyed prune that keeps the newest record per name and drops the superseded ones, amortised on the same one-in-eight cadence as the existing bounded prune, with the orphaned blobs reclaimed by the next sweep. Only the newest record per name was ever readable, so nothing that was reachable becomes unreachable, but the older records are now deleted rather than retained, and the first put per stream after upgrade prunes the history already accumulated. Listing a 1,000-put stream of ten names dropped from 820ms to 9.2ms.
- An over-range cron field costs O(1) instead of an allocation.
Enumerating a field materialised its whole range before checking any
value, so
1-2000000000allocated billions of integers before the first bound check could reject it, which also defeated the promise that an unparseable schedule degrades to prose rather than failing. Enumeration is lazy and raises on the first out-of-range value; every schedule that parsed before yields identical values.
onlyIfLastSucceededmaterialised the full 50-record history window on every scheduled fire to find an outcome that is almost always in the newest few records. It now probes the newestDEPENDS_GATE_PROBE(8) records and widens to the full window only when the probe came back full and held nothing but cancellations and skips, so the skip window is preserved exactly. Evaluating the gate dropped from 64.8ms to 33.3ms.
- Every job-spawned thin client paid for the terminal dashboard.
Registering
cronstable mcpandcronstable tuiimported their modules, and importing the TUI runs a 7,000-line module body and pulls inunicodedata's C table. Everycronstable state get,lockandxcom pulla job runs builds that parser first, so two commands almost never invoked taxed the ones invoked constantly. Both subcommands now register as stubs and import their real modules only when dispatched, with a parity test holding the stub flags in lockstep with the real definitions.cronstable --versionfell from 128ms to 113ms, and against the interpreter's own floor cronstable's share of it fell from 108ms to 93ms.
- The fleet matrix rebuilt its nodes-by-jobs grid on every poll.
Relative ages moved out of the built markup into spans the
once-a-second tick refreshes, which lets the render skip the whole
innerHTMLrebuild and reflow when the payload, the failures-only filter and the owner set are all unchanged. - Log search match counts update incrementally as lines arrive instead of rescanning the buffer, with a full rescan only when the query itself changes.
- A hidden tab drops to a 30-second poll, or stops. A backgrounded viewer kept the daemon serving its full jobs, cluster and dags fan-out every interval forever. A hidden tab now stops polling entirely and resyncs once immediately on return rather than waiting out an interval, unless one of the three opt-in features that read the poll is on: desktop notifications, the audible alarm, or the run ledger. With any of those armed it keeps polling at 30 seconds. All three refresh only off the poll and all three exist for the tab nobody is watching, so pausing outright silenced new-failure notifications, left the alarm sounding on whatever the last poll saw, and punched a hole in the ledger for as long as the tab was backgrounded.
- A failed job launch no longer logs the child's environment. The spawn arguments carry a full copy of the daemon's own environment plus the loopback state-API token, and both the debug line on every launch and the error path when a spawn fails formatted that whole structure into the log record. Any secret the operator exported to cronstable, and a live credential for its own state API, reached journald or syslog and anything shipping from them, at a level no production configuration filters out. The environment is now summarised as a count; the argv, the failure and the encoding that make the message useful are unchanged. Output redaction covers archived job output only and never applied here.
- DAG task ids reject control characters. An id reaches log sinks and durable keys verbatim, so an embedded CR or LF could forge or split daemon log lines. The C0 range and DEL are now refused at config load; the printable set is unchanged.
- A statsd prefix cannot inject samples. CR, LF,
:and|are stripped from the configured prefix, none of which is legal in a statsd metric name, so a working prefix is unchanged. - Prometheus label values escape CR as well as LF, backslash and quote. LF remains the only line delimiter, but a raw CR inside a quoted value is not valid in the exposition grammar and can confuse strict scrapers.
- Blob digests are validated before they become paths. A digest is
content-addressed lowercase sha256 hex by construction; anything else
is now refused rather than joined into a filesystem path, so a
crafted
sha256field in a restored archive cannot escape the blob directory.
- A job can no longer brick a named lock for the entire fleet.
lock acquirecoerced its caller-supplied TTL with a barefloat(), so--ttl inf(which argparse accepts without complaint) flowed intoexpires_at = now + inf; orjson persists that asexpiresAt: null, every later read then failed as unreadable, acquire failed closed, release could not repair it and the sweeper refused to reclaim it -- one request left the lock denied to everyone, forever. A non-finitettlorblockSecondsis now a 400, a finite TTL is clamped into [5s,MAX_LOCK_TTL] (30 days), and the idempotency-claim TTL gets the same finiteness check. - Lone surrogates are stopped at the portability gate.
ensure_portablehad no string branch at all, so a value holding an unpaired surrogate (surrogateescape-decoded OS data is enough) passed the gate on every host -- and a stdlib node then persisted a record no orjson node could ever write or parse, splitting the fleet. Strings and object keys are now walked and a codepoint in U+D800..U+DFFF is rejected at write time, identically on both backends. - Durable JSON gains a nesting-depth ceiling. No depth bound was
defined anywhere, so the accepted-document set was whatever each
backend's encoder happened to tolerate: orjson's encoder hard-fails
at 256 while its parser reads to 1024 and the stdlib reaches ~1000
both ways, so a stdlib node could persist a 256..1023-deep document
every orjson node could read but never write back -- permanently
wedging any read-modify-write path, a DAG run advance included. The
gate itself recursed unboundedly and blew the stack (a ~2KB body is
1,000 levels deep). One
MAX_DEPTH(128) is now enforced with an explicit counter on both backends, so a too-deep value is a clean rejection -- never a RecursionError, and never a document only half the fleet can rewrite. - An integer wider than 64 bits is a parse error on every host.
The stdlib parser preserves
18446744073709551616as an exact int the portability gate rejects, while orjson silently narrowed it to a lossy float the gate happily accepted -- so the same upstream bytes produced a mapped fan-out on one node and an empty one on another, and the very pre-check built to catch the value could not see it.loads()now rejects out-of-window integer literals uniformly (a cheap 19-digit prescan gates a verified parse); every boundary value, 19-digit timestamp and big float literal parses exactly as before. - A state scope is matched exactly, never normalised. The scope --
the isolation boundary between jobs' durable state -- was
strip()ed after authorization checked the raw string and before the store named the on-disk path, soreport,report␠andreport\xa0silently collapsed onto one namespace (a job could read and overwrite another job's private state without appearing in anystateAllowedScopes), and a whitespace-padded job's on-disk scope diverged from the GC keep-set built from its exact config name, letting collection delete a live job's artifacts. A scope that is not equal to its ownstrip()is now refused with a clear 400; distinct scope strings can never share a namespace. A job whose YAML name carries a quoted leading/trailing space loses state access until renamed -- previously it was silently sharing another job's.
prev()no longer starves the event loop for a far-future query. For an aware anchor past the 2099 horizon the backward scan stepped one fire instant at a time from the anchor down to the horizon -- measured ~1.8s of CPU per year for a per-minute schedule, roughly four hours forat: 9999-01-01-- andcron_why_no_runruns it synchronously on the scheduler's event loop, so one schema-valid MCP argument (an LLM's hallucinated year is enough) froze dispatch, the web server and cluster heartbeats until the node lost leadership. The scan now starts at the horizon's edge, where the answer lives.occurrences()honours the fold of its start.datetime.now()in a DST zone returnsfold=1for one hour every autumn, and the iterator dropped it, re-offering the repeated hour's first-leg fires -- instants up to a full DST shift in the past, whilenext()correctly named the future one. The schedule preview, the job-explain payload and the TUI all readoccurrences(), so the dashboard showed a "next" fire that had already happened and missed-run detection reported a run the scheduler never performed. Candidates are now compared through UTC against the fold-aware start, exactly asnext()compares them.- A spring-forward no longer swallows the shifted fire. A fire
whose civil label sits inside the gap (
59 2 * * *on changeover day) really happens one shift later -- but oncenow's wall label passed the schedule's label,next()abandoned the day even though the real instant was still ahead, silently dropping that day's run for every daemon start, reload or re-seed landing in the window; 252 of 598 IANA zones were affected, by up to three hours. The scan now rewinds its seed by the offset jump when a recent transition makes that necessary (one extra offset probe otherwise), and discards candidates through the same resolved-UTC guard, so an already-past instant is never returned.next(),occurrences()andprev()now agree on one fire set in absolute time, and a regression test holds that identity across both transition days, both folds and five zones.
- The credential guard redacts what the detector detects. The two
URL-userinfo helpers -- written specifically to keep passwords out of
logs -- located the authority by splitting on a literal
://, with two subtly different fallbacks: a protocol-relative//user:pass@hostendpoint was invisible to both (the password went verbatim into a ConfigError the reload loop re-logs every cycle), and a trailing-://shape made the detector and redactor disagree, so the "always redacted" error path echoed the cleartext secret. Both helpers now share one RFC 3986 authority locator, so nothing the detector flags can reach a log unredacted; the campaign's enumeration of 81 credentialed endpoint shapes had found 31 leaking. - Output redaction is linear again -- and off the event loop. The
URL-password pattern backtracked quadratically on a long
scheme-character run followed by
://and an@-less tail: a clean 4.0x runtime per input doubling on job-controlled stdout, blocking the entire event loop from inside the archive path (extrapolating to days at the default 16MiB line cap). The scheme is now anchored and bounded, restoring amortised-linear scans (2.0x per doubling, 1.35s to 2.5ms at 32KB), and the archive scrub runs in a thread executor so a future pattern regression degrades that one write, not job dispatch. PGPASSWORD=and friends are recognised as the credentials they are. The key pattern's word-boundary lookbehind rejected an alphanumeric prefix, so the separated compound forms (MY_PASSWORD=) redacted while the unseparated vendor forms libpq and friends actually use --PGPASSWORD=,DBPASSWORD=,MYSQLPWD=, exactly whatset -xechoes -- passed through verbatim into an archive stampedredacted: true. The key may now carry a compound prefix (with the scan still provably linear), andREDISCLI_AUTHjoins the list explicitly.
- Crontab lines are physical, LF-delimited lines. Parsing split on
str.splitlines(), whose Unicode line-boundary set (VT, FF, FS/GS/RS, NEL, U+2028/U+2029) is wider than cron's LF-only model -- so text sitting inside a#comment (one line in every editor,cat,git diffand code review) became a live job, a mid-line FF silently truncated a command, and error line numbers drifted from the file's. FF is the classic Emacs page separator, so this needed no malice -- and doubled as a review-evasion vector. Both the parser and the content sniffer now split on LF alone (one trailing CR tolerated), so a comment is a comment for its whole length and every job name and error names the physical line. - Control characters in a live crontab line are refused, with
file:line. A NUL in a command, environment value or
SHELL=built a job the OS can never spawn, and theValueErrorout of the spawn call escaped every guard on the launch path and killed the whole scheduler -- re-fired on every tick. The crontab front end (the sole route to that shape; the YAML reader already refuses control characters) now rejects C0/C1 controls (tab excepted) in live lines with a clear per-line error, and the spawn guard additionally treatsValueErroras an ordinary start failure so any future unspawnable argv is retried by the reaper rather than fatal. A command with a mid-line FF, previously truncated silently, is now refused rather than half-honoured.
- A schedule-object value must be exactly one cron field. The
object form rendered its values into a whitespace-joined line that
the engine re-splits, so a blank value deleted its column and
shifted every later field left --
second: "0"plus a leftoveryear:turned "every second at :00" into "hourly at minute 0", a silent 60x rate change with no error, no warning and no lint finding -- and an embedded space injected a column and shifted fields right. Every provided value must now render as one non-empty, whitespace-free token or the parse fails naming the offending key. - Secret resolution cannot crash-loop the daemon.
_resolve_secretcaughtOSErrorandUnicodeDecodeError, but a NUL in afromFilepath raisesValueErrorand a lone-surrogate path or env-var name raisesUnicodeEncodeError(reachable from a pure-ASCII file via YAML\uescapes) -- and on the per-fire job-secret staging path anything but ConfigError escapes the scheduler loop and recurs at every fire of that job. Both source branches now catch broadly and re-raise as ConfigError naming the key. - strictyaml's own validators can no longer escape config parsing.
Its numeric prefilters accept
'','.','-'and'_', whichfloat()/int()then reject with a bareValueError-- so anexecutionTimeout:left with no value (an ordinary typo, reaching every one of ~26 Float/Int keys) surfaced as a raw traceback at startup and as "please report this as a bug" in the reload log, and a control character anywhere in a file aborted a whole config-directory load via a strictyamlAttributeError. Both are now translated to the ConfigError every caller already handles.
- A destructive MCP gate tests identity, not truthiness.
cron_backfill_dagreadargs.get("dry_run", True), which applies the default only when the key is absent -- anddry_run: nullis precisely how an MCP client or LLM encodes "unspecified", so the two encodings of the same intent diverged into opposite outcomes and a present-but-falsy value started a real backfill on a tool the registry itself marks destructive. Only the literalfalsenow takes the executing branch, mirroring the existing confirm gate. - Non-finite numeric arguments clamp instead of erroring.
1e999is a well-formed JSON number the stdlib parser reads as infinity, andint(inf)raisesOverflowError-- which none of the MCP numeric coercion helpers caught, turning a schema-validlimit/offset/tailon fifteen tool/argument pairs into an opaque -32603 protocol fault the model cannot self-correct from. All three helpers now fall back to their documented clamp or default. - A task retyped to mapped mid-flight fails cleanly instead of
wedging its run forever. Adding
expand:to a task while an entry sat parkedup_for_retryrouted it exclusively through the mapped placeholder path, which only handles fresh pending entries -- so the retry never fired, the run never terminalised, its advance lease was held for the life of the daemon and every advance paid a full document copy to do nothing. Such a stale-shaped entry is now failed with an explanatory reason so the run finishes and the next run expands normally; genuinely in-flight attempts and parked approval gates keep their own recovery paths. - The object-form
schedule:reaches the shell reporter. The reporter's child environment carriedschedule_unparsedverbatim, and for the object form that is a dict -- the spawn died inos.fsencode, soonFailure/onSuccessshell reports silently never executed for any job whose schedule was spelled as an object. The environment now carries the rendered crontab line, as the status payload and Prometheus already do, and the reporter's spawn guard catches the type errors the "never propagated" contract promises to absorb. - The TUI sorts like the web dashboard. The terminal port compared
names by code point -- every uppercase-initial name first -- where
the web uses
localeCompare's root collation, so the two frontends' default first screen disagreed for any mixed-case fleet, on the name column and on every column's tie-break. The TUI now sorts on a collation-equivalent key (case-insensitive primary, lowercase first on a pure case tie), verified against the browser's collation for mixed-case fleets.
- A regression must now clear the measurement's own scatter to gate. Each side's round-to-round coefficient of variation is estimated (from three rounds up, by a median absolute deviation, so one throttled round cannot inflate the band and hide a real regression behind it), the two sides are combined in quadrature, and a change over its declared limit gates only if it also exceeds two of those bands. Unknown scatter never suppresses, so the guard can only make the gate more conservative. Suppressed changes are reported rather than silently dropped, and every metric's band is shown alongside its delta.
- Benchmarks split into two tiers with their own round counts. The cheap deterministic in-process metrics run five rounds per side, which tightens the noise estimate they are judged against; the nine subprocess metrics (cold start, imports, peak RSS) run two, since each round spawns processes.
- Startup metrics are gated on cronstable's own share. Each side's
measured interpreter floor is subtracted before the delta is computed,
so a change is judged against the part cronstable controls rather than
diluted by process spawn and interpreter init. Displayed values stay
the raw totals. The absolute floor a change must also clear is capped
on that same subtracted scale: the 10ms default is sized for the ~40ms
raw totals, and applied to a single-digit-millisecond own share it
swamped the percentage limit entirely, so
startup.import_cronexprcould double cronstable's own import cost and still pass. A startup metric now gates where its declared limit says it does, at 25% of the own share, with a 2ms floor below which the own share's scatter is too large to judge. - A metric that ran on neither side is reported as ungated. A metric skipped on both the baseline and the current side produced no row, no violation and no mention anywhere in the summary, so a gate that had compared nothing still printed an unqualified pass. Both sides' skips are now listed in the report and warned about in the job log, and the pass line carries the count of metrics actually compared.
- The web-dashboard renders are gated rather than recorded.
Playwright was installed only into the current side's environment, so
every
webui.*metric skipped on the baseline, and a metric with no baseline is reported without a gate; the three render benchmarks could not have failed a release at any magnitude. Both sides now install it. Releases predating the?perf=1hook still skip on the baseline, so these gate from the next release onward rather than immediately. - Nine benchmarks were added and three rescaled, covering the DAG
fan-in and claim paths, the artifact stream, the dependency gate, the
TUI's log restyle and search, and the schedule duplicate scan. Three
more time the web dashboard's row, fleet and log-count renders in a
headless browser through a
?perf=1hook that is inert without it. - Regression labels no longer fall outside the chart. A large improvement clamped to the axis put its percentage label on top of the metric-name gutter; a label that will not fit outside its bar is now drawn inside the bar's end, in a near-black that clears contrast against both fills in light and dark themes.
- The release chart draws every compared metric. The diverging bar chart cut the suite to a top-N of movers, so the release image showed a slice rather than the run; it now grows to fit all compared rows, with an alternate-row wash to carry a name across the gutter, tick labels at both ends of the (now taller) plot, and a footnote counting metrics measured on only one side, whose numbers stay in the release-notes table.
- The repository is LF-only and CI enforces it.
.gitattributesnow stores and checks out every text file with LF on all platforms including Windows undercore.autocrlf=true, images are declared binary, and a static-analysis step fails the build if a CRLF or mixed-ending blob ever reaches the index. - The coverage floor rises from 85% to 90%, against a measured 91.9% on the lowest matrix cell, and the suite grows by roughly 336 tests (about 7,300 lines) covering the filesystem state backend's garbage collection, migration, blob and lease internals, the DAG state machine and scheduler, the web payload builders and state inspector, cluster gossip and leadership renewal, resource sampling, the CLI entry point and cron parsing edge cases.
- Images and binaries shed dead weight. The eight container builds uninstall pip from the virtualenv after the last step that needs it, removing 11.4MB per image, and the frozen binary excludes sixteen stdlib modules it never loads (GUI toolkits, curses, readline, sqlite3 and the developer stdlib), removing a further 172KB.
wingetmanifest updates are temporarily warning-only, sincewingetcreate updatebumps an existing manifest and cannot succeed until the initial submission merges intomicrosoft/winget-pkgs.- Build tooling moves to uv 0.11.29 and
pypa/gh-action-pypi-publish1.14.1.
A second performance pass, aimed at the work a fleet repeats continuously: terminal painting, cluster gossip and election gates, durable-cursor reads, DAG document traffic, and the trends endpoint. All figures below are from an A/B benchmark of this release against 1.2.23 on the same machine, with the scenario sizes noted per figure; medians of repeated runs, variance within a few percent.
- The job drawer's log pane used to re-run its ANSI rewrite over every buffered line on every paint (a full 5,000-line tail is 10,000 regex passes per frame, repeated once a second while idle and up to 30 times a second under output floods), then slice out the ~40 visible rows. It now computes the visible window first and transforms only those rows, through a bounded per-line cache that a theme change empties: a steady-state paint over a full tail dropped from 8.6ms to 0.02ms. The DAG logs tab gets the same treatment.
- The multi-tail console merged and sorted every buffered line of every tail per paint (20,000 tuples for four full tails); it now merges only the slice of each tail that can reach the visible window, which is exact because each tail is already time-ordered: 1.8ms to 0.33ms per paint.
- Log search match counts recompute only when the query or the buffer changed, instead of rescanning the whole tail once per frame while the search box is open.
- The election-derived peer sets (mutual agreement, eligibility, bridge
and vouched contenders, the conflict scans) are pure functions of the
last poll round's observations, yet every owner and leader gate check
re-derived the whole cascade, about six times per
job_ownercall. Each derived set is now memoized against a mutation generation that every peer-state write rolls, so the cascade runs once per observation change: ajob_ownercheck against 50 agreed peers dropped from 185us to 36us, and the saving repeats per due job at every fire, per job in the once-a-minute claim scan, and per dashboard poll. - Serving
/peerrebuilt the full payload and its ETag before it could answer a conditional poll with a 304, so a converged round was only cheap for the network, never the CPU. The handler now reuses the last built payload and tag while the election state is unchanged and the build is under a second old (a bound indistinguishable from the poller having polled a second earlier): 256us to 1.2us per steady-state conditional response, once per configured peer per poll interval, with the node-stats header still sampled fresh on every request.
derive_max(the cursor behind catch-up and cross-node retry claims) parsed every retained run record on every call to recompute a maximum that is monotonic by contract. The filesystem backend now keeps a per-stream watermark and parses only records appended since the last call; a wholesale stream wipe (unlike a bounded prune, which always leaves the newest record standing) is detected and drops the memo, so a recreated stream defines its cursor alone. A repeat derive over 1,000 ten-KB records: 64ms to 1.0ms, paid per catch-up job per service pass at boot.
- Reconcile and claim share one read-modify-write: an advance used to pay one locked full-document pass to recover crash-interrupted tasks and a second to expand, propagate, and claim, even when nothing had changed. The two halves now compose into a single transform in the common case (mapped tasks awaiting expansion still pre-read their upstream lists between two RMWs, the old shape), and a read-only pre-scan skips the transform's full-document copy when it can prove no task is claimable, due, recoverable, or terminalisable, erring toward the full pass on any doubt. Advancing a quiescent 1,000-instance fan-out dropped from 4.4ms (two locked parses, two deep copies) to 1.8ms (one locked parse, no copy, no rewrite), at least once a minute per active run plus after every task completion.
- Launch batches stamp their pids in one write: recording each launched task's OS pid cost a full-document rewrite per subprocess, up to 32 per pass against a document holding up to 1,000 task entries. The whole batch now lands in one RMW under the same per-entry proc-token and attempt fences (a stale entry is dropped alone while the rest applies): stamping 32 launches into a 1,000-entry document fell from 364ms to 11ms of store traffic per pass.
GET /jobs/{name}/trends(and MCPcron_get_job_trends) parsed up to 5,000 ledger records and made four full window-filter passes on the scheduler's event loop, stalling launches, gossip, and output pumping for the duration: a 13.8ms stall per request against a 5,000-record ledger. The parse and aggregation now run on the executor like the pressure and calendar payloads, each record's age is computed once for all nested windows in a single pass, and the per-record output-stream allocation is gone. End-to-end latency edged down (244ms to 221ms, the read itself dominating); the loop now spends only thread-switch granularity on the request instead of the whole aggregation.
A performance release: the hot paths flagged by a full efficiency review now do their work once instead of per poll, per map instance, or per stored record. All figures below are from an A/B benchmark of this release against the previous one on the same machine.
/jobs,/status, and the gossiped fleet summaries no longer re-run a cron-engine search per job per request: they read the next-fire index the run loop already maintains (the same source the Prometheus next-run gauge reads), keeping the engine search only for the startup window before the index is seeded. Builds 4-7x faster on a 300-job fleet, and the saving repeats on every dashboard poll and every cluster gossip exchange.- A running job with a dead schedule used to earn its
never_firesflag by searching the remaining calendar horizon on every poll (about a millisecond each time for0 0 31 2 *); the answer now comes from the dead-schedules latch in constant time. A schedule that dies while the daemon runs (a fixed year slipping into the past) now joins that latch with the same warning a seed-time-dead schedule gets, instead of leaving the fire index silently.
- One document read fewer per advance: the crash-reconcile step already observes the run document inside its locked read-modify-write, so the advance reuses that body instead of re-reading the same document before the claim.
- Completion bursts coalesce: when many task completions land at once (a mapped fan-in), the spawned advances collapse into at most the pass already running plus one follow-up that observes the whole burst, instead of one full pass per completion. A burst of 20 completions measured 20 advance passes and 60 durable document operations before, 2 passes and 4 operations now.
- The claim transform is cheaper on large fan-outs: the dependency
verdict is resolved once per task instead of once per map instance, and
the transform's working copy of the run document is cloned through the
orjson fast path when the
speedupsextra is installed. A claim pass over a 1000-instance fan-out dropped from 3.2ms to 1.5ms, all of it time spent holding the document lock.
- Reading an artifact by name probes the newest 64 records first and
only then falls back to the full scan, so the common lookup (a recently
published name) no longer reads the scope's entire publish history: 65ms
to 5ms against a 1500-record stream, for every
cronstable artifact getand every mapped-task expansion. Looking up a name that was never published costs one probe page on top of the old full scan. - The dashboard's XCom tab fetches each value directly by the blob digest its own listing already holds, instead of re-listing the run's whole artifact stream per entry: at 200 entries that is one stream read instead of 201, and the tab renders 2.5x faster.
- Reporters no longer run inline on the reaper: each finished run's report and retry-arm sequence is spawned as its own tracked task, chained per job so overlapping instances of one job are still handled in finish order, and drained in full on graceful shutdown. One slow SMTP server or webhook used to delay every other job's completion, slot release, and retry arming daemon-wide: with a reporter that takes half a second, the next job's completion waited 524ms before and 0.1ms now. Mail reports also gained an overall 60-second bound (aiosmtplib's default only bounds each individual protocol step).
- The reaper is fully event-driven while jobs run: the job-launch signal joins its wait set, replacing the 1-second poll that re-counted the running set. A daemon babysitting one long-running job used to wake 86,400 times a day for that; now it wakes only when a job starts or finishes.
- Job output passthrough batches per pipe drain: mirroring a job's stdout/stderr to the daemon's own used to cost a blocking write and flush on the event loop for every line; lines are now coalesced and written once per drained read. A 20,000-line job drains in 55ms instead of 132ms, and a stalled daemon-stdout pipe no longer freezes the event loop once per line (a broken pipe is logged and skipped; capture and the live web tail are unaffected).
- The bounded-stream prune rides inside the append: every run, retry, in-flight, manifest, counter, and archive append used to be paired with a separate re-list-and-sort prune call that usually deleted nothing. The bound now travels with the append (one store dispatch instead of two) and the actual re-list runs on one append in eight per stream, so a stream may briefly hold up to seven records beyond its bound before the next due prune trims it. Recording one bounded run record: 6.0ms to 4.9ms, half the worker dispatches.
- Serialization sheds its pre-flight walk where it can: with the
speedupsextra installed, the fleet-portability check now walks payloads only for the one hazard orjson cannot catch itself (a NaN or infinite float); oversized integers and non-string keys are refused by the serializer and reported as the same error as before. A run record with a full resource series serializes in 69us instead of 93us, and the lease record written on every ~10s renew, built entirely from the daemon's own values, skips the walk outright: 0.18us instead of 0.89us.
- The etcd backend re-reads the
@reboot-rankey every 60 seconds instead of on every ~5-second renew round once it is synced and has no marks to persist, matching the filesystem backend's cadence: roughly 17,000 fewer etcd reads per node per day. A leadership gain or a known lease loss still forces an immediate unthrottled read-back, so the failover double-fire guard is unchanged. - The DAG adopt scan stops re-reading finished runs: run terminality is
monotonic, so each node remembers which run documents it has seen finish
and the 30-second orphan scan reads only the runs it does not know yet,
discovering the rest from a single key listing (a new backend capability
that lists document keys without opening any body). With
retainRuns: 50that is one directory listing instead of 50 document reads and parses per dag per cycle: 5.8ms to 2.6ms per scan on local disk, with the gap growing on networked stores where each read is a round trip. A periodic full pass and the hourly GC rebuild the cache from actual bodies.
- The terminal dashboard's string machinery gained ASCII fast paths: width measurement and row cutting now short-circuit for escape-free ASCII (nearly every character of every frame), the ANSI matcher runs only at escape characters instead of at every position, and theme colours are compiled to SGR fragments once per theme instead of re-parsing hex per span. Assembling a 60-row frame's strings dropped from 6.2ms to under 0.1ms.
- Concurrently monitored jobs share one process-table snapshot: per-run resource monitors used to each walk the entire process table every sampling tick; a shared ticker now takes one snapshot per tick and each due monitor derives its own process tree from it, keeping per-monitor intervals and the per-member CPU/memory reads unchanged. Eight monitored jobs sample in 2.8ms per tick instead of 27.3ms.
The schedule dialect learns to speak business days, and the scheduler's own fire enumeration becomes something you can put on a calendar.
- Four additive day forms, straight from the Quartz family the dialect
already borrows
?from:L-<n>in day-of-month counts back from the month's final day (L-3= three days before it),<n>Wfires on the weekday nearest dayn(a Saturday 15th resolves to Friday the 14th, a Sunday 15th to Monday the 16th, flipped inward at the month's edges so the fire never leaves the month),LWis the month's last weekday, and<d>#<n>in day-of-week is the month's n-th such weekday (5#3= third Friday, the siblingL5always wanted). The golden compatibility corpus pins the new forms from the engine's own recorded vectors (flagged"extension": true) alongside the legacy ones. - Every schedule surface understands them from parsed ground truth: the
plain-English describers (server and dashboard), the linter (month
reachability generalizes:
31Wwarns about April,L-28in February earns the leap-day-only note), the no-run explainer, semantic duplicate grouping (fri#3equals5#3), pressure, previews, and the MCP schedule authoring tools. Wrong-field uses keep the hint machinery:#outside day-of-week andWoutside day-of-month name the right field, and Quartz's trailing-L (5L) points at theL5spelling. - The dashboard's client engine reaches full parity with the daemon's:
it now parses the whole day-form family (including the legacy
LandL<n>forms it never previewed before), and a differential test replays the entire golden corpus through both engines instant-for-instant. Two degenerate-quirk gaps fixed on the way: a day-of-week range ending in 0 now wraps like the engine (sun-sunfires daily), and bare-start steps in day-of-week expand over 0-6.
GET /calendar.icsandGET /jobs/{name}/calendar.icsserve upcoming fires as standard iCalendar feeds: one event per fire from the scheduler's own occurrence walk in each job's timezone, emitted as UTC instants with stable UIDs (subscribed clients update in place), block lengths from run history (5-minute floor),TRANSP:TRANSPARENTso maintenance windows do not mark on-call busy, and descriptions that carry the schedule and its plain-English reading but never the command line.?days=(default 14, max 60) and?per_job=bound the feed; truncation is announced in-band.- Calendar clients cannot send bearer headers, so with
web.authTokenset the.icspaths (only) accept the token as atokenquery parameter, the secret-address model calendar services use; every other path still refuses query tokens. - A week calendar in the dashboard (the
◫ weektoolbar button): seven browser-local day columns of hue-keyed fire chips with a live now-line, quarter-hour collision splitting, and click-through to each job's Schedule tab, which now links the per-job.icsfeed. Jobs firing more than ~8x/day summarize into a "background hum" strip under the grid instead of flooding it. The demo mirror carries the full view, and its synthetic fleet gained business-day showcase jobs. - The terminal dashboard keeps feature parity: the same panel under
the same palette command ("Toggle week calendar"), terminal-shaped: a
seven-day by 24-hour shaded fire grid, a scrollable UTC agenda of the
calendar-worthy fires, and the identical background-hum rule, computed
locally from the
/jobssnapshot like the pressure panel, so it works against older daemons too.
The in-house cron engine grows a safety net and a toolbox. A schedule that
can never fire again (a fixed past year, 0 0 30 2 *) used to vanish
silently: it simply never entered the fire index. Now it is loud everywhere:
config load logs a never-fires warning, the scheduler warns once when it
drops the job, /status and /jobs report never_fires, and the dashboards
badge it. It stays a warning rather than an error on purpose (a past year is
also the working idiom for parking a job).
- Advisory findings for legal-but-suspect schedules, computed by the new
shared
cronstable/croninfo.pyand reported identically by config-load logging,GET /jobs(schedule_findings), the TUI cron sandbox, and the TUI job drawer's schedule tab (which renders the daemon-computed findings straight from the payload):never-fires, whose message tells an exhausted year column apart from a date that never exists;day-fields-both-restricted(this dialect's AND rule vs. Vixie's OR, the classic crontab-import surprise);uneven-step(*/7minutes fires at :56 then :00 four minutes later);skipped-months(day 31 never occurs in April);leap-day-only; and DST notes (dst-skipped-time,dst-repeated-time) with the actual transition dates, computed in the job's own timezone.
GET /schedule/previewparses, describes, previews and lints any expression with the daemon's own engine, so tooling can ask the daemon itself instead of re-implementing cron.describe_cron/next_firesmoved from the TUI intocronstable/croninfo.py(the TUI re-exports them), so the terminal sandboxes compute with literally the scheduler's code. The web page's client-side preview stays a convenience, but it now applies the engine's AND day rule when day-of-month and day-of-week are both restricted (it used to preview Vixie's OR: fire days the daemon would never run) and phrases its description the way the daemon does.
CronTab.prev(): the backward mirror ofnext(): seconds since the most recent occurrence strictly before now, for missed-run and late-run reasoning without replaying the schedule forward. Bothprev()and the timezone-awarenext()resolve DST edges through real instants (theoccurrences()policy below), so neither reports a negative delay across a spring-forward gap nor mishandles the second leg of a fall-back hour. The naive golden vectors against the replaced parse-crontab library still pass byte-for-byte; the aware DST-edge vectors are deliberately corrected, because the legacy library answers those with civil arithmetic (in the worst case, a negative delay).CronTab.occurrences(): iterate the exact instants the scheduler would fire. Steps through real instants across DST: a spring-forward wall time is yielded once at its shifted label, a fall-back repeat fires its first occurrence only.next_firespreviews now ride this iterator.- Read-only field-set properties (
minutes,hours,days_of_month,last_day_of_month,months,days_of_week,last_days_of_week,years,seconds), so tooling works from the engine's ground truth instead of re-parsing expression text. ?accepted standing alone in the day fields (the Quartz spelling of "unrestricted"; a 7-field Quartz expression now parses verbatim), and parse errors that smell of Quartz (#,W, the seconds-first 6-field layout) carry a hint naming the dialect and how to convert.
- Jenkins-style
Hfields (H,H(a-b),H/n,H(a-b)/n, any field but the year) hash the job's name to a stable slot, killing the:00thundering herd without random jitter, which would break the "was this run late?" question a monitoring product must keep answerable. The hash is a per-field-salted SHA-256 of the job name: identical across restarts, reloads, replicas and versions (the concrete slots are pinned by tests), soH H * * *picks an uncorrelated minute and hour, and a job's bareHminute agrees with itsH/15phase. In day-of-month, every rangelessHform (bareHandH/nalike) hashes over 1 to 28, never skipping short months;H(1-31)opts back in. Renaming a job re-hashes its slots. Resolution happens at parse time, so matching, the next-fire search and semantic equality see plain values; the linter attaches ahashed-slotnote naming the resolved expression,GET /jobsserves it asschedule_resolved, andGET /schedule/previewgrew aseedparameter so sandboxes can resolve prospectiveHschedules. Classic crontab files acceptHlines too (seeded by their line-derived names). Both sandboxes know the form: the web page explains a validHschedule while still flagging an invalid one, the TUI does the same instead of claiming the daemon would reject it, and the TUI job drawer analyzes the resolved spelling from the payload, so anHjob gets the same description, preview and lint as any other.
GET /schedule/pressure, the collision heatmap: every enabled schedule's fires over the next 24h (up to 168h), enumerated with the scheduler's own engine (timezone- and DST-exact, sub-minute schedules weighted, DAG schedules included) and bucketed into an hour-by-minute grid with per-minute histograms, the busiest-minute headline, the empty minutes, and the heaviest cells with their jobs. Disabled and @reboot jobs are excluded and counted. The walk runs on a worker thread over an immutable snapshot, so a big fleet's enumeration cannot stall the scheduler loop.GET /schedule/duplicates: groups of jobs whose schedules fire on the identical instants, by the engine's own semantic equality (*/5equals0-59/5,@hourlyequals0 * * * *) and the resolved timezone, so two midnight jobs in different zones are not called duplicates.GET /schedule/suggest: the least-loaded minute (period=hourly) or minute and hour (period=daily) for a new job, scored on the same 24h fire walk, deterministic ties breaking circularly away from the busiest slot (an idle fleet is told:30, not:00), with runners-up and theHspelling that keeps future jobs spreading themselves.- The web dashboard grew a schedule-pressure card (grid heatmap,
minute histogram, duplicate groups as clickable chips, suggest-a-slot
buttons, UTC/local display toggle) plus a compact pressure strip on the
wallboard; the job drawer shows what an
Hschedule resolved to, and every client-side preview computes from the resolved form. The TUI has the same panel as an overlay, computed on a worker thread from its/jobsand/dagssnapshots with the identical shared analyzers (one fire walk feeds the heatmap and both suggestions), so it works against older daemons. Three read-only MCP tools (cron_schedule_pressure,cron_schedule_duplicates,cron_suggest_slot) serve the same payloads to agents.
GET /schedule/whyanswers "why didn't this job run at 09:00?" from ground truth: given a job and a timestamp, the newcroninfo.why_no_rundecomposes the engine's own match test field by field ("minute matched; day-of-week Tuesday is not in Monday and Friday"), renders each field's accepted values in prose (ranges collapse, weekday and month names, theLforms spelled out), and brackets the probe with the nearest real fire on each side. Notes call out the two semantics that make a miss genuinely confusing: the dialect's day-field AND rule (when exactly one restricted day field matched, classic Vixie cron would have fired) and DST transitions (a skipped wall time fired at its shifted label instead; a repeated one fired once). Aware timestamps convert into the job's own timezone, naive ones read as wall time there;@rebootand disabled jobs answer honestly, and a DAG'sdag:<name>schedule job resolves too.- Three more read-only MCP tools make an agent a schedule author,
not just a reader:
cron_validate_schedule(parse and lint an expression before it becomes a job: the engine's exact error with its Quartz dialect hints, advisory lint findings, the first upcoming fire, and prospectiveHresolution viaseed),cron_explain_schedule(plain-English description plus the next N fires in a chosen zone plus lint, for round-tripping a proposed schedule to a human before it ships), andcron_why_no_run(the explainer above, with a one-line verdict that points atcron_list_runswhen the schedule DID select the instant). All three ride theobservetoolset; the server'sinitializeinstructions steer agents to validate before proposing.
- winget:
winget install ptweezy.cronstableinstalls the self-contained Windows release binary (amd64orarm64), no Python required. A newwingetrelease job updates the manifest inmicrosoft/winget-pkgsautomatically on every release, the same way the existinghomebrewjob keeps the Homebrew tap current.
This release gives the web dashboard a terminal twin: cronstable tui
opens the same board in a terminal -- an SSH session, a tmux pane, a box where
a browser is one window too many. It is a pure client of the daemon's
existing HTTP control API (the daemon itself is untouched by this release),
it adds zero new dependencies (the standard library plus the already-core
aiohttp), and it keeps the web page's muscle memory: the shortcut table is
the same sixteen keys, now enforced by a test that parses the rows out of the
web page's own source. The final cut also reflects an adversarial review of
the whole surface -- the notable outcomes are documented below because they
are load-bearing guarantees, not incidental polish.
-
The whole board makes the trip. The jobs table (status glyphs, next-fire countdowns, duration sparklines, live CPU/memory chips, the owner column under a spread cluster, filter/sort/status segments); the job drawer with the live SSE log tail (search with
n/N, follow/wrap/timestamps toggles, save-to-file), run history with success rate and per-run bars, resources for monitored jobs, and the schedule tab; the fuzzy command palette; the verdict bar, incident timeline, and mitigate console with its Markdown writeup; the multi-tail merging up to four live logs; the DAG drawer (runs, an ASCII task graph, per-task states, approval gates decided witha/R, XCom, task logs, trigger and backfill); the cluster panel, fleet matrix, node resources, activity heatmap, and next-fire radar; the durable-state inspector; the cron sandbox; the wallboard with itsNO SIGNALbanner and zen screensaver; and the BIOS-style boot self-test, probing the daemon for real (at most every 12 hours; any key,--no-boot, or a settings toggle skips it). -
The schedule tab cannot disagree with the scheduler. The next-fire preview runs the daemon's own
CronTabengine, stepping in absolute time so a window that crosses a DST transition shows the true fire instants (no phantom or hour-shifted entries), and the plain-English description states the engine's deliberate rule that a day must satisfy both day-of-month and day-of-week when both are restricted ("on the 13th, and only on Friday") -- the daemon's documented parse-crontab semantics, not standard cron's OR. An expression the engine would reject (month13, weekday8) degrades to prose in the sandbox instead of raising. -
Hostile bytes die before the frame. A TUI paints raw bytes into the operator's terminal, so everything remote is scrubbed in layers: log content keeps only SGR colour (re-inked per theme) while every other escape family -- CSI with private parameters, OSC (title, clipboard), DCS/SOS/PM/APC strings, single-character escapes like a hard reset, bare trailing ESC -- is stripped; every other API-derived string (job and DAG names, node/peer names arriving over cluster gossip, XCom keys, server error text in toasts) passes the same scrub in the text helpers; and the final row assembly drops any non-SGR escape as a last line of defense. A malicious peer advertising an OSC 52 node name cannot write the operator's clipboard; a job that runs
resetcannot tear down the board from inside its own log pane. -
Tails do not repeat themselves. The daemon replays a finished run's retained buffer on every SSE re-attach, and the stream carries no run identity -- so the tail holds a replay aside until it diverges from what is already on screen: an identical replay that simply ends again is the old run repeated and is dropped whole, while divergence is the next run's output and flushes through (runs stack up behind their end markers, like the page). Idle re-attaches back off geometrically, so an open drawer on a finished job stops re-downloading its log every five seconds.
-
Responsive from the first frame. The input and paint loops start before the first data load, boot-probe API calls race the skip key, and every HTTP call carries a bounded connect timeout (with a read timeout above the daemon's SSE keep-alive cadence on streams) -- so against an unreachable daemon the header says "disconnected" and
q/Ctrl-C work, rather than a blank, un-quittable screen while probes time out. Manual refresh (g, and the refresh after every action or token entry) performs a fetch even with polling paused (--poll 0), and Ctrl-K on the wallboard stays inert rather than opening an invisible palette that could fire unseen actions. -
Same themes, same accessibility, same keys. The five hues in phosphor (dark) and paper (light) variants,
t/Tcycling, the colour-vision-safe remaps, and an--asciiglyph mode for limited fonts; preferences persist in a small JSON file (%APPDATA%\cronstable\tui.jsonon Windows,$XDG_CONFIG_HOME/cronstable/tui.jsonelsewhere). Flags mirror the page's hash routes:--tv(the wallboard),--job NAME(deep-link a drawer), plus--url,--token/--token-env(defaultCRONSTABLE_WEB_TOKEN; a401opens the token prompt, and the token is kept for the session only, never written to the prefs file),--theme,--poll,--boot/--no-boot. Works on Linux, macOS, and Windows (a msvcrt reader thread and VT-mode enablement stand in for termios), and ships in the same package and binaries as the daemon. -
The CLI stays light.
cronstable.tuidefers itsaiohttpimport until the app actually starts, so registering the subcommand costs every othercronstableinvocation nothing.
-
A README section and a Terminal-Dashboard wiki page (options, every key, the panel tour), with screenshots captured from the real TUI driven against the running grand-tour fleet by a new
docs/screenshots/capture_tui.py, alongside the web dashboard's existing pipeline. -
A headless, tty-free test harness boots the real app against a fake daemon on a loopback port, drives it with a scripted key queue, and asserts on painted frames -- the same suite runs on POSIX CI and a Windows checkout. The keyboard-parity test extracts the shortcut table from
fillHelp()in the web page's source (failing loudly if the parse finds nothing), so the two frontends cannot drift apart silently.
A docs-website-and-CI release: no functional changes -- the package, the CLI, and every shipped binary behave exactly as in 1.2.18. The feature-comparison chart grows to cover the orchestration and fault-tolerance surface the daemon already ships, and the wiki stops drifting by hand -- CI now publishes it.
-
The comparison chart expands from 24 capabilities to 35. The matrix now scores the rows that were cronstable's strongest differentiators and simply went unlisted: sub-minute schedules, the extended cron dialect,
@reboot, configurable failure conditions, and the depends-on-past gate under scheduling; dynamic task mapping and poll-until-true sensors under orchestration; cluster-wide concurrency scope and crash-resume of in-flight runs under fault-tolerance; archived-output secret redaction; and state-store backup/restore/migrate. cronstable is native on all 35; the runner-up (Airflow) is at 18. The— not availablelegend entry is dropped -- a blank cell already reads as "no" -- and both renders (docs/comparison.mdand thedocs/comparison.htmlpage) carry the identical 35-row matrix. -
The wiki is published by CI instead of by hand. A new ungated
wikijob in the pipeline mirrorswiki/*.mdonto the project's GitHub wiki (a separate.wiki.gitrepo) on every push todevelop, makingwiki/in this repo the single source of truth. The previous manual clone-copy-push had drifted -- pages a week stale -- exactly the way an unautomated step always does. The mirror is authoritative, so it deletes: a page edited from the wiki's web UI is reverted on the nextdeveloppush, and the job prints every add/modify/delete to the run log. It publishes fromdevelop(nevermain, whose merges would race the per-branchconcurrencykey), is guarded to the canonical repo so forks don't redden, and needs no PAT -- aGITHUB_TOKENwithcontents: writecan push a repo's own wiki.CONTRIBUTING.mdand the releasing wiki page gain an Editing the wiki section documenting the flow and the deliberately-dead bare[Page](Page)links. -
The pendulum-logo tooltip drops its interaction hints. The header wordmark's hover title no longer spells out "sweep your cursor through it to nudge it, right-click to knock it over"; it now just names the mark ("the cronstable logo, the l is a self-balancing double pendulum"). The physics is unchanged -- only the tooltip copy -- and the edit is applied identically to the dashboard (
cronstable/web/index.html) and its demo mirror (docs/demo/index.html). -
CI housekeeping. Dependabot bumped
softprops/action-gh-releasefromv3.0.1tov3.0.2in the release job.
A reliability release: correctness and hardening fixes across the job runner,
the cluster backends, the DAG scheduler, and the web API, with no config
migrations and no behavior change for a healthy single-node install. Two
scheduler-crash paths are closed, a clustered @reboot failover double-fire is
sealed on etcd, the control API gains an always-on cross-site defense, and a
family of unbounded or undecodable inputs can no longer wedge the daemon.
-
Jobs run in their own process group, and cancellation takes the whole tree down. A job that leaves a helper behind (
sh -c 'helper & main') used to strand its run forever: terminating only the process cronstable spawned left the helper holding the job's stdout/stderr write-ends, so the pipe never reached EOF,wait()never returned, the slot was never released, and underconcurrencyPolicy: Forbidthe job never ran again. Jobs now spawn in a fresh session/process group (start_new_sessionon POSIX; the process tree is walked bytaskkill /Ton Windows), andcancel()signals the group --SIGTERM, then an unconditionalSIGKILLafterkillTimeout-- so descendants that outlive a killed shell go down with it andexecutionTimeoutbounds the run's work rather than just its root process. As defense in depth for a descendant that escaped the group (it calledsetsiditself, or Windows lost the orphan from the tree), the post-kill stream drain is now bounded, so the run always leavesrunning_jobs-- at the cost only of output already lost -- and cronstable closes its end of the job's stdout/stderr pipe once that drain returns, so such a run does not leak the pipe's read-end file descriptor until garbage collection. -
Cancelling a never-spawned run is a no-op, not a scheduler crash. A job whose command failed to spawn registers with
proc=None; the next fire'sReplacebranch -- and the cluster slot-renewer -- then cancel whateverrunning_jobsholds, and both run outside the scheduler loop'stry/except.cancel()raisingRuntimeError("process is not running")there could take down the whole daemon on the second fire after a bad deploy; it now logs and returns, and the reaper still completes the run through itsstart_failedpath. -
A clustered
@rebootcan no longer double-fire across an etcd failover. Leadership and the persisted@reboot-ranrecord live at separate etcd keys read by separate requests, so a failover leader whose ran-set read blipped could answer "not run yet" from a stale cache and re-run a one-shot the previous leader had already marked. The etcd backend now applies the same conservative read-side gate the filesystem backend uses: between gaining leadership (or a known lease loss) and the first completed read-back, it answers@reboot-ranqueries by deferring -- the one-shot stays pending and is re-asked next wakeup -- rather than risking a second run. The sharedRebootRanUnknownErrormoves tocronstable.leadership(re-exported frombackends.filesystemfor compatibility); Kubernetes needs no gate, since its ran-set rides the very Lease read that wins leadership. -
A transient store blip no longer freezes a mapped DAG task into an empty fan-out. A mapped task's expansion is recorded once and never recomputed, so reading its upstream list at an instant the store could not answer (an
ESTALE/EIOon a shared NFS/EFS mount) used to be indistinguishable from "published nothing" -- silently skipping the task's entire fan-out while reporting success downstream.artifact_get/artifact_get_recordgain astrict=mode that propagates an unreadable record instead of skipping it, and the expansion read now maps a store error to unknown (stay unexpanded, retry next pass), reserving the empty fan-out for a definitively absent, non-list, or blob-gone (410) result. -
Cross-site request defense for the control API. An always-on middleware refuses cross-site browser requests to the mutating endpoints (
POST /jobs/{name}/start,/cancel,/dags/{name}/trigger,/backfill, task decisions). Those POSTs are CORS "simple requests" -- sent without a preflight -- so without this any web page an operator happened to visit could fire them at a localhost-bound daemon (classic CSRF, and the DNS-rebinding variant). Same-origin requests, and clients that send noOrigin(curl, monitoring), always pass; a foreignOriginis refused403. A newweb.allowedOriginsallow-lists trusted cross-origin dashboards, a specificAccess-Control-Allow-Originresponse header is folded in automatically, andAccess-Control-Allow-Origin: *disables the gate (logged loudly)./mcpkeeps enforcing its ownmcp.allowedOrigins. The gate complements, not replaces,web.authToken, and is the default posture when no token is set. -
The shell reporter is bounded. Reports run inline on the reaper -- the daemon's single job-completion loop -- so a notify command that never exits (
curlwith no--max-time, a script that reads stdin) would freeze completion handling for every job daemon-wide. A newreport.shell.timeout(default 60 s) kills the reporter's whole process group on expiry and lets completion proceed. -
Undecodable secret and token files fail cleanly instead of crash-looping. A
fromFilesecret pointing at binary data (a.p12bundle, a gzip, a key with a stray high byte) raisedUnicodeDecodeErrorfrom the read, which onlyConfigErrorcallers handle -- so it escaped the scheduler loop and crash-looped the daemon at every fire of that job, or 500'd web startup.config._resolve_secretandweb.authToken.fromFilenow surface a cleanConfigError. -
Smaller hardening across the surface. Bearer-token comparison now runs on bytes, so a non-ASCII
Authorizationheader is a clean401rather than a500; secret redaction now catches the space-lessAuthorization:Basic <b64>form (still requiring a separator, so ordinary prose is untouched); the job-state base URL brackets IPv6 literals soCRONSTABLE_STATE_URLis parseable (http://[::1]:8080); the semaphoreacquireendpoint capspermitsat 1024, rejecting an absurd count up front instead of launching up to a billion sequential store probes; the state CLI's binary artifact/xcom verbs tolerate a non-JSON error body (a bare plaintext401) instead of printing aJSONDecodeErrortraceback. -
Two in-memory leaks pruned. A reload now drops the
last_run/run_historydisplay data of removed jobs -- unreachable once the job is gone, and worst under classic crontabs, whose<file>:<line>job names are reminted by every line added or removed above them -- and the DAG scheduler sweeps the per-run advance locks it accumulated for every run it ever touched, keeping only those still owned, held, or awaited.
A docs-and-examples release: no functional changes -- the package, the CLI, and every shipped binary behave exactly as in 1.2.16.
-
Each example owns its compose file. The eight
docker-compose-*.ymlfiles that lived in the repo root now sit in the example they belong to, asexample/<name>/docker-compose.yml, next to that example's config and README. Commands change accordingly --docker compose -f example/cluster/docker-compose.yml upinstead ofdocker compose -f docker-compose-cluster.yml up-- and the READMEs, wiki, and in-file comments are updated to match. The rootdocker-compose.ymlis untouched:docker compose upstill boots thedemoquickstart from a fresh clone. -
The MCP example joins the gallery.
example/mcp-- an agent driving the scheduler overPOST /mcp-- was missing from the README's example table.
The dashboard's public-face release. The wallboard is redesigned around what
a wall viewer actually glances for, the header wordmark becomes a live
control system, and the project gains a feature-comparison chart plus a
zero-install live demo of the dashboard. Every product change is contained
to the dashboard page (cronstable/web/index.html); the daemon, scheduler,
CLI, configuration, and APIs are untouched.
-
Wallboard tiles lead with the glanceable fact. Instead of showing every job the same next-fire countdown, each tile now leads with the datum its state makes urgent: a failing tile shows when it failed and its exit code, with the age counting up live between polls; a running tile shows elapsed time, turning amber once the run exceeds twice its longest recent duration (with 60 s of slack for short jobs); healthy tiles keep the countdown. Elapsed times are an honest observed-since lower bound (stamped when this browser first saw the run -- the
/jobspayload has no start field), never an invented start time. The status hue now fills the whole tile rather than just a glyph and a 1 px border, so state reads at TV distance, and the footer tally represents every health bucket -- pending, unknown, and cancelled jobs are counted instead of silently dropped. -
A verdict headline at TV scale. The dashboard's correlation verdict (single failing job + exit code, correlated fleet event, or cluster alert) used to vanish on the wallboard along with the header that carries it. It now renders as a full-width headline strip above the grid, distilling scope and likely cause into one sentence instead of "count the red tiles". The strip is rewritten only on a real change, so its screen-reader status region announces each verdict once, not once per poll, and its age ticks live between polls (the main verdict bar's age now ticks too).
-
The grid fits itself to the glass. A TV has no one to scroll it, so a new fit governor re-decides the layout on every paint, resize, and zoom change: a handful of jobs on a big screen grows its tiles and type proportionally to use the glass; a crowded fleet first scales full tiles down -- name, glance line, and run-history sparkline all kept -- before conceding anything; only past the readability floor does it step down to compact tiles, and even those keep the sparkline whenever their stretched rows leave it room; and if even compact overflows, the healthiest tail is cut behind an explicit footer chip computed from what was actually cut (
+22 offscreen . none failing, turning red with exact counts if failing or unknown tiles ever overflow a whole screen). The board never silently clips a failure behind a scrollbar nobody can reach. -
Wallboard correctness fixes. The
unknownstate was missing from the worst-first sort order, which left the entire tile ordering unspecified whenever an interrupted job was on the board -- fixed. Grid rebuilds are now skipped when the structure is unchanged, so tile animations no longer restart on every poll and the exit button keeps keyboard focus. The escalation INCIDENT stamp moved from an absolutely-positioned overlay -- which could cover the worst (first) tile or hide beneath the NO SIGNAL banner -- to an in-flow strip that can do neither, and leaving the wallboard now tears escalation state down symmetrically, so re-entering after a recovery no longer flashes a stale stamp and vignette over a healthy grid. Navigating away from#tvby URL (back button, address-bar edit, kiosk script) actually exits the wallboard and preserves the target deep link, and a DAG drawer left open can no longer desync the#tvhash. Keyboard shortcuts ignore modifier chords -- Ctrl+A is select-all again, not a silent alarm acknowledgement. -
The l balances itself now. The block glyph that used to spin beside the wordmark is retired: the
lin the header's "cronstable" is a live cart-and-double-pendulum simulation -- the full nonlinear dynamics integrated at 240 Hz (RK4) and balanced by an LQR controller whose gains are computed in your browser at page load from a numerical linearization. Not a canned animation: while the daemon is live the letter stands upright, riding out little gusts; sweep your cursor through the header to brush it aside, right-click to knock it clean over. Lose the daemon and its motor cuts -- the letter collapses out of the word and swings. When the signal returns, an energy-shaping swing-up threaded by a receding-horizon cross-entropy planner carries it back into the balance controller's basin, and every catch is verified by a two-second closed-loop rollout before it is committed; a hard recovery gets running room (the right end of the track swings open toward mid-page, then eases home once the letter stands). Reduced motion parks a still pose that stays honest about daemon state -- upright when live, hanging when not; without JavaScript the span prints a plain roman l; the SVG is absolutely positioned, so none of this moves the layout. Tuned and Monte-Carlo-tested headlessly: recovery median ~15 s, ~100% by 90 s, zero unverified catches in a 10-minute soak. -
How cronstable compares. A new
docs/comparison.mdscores 24 capabilities -- AI and agent control, scheduling core, orchestration, distribution and fault tolerance, observability, platform -- against yacron, supercronic, Ofelia, dkron, Cronicle, Kubernetes CronJob, and Apache Airflow. Every competitor cell was checked against that project's official docs and adversarially re-verified; the MCP server remains the row nothing else in the field ships natively, Airflow included.docs/comparison.htmlis the same chart as a standalone styled page ("As simple as cron, as capable as Airflow, in one daemon") with an MCP spotlight and a path into the live demo. -
A live demo with nothing to install.
docs/demo/index.htmlis the real dashboard page with a synthetic backend injected ahead of it: it patcheswindow.fetch, so the untouched SPA runs against a deterministic nine-node "meridian" fleet derived from the wall clock -- countdowns tick, long-runners stay running, a staged incident flares and self-clears, and every view works (jobs, drawers, DAGs, cluster, fleet, wallboard, state, boot self-test). No network request ever leaves the page, and it serves from GitHub Pages or any static host. -
Assets, docs, and capture tooling. The README wears the new mark:
logo-balance.gifreplaceslogo-spin.gif, recorded by stepping the real simulation deterministically at 50 fps, so the loop plays the product story with the real controller -- theme glitches physically knock the pendulum, the big one cuts the signal, and the word heals through a verified catch. The social card lifts the logo engine straight out of the dashboard page so it can never drift from the shipped mark, the still-capture scripts park the pendulum at exact upright so screenshots stay pixel-identical across themes, and a newdocs/logo-lab.htmlpreserves the standalone harness the mark was tuned in. The wiki's wallboard section is rewritten for the new tiles, the failure glyph across the dashboard and docs swaps the heavy ballot cross for a symmetric one that sits better next to the check mark, and the full screenshot set is regenerated.
A maintenance release: no functional changes to cronstable itself -- the package, the CLI, and every shipped binary behave exactly as in 1.2.14. It consolidates the project's separate CI workflows into one gated pipeline, hardens the release automation and updates the contributor docs to match.
-
One CI/CD pipeline. The former
build.yml,docker.yml, andtox.ymlare folded into a singlerelease.ymlthat builds and tests the whole product on every push and pull request -- the tox lint/mypy/pytest matrix, the wheel + sdist, every self-contained binary (Linux glibc and musl across the full arch set, macOS arm64/amd64, Windows amd64/arm64), and all eight Docker images -- and gates a release on all of it. Nothing publishes on an ordinary commit; a release still ships the PyPI upload, the GitHub Release with every binary and a singleSHA256SUMS, the container images, and the Homebrew tap update, but only after the entire build + test matrix is green. -
Reliable release publishing. The consolidated pipeline attaches every release asset in one shot as the GitHub Release is created (no separate attach step that could fail against an already-published, immutable release), and the Homebrew tap update is no longer best-effort -- if the tap cannot be updated (for example, a missing token) the release now fails loudly instead of finishing green with a stale formula.
-
Hardened release trigger. A
1.3.0was published in error before this release: the trigger substring-matched whole commit messages, so a commit body that merely discussed the bare[release]marker out-bumped the intended[release:patch].1.3.0is withdrawn (yanked on PyPI, its GitHub release and container tags removed) and contains exactly what ships here as1.2.15. The trigger now scans only commit subject lines, only honors a marker at the very start of the subject, and when several commits carry one, the latest commit's marker wins. -
Contributor docs.
CONTRIBUTING.mdand the "Contributing and Releasing" wiki page are rewritten for the single-pipeline flow.
cronstable becomes drivable by an AI agent. A new, opt-in MCP server
exposes the scheduler over the Model Context
Protocol, so Claude, Cursor, VS Code
Copilot, or any MCP client can observe every job, DAG, the cluster/fleet,
metrics, and the durable state store the way an operator reads the dashboard
-- and, when you opt in, act (run or cancel a job, trigger / backfill /
approve a DAG). It is read-only by default, served two ways from one
implementation -- a POST /mcp endpoint on the existing web listeners and a
cronstable mcp stdio bridge for desktop clients -- and, like the rest of
cronstable, hand-rolled in pure Python with no new dependencies. It stays
off unless an mcp: section sets enabled: true, so a plain install pays
nothing and behavior is unchanged without it.
-
The
mcp:section and its two transports. Configuration lives under a new optional top-levelmcp:block that rides theweb:listeners -- awebsection is required, because there is nowhere else to serve it.enabled: trueturns on a stateless Streamable-HTTP JSON-RPC 2.0 endpoint atPOST /mcp, pinned to MCP revision2025-11-25(noMcp-Session-Id;GET /mcpis405), and exposes thecronstable mcpstdio bridge that desktop clients launch as a subprocess. Both paths run the same server code. -
Tools, grouped into opt-in toolsets. The default
observetoolset is read-only -- status, jobs, per-job runs / trends / resources, cluster, fleet, node load, a metrics query, version, and live log tails (twelve tools).dagsadds DAG, run, and XCom reads plus task-log tails;stateadds a redacted durable-state inspector;actadds mutating job control (cron_run_job,cron_cancel_job), anddagsgains DAG control (cron_trigger_dag,cron_backfill_dag,cron_decide_gate) once writes are enabled -- 23 tools with every toolset on andreadOnly: false. Mutating tools require an explicitconfirm: true, carry honestdestructiveHintannotations, re-check the same authorization as the REST route, andcron_backfill_dagpreviews as a dry run unless it is called with bothdry_run: falseandconfirm: true. -
Resources and prompts, both read-only and on by default. Resources are URI-addressable read-only snapshots a client can attach as context --
cronstable://status,cronstable://cluster,cronstable://fleet,cronstable://version, and templates for jobs, runs, DAGs, and state namespaces -- and every critical read is also a tool, because client support for resources is uneven. Prompts are canned triage playbooks that chain the read tools:triage_job_failure,why_did_dag_run_fail,blast_radius,fleet_health_summary, andbackfill_plan. Both are scoped by the enabled toolsets (a DAG resource appears only with thedagstoolset) and can be turned off (resources: false,prompts: false) for a tools-only client. -
Safe by default.
readOnly: trueis the default and strips every mutating tool regardless of toolset, so an agent gets look-but-don't-touch until you opt in./mcpinheritsweb.authTokenexactly like the data routes and is never in the public set; enable it with no token on a routable (non-loopback, non-socket) listener and cronstable fails closed at config load (without a token the web app installs no auth middleware at all, so/mcpwould be wide open) -- restrictweb.listento loopback/sockets, set a token, or setmcp.allowUnauthenticated: truewhen a proxy terminates auth. A present, non-allow-listedOriginis refused403(a DNS-rebinding defense; browser clients go onmcp.allowedOrigins), an oversized body is refused413(maxBodyBytes, 1 MiB default), and any list tool'slimitis capped atmaxRows(200) with an opaque cursor for the remainder.cron_inspect_statemirrors the dashboard's metadata-only stance: KV values collapse to a size/type summary and secret names appear without their values. -
One source of truth for REST and MCP. The web layer's read handlers are refactored so each endpoint's data is produced by a reusable
*_payloadmethod --status_payload,jobs_payload,cluster_payload,fleet_payload,node_payload, the per-job runs / resources / trends projections,dags_payload, thestate_*_payloadfamily, and new poll/cursor log-tail projections -- and both the REST routes and the MCP tools call those same methods, and the samestart_job_by_name/cancel_job_by_nameaction paths, soGET /jobsandcron_list_jobscan never drift apart. The web app now also rebuilds when only themcpconfig changes, so flippingreadOnlyor adding a toolset takes effect on the next reload. -
The
cronstable mcpstdio bridge. A featherweight stdlib client -- imported lazily so it never pulls the daemon graph into CLI start-up -- reads newline-delimited JSON-RPC on stdin and forwards each frame to a running daemon's/mcp, writing replies to stdout and logs to stderr.--url,--token/--token-env(defaulting to theCRONSTABLE_WEB_TOKENenv var), and a--checkhandshake that runsinitialize+tools/list, prints the negotiated protocol and tool count, and exits. It needs a reachable running daemon -- the right model for an ops tool. -
Docs, an example, and a rebuilt README. New wiki pages (MCP and MCP-Server-Design), an
mcprow and full option table on Configuration-Reference, and aPOST /mcpsection on HTTP-API.example/mcp(withdocker-compose-mcp.yml) boots a single node with the server on and every toolset enabled -- a steadyheartbeat, an intentionally failingflaky-export, a longslow-report, and an on-demandon-demand-sync-- and walks through wiring Claude Code / Desktop, Cursor, and VS Code to it. The README now leads with an animated dashboard reel (every frame a real running fleet, WebP with a GIF fallback) and a ten-theme / two-font showcase, adds a written tour of the dashboard's accessibility options (interface font, UI scale, color-vision-safe palettes, reduced motion), lists the MCP server among the features, moves the yacron fork attribution to the foot of the page, and -- at last -- says how to pronounce the name: kraahn-stuh-bl, like constable. -
Internal: the MCP server ships with a dedicated
tests/test_mcp.pysuite -- theinitialize/capability handshake, toolset andreadOnlygating, theconfirmand dry-run write guards,maxRowsclamping and pagination, the fail-closed no-token check and the Origin / body-size / batching HTTP defenses, resource and prompt scoping, and the featherweight-bridge import cost -- and the release workflow's finalize job, which has noactions/checkout(sogh release download/uploadcould not infer the repository and died with "not a git repository"), now setsGH_REPOexplicitly.
The package finishes becoming cronstable. Everything that was still
named yacron2 -- the import package, the python -m entry point, the
console-script command, and the PyPI distribution -- is now cronstable, so
the name is consistent all the way from pip install through import to the
CLI. The fork has shipped as cronstable in its repo and docs for a while;
this release moves the code's own identifiers to match, so nothing user-facing
still answers to the old name.
-
The
yacron2package is renamed tocronstable. The source tree moves fromyacron2/tocronstable/and every intra-package import follows, soimport yacron2...becomesimport cronstable...andpython -m yacron2becomespython -m cronstable. The web assets, backends, and cluster modules move with it; no module contents change, only their home. -
The CLI command and PyPI distribution are renamed. The console script the wheel installs is now
cronstable(it wasyacron2), and the project publishes to PyPI ascronstable--pip install cronstable. The repo, Docker image, and container registry areptweezy/cronstable. Update any scripts, service/unit files, orpip/import references that still sayyacron2; there is no compatibility shim, so the old names stop resolving. -
Release-pipeline hardening. The release workflow now pushes the version tag with a scoped
RELEASE_TOKENrather than the defaultGITHUB_TOKEN-- GitHub refuses to let the Actions app push a tag whose commit touches.github/workflows/without theworkflowsscope it cannot be granted -- and the PyPI publish runs withskip-existing, so a release interrupted after the upload (before the tag and GitHub Release) can be retried without burning the version.
cronstable now parses cron expressions itself. This release retires the
third-party crontab (parse-crontab) library in favor of a small,
stdlib-only engine that lives in the tree -- so the scheduler owns the one
piece of syntax every job depends on, the dialect is documented and tested
where it is implemented, and the install carries one dependency fewer.
Compatibility is not aspirational: it is pinned by golden vectors recorded
from the old library across 180+ expressions and fixed instants (DST
transitions, leap days, month/year boundaries, the year cap, ambiguous
fall-back folds). Nothing changes for existing configs -- the dialect, the
timezone model, and the strictly-future fire semantics are all preserved
vector-by-vector.
-
cronstable/cronexpr.py: the built-in cron engine. A newCronTabclass parses the crontab dialect cronstable has always accepted (5/6/7 fields, ranges, steps including bare-start5/15, lists, case-insensitivejan/monnames,0-7weekdays with6-0wrap,L-last-day andL5-last-Friday forms,@-nicknames) and answers the scheduler's two questions:next()(strictly future, DST-correct across UTC-offset changes, capped at the 2099 horizon so dead schedules drop from the index) andtest(). A stdlib-only leaf module --calendaranddatetime, no third-party imports. -
The
crontabdependency is dropped. Removed frompyproject.tomland every import site (config.py,cron.py,crontabs.py,dagrun.py,prometheus.py); classic crontab-file loading and YAMLschedulestrings now share the same in-house engine, so both formats still accept identical expressions. -
Golden-vector compatibility harness.
tests/gen_cron_golden.pyrecordsnext()/test()answers from the original parse-crontab library intotests/data/cron_golden.json;tests/test_cronexpr.pyreplays them against the new engine so any behavioral drift fails the suite. This is the proof that "behavior-compatible" is a fact, not a hope. -
mergedictsreimplemented. The config defaults-merge helper is rewritten with the identical semantics -- dicts merge recursively, an empty YAML section (None) never wipes a populated default,environmentandsecretslists merge by key/name, sentryfingerprintreplaces rather than appends, and all other lists concatenate -- and now returns adictdirectly, retiring thedict(mergedicts(...))wrappers at every call site. -
Dashboard accessibility pass. The web UI gains an interface font toggle (the terminal monospace, or a proportional sans "reader mode" for easier reading, with logs and cron strings kept monospace either way), app-level color-vision palettes (deutan / tritan status-color remaps, status glyphs always distinct too), whole-UI zoom, and a reduce-motion switch, each reachable from Settings and the command palette and persisted across sessions.
-
Docs. The "Schedules and Timezones" wiki page now documents the full dialect the engine owns -- the day-of-month-AND-day-of-week rule (Friday the 13th, deliberately kept over Vixie's OR), the
Lforms, the 1970-2099 year horizon -- and Installation, Migration-from-yacron, Classic-Crontabs, Architecture-and-Internals, and the README are updated to point at the built-in engine instead of the removed library.
Resource monitoring grows a time axis. 1.2.8 answered "what did that run
use?" with two numbers per run; this release records how the run used it
-- a per-run CPU/memory chart series with a user-tunable sampling cadence --
and puts proper charts in the dashboard: a live view of the running
instance, the recorded profile of any recent run, per-run trend strips, and
a whole-node history chart behind the header meter. Everything rides the
existing HTTP+JSON surface and the durable run ledger; nothing changes for
existing configs (monitorResources: true still means what it meant, with
the same 1s cadence).
-
monitorResourcesmap form. Alongside the bool, the option now takes a map:enabled(default true -- writing the map at all opts in),interval(seconds between process-tree samples, default1.0, floor0.1), andhistory(chart points kept per run, default240,0for summary-only, ceiling2000). Validated at load time with the other numeric ranges, merged normally underdefaults:, and accepted by DAG tasks. Not fingerprinted, like the bool before it. -
Per-run chart series. Each monitored run now records a
[t, cpu%, rss]point per sample, downsampled in place once it exceeds thehistorycap: adjacent buckets merge with mean CPU but peak RSS, so the memory spikes people monitor for survive downsampling, and a run of any length stays a few KB with uniform bucket widths. The series is embedded in the durable run record'sresources.series-- charts survive restarts and are bounded by the existingstate.maxRunsPerJobpruning -- and is deliberately excluded from the polled/jobsand/jobs/{name}/runspayloads, which keep their summary-only shape. -
GET /jobs/{name}/resources. A lazy, chart-grade endpoint: the run-so-far series of every currently-running monitored instance (plus its live instantaneous readings) and the recorded series of recent finished monitored runs, capped by arunsquery parameter.monitored: falsewith empty lists distinguishes "never opted in" from "no data yet". -
GET /node/history+web.nodeHistory. A background sampler records the node's CPU/memory (the same cgroup-aware percentagesGET /nodereports) into an in-memory ring -- every 5s, keeping the last hour, by default. It follows the web app's lifecycle, is on whenever the web API is on, and is tuned or disabled viaweb.nodeHistory(interval/points, orfalse). A gap wider than the cadence in the returned points means the daemon was down, not idle. -
Dashboard: the Resources tab and the node card. The job drawer gains a Resources tab: CPU and memory drawn as separate small-multiple charts (one honest 0-based axis each, never dual-axis) with a synced crosshair and tooltip, chips to flip between the live instance and recent runs, a refresh-cadence selector for the live view (1s-10s, persisted), and clickable per-run CPU-time / peak-memory trend strips. Clicking the header node meter opens a node resources card charting the retained node history. Chart inks are the theme-aware blue/amber pair (colorblind-safe and contrast-checked against every theme surface, light and dark), gaps in sampling break the line rather than lying across it, and unmonitored jobs get a pointer at the config instead of an empty chart.
-
Examples. The grand tour (
_defaults.yaml+platform.yaml) and the large-cluster demo now use the map forms -- a 0.5s sampling cadence with the default 240-point series, and a 2s x 1800-point node history ring -- so their resource-heavy jobs light the new charts up out of the box.
This release answers "what is this actually using?" Opt-in per-job
resource monitoring records every run's CPU time and peak memory and
carries the numbers everywhere a run already reports -- the dashboard, the
HTTP API, Prometheus, statsd, and failure reports -- while a new GET /node
endpoint and a cluster.observability block put live whole-node load beside
the jobs, on every node in the fleet. One footprint note: psutil joins the
core dependencies, the fork's first addition to the core install (it ships
wheels for the mainstream targets and builds from source elsewhere).
Behavior is unchanged without the new config: monitorResources is off by
default and the observability overlay is opt-in.
-
Per-job resource monitoring (
monitorResources: true). A psutil-backed sampler polls the run's whole process tree and records its total CPU time (user and system) and its sampled peak resident memory. Accounting is best-effort by design: a process that exits mid-sample, a platform that denies the read, or psutil failing outright simply yields whatever was captured so far -- monitoring never crashes a job, never delays it, and never changes its success/failure verdict. Peak RSS is a sampled high-water mark and per-member CPU is banked as the tree shrinks, so the long, heavy runs that matter are measured well; only a child that spawns and exits within a single sampling gap escapes entirely. -
The numbers surface everywhere a run does. The dashboard overview shows live CPU/memory chips on a running job, and the history tab adds per-run CPU and peak-memory columns and stats. The HTTP API carries
resourceson each run in the history, liverunning_resourceson a running job, and windowed CPU/RSS aggregates in the job stats. Prometheus growscronstable_job_cpu_seconds_total{job_name, mode}(user/system),cronstable_job_peak_rss_bytes, and last-run CPU/RSS gauges -- emitted only once a job has a monitored run, and persisted across restarts by the durable metrics snapshot. A monitored run's statsd stop datagram gains acputimer and amax_rssgauge (an unmonitored job's datagram is unchanged), and failure reports getcpu_seconds/max_rss_bytes(and friends) plusCRONSTABLE_CPU_SECONDS/CRONSTABLE_MAX_RSS_BYTEStemplate variables. -
GET /node: the node's own live load. A new endpoint samples the serving host's CPU and memory fresh per request -- whole-host utilisation plus the daemon's own footprint -- and drives a node meter in the dashboard header. It is container-aware: under a cgroup v2 limit (Docker/Kubernetes limits, systemd slices) the numbers describe the daemon's slice -- the effective memory limit with reclaimable page cache excluded (the same accountingdocker statsshows) and utilisation of the CPU quota -- with memory and CPU switching over independently. Unlimited cgroups, cgroup v1 hosts, and non-Linux platforms report whole-host numbers, and the response shape never changes. -
cluster.observability: gossip as a secondary data plane. Opt in and every node shares its whole-node CPU/memory across the cluster. Underbackend: gossipthe reading rides the election mesh as a smallX-Cronstable-Node-Statsresponse header on full and304responses alike, so a sharing cluster's steady-state round still costs headers only. The lease backends (kubernetes/etcd/filesystem), which have no node-to-node channel of their own, can stand up a second, election-inert gossip mesh purely for observability data -- which also brings the fleet view to lease-backed clusters. The dashboard's cluster panel gains per-peer load meters and the fleet view puts each node's live load in its column header. Like the run summaries, node stats are best-effort display data: a malformed peer payload degrades to "no data", never poisoning the view or any decision. -
Dashboard themes. A new
carolina-lighttheme joins the palette, andcarolinareplacesamberas the default. -
Docs and examples. The README is rebuilt around a sixty-second quick start, four tutorials (alerting and retries, durable restarts, a first DAG, two-replica leader election), and a screenshot tour of the dashboard; the screenshots themselves are now reproducible via a scripted pipeline under
docs/screenshots/that captures a live grand-tour fleet. The grand tour gains resource-monitored CPU- and memory-heavy demo jobs and the observability overlay, and the new features are documented on the wiki's Configuration-Reference, HTTP-API, Clustering, Metrics, and Reporting pages.
This release makes cronstable stateful. An opt-in durable state store lets
the scheduler remember across restarts -- retries that survive a daemon
restart, @reboot that really means once per boot, Prometheus counters that
do not reset -- and turns a shared directory into fleet-wide coordination:
cluster-scoped concurrency, cross-node retry takeover, and leader election
with nothing but a mount both nodes can reach. On top of the store sit a
state API handed to every job (key-value, cursors, fleet locks, idempotency
claims, artifacts, run-scoped secrets) and durable DAG orchestration
(dependencies, XCom, fan-out, sensors, approval gates, backfills) with
crash-resume. All of it is opt-in: without a state: block (and a dags:
block for pipelines) nothing changes -- no new behavior, no new files on
disk, and the zero-new-dependency, architecture-portable core install is
untouched.
-
A durable state store behind a single
state:block.state.pathnames a directory -- a local disk or a shared NFS/EFS-style mount -- and the daemon keeps everything under<path>/<deploymentId>: append-only JSON record streams, mutable documents, content-addressed blobs, andflock-guarded TTL leases with monotonic fence counters. The write discipline is crash-safe on POSIX and Windows alike (atomic temp-plus-rename with directory fsyncs; a record that cannot be parsed is quarantined, never trusted and never fatal), files are owner-only (0o700/0o600-- archived job output is exactly where secrets live), and archived output additionally passes through a conservative best-effort secret redactor before it is written. A store outage degrades the stateful features, never scheduling: durable writes are fire-and-forget, reads on scheduling paths are bounded and fall back, store calls run on abandonable worker threads so a hung hard mount cannot wedge the daemon or its shutdown, andstate.maxOpsPerSecondthrottles everything except lease renewals, which must never queue behind bulk work. The full model is documented on the wiki's Durable-State page. -
Restart-surviving scheduling. With a store configured, a pending retry re-arms after a daemon restart instead of vanishing;
@rebootdistinguishes a real boot from a mere restart (and, under election, runs once per fleet); Prometheus counters persist across restarts; the run ledger, optionally archived output (archiveOutput), and catch-up checkpoints are durable; and in-flight run records let a restarted daemon settle the runs that died with it, so failure handlers and retries fire for work a crash orphaned. -
Garbage collection that can prove absence. Every node periodically writes a manifest of the jobs, scopes, and dags it carries; GC deletes a stream only when no recent manifest references it and its newest record is older than
state.gcGraceSeconds(default seven days), and it defers wholesale until the retained manifest history spans a full grace window -- "nobody has manifested yet" never reads as "nobody wants this". Artifact streams and payload blobs age out with their scope, run documents of removed dags are collected by the daemon that owned them, and of the lease files only the per-run DAG advance class is ever reclaimed: every other lease carries fences that persist in durable records, so it is never deleted at any age. A node that is merely down loses nothing. -
Fleet HA through a shared directory. A new
cluster.backend: filesystemruns leader election over the same flock-and-lease machinery -- no gossip ports, no Kubernetes, no etcd -- and composes with everything clustering shipped in 1.2.1.concurrencyScope: clustermakesconcurrencyPolicy: Forbid/Replacehold fleet-wide through per-job slot leases (aReplacefired anywhere cancels the run wherever it lives; a crashed holder's slot frees by TTL), a pending retry left by a dead node can be claimed and resumed by a survivor -- serialized on a claim lease and re-checked under it; the contract is at-least-once, honestly -- and@rebootunder election survives leader failover in the safe direction: a takeover can delay a one-shot, never double-run it. -
Every job gets a state API. With
state.jobApi(on by default once a store is configured) the daemon serves a loopback-only HTTP endpoint and injects its address and a per-run bearer token into each job's environment; thecronstablebinary doubles as the client.cronstable state get/set/delete/keysis durable KV;cronstable cursorkeeps resumable positions;cronstable lockgives fleet-wide mutexes and semaphores backed by the same TTL leases the cluster uses, with fencing tokens and alock run --wrapper;cronstable idempotentmakes run-once guards honest (exit0fresh,5duplicate,1transport or store error);cronstable artifactstores content-addressed payloads under configurable size caps. A job'ssecrets:block stages secrets over the endpoint for exactly one run -- resolved fresh, served only to that run, never in the environment and never in the durable store -- read back withcronstable secret get. Scopes default to the job's own name;stateAllowedScopesopens shared ones. -
DAG orchestration. A new
dags:section defines multi-step pipelines on the job grammar: tasks withdependsOnand per-task retries, XCom hand-off between tasks (cronstable xcom push/pull), mapped fan-out over a pushed list (capped, launched in bounded batches), sensors that poke on an interval, and approval gates a human resolves from the dashboard or API. Dags run on cron schedules (the job schedule grammar, minus@reboot), manual triggers, and date-range backfills. A per-run advance lease makes exactly one node drive each run; when a driver dies, the lease lapses and a peer adopts the run mid-flight, reconciling exactly what was and was not still running. Run history is retained per-dag and collected under the same grace rules. See the wiki's Orchestration-and-DAGs page. -
Dashboard and HTTP API. The dashboard gains DAG cards with a run drawer and task graph (trigger, backfill, and approve/reject inline), cluster/HA chips, per-job durable run history, and a metadata-only state inspector -- streams, documents, leases, and blob inventory by name and size, never values. The HTTP API adds the matching
/dags/...routes (runs, XCom, live task logs) and/stateinventory routes, and the Prometheus endpoint grows state-store and DAG metric families. All routes are documented on the wiki's HTTP-API page. -
Store administration.
cronstable state backupwrites an owner-only.tar.gzof the whole store, safe against a live daemon;state restoremerges it back atomically (fence-aware, refuses a non-empty store without--force, and is not safe while a daemon uses the store);state migratecopies a store across paths or mounts without a reader ever seeing a torn record;state gc [--dry-run]runs or previews a collection pass;state checkverifies the store is usable and prints an inventory;state migrate-schemarewrites records of older known schemes. -
Packaging and examples. orjson joins uvloop in the
speedupsextra, accelerating the durable-state and cluster-gossip JSON paths throughcronstable._json, whose stdlib fallback is behavior-identical -- the core install stays zero-new-dependency and architecture-portable, and the prebuilt binaries bundle orjson wherever a real wheel or verified source build exists, with a verify-or-strip step mirroring uvloop's. New examples:example/job-state(the CLI primitives),example/dagandexample/dag-cluster(pipelines, single-node and fleet), andexample/grand-tour(a docker-compose fleet exercising the whole feature set end to end).
A toolchain and packaging release. There are no behavior, API, or configuration changes and no change to the core dependency footprint: the published wheel/sdist and the release binaries are built from the same sources as before and install exactly the same runtime dependencies. The work adopts uv across the paths where it pays off -- CI, the build/release pipeline, and the local dev loop -- refreshes the container base images, and pins the CI action versions.
-
uv on the runner-native CI, build, and dev paths. The dist build (
uv build), the runner-native PyInstaller binaries (macOS, Windows, and the all-wheels Linux arches, each installed into a throwawayuv venvand frozen withuv run), the version probe (uv run --no-project --with setuptools-scm), andtwine check(uvx twine) all run through uv now: parallel downloads and a shared global wheel cache make them markedly faster, and the results are behavior-identical.UV_PYTHON_DOWNLOADS=neverkeeps uv on the exact interpretersetup-pythonpinned rather than fetching a managed one, andUV_HTTP_TIMEOUTcarries the same transient-network hardening the pip paths had. -
The emulated foreign-arch binary legs stay on pip, on purpose. The musl and glibc-extra binary jobs (armv7/armv6, ppc64le, s390x, riscv64, i686) build inside
docker runcontainers under QEMU, where uv is not a fit: its official image is amd64/arm64 only and it has no musl ppc64le/s390x wheels, so pip remains the arch-portable choice there and keepsPIP_RETRIES/PIP_TIMEOUThardening. The uvloop bundling and per-arch--versionsmoke test are unchanged on every leg. -
uv in the local dev loop.
tox.ininow declaresrequires = tox-uv, so a plaintoxauto-provisions its environments and installs dependencies with uv (much faster, behavior-identical);tox-uvis added to thedevextra andrequirements_dev.txt.CONTRIBUTING.mddocuments the uv quickstart (uv venv,uv pip install -e ".[dev]") alongside the unchanged stockvenv+pippath, and notes thetox --runner virtualenvescape hatch for anyone who wants the legacy runner. -
Refreshed container base images. The Docker variant matrix moves to current bases:
ubuntu24.04 -> 26.04 (Python 3.12 -> 3.14),rhelUBI9 -> UBI10,fedora41 -> 44 (3.13 -> 3.14),opensuseLeap 15.6 -> 16.0 (3.11 -> 3.13), anddistrolessto Python 3.13. The Debian/Alpine images and every image tag stay as they were. -
Internal: every CI-consumed action is pinned to an exact version (
checkout@v7.0.0,setup-python@v6.3.0,setup-uv@v8.2.0, the docker/* actions, etc.), and adependabot.ymlis added to keep those pins and the Python dev dependencies current.
A performance and footprint release. There are no behavior or configuration changes: every schedule fires exactly as before, the metrics endpoint renders byte-for-byte identically, and the core install stays zero-new-dependency. The work trims CPU on the daemon's hottest repeating paths -- the once-a-minute config reload, every Prometheus scrape, and each cluster poll / gossip round / lease renew -- lowers steady-state memory, and adds an optional faster event loop.
-
Optional uvloop event loop (
speedupsextra).pip install cronstable[speedups]swaps asyncio's selector loop for uvloop's faster libuv-based one, speeding every I/O path cronstable drives: cluster gossip and lease HTTP, the web dashboard, and the Prometheus scrape. It is entirely opt-in and best-effort --__main__selects uvloop lazily on POSIX and falls back to stock asyncio, behavior unchanged, whenever it is absent or unimportable -- so it stays off the core install to keep the baseline architecture-portable. Windows always uses its Proactor loop (there is no uvloop build there, and the Proactor loop is required for subprocess support anyway). The prebuilt POSIX binaries now bundle uvloop wherever it builds: a wheel where one exists, an otherwise verified source build, with a start-up self-test (verify_uvloop.py) that uninstalls a miscompiled build (a real risk under QEMU emulation) before freezing so the binary cleanly runs on asyncio instead. An arch where uvloop cannot build ships the asyncio binary exactly as before. -
The once-a-minute reload no longer reparses an unchanged config. The scheduler rereads and reparses the config every minute so an on-disk edit is picked up promptly, but strictyaml is a slow pure-Python parser and reparsing an unchanged file was pure wasted work (in a worker thread, but still real CPU plus thread-pool churn).
reload_confignow compares a cheapos.statfingerprint --(path, mtime_ns, size)per file, plus the config directory's own mtime -- of exactly the files the last parse read (the top-level config, every transitivelyincluded file, and each job'senv_file) and skips the reparse entirely when nothing has changed, returning the already-loaded config. A genuine edit, a vanished file, or a new entry dropped into a config directory still reparses on the next pass. -
Cheaper Prometheus scrapes. The job-set fingerprint (
job_set_id, queried on every scrape and every cluster poll / gossip round / lease renew) is a pure function of the loaded jobs, so it is now computed once per reload and memoized rather than re-deriving its per-job deepcopy / JSON / SHA-256 each time. Thejob_next_run_timestampgauge reads the scheduler's authoritative next-fire index instead of re-walking every crontab and building two aware datetimes per job per scrape (falling back to a direct computation only in the brief start-up window before the index is seeded). The histogramlelabel strings are precomputed once from the bucket bounds rather than re-rendered for every bucket of every job on every scrape. -
Lower steady-state memory and faster attribute access.
JobConfig-- one instance per configured job for the life of the process -- now declares__slots__, trimming its per-instance__dict__and speeding the attribute reads on the scheduling hot path. Fingerprint redaction is now copy-on-write instead ofdeepcopy, so the long immutable report templates (the sentry body, the webhook body) are shared by reference rather than duplicated on each fingerprint. The PyInstaller binaries are now built withoptimize=2, which strips docstrings and (side-effect-free, internal-invariant) asserts from the frozen bytecode -- cronstable's modules are deliberately docstring-dense, so this shrinks the binary and lowers resident memory for the life of the daemon. -
The idle scheduler no longer polls once a second. The job reaper waited on its "any job running?" event with a one-second timeout, waking every second even when nothing was running. It now blocks on the event outright -- the wait condition can only change when a job launches or shutdown is signalled, both of which set the event (shutdown now does so explicitly, so the reaper exits promptly) -- so a fully idle daemon does no per-second work. A related fix reads the running-jobs map with
.get()so a concurrency check can no longer leave a phantom empty entry that would spin the reaper hot at shutdown. -
Internal: the reload skip cache (change detection, the failed-parse and worker-thread paths), the memoized fingerprint, and the scrape reading the seeded next-fire index are covered by new tests; the uvloop bundling is gated behind a build-time verification step and the per-arch
--versionsmoke test.
This release re-implements the scheduler core added in 1.2.3 without changing what it does: every schedule fires exactly when it did before, and there are no configuration changes. The daemon no longer wakes on a fixed cadence and tests every job against the clock; it keeps each job's next fire time in an index and sleeps until the soonest one is due, servicing only the jobs whose moment has arrived.
-
Per-wake cost scales with jobs due, not jobs configured. The previous loop matched every enabled job against the clock on every tick -- and with a second-level job present that tick was once a second, so the whole job set was scanned every second. The scheduler now maintains a next-fire index (each job's next-fire instant, mirrored in a min-heap), sleeps until the earliest entry, and touches only the jobs actually due. An idle wake over a large fleet is an O(1) heap peek that runs zero cron matching; a wake with a cohort due matches only that cohort. A deployment running thousands of sparsely-scheduled jobs pays dramatically less per wake, and adding a second-level job no longer imposes a per-second scan of everything else.
-
Robust across wall-clock and NTP steps. The sleep is realized against the event loop's monotonic clock, and firing compares the wall clock against fixed, forward-only next-fire instants. A clock step backward (an NTP correction or a manual set) now defers the pending fire instead of re-running a slot that already fired; a large step forward, a resume from suspend, or an RTC-less boot corrected far ahead resumes at the current slot in O(1) instead of enumerating and replaying every occurrence in the skipped span. The bounded catch-up for a genuinely overrun pass -- a slow config reload, a burst of simultaneous launches -- is retained, still capped at the ten-second
CATCHUP_LIMIT, and now covers minute- and second-level jobs by the same path. -
De-duplication is now structural. A fired slot cannot fire twice because advancing the index moves a job's next fire strictly past the slot it just fired; the old per-slot "did this already run?" gate is gone (
_last_run_slotis kept only for status/introspection). All the surrounding guarantees are preserved: a job fires exactly once per matching slot, a mid-period restart skips the period already under way (the index is seeded strictly-future at start-up),@rebootjobs run once at boot, a config reload landing on a job's own boundary minute does not skip that fire, and housekeeping (config reload, cluster and web upkeep, logging) still runs at most once a wall-clock minute. -
Internal: the next-fire index, monotonic sleep, clock-step handling, reload reconciliation, and a fleet-scale performance demonstration are covered by a new batch of scheduler tests; the wiki's "How the scheduler ticks" section is rewritten to describe the index.
This release brings second-level (sub-minute) scheduling: a job can now
fire at second granularity, either through a new second field on the schedule
object or a full seven-field crontab string. The scheduler keeps its historical
once-a-minute cadence -- and its zero overhead -- until some enabled job
actually asks for seconds, at which point it ticks once a second, firing
second-level jobs on time while every minute-level job still fires exactly once
in its minute. The release also starts honoring the schedule object's year
key (previously accepted but silently dropped, a behavior change for the few
configs that set it), surfaces a malformed schedule as a named ConfigError at
reload instead of an anonymous traceback, teaches the web dashboard to parse,
describe, and preview five-, six-, and seven-field expressions, and ships two
runnable examples (pulse-monitor and its clustered sibling pulse-cluster)
built around second-level probing. Sub-minute scheduling is entirely opt-in;
see the upgrade notes below for the one behavior change that can affect an
existing deployment.
-
New
secondfield and seven-field crontab strings. parse-crontab reads extra columns from the ends of a crontab line, so the field count selects the dialect: a five-field line has an implicit second of0and any year, a six-field line adds a trailing year column, and a seven-field line adds a leading second column too (second minute hour dayOfMonth month dayOfWeek year). So the objectsecond: "*/15"and the seven-field string"*/15 * * * * * *"both fire every 15 seconds, while a six-field string pins a year and stays minute-granular. Thesecondfield takes the same syntax as any other (*,*/5,0,30,10-20);second: "*"fires every second. Second-level scheduling is a YAML feature: classic crontab files stay five-field and minute-granular. -
Adaptive cadence, zero cost when unused. The scheduler ticks once a second only while some enabled job pins a second (
Cron._needs_subminute()); otherwise it keeps the historical once-a-minute cadence, aligned to the top of each UTC minute, byte-for-byte as before. A disabled second-level job never forces the per-second cadence. The cadence is re-evaluated every tick, so a reload that adds or removes a second-level job switches modes on that same tick. -
Exactly once per slot; mixed cadences. Each pass reads the clock once and tests every job against a single scheduling "slot" truncated to that job's own resolution -- the whole second for a second-level job, the top of the minute otherwise. Launches are de-duplicated per slot (
_last_run_slot), so a minute-level job now tested up to 60 times in its due minute still fires exactly once, and a second-level job fires once per matching second even if two ticks land in the same second. A leader-gated job is evaluated once per slot regardless of which node runs it. Sub-minute and per-minute jobs mix freely in one config;concurrencyPolicystill governs overlap as before. -
Catch-up for overrun seconds, bounded. In sub-minute mode, if a pass runs long -- many simultaneous launches, or the once-a-minute config reload -- and the clock crosses one or more whole seconds before the next pass, the skipped seconds are serviced after the fact, so a second-level job due in the gap still fires (once) rather than being silently dropped. The replay is bounded by a ten-second
CATCHUP_LIMIT: a larger gap is treated as a stall, suspend, or clock jump and resumed past with a warning, never replayed as a burst of backdated launches (matching cron's no-catch-up-after-an-outage behavior). Minute-level jobs need no catch-up: their minute-truncated slot already absorbs any sub-minute overrun. -
No spurious run at a mid-period restart. On startup the de-dup map is seeded with the in-progress slot for every scheduled job, so a job whose minute (or second) is already under way does not fire immediately on the first tick; it first fires at the next matching boundary, exactly as in minute-only mode. Without this, merely having any second-level job present would have made every minute-level job fire about a second after a mid-minute restart.
@rebootjobs are unaffected and still fire once at startup. -
Concurrent launches within a slot. When several jobs are due in the same slot,
spawn_jobsnow launches them concurrently instead of one at a time. With N jobs sharing a slot the old serial form cost N times a subprocess spawn -- the dominant source of same-second overrun -- which now collapses to about a single spawn. The single-job case (the norm) still takes a direct await and is byte-identical to before, and the de-dup and cluster-gate decisions are still made sequentially, so only the per-job "Starting"/"spawned" log lines may now interleave. -
Config reload moved off the event loop. The once-a-minute reload now runs its disk read and full reparse in a worker thread (
reload_config), so a slow parse no longer freezes the event loop -- web API, cluster gossip, job-output pumping -- for its whole duration. The parsed job set is still applied on the loop thread and before jobs are serviced, so the cluster leader-gate is always current for the tick. Housekeeping (config reload, cluster and web (re)start, logging config) is gated to run at most once per wall-clock minute even while the loop ticks per second; in pure minute-tick mode it runs every iteration, exactly as before.
-
yearrestricts the schedule to specific years. Earlier releases accepted ayearkey on the schedule object but built only a five-field crontab string from it, silently droppingyearso it had no effect -- a job with an object-formyearran every year. It is now emitted as parse-crontab's trailing year column and honored, soyear: "2017"really does pin the schedule to 2017. (String schedules were always passed to parse-crontab verbatim, so a six-field string already honored its year; only the object form changes.) This is a behavior change -- see the upgrade notes below. -
Honoring
yearchanges that job's job-set fingerprint, so during a rolling upgrade of a cluster the old and new binaries compute differentjob_set_ids for the identical config and will not treat each other as agreed peers until every node is upgraded -- the same transient, self-healing drift as any config rollout, and leader election stays at-most-once throughout. Jobs that do not use object-formyearare unaffected: their fingerprint is byte-for-byte identical to before.
-
A malformed schedule now fails the reload with a named error. parse-crontab's
ValueErroron a bad field (an out-of-range value, the wrong field count) is caught and re-raised asConfigError("invalid schedule '...': ..."), naming the offending expression, so a bad schedule fails config load or reload cleanly with a message the reload loop can log, rather than surfacing as an anonymous traceback. -
One object-to-crontab builder, shared everywhere. A single
schedule_object_to_crontabhelper now renders the object form to a crontab line -- five fields normally, six or seven whenyear/secondare used -- and is shared by parsing, the fingerprint, and the dashboard's schedule label, so those three can never disagree on the mapping. The object form still collapses to the exact five-field line as before when neithersecondnoryearis set, so its fingerprint is unchanged. Whether a schedule counts as second-level is derived from the actual rendered field count (seven), not mere key presence, so a blanksecond:value that renders an empty column does not force the whole scheduler onto the per-second cadence.
-
Cron parsing, description, and preview understand five-, six-, and seven-field expressions. The client-side cron engine normalizes any of the three widths (implicit second
0and any year for five fields, a trailing year for six, a leading second for seven), computes next-fire times at second resolution with year restriction (and parse-crontab's 2099 year ceiling), and renders wall-clock times with a seconds component where the schedule has one. Plain-English descriptions gain "Every second", "Every N seconds", "At second(s) ...", and an "in {year}" clause -- and deliberately do not lead with a per-second cadence phrase when a coarser field is restricted, so a schedule like* 30 * * * * *is not described as firing every second. -
Cron sandbox covers the new widths. The palette's schedule sandbox validates 5-, 6-, and 7-field expressions (its error copy and field-breakout labels updated to match, labelling the leading second and trailing year columns correctly), and its next-fire preview shows seconds.
-
Clicking the wordmark spins the logo. The "cronstable" wordmark now triggers the same mark-spin animation as clicking the mark glyph.
-
example/pulse-monitor-- a small, runnable real-time uptime / SLA monitor built entirely on second-level scheduling: it probes a latency-critical service every few seconds, heartbeats every ten, and rolls up a summary once a minute (which still fires exactly once per minute alongside the per-second probes). It watches cronstable's own/statusendpoint, sodocker compose -f docker-compose-pulse.yml upneeds nothing else running. -
example/pulse-cluster-- the clustered sibling: a three-node, mutual-TLS, leader-electing cluster that splits the monitoring work the way a real fleet should --liveness-proberuns on every node (independent vantage points catch a partition outage), whilelatency-sloand the summary run on the leader only. A one-shot service mints throwaway certs, and an optionaldistribution: spreadfans the leader jobs across nodes by rendezvous hashing.docker compose -f docker-compose-pulse-cluster.yml up. -
The wiki and README are updated throughout: a new "Second-level schedules" reference and field-count table in Schedules and Timezones, the
yearkey documented as honored (with an upgrade note), a Troubleshooting entry on the common "six fields is a year, not seconds" mistake, the Configuration Referenceschedulerow, and the Web Dashboard sandbox notes. -
Internal: second-level scheduling ships with a matching batch of tests (
test_cron.py,test_config.py,test_fingerprint.py) covering the object and string spellings, the adaptive cadence and its zero-overhead minute path, per-slot de-duplication, bounded catch-up, the mid-period-restart seeding, theyearfingerprint change, and the malformed-schedule error path.
-
Object-form
yearis now honored (breaking). A schedule object that setsyearpreviously had no effect and now restricts the job to that year, so a past year stops the job firing. This is the only change that can affect an existing deployment, and only one that uses object-formyear; to keep the old "runs every year" behavior, remove theyearkey. During a rolling cluster upgrade such a job's fingerprint changes, so mixed-version nodes will not agree on itsjob_set_iduntil all are upgraded (transient and self-healing; leader election stays at-most-once). All other schedules -- crontab strings, and object schedules withoutyear/second-- behave and fingerprint exactly as before. -
Seven-field crontab strings now fire at second granularity. A seven-field string earlier fired at most once a minute, because the scheduler zeroed the seconds column and only woke per minute; such schedules were effectively meaningless. They now fire on the seconds they specify. This is unlikely to surprise, but audit any seven-field strings already in your configs.
-
New webhook reporter: native Slack/Discord/Teams/ntfy notifications. A fourth reporter joins sentry/mail/shell in every
reportblock:webhooksends an HTTP request (POST by default) to a configured URL with a jinja2-templated body. The default body is a{"text": ...}JSON payload carrying the same subject-plus-body text as the default mail/sentry templates, JSON-encoded with jinja2'stojsonfilter so quotes, newlines, and non-ASCII job output always produce valid JSON -- pointurlat a Slack, Mattermost, or Teams incoming webhook and it works with no further configuration.method,contentType,headers,body, andtimeoutcover everything else (Discord's{"content": ...}shape, ntfy's plain-text body and header-driven priority, or your own endpoint). The URL resolves like the sentry DSN (value/fromFile/fromEnvVar) and is treated as a secret throughout: it is never logged, and the job-set fingerprint redacts the inline URL value and all header values (which commonly carryAuthorizationtokens). No new dependency -- outbound delivery rides the core aiohttp. Note: because every job's effective config gains the new default block, job-set ids change on upgrade; replicas must be on the same version to compare ids, as before. -
Unchanged peers now answer gossip polls with a bodyless
304. Every/peerresponse carries a strongETag(a content hash of the payload), and each polling node echoes the tag of the last full body a peer served it back asIf-None-Match, so a peer whose state has not changed since then skips re-sending the full O(members + jobs) JSON: a converged, idle cluster's steady-state round costs headers rather than bodies. This is a transport optimization, not a protocol change. A304is still a fresh, mutually-authenticated round trip, and because the tag is content-derived, a match proves the peer's payload is exactly the one the poller already holds, so the poller replays its cached observation and every gate (mutual agreement, conflict detection, thecluster.driftAfterdebounce) advances exactly as if the identical body had been re-sent. The one live field, a job's seconds-to-next-fire countdown, is hashed as the absolute next-fire time instead, so the tag stays stable between fires and rolls exactly when a schedule fires. Mixed fleets degrade safely during a rolling upgrade: an older peer ignoresIf-None-Matchand keeps serving full bodies, a tagless response stops the poller from sending the header at all, an unsolicited304is recorded as a failed poll, and an over-long or non-printable tag is never stored or echoed. -
Fleet-view countdowns are aged, not frozen. With
304rounds refreshing a peer's liveness without re-shipping its job summaries, a stored snapshot can now legitimately outlive many polling rounds, soGET /fleetre-derives each peer job's advertisedscheduled_incountdown from the snapshot's age (an elapsed duration measured on the local clock alone, so peer clock offsets never leak in) instead of serving the value the snapshot arrived with, clamping at zero. The fire itself rolls the peer'sETag, so the next poll ships a full body carrying the real successor value. -
/peerbodies that do go out are gzip-compressed once they reach 1 KiB (below that, the per-request CPU spend outweighs the few bytes saved). The polling side already advertised gzip support, and the existing response-size cap applies to the decompressed payload, so compression does not weaken it. -
Dashboard: the version and job-set id header chips copy their value on click. Header text is chrome and is no longer text-selectable; the two values worth grabbing hand themselves out instead. Clicking the version chip copies the version, and clicking the job-set chip copies the full job-set id even though the header shows only a short prefix; both tooltips say so. The command palette carries the same two copies, so both values stay reachable from the keyboard.
-
Dashboard: the quick "power-on sweep" flash between boots is gone. The full POST boot screen still replays once its cooldown elapses, but the visits in between now start the app directly instead of playing a full-screen power-on animation first.
-
Internal: the conditional exchange ships with a matching batch of cluster tests (tag stability across countdown ticks and rollover on a fire, the
304replay path, unsolicited-304and unusable-tag rejection, countdown aging, and an end-to-end mutual-TLS304-plus-gzip round), and the wiki's Architecture and Internals page documents the exchange.
This is the largest cronstable release since the fork. Its headline feature is clustering: several replicas can now verify they hold the same job set, elect a leader so each scheduled job runs on one node instead of every node, spread jobs across the fleet, and fail over when a node dies, coordinating either peer-to-peer over mutual TLS or through a Kubernetes or etcd lease. The release also adds native Prometheus metrics, accepts classic crontab files as configuration, and grows the web dashboard into a small operations console. Clustering is entirely opt-in; see the upgrade notes below for the few behavior changes that apply to existing deployments.
- New optional top-level
cluster:section. Give every replica the same static peer list and a dedicated mutual-TLS listener (cluster.listen,cluster.tls.{ca,cert,key},cluster.peers), and each node polls every peer once per round (cluster.interval, default 30 seconds), comparing job-set ids (see 1.1.8) to attest that all replicas hold an identical job set. Peers are reported asagreed,syncing,drifted(a mismatch that persists forcluster.driftAfterconsecutive rounds, default 3),unreachable,untrusted(TLS verification failed), orconflict. On its own this is observe-only (every node still runs every job), which makes it a safe first rollout step. - Leader election (
cluster.electLeader: true). The leader is the lowestcluster.nodeName(default: the hostname) among the agreeing nodes this node can see, and only when they form a strict majority of the declared cluster size; below quorum a node stands down and runs nothing. Two disjoint majorities cannot exist, so a clean partition produces at most one leader within about one polling interval. Conflicts fail closed: a duplicatenodeName, a cluster-size disagreement (say, a rolling resize from 3 to 5 nodes), or a coordination-policy divergence parksLeaderjobs until the fleet reconverges, and each is logged loudly and shown on the dashboard. A freshly started or reconfigured node holds its jobs until it has polled every configured peer at least once, so a blank-view node cannot elect itself. Two-node election is refused at config load (it is strictly worse than a single replica); even cluster sizes log a warning. - A per-job
clusterPolicy(also settable underdefaults:) decides what election means for each job:Leader(the default: only the elected leader runs it, and it is skipped when in doubt),PreferLeader(never skip: the lowest reachable agreeing node runs it even without quorum, accepting a rare double-run, so reserve it for idempotent jobs), andEveryNode(all nodes run it, for node-local housekeeping). Manual runs viaPOST /jobs/{name}/startare never gated. Automatic retries re-check the gate before every relaunch and abandon a pending retry when ownership has demonstrably moved to another node;@rebootjobs under election are deferred until the cluster converges and then run once, on the owner. - Load-balanced jobs:
cluster.distribution: spread. Instead of one leader running everyLeader/PreferLeaderjob, each such job is assigned an owning node by rendezvous (highest-random-weight) hashing over the quorate agreeing members, spreading work across the fleet. A membership change moves only the departing or joining node's share of jobs. The same quorum gating applies, andGET /jobsreports each job's currentclusterOwner. - Mutual TLS is the entire trust boundary. A peer must present a certificate chaining to the configured CA and matching the host it was reached at, so the CA should be dedicated to cronstable nodes and nothing else. Certificates rotated in place are detected and reloaded without a restart. The peer endpoint is served only on the cluster listener, never on the web API listener.
- The failure semantics are documented, not hand-waved. The built-in
backend is best-effort by design: there are narrow, self-healing windows
where a
Leaderjob can be skipped or aPreferLeaderjob can double-run, and the new wiki page enumerates them. When you need fenced exactly-once execution, use one of the lease backends below. cronstable --validate-configvalidates the entireclustersection (peer list, TLS material, lease timing invariants) without starting anything.
cluster.backend: gossip | kubernetes | etcdpicks the coordination mechanism (defaultgossip, the peer-to-peer mode above). The lease backends replace the peer quorum with a fenced, expiring lease in an external store: exactly one node holds the lease at a time, soLeaderjobs are exactly-once while the store is reachable. With a lease backend, election is always on, there is no peer list or cluster mTLS to manage, anddistribution: spreadis rejected at config load (a single lease cannot express per-job ownership).- Kubernetes (
cluster.kubernetes.*): replicas campaign for acoordination.k8s.io/v1Lease object using the client-go leader-election algorithm (leaseNamedefaults tocronstable-leader;leaseDurationSeconds/renewDeadlineSeconds/retryPeriodSecondsdefault to 15/10/2, with the client-go timing invariants enforced at config load). In-cluster ServiceAccount token, CA, namespace, and API server are detected automatically; akubeconfigand an explicitapiServer(https only) are supported. The backend talks to the API server over the bundled aiohttp by default; installing the new optional extra (pip install cronstable[kubernetes]) switches to the official Kubernetes client (clientLibrary: auto|http|library). The stored holder identity embeds a per-process token, so two pods that accidentally share a name cannot both hold the lease.example/kubernetes/ships a ready-to-apply Deployment with the minimal ServiceAccount, Role, and RoleBinding. - etcd (
cluster.etcd.*): replicas campaign for a lease-bound key (electionNamedefaults tocronstable/leader) through etcd's v3 JSON/HTTP gRPC gateway, using a create-if-absent transaction fenced by the lease id (ttldefaults to 15 seconds, minimum 3). Multipleendpointsfail over in order; optionalusername/password(literal,fromFile, orfromEnvVar) and client TLS are supported, and credentials are refused unless every endpoint ishttps://.example/etcd/ships a compose demo with an etcd and two cronstable replicas. - Failover is fast and clock-safe on both. All fencing runs on the
monotonic clock with a one-second skew margin (no wall-clock or cross-node
clock-sync assumptions): a holder whose renewals stall demotes itself
locally before the store-side lease can expire, without a network call. A
graceful shutdown releases the lease explicitly (Kubernetes clears
holderIdentity, etcd revokes its lease) so a survivor takes over immediately rather than waiting out the TTL. If the store becomes unreachable,Leaderjobs fail closed andPreferLeaderjobs keep running.@rebootbookkeeping is persisted in the store, scoped to the job-set id, so aLeader@rebootjob runs once per job configuration rather than once per fleet restart. - No new runtime dependencies. Both lease backends are plain HTTP over the existing aiohttp core (no etcd or gRPC client; the Kubernetes client is optional), so all packaged architectures keep working.
- Native Prometheus metrics at
GET /metrics, served by the existing web listener whenever the web API is enabled; there is no exporter sidecar and no new dependency (the exposition is generated in-process). Both the classic text format and OpenMetrics 1.0 are served, negotiated via theAcceptheader. - Per-job series cover run outcomes (
cronstable_job_runs_totallabeled byjob_nameandstatus), retries, permanent failures, failures to start, a duration histogram with configurable buckets (web.metrics.durationBuckets), last success/failure/run timestamps and exit code, and live state (enabled, running, next run). Daemon series cover the version, start time, job-set id, job counts, and config-reload health. Cluster series cover size, quorum, leadership, per-peer status counts, conflicts, and leader/quorum transition counters; the wiki suggests alerting onsum(cronstable_cluster_is_leader) > 1and on losingcronstable_cluster_quorate. Metrics are recorded at the same point as the run history, so/metricsand/jobs/{name}/runsnever disagree. web.authToken, when configured, protects/metricslike every other endpoint;web.metrics.public: trueexempts just this endpoint for a scraper, andweb.metrics: falseremoves it entirely.
- Classic (Vixie) crontabs are now accepted as configuration. A file
with a
.crontabor.cronextension, or named exactlycrontab, is parsed as a crontab wherever configuration is loaded: passed to-c, dropped into a config directory alongside*.yamlfiles, or pulled in viainclude:; a neutral-named file given to-corinclude:is content-sniffed. Supported syntax: five-field entries (the same field dialect as YAMLschedulestrings),@keywords(including@rebootand@midnight), position-sensitiveVAR=valueenvironment lines, comments, and the\%escape.SHELLandCRON_TZassignments are honored as the job'sshellandtimezone. - Each entry becomes a normal cronstable job named
<file>:<line>, indistinguishable downstream: it shows up on the dashboard and HTTP API, participates in the job-set id and clustering, and can be run or cancelled on demand. cronstable's defaults apply rather than cron's (UTC unlessCRON_TZsays otherwise; any stderr output marks the run as failed). Deviations are deliberate and loud: an unescaped%is a load-time error instead of silently not feeding stdin,MAILTOis exported to the job but sends no mail, and the six-column system-crontab user field is not supported. Per-entry knobs (retries, reporting, timeouts) still require migrating that line to YAML; the new wiki page shows how.
- A cluster operations view. A cluster panel shows this node's role
(leader, follower, or standing down and why), quorum state, and a
per-peer table with status dots and a per-peer history timeline; lease
backends show the lease holder and expiry instead. A page-level alert bar
calls out a conflict or lost quorum without scrolling, and a fleet view
(backed by the new
GET /fleet) renders a jobs-by-nodes matrix of every node's last outcome per job, with a failing-only filter. - Incident tooling. A verdict bar summarizes active failures and, when
several jobs fail together, says whether they look like one cause or
independent ones; an incident timeline (press
i) orders every job's most recent run; a mitigate console bulk-starts or bulk-cancels jobs and copies a Markdown incident summary; and a merged multi-tail console streams up to four jobs' logs into one color-coded view. - Wallboard and zen mode. A full-screen TV mode (press
w, or open#tv) shows worst-first job tiles with a staleness watchdog that showsNO SIGNALrather than a false all-green; when everything is healthy and idle for a while it drifts into a zen screensaver in which every job pulses at its real next fire time. - More ways to read the schedule. An activity heatmap (a 6h/24h/7d punchcard per job), a cron sandbox that validates a cron expression and previews its next 12 fire times alongside live jobs sharing it, a column picker adding Owner/Policy/TZ/Next-at/Rate columns, and an opt-in, browser-local run ledger (IndexedDB) that keeps run history beyond the daemon's in-memory cap and flags unusually slow runs against each job's own duration baseline.
- Finishing touches. A fourth theme,
carolina(a Carolina-Blue CRT phosphor), joins amber/green/modern; optional audible cues with a volume setting and an escalating failure alarm (pressato acknowledge); and an optional boot self-test splash on load.
- New endpoints:
GET /cluster(this node's cluster view: backend, quorum, leader, conflicts, per-peer detail) andGET /fleet(fleet-wide per-job run summaries carried on the gossip round; observability only), plusGET /metricsabove.GET /jobsgainsclusterPolicyand, underdistribution: spread, each job'sclusterOwner. The HTTP API wiki page now documents every endpoint with full response shapes. - Configured
web.headersare now applied to every successful response, including the new endpoints, and to the409 Conflictbodies ofstart/cancel. - A web-section configuration error found during a reload now leaves the web API down with a clear log line while the rest of the reload (jobs, cluster, logging) still applies.
- A failed job launch can no longer crash the scheduler. Spawning a job
now guards against any
OSError(file-descriptor exhaustion, fork or memory limits, permission errors), not just a missing executable: the run is recorded as an ordinary start failure with exit code 127 (and counted incronstable_job_start_failures_total) instead of the error propagating out of the scheduling loop and taking the daemon down.
/metricsis served by default wherever the web API is enabled. It sits behindweb.authTokenlike every other endpoint when a token is configured; setweb.metrics: falseto remove it.- The job-set id changes once on upgrade:
clusterPolicyis now part of every job's fingerprint, so an unchanged configuration hashes to a differentv1:id than 1.1.8 through 1.1.11 reported. Compare ids only between nodes running the same cronstable version; a mixed-version fleet reads as drift until the rollout completes. - Config directories now load crontab-named files (
*.crontab,*.cron,crontab) that earlier releases silently ignored.
- New
cronstable.backendssubpackage, and a new optional extra:pip install cronstable[kubernetes]installs the official Kubernetes client forclientLibrary: library. A core install gains no new runtime dependency and the supported Python range is unchanged. The PyPI metadata gains prometheus/metrics/monitoring keywords and an updated description.
- Three new wiki pages: Clustering and Leader Election (a full operations guide: both backend families, every failure window, sizing, rollout, and troubleshooting), Metrics with Prometheus, and Classic Crontabs, plus major updates to the HTTP API, Web Dashboard, Production Deployment, Architecture and Internals, and Troubleshooting pages. The README gains matching sections.
- New runnable demos:
docker-compose-cluster.yml(a three-node mutual-TLS cluster with scripted try-it failover scenarios),docker-compose-cluster-large.yml(ten nodes withdistribution: spreadand CPU-heavy jobs, for watching work fan out),docker-compose-acme.yml(a five-node simulated data platform with a mail sink, a statsd exporter, and deterministic scripted incidents that exercise the dashboard's incident tooling),docker-compose-zen.yml(one calm node for the zen screensaver),example/etcd/andexample/kubernetes/for the lease backends, andexample/crontab/for classic crontabs.
- The test suite roughly doubles, with new suites for the cluster manager,
both lease backends, the leadership abstraction, the Prometheus
registry/exposition, crontab parsing, and backend config validation, plus
large additions to the scheduler tests. The
devextra addscryptographyfor the cluster mTLS tests (skipped on Windows ARM64, which has no wheel). A new.gitattributespins*.shto LF line endings so the bind-mounted cluster demos work from a Windows clone.
- Coverage is now published to Codecov.
Every CI matrix cell uploads its own
coverage.xmlunder an<os>-py<version>flag, and Codecov merges them into one combined number, so POSIX-only paths that Windows skips (privilege drop,user/groupresolution) still count toward the published total instead of dragging it down to the lowest single row. tox now also writes the report it consumes (pytest --cov-report=xml). The hard pass/fail gate stays with tox's--cov-fail-under=82(see 1.1.10): Codecov's own project and patch status checks are configured as informational only, so they annotate pull requests without ever blocking them. The upload runs even on failed or cancelled jobs and keepsfail_ci_if_error: false, so a Codecov outage never reds the build, and flagcarryforwardkeeps the combined number stable when a matrix row is skipped on a given run. The README gains a matching coverage badge.
-
Numeric
user/groupis read as a uid/gid, not a login name. In the config schema theuser/grouptype was aStr() | Int()union, and strictyaml matched the always-acceptingStr()first, souser: 1000arrived as the string"1000"and was looked up as a login name (getpwnam("1000")) rather than used as uid 1000. The union is nowInt() | Str(), so a bare number is treated as the uid/gid it looks like; a non-numeric name (user: www-data) still falls through toStr(). (POSIX only; per-jobuser/groupremains rejected with a configuration error on Windows.) -
More resilient container builds. Every image build, across the default Debian image and all seven distro variants (
-alpine,-ubuntu,-rhel,-fedora,-opensuse,-amazonlinux,-distroless), now wraps its package-manager andpipnetwork steps in a retry-with-backoff helper, alongside each manager's native knobs (apt'sAcquire::Retries,dnf's--setopt=retries, an explicitzypper refreshretry, and a longerpip --timeout), so a transient mirror or package-index hiccup retries instead of failing the whole build. The build and test CI workflows get the same hardening viaPIP_RETRIES/PIP_TIMEOUT, withbuild.ymlforwarding them into its emulated cross-architecture binary builds viadocker run -e. -
The
-distrolessimage now builds foramd64/arm64only. Thegcr.io/distroless/python3-debian12base publishes noppc64leors390xmanifest, so requesting those arches aborted the distroless release with "no match for platform in manifest". The RPM-based variants (-rhel,-fedora,-opensuse) still cover the wider arch set. -
The README status badges also gain brand new colors (and logos on the PyPI/Python badges). yay
-
Internal: branch coverage is now measured and gated in CI (tox runs
pytest --cov-fail-under=82), backed by substantially expanded unit tests for config and user/group validation, config reload and graceful shutdown, the job runner, and the job-set-id fingerprint.
- More prebuilt container images. Alongside the default Debian-based image,
every release now also publishes the same build on seven more bases, each
tagged with a
-<distro>suffix:-alpine,-ubuntu,-rhel(Red Hat UBI 9),-fedora,-opensuse(Leap),-amazonlinux(2023) and-distroless, plus an explicit-debianalias for the default. Pick the base that matches your host userland or image-provenance policy; behavior is identical, since cronstable is a pure-Python app (Python >= 3.10) and each image uses its distro's native interpreter. The Debian image still owns the barelatest/<version>tags and the widest architecture coverage. See Distro variants.
- Job-set id. cronstable can now emit a job-set id: an order-independent
hash of every job's effective configuration. Two instances deployed from the
same configuration produce the same id, so replicas can confirm they hold an
identical set of jobs. It is taken over the merged, effective config (so
reordering jobs, or moving
a setting into
defaults, doesn't change it), normalizes equivalent schedule spellings, fingerprintsuser/groupas configured rather than as a host-specific resolved uid/gid, and embeds no secret material (inline reporting secrets are redacted, and onlyenvironmentvariable names are hashed, not their values). Get it from the CLI (cronstable --job-set-id, prints and exits), the web API (GET /job-set-id, alsoapplication/json), and the dashboard header; it is logged once at startup and again whenever a config reload changes it. The scheme is versioned (av1:prefix) so ids are only compared within a scheme.
- Windows support. cronstable now runs natively on Windows, in addition to
Linux and macOS. The core was made portable: the POSIX-only
grp/pwdimports are now lazy and guarded, Ctrl-C / Ctrl-Break shutdown is wired up without the POSIX-only event-loop signal handlers, and subprocess argv is encoded per platform.pip install cronstableworks on Windows, and every release now also ships self-contained binariescronstable-windows-amd64.exeandcronstable-windows-arm64.exe(Python not required on the target).- On Windows a string
commandwith no explicitshellruns through the native command processor (%ComSpec%, i.e.cmd.exe), mirroring the/bin/shdefault on POSIX. Setshell:or passcommandas a list for anything else. - The default config location (
-c) is%APPDATA%\cronstableon Windows (/etc/cronstable.dis unchanged on POSIX). - Two features remain POSIX-only and are reported clearly on Windows: per-job
user/groupswitching (rejected with a configuration error) andunix://web listeners (skipped with a warning; use anhttp://listener).
- On Windows a string
- CI now runs the test suite on Windows (x64 and ARM64) as well as Linux, and the per-commit build plus every release build the Windows binaries.
- Update the README
Platformsbadge to include Windows.
- Add self-contained binaries for two more Linux architectures, bringing
every release to eight Linux architectures plus macOS: 64-bit RISC-V
(
riscv64) in both glibc and musl flavors (cronstable-linux-riscv64andcronstable-linux-riscv64-musl) and 32-bit ARMv6 (armv6, e.g. Raspberry Pi Zero / Pi 1) in musl only (cronstable-linux-armv6-musl). As with the other binaries, Python is not required on the target system. Neither arch has a native GitHub runner, so they build inside a container viadocker run --platformunder QEMU emulation.armv6is musl-only because the Debian/glibc base image ships no 32-bit ARMv6 variant (only ARMv5/ARMv7), so there is no glibcarmv6binary and the container image does not cover it.- Some dependencies ship no prebuilt wheel for these arches
(
multidict/frozenlist/ruamel.yaml.clibonriscv64; the entire C-extension stack onarmv6), so they compile from source during the build.
- The published container image now also covers
linux/riscv64(alongsidelinux/amd64,linux/arm64,linux/386,linux/arm/v7,linux/ppc64leandlinux/s390x), and is build-checked at that full arch set on every commit. - Update the README
Architecturesbadge to list the new targets (amd64,arm64,armv7,armv6,i686,ppc64le,s390x,riscv64).
This is a documentation release; there are no changes to the cronstable
package itself.
- README changes
- Add an
Architecturesbadge to the README summarizing the binary and container targets (amd64,arm64,i686,armv7,ppc64le,s390x).
- Default the manual (
workflow_dispatch) release to apatchbump and listpatchfirst in the bump options, since patch releases are the common case.
- Add self-contained binaries for two more Linux architectures to every
release, in both glibc and musl flavors: little-endian POWER (
ppc64le) and IBM Z (s390x) (cronstable-linux-ppc64le,cronstable-linux-s390x, and their-muslvariants) alongside the existingamd64,arm64,i686andarmv7builds. As with the other binaries, Python is not required on the target system. Neither arch has a native GitHub runner, so they build inside a container viadocker run --platformunder QEMU emulation; both have prebuilt manylinux and musllinux wheels for the aiohttp dependency stack, so nothing compiles from source. - The published container image now covers them too: the multi-arch image
adds
linux/ppc64leandlinux/s390x(tolinux/amd64,linux/arm64,linux/386andlinux/arm/v7), and is build-checked at that full arch coverage on every commit.
- Add self-contained binaries for two more Linux architectures to every
release, in both glibc and musl flavors: 32-bit x86 (
cronstable-linux-i686andcronstable-linux-i686-musl) and 32-bit ARM (cronstable-linux-armv7andcronstable-linux-armv7-musl), alongside the existing 64-bitamd64andarm64builds. As with the other binaries, Python is not required on the target system. The 32-bit binaries are built inside a 32-bit container (i686natively on the x86-64 runner,armv7under QEMU emulation). - The published container image now covers those architectures too: the
multi-arch image is built for
linux/amd64,linux/arm64,linux/386andlinux/arm/v7, and is build-checked at that full arch coverage on every commit.
This is a documentation release; there are no changes to the cronstable
package itself.
- Add a project wiki (under
wiki/) covering installation, the configuration reference, the HTTP API, the web dashboard, schedules and timezones, reporting, statsd metrics, output capturing, concurrency and timeouts, failure detection and retries, includes and defaults, logging, the CLI, architecture and internals, production deployment, migration from yacron, contributing/releasing, and troubleshooting. - Showcase the web dashboard near the top of the README with annotated screenshots of the overview, live log tail, run history, schedule preview, command palette, keyboard-shortcut reference, and the green-phosphor and flat modern themes, linking the dashboard tour in the wiki.
- Slim the README's web-server section to an "Enabling the web dashboard" pointer to that showcase and the wiki, removing the duplicated feature list.
- Add a built-in web dashboard, served at the root path (
/) of anyhttp://listener. It shows each job's latest status with a live countdown to the next run and a trend sparkline, tails job logs live (with in-log search, ANSI-color rendering, optional timestamps, a line-wrap toggle, and a download button), runs or cancels jobs on demand, and reports each job's run history, success rate, and a plain-English schedule with a preview of upcoming run times. It is keyboard-first (?for shortcuts,Ctrl-K/⌘Kcommand palette,/to filter), with configurable themes, a compact density mode, polling interval, and optional desktop failure notifications, all remembered in the browser. - Cancel running jobs over the REST API with
POST /jobs/{name}/cancel, using the same graceful SIGTERM-then-SIGKILL sequence (honoringkillTimeout) as elsewhere. A cancelled run is recorded with acancelledoutcome and is neither reported nor retried; the endpoint returns409 Conflictif the job is not running and404 Not Foundfor an unknown job. GET /jobsnow returns detailed per-job information: schedule, timezone, enabled/running state, time until the next run, a summary of the most recent finished run, and a compact recent-outcome history.- Read a job's retained run history and aggregate statistics (success
rate and average/min/max duration) with
GET /jobs/{name}/runs. - Tail a job's captured output live over Server-Sent Events with
GET /jobs/{name}/logs, replaying the most recent run's buffered output before streaming new lines. - Add a
web.uioption; setui: falseto expose only the REST API and disable the dashboard. - Keep run history and live logs in memory only, so the dashboard does not change cronstable's read-only-filesystem deployment story; history resets when cronstable restarts.
- Ship a
docker-compose.ymland a demo crontab for trying the dashboard against a set of varied example jobs.
- Serve the dashboard with a strict
Content-Security-Policyand additional hardening headers (X-Content-Type-Options,X-Frame-Options,Referrer-Policy); each can be overridden viaweb.headerswhile unset defaults are still applied. - When bearer-token authentication (
web.authToken) is enabled, the dashboard page loads without a token and then prompts for one, storing it only in that browser tab; every data request it makes is authenticated with that token.
- Publish container images to Docker Hub as
docker.io/ptweezy/cronstableon every release, in addition to GHCR. The two registries carry the same multi-arch (linux/amd64+linux/arm64) image, so you can pull from whichever you prefer. - Document the Docker Hub images in the README and add a quick-start
docker runexample and a Docker Hub badge. - Harden the release workflow so Docker Hub publishing is enabled only
when both
DOCKERHUB_USERNAMEandDOCKERHUB_TOKENare configured.
- Lower the minimum required Python version from 3.13 to 3.10; cronstable now supports Python 3.10, 3.11, 3.12, 3.13, and 3.14.
- Add PyPI trove classifiers for Python 3.10, 3.11, and 3.12 so the expanded support is reflected on the package page.
- Expand the test matrix (
toxand CI) to run across all five supported interpreters (3.10–3.14). - Type-check with
mypyagainst Python 3.10 so stdlib APIs that are unavailable on the lowest supported interpreter are caught at lint time rather than at runtime.
Since 1.0.13, the net changes are entirely build/CI hardening (a new build.yml, an arm64 addition to docker.yml). Here's the changelog body:
- Add a per-commit build-verification workflow that builds the wheel,
sdist, and self-contained PyInstaller binaries for Linux (both glibc and musl/Alpine, onamd64andarm64) and macOS (amd64andarm64) on every push, without publishing, so a broken build or bundle is caught at commit time instead of only at release. - Build-verify the Docker image for both
linux/amd64andlinux/arm64on every commit, catching arm64-only breakage (such as a dependency with no arm64 wheel) that the previous amd64-only check would miss.
- Update the bundled Python runtime in the standalone binaries to
3.13.14(from3.13.5), picking up the latest upstream bug and security fixes. - Expand the PyPI package metadata with additional keywords, trove
classifiers, and project links (
Documentation,Source,Changelog,Issues, andContainer) for easier discovery.
- Tidy up
README.md, trimming redundant badges and condensing the macOS code-signing notes.
- Update the GitHub Actions used to build and publish Docker images
(
docker/metadata-action,docker/login-action,docker/setup-qemu-action,docker/setup-buildx-action, anddocker/build-push-action) to their latest major versions. - Update the release workflow's
actions/upload-artifact,actions/download-artifact, andsoftprops/action-gh-releaseactions to their latest major versions.
- The macOS binaries are now Developer ID code-signed and notarized by
Apple, so Gatekeeper accepts them and they run without first clearing
the quarantine attribute (
xattr -d com.apple.quarantineis no longer needed).
- Release binaries now include macOS builds for both Apple Silicon
(
cronstable-macos-arm64) and Intel (cronstable-macos-amd64), alongside the existing Linux glibc and musl binaries. As with the Linux binaries, Python is not required on the target machine. - Document clearing the macOS Gatekeeper quarantine with
xattr -d com.apple.quarantinebefore first running the macOS binaries, which are unsigned and unnotarized. - Fix a typo in the README fork attribution.
- Document that the standalone binary is self-extracting: on each start it unpacks its embedded Python runtime into a temporary directory, so it requires a temp directory that is both writable and executable.
- Add guidance for running the binary under a read-only root
filesystem — mount a small
rw,exectmpfs at/tmp(Docker's--tmpfsdefaults tonoexec, which fails), use a KubernetesemptyDir, or pointTMPDIRat a writable, executable directory. - Clarify that this temp-directory requirement is unique to the
standalone binary: the published container image and
pip/pipxinstalls run cronstable as a normal Python package and need no writable temp directory.
- The official multi-arch (
linux/amd64+linux/arm64) container image is now built and published to GHCR automatically as part of every release, and is build-checked on every commit so a brokenDockerfilefails fast.
- Add self-contained musl binaries to every release for Alpine and
other musl-based systems:
cronstable-linux-amd64-muslandcronstable-linux-arm64-musl, alongside the existing glibccronstable-linux-amd64andcronstable-linux-arm64builds. Python is not required on the target system. - Build the release binaries with Python 3.14.
- GitHub Releases now use the curated
HISTORY.mdsection for the release as the body of the release notes. The matching## X.Y.Zentry is extracted and shown above GitHub's auto-generated "What's Changed" list and changelog compare link, so each release page leads with the human-written changelog instead of only auto-generated notes.
- Release binaries are now published for both
linux/amd64andlinux/arm64. Every GitHub Release attaches a self-containedcronstable-linux-amd64andcronstable-linux-arm64executable, each built natively on its target architecture (previously only a single binary was provided). - The downloadable binaries embed Python, so none is required on the target system, and run on any Linux host with glibc 2.39 or newer (e.g. Ubuntu 24.04) matching the CPU architecture.
- Each binary is smoke-tested with
--versionand built before publishing
- docker builds
- Config reload failures no longer risk crashing the scheduler: if
re-reading the configuration fails (for example, a YAML error introduced
while cronstable is running), the previously-loaded jobs keep running
instead of the main loop failing on an unset
configreference. - A job whose command cannot be launched (for example, the executable does
not exist) is now reported as an ordinary failure with exit code
127, instead of raisingRuntimeError("process is not running")and being logged as an internal "please report this as a bug" error. - statsd reporting is now strictly best-effort: a failure to send the
job_started/job_stoppedmetrics (for example, an unresolvable statsd host) is logged as a warning instead of propagating out of job start/stop. - The mail reporter now always closes its SMTP connection, even when
STARTTLS, login, or sending fails, so a misbehaving mail server can no longer leak one connection per report. - Sentry and e-mail reporting no longer raise
KeyErrorwhen the DSN or password is configured withfromEnvVarbut the environment variable is unset; cronstable logs an error and skips that report instead.
- The Sentry
fingerprintsetting now replaces rather than appends when mergingdefaults: a job (ordefaultsblock) that defines its ownfingerprintoverrides the default entirely, so custom Sentry issue grouping works as configured (previously the three default entries were silently prepended). includecycles are now detected and rejected with a clearConfigError("include cycle detected") instead of recursing until aRecursionError.- Jobs loaded from a configuration directory are now processed in sorted filename order, so job ordering and "first config found" messages are deterministic rather than dependent on the filesystem's directory order.
- Environment files (
env_file) are now read as UTF-8.
- The web API's
Authorizationcheck now treats theBearerauth scheme as case-insensitive (per RFC 7235), while still comparing the token itself in constant time. - The mail reporter no longer logs the name of the configured password environment variable.
- Refactored
JobConfigconstruction into focused helper methods and switchedsend_to_statsdtoasyncio.get_running_loop(); no behavioral change. - Added a
.github/CODEOWNERSfile.
This is a tooling and documentation release; there are no changes to the
cronstable package itself.
- Added an opt-in, marker-driven
releaseGitHub Actions workflow: a push tomainwhose commit message carries a release marker on its own line ([release]/[release:major|minor|patch]), or a manual run, gates ontox, builds at the next version, publishes to PyPI via Trusted Publishing (OIDC), and only after a successful publish tags the commit and cuts a GitHub Release. - Hardened the release trigger to match a whole-line marker, so a
[release]mention inside prose never triggers a publish, and added a localcommit-msghook (scripts/gen_changelog_entry.py) that drafts a changelog entry for release commits. - Set least-privilege
permissions: contents: readdefaults on thetoxandreleaseworkflows.
- Added
CONTRIBUTING.mddocumenting the development setup, the test/lint/type-check workflow, and the release process, and linked it from the README. - Converted the changelog from reStructuredText to Markdown (
HISTORY.rst->HISTORY.md) and pointed the changelog generator, thecommit-msghook, andCONTRIBUTING.mdat the Markdown changelog.
- Promote the PyPI
Development Statusclassifier from4 - Betato5 - Production/Stableto reflect the stable 1.0 release series. No code changes.
- The web API now fails closed when
web.authTokenis configured but resolves to an empty token (an unsetfromEnvVar, or an empty/missingfromFile): cronstable raises aConfigErrorand refuses to start the HTTP server, instead of silently serving the control API without authentication. - The web API now honors
enabled: false.POST /jobs/<name>/startreturns409 Conflictfor a disabled job rather than launching it, andGET /statusreports such jobs asdisabledinstead of an inapplicablescheduled (in N seconds). - Invalid
web.listenURLs (an unsupported scheme, or anhttpurl missing host/port) are now logged as a warning and skipped, instead of being surfaced as an internal "please report this as a bug" error; a bind failure (OSError) on one address likewise no longer aborts the whole config update. The "started listening" message is logged only after the bind actually succeeds. concurrencyPolicy: Replaceno longer reports the replaced (cancelled) job instance as a failure and no longer schedules retries for it; the forced termination is treated as a replacement, not a job failure.
- Removed a dead Windows event-loop branch from
main()(cronstable is POSIX-only because it importsgrp/pwdat load time). naturaltimeno longer relies on anassertfor control flow (which would be stripped underpython -O).- The concurrency-policy test was rewritten to be deterministic (it was
previously an
xfailthat could never exercise a second job instance).
- cronstable 1.0.0 is the first release of the cronstable fork, based on gjcarneiro/yacron 0.19. It carries forward all of upstream yacron's functionality and adds modernized packaging, a Python 3.13+ runtime, new web-API authentication, and a set of security and correctness fixes.
- The project, package, command, config directory, and reporter environment
variables have all been renamed from
yacrontocronstable(see Breaking changes for migration steps).
- The installed command and PyPI distribution are renamed
yacron->cronstable(install withpip install cronstable; runcronstable). The Python import package is nowcronstableand the entry point iscronstable.__main__:main. - The default config directory changed from
/etc/yacron.dto/etc/cronstable.d; operators relying on the default path must move their config directory. - Minimum Python is now 3.13 (
requires-python >=3.13); only Python 3.13 and 3.14 are supported. Python 3.7 through 3.12 are no longer supported. - Reporter shell environment variables were renamed
YACRON_*->CRONSTABLE_*(e.g.CRONSTABLE_JOB_NAME,CRONSTABLE_RETCODE). ExistingonFailure/onSuccessshell scripts must be updated. - mail
validate_certsnow defaults toTrue, so SMTP TLS certificate validation is enabled unless explicitly disabled. Delivery to servers with self-signed/invalid certificates that previously worked silently will now fail unlessvalidate_certs: falseis set. - Privilege drop now drops/sets supplementary groups (
os.initgroups/os.setgroups) beforesetuid, fixing a privilege-escalation bug where root's supplementary group memberships leaked into the child. A numericuserwithout an explicitgroupnow derives its primary gid from the passwd database instead of silently keeping yacron's gid 0. defaults.environmentnow merges by key instead of concatenating: a job overriding a default variable yields a single entry. Configs relying on the old duplicate-key concatenation behave differently.- Dependency pins changed:
crontabjumped from==0.22.8to>=1,<2(major version change),strictyamlto>=1.7,<2,aiohttpto>=3.10,<4,aiosmtplibto>=3,<6(v2+ login API),sentry-sdkto>=2,<3.pytzand the directruamel.yamlpin were dropped;tzdata>=2024.1was added.
- New
web.authTokenoption adds opt-in bearer-token authentication to the HTTP API (literalvalue,fromFile, orfromEnvVar); when set, an aiohttp middleware requiresAuthorization: Bearer <token>on every route, compares it in constant time (hmac.compare_digest), and returns 401 otherwise. - New
web.socketModeoption sets octal permissions onunix://listen sockets, logging a warning rather than failing on invalid values; non-unix schemes are ignored. - Job stderr is now written to the process's stderr instead of stdout, so operators separating cronstable's own stdout/stderr streams get correctly routed output.
- Config now validates numeric ranges at load time and raises a clear
ConfigErrorfor invalid values (saveLimit>=0,maxLineLength>0,killTimeout>=0,executionTimeout>0, andonFailure.retryconstraints) instead of failing obscurely at runtime. - Multi-file config directories now aggregate jobs, defaults, and logging
across all files instead of using only the last file's settings. Duplicate
weborloggingblocks across the directory raise aConfigError, an empty/all-skipped directory yields an empty config (noUnboundLocalError), and a missing/unreadable single config file now raises a clearConfigError. - Logging configuration is now re-applied on reload when it changes and is only marked applied on success, so a logging section fixed after an error or changed at runtime is picked up without a restart.
- Scheduling a retry for a job that was removed from the configuration mid-retry no longer crashes; the stale retry state is cleared and the retry is skipped.
- Job stop metrics (statsd
job_stopped) are now emitted exactly once per run; a guard makes_on_stopidempotent, preventing duplicate metrics whencancelraceswait(e.g.concurrencyPolicy=Replace). - Non-UTF-8 job output no longer crashes the stream reader (output is decoded
with
errors='replace'). - A job with an empty environment list now gets its environment assigned
correctly (previously left
None). - Email reports now set an RFC 5322
Dateheader (email.utils.format_datetime), encode HTML bodies with the correct charset/transfer-encoding (set_contentsubtypehtml), and callsmtp.loginpositionally for aiosmtplib v2+ compatibility. - The Sentry client is now initialized once per
(dsn, environment)and cached instead of on every report, and usessentry_sdk.new_scope()(replacing the deprecatedpush_scope()). - Report templates (sentry/mail body, subject, fingerprint) are now compiled
and cached via an
lru_cache, and the three report blocks (onFailure,onPermanentFailure,onSuccess) deep-copy their defaults so they no longer alias one shared mutable object. - The shell reporter now logs a nonzero reporter exit code via
logger.error(clean message) instead oflogger.exception(which logged a bogusNoneType: Nonetraceback). - statsd UDP errors are now logged with their detail (
UDP error received: %s) instead of being dropped due to a missing format placeholder.
- Timezone handling migrated from third-party
pytzto the standard-libraryzoneinfo; invalid timezones now raiseConfigError. - Added
tzdata>=2024.1sozoneinfocan resolve timezones on slim/minimal container images that don't ship the system tz database. - The asyncio event loop is now created with
asyncio.new_event_loop()instead of the deprecatedasyncio.get_event_loop()(carried from upstream). - Internal logger and argparse program name updated to
cronstable; CLI error/version output now readscronstable.
- Migrated from legacy
setup.py/setup.cfgto a PEP 621pyproject.tomlusing the setuptools build backend (setuptools>=77,setuptools_scm>=8);setup.pyandsetup.cfgwere removed. - Versioning continues via setuptools_scm, now configured under
[tool.setuptools_scm]writingcronstable/version.py. - Adopted a PEP 639 SPDX license expression (
license = "MIT") withlicense-files, and updated the LICENSE with aCopyright (c) 2026, the cronstable developersline alongside the original 2019 copyright. - Added a
[project.optional-dependencies]devextra (mypy, mypy-extensions, pytest, pytest-asyncio, pytest-cov, ruff, tox) and trimmedrequirements_dev.txtto match (dropped flake8, types-pytz, and stale pins; added ruff). - Consolidated mypy and pytest configuration into
pyproject.tomland bumped the black/ruff target-version topy313. MANIFEST.inand packaging metadata updated for theREADME.rst->README.mdswitch.
- Removed Travis CI configuration (
.travis.yml). - Switched linting from black + flake8 to ruff (
ruff check+ruff format) with bugbear/mccabe/pycodestyle/pyflakes/import-sorting rules and a mccabe complexity limit; added a bandit config and a.pre-commit-config.yamlrunning bandit and ruff hooks (carried from upstream). - Modernized the GitHub Actions tox workflow: bumped
actions/checkout(v3 -> v7) andactions/setup-python(v3 -> v6.2.0), renamed the lint job, and trimmed the test matrix to Python 3.13 and 3.14. - Modernized
tox.ini(envlistpy313, py314, lint, mypy), removed the Travis mapping section, addedskip_installto the lint/mypy envs, droppedtypes-pytzfrom the mypy env, and pointed lint/mypy commands at thecronstablepackage. - Bumped pre-commit hook revisions (ruff-pre-commit and bandit).
- Converted the README from reStructuredText to Markdown (
README.rst->README.md) and rebranded it to cronstable, with a new intro noting it is a fork of gjcarneiro/yacron continuing from 0.19. The content is otherwise the same as upstream 0.19, not a rewrite; install docs now require Python >= 3.13, the prebuilt binary targets glibc 2.39 / Ubuntu 24.04, and releases come from github.com/ptweezy/cronstable. HISTORY.rstgained a fork-attribution preamble; older entries are retained as upstream yacron history.- Modernized the Docker example: base image
python:3.14-slimwithpip install cronstable(replacing ubuntu:xenial + virtualenv), config copied into/etc/cronstable.d, andENTRYPOINT ['cronstable']. - Updated the Kubernetes example to the
apps/v1Deployment API with the now-required selector, rebrandedyacrondemo->cronstabledemo. - Rebranded the ad-hoc example config directory, example tab file, PyInstaller
spec/launcher, and listen socket paths (
/tmp/yacron.sock->/tmp/cronstable.sock) to cronstable.
web.headersoption to control HTTP response headers on all web endpoints, by Gustavo Carneiro (gjcarneiro), commit bde0f0b; merged upstream but never released in yacron 0.19.0.- Python 3.14 compatibility, including
asyncio.new_event_loop()and modern-Python lint/format fixes, by Gustavo J. A. M. Carneiro (gjcarneiro), commit 27a32bc (#100). - Switch from black/flake8 to ruff, plus bandit and pre-commit configuration, by Gustavo Carneiro (gjcarneiro), commits c656fa6 and 4f7936a.
- Removal of Travis CI and modernization of the Python/PyInstaller version matrices, by upstream yacron (gjcarneiro), commits d9b1ca6, 8d28816, 4e6892a, 2941dcf.
- README logging example fix adding
datefmt: '%Y-%m-%d %H:%M:%S'to the custom-logging formatter, by andreas-wittig, commit 931b186.
- Add ability to configure yacron's own logging (#81 #82 #83, gjcarneiro, bdamian)
- Add config value for SMTP(validate_certs=False) (David Batley)
- fixes "Job is always executed immediately on yacron start" (#67)
- add an
enabledoption in jobs (#73) - give a better error message when no configuration file is provided or exists (#72)
- Support Additional Shell Report Vars (RJ Garcia)
- Shell reporter: handle long lines truncatation (Hannes Hergeth)
- exe: undo pyinstaller LD_LIBRARY_PATH changes in subprocesses (#68, Gustavo Carneiro)
- make the capture max line length configurable and change the default from 64K to 16M (#56)
- Add config option to change prefix of subprocess stream lines (#58, eelkeh)
- Fix a bug in the --validate option (#57, Leonid Repin)
- Allow emails to be html formatted
- Fix an error when reading cmd output with huge lines (#56)
- Sentry: increase the size of messages before getting truncated #54
- Sentry: allow specifying the environment option #53
- Minor fixes
- unicode fixes for the exe binary version
- Add ability for one config file to include another one #38
- Add shell command reporting ability (Hannes Hergeth, #50)
- constrain ruamel.yaml to version 0.17.4 or below, later versions are buggy
- blacklist ruamel.yaml version 0.17.5 in requirements #47
- web: don't crash when receiving a web request without Accept header (#45)
- add env_file configuration option (Alessandro Romani, #43)
- email: add missing Date header (#39)
- Add back a self contained binary, this time based on PyInstaller
- Fix email reporting when multiple recipients given
- reporting: add a failure reason line at the top of sentry/email (#36)
- mail: new tls, startls, username, and password options (#21)
- allow jobs to run as a different user (#18)
- Support timezone schedule (#26)
- Minor bugfixes
- HTTP remote interface, allowing to get job status and start jobs on demand
- Simple Linux binary including all dependencies (built using PyOxidizer)
- Build Linux binary inside Docker Ubuntu 16.04, so that it is compatible with older glibc systems
- Build a standalone Linux binary, using PyOxidizer
- Switch from raven to sentry-sdk
- Added an option to just check if the yaml file is valid without running the scheduler.
- Fix missing
bodyin the schema for sentry config
- Fix a bug handling
@rebootin schedule (#22)
- Sentry: add new
extraandleveloptions.
- Added the
utcoption and document that times are utc by default (#17); - If an email body is empty, skip sending it;
- Added docker and k8s example.
- Add custom Sentry fingerprint support
- Ability to send job metrics to statsd (thanks bofm)
alwaysflag to consider any cron job that exits to be failed (thanks evanjardineskinner)maximumRetriescan now be-1to never stop retrying (evanjardineskinner)schedulecan be the string@rebootto always run that cron job on startup (evanjardineskinner)saveLimitcan be set to zero (evanjardineskinner)
- Templating support for reports
- Remove deprecated smtp_host/smtp_port
- Bug fixes
- Bug fixes
- More polished handling of configuration errors;
- Unit tests;
- Bug fixes.
- New option
executionTimeout, to terminate jobs that get stuck; - If a job doesn't terminate gracefully kill it. New option
killTimeoutcontrols how much time to wait for graceful termination before killing it; - Switch parsing to strictyaml, for more user friendly parsing validation error messages.