chore(release): 0.31.0

[client-software-ci]

0.31.0 (2026-08-10)

Features

• Rust session runtime adapter: sessions can run on the OpenJD v1 Rust runtime as an alternative to the Python runtime. Select it by setting session_runtime in worker.toml to python, rust, or service-selected. (feat: add RustSessionRuntime adapter for OpenJD v1 sessions #1002)
• With service-selected, the session runtime (Python or Rust) is chosen from a runtimeHint provided by the service, defaulting to Python when no hint is given. (feat: add select_runtime() to resolve session runtime mode #1009, feat: consume runtimeHint from UpdateWorkerSchedule responses #1016)
• Runtime selection and failure telemetry events added. To opt out, set opt_out = true under [telemetry] in worker.toml, pass --telemetry-opt-out to the installer, or set the DEADLINE_CLOUD_TELEMETRY_OPT_OUT=true environment variable. (feat: emit runtime selection and failure telemetry events #1021)

Bug Fixes

• Wrap-environment jobs failed on both the Python and Rust runtimes because step_name wasn't forwarded, leaving RFC 0008's WrappedStep.Name unresolved; both runtime paths now forward it. (fix: thread step_name to run_task for wrap-action support #1039, fix: forward step_name on the Rust runtime and add e2e test-rust script #1040)
• Rust runtime panics no longer silently kill the session thread; they are now reported as a failed session with proper cleanup and telemetry. (fix: convert runtime crashes at the adapter boundary and emit failure telemetry #1026)
• Transient network errors (connection closed, connect/read timeout, endpoint connection) are now retried with exponential backoff instead of terminating the agent. (fix: retry transient network errors in UpdateWorkerSchedule #1013)
• Credentials expiring mid-call during hibernate/sleep no longer cause an unrecoverable exit; the agent now detects the time jump and retries with bootstrap credentials. (fix(credentials): Retry with bootstrap creds on hibernate/sleep expiry #1014)

[github-actions]

The 0.31.0 notes don't mention the dependency floor bumps that landed in this release range. Between 290e493 (0.30.2) and ab7b5ec, pyproject.toml changed:

• openjd-model: >= 0.8.1, < 0.11 → >= 0.11.1, < 0.12
• openjd-sessions: 0.10.10 → 0.10.13
• deadline-job-attachments: 0.1.2 → 0.1.3

The openjd-model change raises the minimum by three minor versions and moves the exclusive upper bound past the old ceiling, so the new range does not intersect the old one at all. Any environment that co-installs the worker agent with another package pinned to openjd-model < 0.11 will now fail to resolve on upgrade to 0.31.0. That is a user-visible consequence of upgrading, which is exactly what a release changelog is for — the referenced PRs (#1015, #1040, #1024) carry the rationale but a reader of the changelog has no way to find it.

Suggest adding a line under Features (or a ### Dependencies subsection, matching whatever this repo prefers) noting the openjd-model/openjd-sessions minimums. Worth double-checking whether the non-intersecting range is significant enough for the maintainers to consider it a BREAKING CHANGES entry rather than a plain note.

[github-actions]

The Features section describes runtime selection in terms of internals (select_runtime(), runtimeHint, adapters) but never names the knob a user actually sets. The user-facing surface is the session_runtime setting — worker.toml [worker] session_runtime = "python" | "rust" | "service-selected" and the --session-runtime CLI flag (src/deadline_worker_agent/config/cli_args.py:176, config_file.py:201).

That flag landed in #982, which is an ancestor of the 0.30.2 release commit 290e493, yet the 0.30.2 notes only list #997 and #1005 — so it was never documented in any release. The result is that 0.31.0 ships a fully-wired feature with no changelog mention of how to enable it. select_runtime() is an internal function; a changelog reader has no way to act on it.

Also note the entry for #1009 says the modes are python, rust, or service-selected — correct for the config vocabulary, but the immediately following Bug Fixes entry correctly explains that the wire vocabulary is pythonexpr/rust. Since both appear in the same release notes, it is worth making explicit which of the two a user writes into worker.toml (python, not pythonexpr), otherwise the two bullets read as contradictory.

Suggest recasting the select_runtime() bullet around the setting, e.g. "The session_runtime setting (worker.toml / --session-runtime) selects the session runtime: python (default), rust, or service-selected to defer to the service's per-session hint."

[godobyte]

This PR showed up in Fixes too

[godobyte]

As a followup, maybe we can add steering and/or review github actions rule for PR titles.

[github-actions]

This entry overstates the scope of #1013. The retry handling added in af02f3f is wired into exactly one call site — update_worker_schedule() in src/deadline_worker_agent/aws/deadline/__init__.py:858. Every other Deadline API call in that module (assume_fleet_role_for_worker, create_worker, batch_get_job_entity, update_worker, …) still funnels transport errors into the generic except Exception → DeadlineRequestUnrecoverableError path. As written, an operator would reasonably read this as agent-wide resilience to network blips, which is not what shipped.

Also, the parenthetical list of error types is narrower than the code: the implementation catches the botocore base classes ConnectionError and HTTPClientError (_TRANSIENT_NETWORK_EXCEPTIONS, line 57), which additionally covers SSLError, ProxyConnectionError, and ResponseStreamingError.

Suggested rewording:

• Transient network errors during UpdateWorkerSchedule (connection, timeout, and SSL/proxy transport failures) are now retried with exponential backoff instead of terminating the agent. (fix: retry transient network errors in UpdateWorkerSchedule #1013)

[github-actions]

This entry omits the operator-facing prerequisite, which makes the behaviour sound automatic. Per src/deadline_worker_agent/sessions/runtime/_select.py:41-48, the service runtimeHint is only consulted when the worker is explicitly configured with session_runtime = "service-selected". The default config is "python", and in that mode select_runtime() returns PYTHON unconditionally and ignores the hint entirely. So on a default install this feature is inert — no hint is ever honoured.

Suggest making the opt-in explicit, e.g.:

• When the worker is configured with session_runtime = "service-selected", sessions select their runtime (Python or Rust) from the service-provided runtimeHint, defaulting to Python when no hint is present. Workers keep the default python configuration unless changed. (feat: add select_runtime() to resolve session runtime mode #1009, feat: consume runtimeHint from UpdateWorkerSchedule responses #1016)

Two smaller accuracy notes on the same line:

• The hint wire values are "pythonexpr" and "rust" (_RUNTIME_HINT_MAP, _select.py:19-22), not "python"/"rust". Worth noting since the wire vocabulary intentionally differs from the config strings and operators reading dashboards/logs will see the raw value.
• An unrecognised hint is not treated as "no hint": select_runtime() raises ValueError, and the scheduler fails all of that session's actions (scheduler.py:1185). The "defaulting to Python" clause only covers a genuinely absent hint.

[github-actions]

#1021 (1d9401c) shipped more than telemetry, and the second half is the part operators actually care about: Session() construction is now wrapped in a try/except (ValueError, NotImplementedError, OSError) (scheduler/scheduler.py:1225) so a per-session runtime construction failure fails just that session's actions instead of propagating out and taking down the scheduler thread. Previously an unavailable runtime adapter on one host could kill scheduling for the whole worker.

That is a distinct, user-visible reliability fix and is currently invisible in these release notes. Suggest a separate entry under Bug Fixes:

• A session whose runtime fails to construct (e.g. the selected runtime's adapter is unavailable on the host) now fails only that session's actions instead of terminating the scheduler. (feat: emit runtime selection and failure telemetry events #1021)

[github-actions]

"the agent detects the time jump" overstates what 528cd41 actually does, and the implementation's own comment says so. The retry in aws_credentials/worker_boto3_session.py triggers on any DeadlineRequestUnrecoverableError from AssumeFleetRoleForWorker whenever credentials_object.are_expired() happens to be true afterwards — there is no time-jump detection. Per the in-code note: "are_expired() is a heuristic, not proof of the failure's cause… A genuine AccessDeniedException (e.g. policy change, role deletion) that happens to coincide with expiry will also be retried here rather than surfacing immediately."

That matters for operators debugging a real permissions regression: the failure is now retried with bootstrap credentials before surfacing, so the symptom and the log line they see ("possible hibernate/sleep") can point away from the actual cause. Worth wording so the heuristic is visible, and scoping it to the credential-refresh path:

• Fleet-role credentials expiring mid-call during hibernate/sleep no longer cause an unrecoverable exit — credential refresh retries once with bootstrap credentials when the credentials are found expired after the call fails. (fix(credentials): Retry with bootstrap creds on hibernate/sleep expiry #1014)

[github-actions]

Two accuracy problems with this entry.

1. Scope is agent-wide but the fix is not. 528cd41 touches only WorkerBoto3Session.refresh_credentials() (src/deadline_worker_agent/aws_credentials/worker_boto3_session.py:101-122), i.e. fleet-role credentials. QueueBoto3Session.refresh_credentials() (queue_boto3_session.py:265+) has no equivalent re-check — a hibernate/sleep that expires queue-role credentials mid-AssumeQueueRoleForWorker still propagates DeadlineRequestUnrecoverableError unchanged. As worded, an operator would read this as covering credential refresh generally.

2. "detects the time jump" is not what the code does. There is no clock-delta or monotonic-vs-wall comparison anywhere in the fix. It simply re-calls credentials_object.are_expired() in the except handler and retries if the credentials are now expired and bootstrap was not already in use. The implementation's own comment says so explicitly (line 107): "are_expired() is a heuristic, not proof of the failure's cause… A genuine AccessDeniedException (e.g. policy change, role deletion) that happens to coincide with expiry will also be retried here." The changelog asserts a detection capability the code does not have.

Suggested rewording:

• AssumeFleetRoleForWorker failures that coincide with fleet-role credential expiry (e.g. the host hibernated or slept mid-call) are now retried once with bootstrap credentials instead of causing an unrecoverable exit. (fix(credentials): Retry with bootstrap creds on hibernate/sleep expiry #1014)

[github-actions]

This entry describes a user-facing failure ("Wrap-environment jobs failed") that, per the implementing commit itself, no user could actually hit. 66ecb5c's message states: "No wrap-action job can reach either runtime today, since the service does not yet accept extension requests at job submission." Filing it under Bug Fixes with past-tense "jobs failed" will send operators hunting for a regression they never experienced, and may prompt unnecessary upgrades.

Two further inaccuracies in "both runtime paths now forward it":

• The Python adapter forwards step_name from run_task (sessions/runtime/python.py:90) but deliberately not from _run_task_without_session_env (line 100-109, with an explanatory comment). The attachment-sync path is therefore still unforwarded on the Python runtime.
• The Rust adapter falls back to the literal "Placeholder" when step_name is None (sessions/runtime/rust.py:324), so WrappedStep.Name still resolves to a bogus value for any caller that omits it.

Consider moving this under Features (or a "Preparatory work" note) and stating the forward-looking framing, e.g.:

• step_name is now threaded through Session.run_task to both runtime adapters so RFC 0008's WrappedStep.Name resolves once the service accepts wrap-action extension requests. (fix: thread step_name to run_task for wrap-action support #1039, fix: forward step_name on the Rust runtime and add e2e test-rust script #1040)

[github-actions]

"no longer silently kill the session thread" is broader than what 6080bc9 actually protects. convert_runtime_crashes is applied only to the adapter's instance methods (sessions/runtime/rust.py:258, 287, 301, 336, 406, 414, 423, 428, 433). Two PyO3 boundaries are left undecorated:

• RustSessionRuntime.__init__ (line 206). The OpenJDRustSession(...) construction at line 246 is not wrapped. A panic there escapes as a BaseException, and the only guard at the call site — scheduler.py:1249 — catches (ValueError, NotImplementedError, OSError). So a construction-time panic still propagates out of the scheduler's loop uncaught, which is a strictly worse outcome than the session-thread death this entry claims to have fixed.
• _rust_action_callback (line 243). This runs on a Rust-owned thread. A panic raised there does not traverse any decorated method, so it is neither converted nor attributed.

Either narrow the wording to the boundary that is actually covered, e.g.:

• Rust runtime panics raised from session runtime operations (enter/exit environment, run task, cancel, cleanup, status) are now converted at the adapter boundary and reported as a failed session with cleanup and telemetry, instead of silently killing the session thread. (fix: convert runtime crashes at the adapter boundary and emit failure telemetry #1026)

…or, preferably, close the __init__ gap so the original claim holds.

[github-actions]

This release changes required dependency ranges in an install-breaking way, and none of it is listed. Between 290e493 (the 0.30.2 release commit) and this head, pyproject.toml changed:

-    "openjd-sessions == 0.10.10",
-    "openjd-model >= 0.8.1, < 0.11",
+    "openjd-sessions == 0.10.13",
+    "openjd-model >= 0.11.1, < 0.12",

(plus deadline-job-attachments == 0.1.2 → 0.1.3)

The openjd-model change is the significant one: the new floor (>= 0.11.1) sits above the previous ceiling (< 0.11), so the ranges are disjoint. Any environment that currently pins or resolves openjd-model in the 0.8–0.10 band — a co-installed submitter, plugin, or vendored constraints file — cannot satisfy both and will fail to resolve on upgrade to 0.31.0. openjd-sessions is an exact pin, so it too must move in lockstep.

Given the repo's own note that the openjd-sessions pin exists "due to Host Config Script runner usage of private OpenJD Sessions API", these bounds are load-bearing and worth surfacing to operators rather than leaving them to discover at pip install time. Suggest adding under a Dependencies heading (or BREAKING CHANGES if the disjoint range is considered breaking):

Dependencies

• openjd-model requirement moved to >= 0.11.1, < 0.12 (previously >= 0.8.1, < 0.11) and the openjd-sessions pin to 0.10.13, required by the Rust session runtime and step_name support. Environments pinning openjd-model below 0.11.1 will need to upgrade. deadline-job-attachments pinned to 0.1.3.

[github-actions]

This describes the happy path but omits the failure mode an operator who opts into service-selected is taking on: an unrecognized runtimeHint does not fall back to Python — it fails the session.

select_runtime() treats _RUNTIME_HINT_MAP as a strict allowlist and raises ValueError on any value outside {"pythonexpr", "rust"} (sessions/runtime/_select.py:52-56). The scheduler catches that and calls _fail_all_actions() (scheduler.py:1173-1178), so the first action reports FAILED and the rest NEVER_ATTEMPTED. Only an absent hint defaults to Python.

That matters because the failure is triggered by a service-side value the operator does not control, and the commit's own comment names version skew as the expected cause. An operator reading "defaulting to Python when no hint is given" would reasonably assume service-selected is fail-safe against a service that starts sending a new RuntimeMode value this agent version predates; it is not.

Worth stating explicitly, e.g.:

• With service-selected, the session runtime is chosen from the per-session runtimeHint returned by UpdateWorkerSchedule, defaulting to Python when no hint is present. A hint value this agent version does not recognize fails that session's actions rather than falling back, so keep agents current when using service-selected. (feat: add select_runtime() to resolve session runtime mode #1009, feat: consume runtimeHint from UpdateWorkerSchedule responses #1016)

[github-actions]

The opt-out instructions here are accurate, but the entry says nothing about what the new events transmit — and this release starts sending identifiers that prior telemetry did not.

Both new events carry farm_id, queue_id, session_id, and region in event_details (aws/deadline/__init__.py:938-947 for runtime_selection, and the same set in record_runtime_failure_telemetry_event). runtime_selection fires once per session (scheduler.py:1214), so this is a per-session stream of farm/queue/session identifiers, not an aggregate counter.

For a public release note on an opt-out telemetry system, the collected fields are the part operators need in order to make the opt-out decision the entry is inviting them to make. Note the implementation is visibly careful here — exception text is deliberately replaced with constants specifically to keep filesystem paths and other free text out of telemetry (see the comments at scheduler.py:1264-1270 and 1188-1191) — which is worth the credit, and reinforces that the fields that are sent were a deliberate choice worth documenting.

Suggested addition:

• Runtime selection and failure telemetry events added. Each event records the selected runtime, the reason for the selection, and the associated farm, queue, session, and region identifiers; one selection event is emitted per session. To opt out, set opt_out = true under [telemetry] in worker.toml, pass --telemetry-opt-out to the installer, or set DEADLINE_CLOUD_TELEMETRY_OPT_OUT=true. (feat: emit runtime selection and failure telemetry events #1021)

[github-actions]

Separate point from my dependency comment on this line: this bullet presents the Rust runtime as a drop-in peer of the Python runtime ("as an alternative to"), but the adapter carries documented functional gaps that an operator flipping session_runtime = "rust" in production would want to know about up front.

From sessions/runtime/rust.py:

• Attachment sync runs through a stopgap path. _run_task_without_session_env (line 336) does not delegate to the runtime at all — it materializes embedded files itself, resolves {{ Task.File.* }} by regex, and shells out via run_subprocess. The comment at line 353-357 labels it "STOPGAP" and notes that delegating properly "would also restore the library's cross-platform file permission handling", i.e. permission handling on this path is currently hand-rolled and POSIX-conditional (line 375-381).
• Unsupported OpenJD extensions are silently skipped, not rejected. Line 224-232 logs a warning and drops any extension the Rust crate does not know (WRAP_ACTIONS is named as an example). Since Session.__init__ requests all extensions unconditionally (sessions/session.py:226), the drop is routine, and the failure surfaces later at template-decode time rather than at selection.
• Only OpenJD 2023-09 is supported (_SPEC_REVISIONS, line 62); any other revision raises ValueError at construction.

Suggest tempering the wording so rust reads as opt-in/preview rather than equivalent, e.g. appending: "The Rust runtime supports OpenJD revision 2023-09, silently ignores model extensions the Rust crate does not implement, and uses an interim implementation for the job-attachment sync path; the Python runtime remains the default."

[godobyte]

Not blocking - I remember we used to have Experimental for in-progress code path not active yet.