Skip to content

fix(aziot): remove the identity recovery dead end after a super config outage - #680

Merged
JanZachmann merged 6 commits into
omnect:mainfrom
JanZachmann:jz-2026-08-03-aziot-identity-recovery
Aug 4, 2026
Merged

fix(aziot): remove the identity recovery dead end after a super config outage#680
JanZachmann merged 6 commits into
omnect:mainfrom
JanZachmann:jz-2026-08-03-aziot-identity-recovery

Conversation

@JanZachmann

@JanZachmann JanZachmann commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • disable the trigger limit on the aziot sockets (identityd, keyd, certd, tpmd, edged mgmt+workload): a service skipped by its ConditionPathExists can no longer fail its socket, so the first client connect after the super config is back activates the service
  • aziot-identityd-precondition.service starts aziot-identityd.service (and aziot-edged.service on iotedge distros) from ExecStartPost after a successful config apply, so the after-boot path behaves like the boot path
  • deviceupdate-agent.service retries forever with backoff (5 s growing to 60 s) instead of parking in failed after its start limit

Reason

While /etc/aziot/identityd/config.d/00-super.toml is missing and identityd is not running, one reconnecting client is enough to push aziot-identityd.socket into failed via its trigger limit (20 activations in 2 s, each skipped instantly). From there nothing recovers on its own: a failed socket is never activated again, a later successful config apply writes the config but starts nothing, and the ADU agent - the only client that restarts its unit - exhausts its start limit after ~50 s and is only revived by its timer minutes later. A one-minute config outage costs up to ten minutes of identity and update capability, and the socket stays failed even after the config is back.

Recovery before this change is only a side effect of the ADU agent's retry timer: the timer start pulls identityd up through the agent's Wants=; without the agent, or with its timer disabled, nothing starts identity again. And each timer firing while the config is still missing burns another restart burst and fails again, so recovery is postponed in steps of the 10-minute timer interval.

Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
While the aziot super config is missing, aziot-identityd.service (and
keyd/certd/tpmd) are skipped by their ConditionPathExists. Each skipped
activation counts against the socket's default trigger limit (20 in 2s),
so one reconnecting client fails the socket, and a failed socket is never
activated again - the device keeps no path back to a working identity.

Disable the trigger limit on the aziot sockets. The socket then stays
listening while the service is skipped, and the first client connect after
the config is back activates the service. Rapid re-triggering is throttled
by systemd's poll limit instead of failing the unit.

Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
aziot-edged.service carries the same ConditionPathExists on the aziot
super config as the identity services, so its mgmt and workload sockets
can fail through the trigger limit the same way. Disable the limit so the
sockets stay listening and the first connect after the config is back
activates edged.

Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
When the precondition unit runs after boot (timer, update-validation
flags), a successful 'config apply' only writes the configuration and
nothing starts the identity services. Start them from ExecStartPost so
the after-boot path behaves like the boot path. identityd's connects to
the keyd/certd/tpmd sockets activate those services; edged needs its own
start on iotedge distros.

Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
The agent exits when its startup health check cannot reach identityd,
and ~50 s of identity outage used to park the unit in 'failed' until the
retry timer fired minutes later. Drop the start limit and back the
restart delay off from 5 s to 60 s: the agent recovers within one delay
once identity is back, and each restart pulls identityd up again through
its Wants=. A persistent loop is still flagged by the crash-loop check,
which keys on the auto-restart sub-state.

Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
Working documents (design specs, implementation plans) stay local; the
repo's tracked documentation lives in doc/.

Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves recovery behavior of the Azure IoT (aziot) stack after a “super config” outage by preventing socket activation from getting stuck in a failed state, ensuring key services are (re)started after a successful config apply, and making the ADU agent retry indefinitely with increasing restart delay.

Changes:

  • Disable systemd socket trigger limits on aziot sockets so transient ConditionPathExists failures don’t permanently fail sockets.
  • Start aziot-identityd.service (and aziot-edged.service on iotedge distros) via ExecStartPost after successful config apply.
  • Remove deviceupdate-agent start limiting and add restart backoff parameters.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
recipes-azure-iot/iot-hub-device-update/iot-hub-device-update/deviceupdate-agent.service Disables start limiting and adds restart backoff parameters to keep retrying after outages.
recipes-azure-iot/azure-identityd/aziot-identityd/aziot-identityd-precondition.service Adds post-config apply start of stack units to mimic boot-time bring-up behavior.
recipes-azure-iot/azure-identityd/aziot-identityd.inc Applies trigger-limit disabling to installed aziot sockets; fills new @@STACK_UNITS@@ placeholder.
dynamic-layers/virtualization/recipes-azure-iot/iotedge/aziot-edged/aziot-edged.inc Disables trigger limits on edged mgmt/workload sockets to avoid permanent socket failure.
classes/aziot.bbclass Introduces helper to inject TriggerLimitIntervalSec=0 into socket unit files.
.gitignore Ignores a top-level /docs directory.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (2)

recipes-azure-iot/iot-hub-device-update/iot-hub-device-update/deviceupdate-agent.service:10

  • The new comment mentions a “crash-loop check”, but StartLimitIntervalSec=0 disables systemd start-rate limiting for this unit (and StartLimitBurst was removed). With the limit disabled, the service will not be stopped by systemd’s usual start-limit protection; only the restart backoff delays the loop. Consider updating the comment to accurately describe the behavior so operators aren’t misled when debugging restarts.
# never park in 'failed': each restart also pulls identity up again via
# Wants=, and a persistent loop is caught by the crash-loop check
StartLimitIntervalSec=0

recipes-azure-iot/azure-identityd/aziot-identityd/aziot-identityd-precondition.service:33

  • ExecStartPost uses /bin/systemctl, which may not exist on systems without /usr-merge (systemctl is typically installed under /usr/bin). Because this line is also prefixed with -, a missing binary would silently skip the intended post-config apply start of the stack units. Using the canonical path reduces the chance that recovery acceleration is accidentally disabled.
ExecStartPost=-/bin/systemctl start --no-block @@STACK_UNITS@@

@JanZachmann

Copy link
Copy Markdown
Contributor Author

On the two suppressed review comments:

deviceupdate-agent.service comment: the comment does not claim systemd stops the loop - disabling the start limit is the point of the change ("never park in 'failed'"). The "crash-loop check" is omnect's health check (check_no_crash_loops in the omnect-os tests), which flags a unit in the auto-restart sub-state; the unit sits in that state during every backoff delay, so a persistent loop stays visible without systemd's start-limit protection. Keeping the comment.

/bin/systemctl path: on omnect-os it is the other way around: usrmerge is not in DISTRO_FEATURES, and without it OE installs systemctl at /bin/systemctl - verified in the built rpi4 image rootfs (/bin/systemctl is a regular file, /usr/bin/systemctl does not exist). With usrmerge /bin is a symlink into /usr, so the path resolves there too. /usr/bin/systemctl would break the current images. Keeping the path.

@JanZachmann
JanZachmann merged commit ec7a7d8 into omnect:main Aug 4, 2026
4 checks passed
@JanZachmann

Copy link
Copy Markdown
Contributor Author

FYI @mlilien

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants