fix(aziot): remove the identity recovery dead end after a super config outage - #680
Conversation
Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
While the aziot super config is missing, aziot-identityd.service (and keyd/certd/tpmd) are skipped by their ConditionPathExists. Each skipped activation counts against the socket's default trigger limit (20 in 2s), so one reconnecting client fails the socket, and a failed socket is never activated again - the device keeps no path back to a working identity. Disable the trigger limit on the aziot sockets. The socket then stays listening while the service is skipped, and the first client connect after the config is back activates the service. Rapid re-triggering is throttled by systemd's poll limit instead of failing the unit. Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
aziot-edged.service carries the same ConditionPathExists on the aziot super config as the identity services, so its mgmt and workload sockets can fail through the trigger limit the same way. Disable the limit so the sockets stay listening and the first connect after the config is back activates edged. Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
When the precondition unit runs after boot (timer, update-validation flags), a successful 'config apply' only writes the configuration and nothing starts the identity services. Start them from ExecStartPost so the after-boot path behaves like the boot path. identityd's connects to the keyd/certd/tpmd sockets activate those services; edged needs its own start on iotedge distros. Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
The agent exits when its startup health check cannot reach identityd, and ~50 s of identity outage used to park the unit in 'failed' until the retry timer fired minutes later. Drop the start limit and back the restart delay off from 5 s to 60 s: the agent recovers within one delay once identity is back, and each restart pulls identityd up again through its Wants=. A persistent loop is still flagged by the crash-loop check, which keys on the auto-restart sub-state. Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
Working documents (design specs, implementation plans) stay local; the repo's tracked documentation lives in doc/. Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR improves recovery behavior of the Azure IoT (aziot) stack after a “super config” outage by preventing socket activation from getting stuck in a failed state, ensuring key services are (re)started after a successful config apply, and making the ADU agent retry indefinitely with increasing restart delay.
Changes:
- Disable systemd socket trigger limits on aziot sockets so transient
ConditionPathExistsfailures don’t permanently fail sockets. - Start
aziot-identityd.service(andaziot-edged.serviceon iotedge distros) viaExecStartPostafter successfulconfig apply. - Remove
deviceupdate-agentstart limiting and add restart backoff parameters.
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| recipes-azure-iot/iot-hub-device-update/iot-hub-device-update/deviceupdate-agent.service | Disables start limiting and adds restart backoff parameters to keep retrying after outages. |
| recipes-azure-iot/azure-identityd/aziot-identityd/aziot-identityd-precondition.service | Adds post-config apply start of stack units to mimic boot-time bring-up behavior. |
| recipes-azure-iot/azure-identityd/aziot-identityd.inc | Applies trigger-limit disabling to installed aziot sockets; fills new @@STACK_UNITS@@ placeholder. |
| dynamic-layers/virtualization/recipes-azure-iot/iotedge/aziot-edged/aziot-edged.inc | Disables trigger limits on edged mgmt/workload sockets to avoid permanent socket failure. |
| classes/aziot.bbclass | Introduces helper to inject TriggerLimitIntervalSec=0 into socket unit files. |
| .gitignore | Ignores a top-level /docs directory. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 6 changed files in this pull request and generated no new comments.
Suppressed comments (2)
recipes-azure-iot/iot-hub-device-update/iot-hub-device-update/deviceupdate-agent.service:10
- The new comment mentions a “crash-loop check”, but
StartLimitIntervalSec=0disables systemd start-rate limiting for this unit (andStartLimitBurstwas removed). With the limit disabled, the service will not be stopped by systemd’s usual start-limit protection; only the restart backoff delays the loop. Consider updating the comment to accurately describe the behavior so operators aren’t misled when debugging restarts.
# never park in 'failed': each restart also pulls identity up again via
# Wants=, and a persistent loop is caught by the crash-loop check
StartLimitIntervalSec=0
recipes-azure-iot/azure-identityd/aziot-identityd/aziot-identityd-precondition.service:33
ExecStartPostuses/bin/systemctl, which may not exist on systems without/usr-merge (systemctl is typically installed under/usr/bin). Because this line is also prefixed with-, a missing binary would silently skip the intended post-config applystart of the stack units. Using the canonical path reduces the chance that recovery acceleration is accidentally disabled.
ExecStartPost=-/bin/systemctl start --no-block @@STACK_UNITS@@
|
On the two suppressed review comments: deviceupdate-agent.service comment: the comment does not claim systemd stops the loop - disabling the start limit is the point of the change ("never park in 'failed'"). The "crash-loop check" is omnect's health check (
|
|
FYI @mlilien |
Summary
ConditionPathExistscan no longer fail its socket, so the first client connect after the super config is back activates the serviceaziot-identityd-precondition.servicestartsaziot-identityd.service(andaziot-edged.serviceon iotedge distros) fromExecStartPostafter a successfulconfig apply, so the after-boot path behaves like the boot pathdeviceupdate-agent.serviceretries forever with backoff (5 s growing to 60 s) instead of parking infailedafter its start limitReason
While
/etc/aziot/identityd/config.d/00-super.tomlis missing and identityd is not running, one reconnecting client is enough to pushaziot-identityd.socketintofailedvia its trigger limit (20 activations in 2 s, each skipped instantly). From there nothing recovers on its own: a failed socket is never activated again, a later successfulconfig applywrites the config but starts nothing, and the ADU agent - the only client that restarts its unit - exhausts its start limit after ~50 s and is only revived by its timer minutes later. A one-minute config outage costs up to ten minutes of identity and update capability, and the socket stays failed even after the config is back.Recovery before this change is only a side effect of the ADU agent's retry timer: the timer start pulls identityd up through the agent's
Wants=; without the agent, or with its timer disabled, nothing starts identity again. And each timer firing while the config is still missing burns another restart burst and fails again, so recovery is postponed in steps of the 10-minute timer interval.