From 81029f87f80ff27ba8c10e2073a0993a35c68d7a Mon Sep 17 00:00:00 2001 From: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com> Date: Mon, 3 Aug 2026 18:48:48 +0200 Subject: [PATCH 1/6] docs(aziot): add identity recovery design and implementation plan Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com> --- .../2026-08-03-aziot-identity-recovery.md | 412 ++++++++++++++++++ ...26-08-03-aziot-identity-recovery-design.md | 88 ++++ 2 files changed, 500 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-03-aziot-identity-recovery.md create mode 100644 docs/superpowers/specs/2026-08-03-aziot-identity-recovery-design.md diff --git a/docs/superpowers/plans/2026-08-03-aziot-identity-recovery.md b/docs/superpowers/plans/2026-08-03-aziot-identity-recovery.md new file mode 100644 index 00000000..1a891fc6 --- /dev/null +++ b/docs/superpowers/plans/2026-08-03-aziot-identity-recovery.md @@ -0,0 +1,412 @@ +# aziot identity recovery implementation plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Remove the identity dead end: sockets survive skipped activations, a late `config apply` starts the stack, the ADU agent retries forever with backoff. + +**Architecture:** Unit-file changes only, applied through the Yocto recipes in meta-omnect. No source patches. Spec: `docs/superpowers/specs/2026-08-03-aziot-identity-recovery-design.md`. + +**Tech Stack:** BitBake recipes (`.inc`, `.bbclass`), systemd unit files, bash. + +## Global Constraints + +- Branch: `jz-2026-08-03-aziot-identity-recovery` (off `upstream/main`), repo `/home/jzac/projects/meta-omnect`. +- Target systemd is 255.21 (scarthgap); `RestartSteps`/`RestartMaxDelaySec` (need >= 254) are available. +- `systemctl` lives at `/bin/systemctl` on the target (no `usrmerge` in `DISTRO_FEATURES`). +- Every commit message ends with `Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>`. +- Comments in unit files/recipes: short, explain only the why, no PR/issue references, no version numbers. +- There is no unit-test suite in this repo. Each task verifies its sed/edit against a fixture in the scratchpad (`/tmp/claude-1000/-home-jzac-projects-omnect-os/1d9ec82f-888d-46f1-9563-aeaf209297ae/scratchpad`), the final task builds an image and verifies on a device. + +**Out of scope:** the omnect-os test change (drop `repair_identity_units` from `ci/tests/precondition_recovery_test.sh`, raise `RECOVER_DEADLINE`, whitelist check) — separate PR in omnect-os after this is merged. + +--- + +### Task 1: identity sockets survive skipped activations + +**Files:** +- Modify: `classes/aziot.bbclass` (add helper at end of file) +- Modify: `recipes-azure-iot/azure-identityd/aziot-identityd.inc` (4 socket install sites) + +**Interfaces:** +- Produces: shell function `aziot_disable_socket_trigger_limit `, defined in `classes/aziot.bbclass`, usable from `do_install` of every recipe that inherits `aziot`. Task 2 calls it. + +Background: `aziot-identityd.service` (and keyd/certd/tpmd) carry `ConditionPathExists=` on the aziot super config. While the config is missing, every socket activation is skipped instantly and counts against the socket's default trigger limit (20 in 2 s); when hit, the socket fails and is never activated again. `TriggerLimitIntervalSec=0` disables the limit, so the socket stays listening and the first connect after the config is back activates the service. On systemd 255 the poll limit (on by default) throttles rapid re-triggering by pausing polling, so no busy loop. + +- [ ] **Step 1: Add the helper to `classes/aziot.bbclass`** + +Append at the end of the file: + +```sh +# a service skipped by its condition must not fail its socket through the +# trigger limit: without the limit the socket stays listening, and the first +# connect after the condition turns true activates the service +aziot_disable_socket_trigger_limit() { + sed -i 's/^\[Socket\]$/[Socket]\nTriggerLimitIntervalSec=0/' "$1" +} +``` + +- [ ] **Step 2: Call it for the four sockets in `aziot-identityd.inc`** + +After each socket's `fill_placeholders` line, add the helper call. The four sites (line numbers from current `upstream/main`): + +`aziot-identityd.inc:133` (certd): +```sh + fill_placeholders ${D}${systemd_system_unitdir}/aziot-certd.socket + aziot_disable_socket_trigger_limit ${D}${systemd_system_unitdir}/aziot-certd.socket +``` + +`aziot-identityd.inc:149` (identityd): +```sh + fill_placeholders ${D}${systemd_system_unitdir}/aziot-identityd.socket + aziot_disable_socket_trigger_limit ${D}${systemd_system_unitdir}/aziot-identityd.socket +``` + +`aziot-identityd.inc:161` (keyd): +```sh + fill_placeholders ${D}${systemd_system_unitdir}/aziot-keyd.socket + aziot_disable_socket_trigger_limit ${D}${systemd_system_unitdir}/aziot-keyd.socket +``` + +`aziot-identityd.inc:174` (tpmd, inside the `tpm2` branch): +```sh + fill_placeholders ${D}${systemd_system_unitdir}/aziot-tpmd.socket + aziot_disable_socket_trigger_limit ${D}${systemd_system_unitdir}/aziot-tpmd.socket +``` + +- [ ] **Step 3: Verify the sed against a fixture** + +The upstream socket files all have a plain `[Socket]` line (verified at SRCREV `833381a` for identityd/keyd/certd/tpmd). Reproduce one and run the exact sed: + +```bash +SCRATCH=/tmp/claude-1000/-home-jzac-projects-omnect-os/1d9ec82f-888d-46f1-9563-aeaf209297ae/scratchpad +cat > ${SCRATCH}/fixture.socket <<'EOF' +[Unit] +Description=Azure IoT Identity Service API socket +PartOf=aziot-identityd.service + +[Socket] +ListenStream=@socket_dir@/identityd.sock +SocketMode=0660 +DirectoryMode=0755 +SocketUser=@user_aziotid@ +SocketGroup=@user_aziotid@ + +[Install] +WantedBy=sockets.target +EOF +sed -i 's/^\[Socket\]$/[Socket]\nTriggerLimitIntervalSec=0/' ${SCRATCH}/fixture.socket +grep -A1 '^\[Socket\]$' ${SCRATCH}/fixture.socket +``` + +Expected output: +``` +[Socket] +TriggerLimitIntervalSec=0 +``` + +- [ ] **Step 4: Commit** + +```bash +cd /home/jzac/projects/meta-omnect +git add classes/aziot.bbclass recipes-azure-iot/azure-identityd/aziot-identityd.inc +git commit -m "fix(aziot-identityd): keep sockets active while the service is skipped + +While the aziot super config is missing, aziot-identityd.service (and +keyd/certd/tpmd) are skipped by their ConditionPathExists. Each skipped +activation counts against the socket's default trigger limit (20 in 2s), +so one reconnecting client fails the socket, and a failed socket is never +activated again - the device keeps no path back to a working identity. + +Disable the trigger limit on the aziot sockets. The socket then stays +listening while the service is skipped, and the first client connect after +the config is back activates the service. Rapid re-triggering is throttled +by systemd's poll limit instead of failing the unit. + +Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>" +``` + +--- + +### Task 2: edged sockets survive skipped activations + +**Files:** +- Modify: `dynamic-layers/virtualization/recipes-azure-iot/iotedge/aziot-edged/aziot-edged.inc` (2 socket install sites in `do_install`) + +**Interfaces:** +- Consumes: `aziot_disable_socket_trigger_limit ` from `classes/aziot.bbclass` (Task 1). The recipe already inherits `aziot`. + +`aziot-edged.service` carries the same two `ConditionPathExists` lines (added by this recipe's `do_install` sed), so its two sockets have the same dead end. + +- [ ] **Step 1: Add the helper calls** + +In `do_install`, directly after the two socket installs: + +```sh + install -m 0644 ${S}/edgelet/contrib/systemd/debian/aziot-edged.workload.socket ${D}${systemd_system_unitdir}/aziot-edged.workload.socket + install -m 0644 ${S}/edgelet/contrib/systemd/debian/aziot-edged.mgmt.socket ${D}${systemd_system_unitdir}/aziot-edged.mgmt.socket + aziot_disable_socket_trigger_limit ${D}${systemd_system_unitdir}/aziot-edged.workload.socket + aziot_disable_socket_trigger_limit ${D}${systemd_system_unitdir}/aziot-edged.mgmt.socket +``` + +- [ ] **Step 2: Verify the sed against the edged fixture** + +The upstream edged sockets also have a plain `[Socket]` line (verified at SRCREV `306856c`): + +```bash +SCRATCH=/tmp/claude-1000/-home-jzac-projects-omnect-os/1d9ec82f-888d-46f1-9563-aeaf209297ae/scratchpad +cat > ${SCRATCH}/edged-fixture.socket <<'EOF' +[Unit] +Description=Azure IoT Edge daemon management socket +Documentation=man:aziot-edged(8) +PartOf=aziot-edged.service + +[Socket] +ListenStream=/var/run/iotedge/mgmt.sock +SocketMode=0660 +DirectoryMode=0755 +SocketUser=edgeagentuser +SocketGroup=iotedge +Service=aziot-edged.service + +[Install] +WantedBy=sockets.target +EOF +sed -i 's/^\[Socket\]$/[Socket]\nTriggerLimitIntervalSec=0/' ${SCRATCH}/edged-fixture.socket +grep -A1 '^\[Socket\]$' ${SCRATCH}/edged-fixture.socket +``` + +Expected output: +``` +[Socket] +TriggerLimitIntervalSec=0 +``` + +- [ ] **Step 3: Commit** + +```bash +cd /home/jzac/projects/meta-omnect +git add dynamic-layers/virtualization/recipes-azure-iot/iotedge/aziot-edged/aziot-edged.inc +git commit -m "fix(aziot-edged): keep sockets active while the service is skipped + +aziot-edged.service carries the same ConditionPathExists on the aziot +super config as the identity services, so its mgmt and workload sockets +can fail through the trigger limit the same way. Disable the limit so the +sockets stay listening and the first connect after the config is back +activates edged. + +Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>" +``` + +--- + +### Task 3: late `config apply` starts the stack + +**Files:** +- Modify: `recipes-azure-iot/azure-identityd/aziot-identityd/aziot-identityd-precondition.service` +- Modify: `recipes-azure-iot/azure-identityd/aziot-identityd.inc` (the `@@AZIOTCLI@@` substitution block, lines 188-192) + +**Interfaces:** +- Produces: placeholder `@@STACK_UNITS@@` in the precondition unit, substituted by `aziot-identityd.inc`'s `do_install`. Internal to this task. + +Background: on a normal boot the precondition is ordered `Before=` the aziot services and they start in the same transaction. When the unit runs after boot (its timer, the update-validation flags), a successful `config apply` only writes the config - nothing starts the services, and a client connect is the only trigger left. Starting identityd from `ExecStartPost` closes that gap; identityd's connects to the keyd/certd/tpmd sockets activate those services. edged is not pulled by identityd and needs its own start on iotedge distros. + +- [ ] **Step 1: Add `ExecStartPost` to the precondition unit** + +In `aziot-identityd-precondition.service`, `[Service]` section, after the `ExecStart` line: + +```ini +[Service] +Type=oneshot +ExecStart=@@AZIOTCLI@@ config apply +# a run after boot only writes the config; bring the services up like the +# boot transaction would. --no-block is required: this unit is ordered +# Before= the started units, a blocking start would deadlock. On a normal +# boot the request merges with the already queued jobs. +ExecStartPost=-/bin/systemctl start --no-block @@STACK_UNITS@@ +Restart=on-failure +RestartSec=5 +RemainAfterExit=true +``` + +- [ ] **Step 2: Substitute `@@STACK_UNITS@@` in `aziot-identityd.inc`** + +Extend the existing `DISTRO_FEATURES` branch (currently only substituting `@@AZIOTCLI@@`): + +```sh + if ${@bb.utils.contains('DISTRO_FEATURES', 'iotedge', 'true', 'false', d)}; then + sed -i "s/@@AZIOTCLI@@/iotedge/" ${D}${systemd_system_unitdir}/aziot-identityd-precondition.service + sed -i "s/@@STACK_UNITS@@/aziot-identityd.service aziot-edged.service/" ${D}${systemd_system_unitdir}/aziot-identityd-precondition.service + else + sed -i "s/@@AZIOTCLI@@/aziotctl/" ${D}${systemd_system_unitdir}/aziot-identityd-precondition.service + sed -i "s/@@STACK_UNITS@@/aziot-identityd.service/" ${D}${systemd_system_unitdir}/aziot-identityd-precondition.service + fi +``` + +- [ ] **Step 3: Verify both substitution variants** + +```bash +SCRATCH=/tmp/claude-1000/-home-jzac-projects-omnect-os/1d9ec82f-888d-46f1-9563-aeaf209297ae/scratchpad +src=/home/jzac/projects/meta-omnect/recipes-azure-iot/azure-identityd/aziot-identityd/aziot-identityd-precondition.service +sed "s/@@STACK_UNITS@@/aziot-identityd.service aziot-edged.service/" $src | grep ExecStartPost +sed "s/@@STACK_UNITS@@/aziot-identityd.service/" $src | grep ExecStartPost +grep -c "@@" $src +``` + +Expected: +``` +ExecStartPost=-/bin/systemctl start --no-block aziot-identityd.service aziot-edged.service +ExecStartPost=-/bin/systemctl start --no-block aziot-identityd.service +2 +``` +(the `2` = the two remaining placeholders in the source file: `@@AZIOTCLI@@` and `@@STACK_UNITS@@`) + +- [ ] **Step 4: Commit** + +```bash +cd /home/jzac/projects/meta-omnect +git add recipes-azure-iot/azure-identityd/aziot-identityd/aziot-identityd-precondition.service recipes-azure-iot/azure-identityd/aziot-identityd.inc +git commit -m "fix(aziot-identityd): start the identity stack after a late config apply + +When the precondition unit runs after boot (timer, update-validation +flags), a successful 'config apply' only writes the configuration and +nothing starts the identity services. Start them from ExecStartPost so +the after-boot path behaves like the boot path. identityd's connects to +the keyd/certd/tpmd sockets activate those services; edged needs its own +start on iotedge distros. + +Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>" +``` + +--- + +### Task 4: ADU agent retries forever with backoff + +**Files:** +- Modify: `recipes-azure-iot/iot-hub-device-update/iot-hub-device-update/deviceupdate-agent.service` + +**Interfaces:** none (self-contained unit change). + +Background: the agent exits with code 1 when its startup health check cannot reach identityd. With `StartLimitBurst=10`/`StartLimitIntervalSec=120` an identity outage of ~50 s parks the unit in `failed`, and only `deviceupdate-agent.timer` (minutes) starts it again. Retry forever instead: each auto-restart also re-queues `Wants=aziot-identityd.service`, so the agent is itself a recovery path for identity. A persistent loop stays visible: the crash-loop check keys on the `activating (auto-restart)` sub-state, which the unit occupies during every backoff delay. + +- [ ] **Step 1: Edit the unit** + +In the `[Unit]` section, replace + +```ini +StartLimitBurst=10 +StartLimitIntervalSec=120 +``` + +with + +```ini +# never park in 'failed': each restart also pulls identity up again via +# Wants=, and a persistent loop is caught by the crash-loop check +StartLimitIntervalSec=0 +``` + +In the `[Service]` section, replace + +```ini +Restart=always +RestartSec=5 +``` + +with + +```ini +Restart=always +RestartSec=5 +RestartSteps=5 +RestartMaxDelaySec=60 +``` + +- [ ] **Step 2: Verify the resulting unit** + +```bash +grep -E "StartLimit|Restart" /home/jzac/projects/meta-omnect/recipes-azure-iot/iot-hub-device-update/iot-hub-device-update/deviceupdate-agent.service +``` + +Expected (exactly these, no `StartLimitBurst` left): +``` +StartLimitIntervalSec=0 +Restart=always +RestartSec=5 +RestartSteps=5 +RestartMaxDelaySec=60 +``` + +- [ ] **Step 3: Commit** + +```bash +cd /home/jzac/projects/meta-omnect +git add recipes-azure-iot/iot-hub-device-update/iot-hub-device-update/deviceupdate-agent.service +git commit -m "fix(adu): retry forever with backoff instead of reaching the start limit + +The agent exits when its startup health check cannot reach identityd, +and ~50 s of identity outage used to park the unit in 'failed' until the +retry timer fired minutes later. Drop the start limit and back the +restart delay off from 5 s to 60 s: the agent recovers within one delay +once identity is back, and each restart pulls identityd up again through +its Wants=. A persistent loop is still flagged by the crash-loop check, +which keys on the auto-restart sub-state. + +Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>" +``` + +--- + +### Task 5: build and device verification + +**Files:** none (verification only, no commit). + +- [ ] **Step 1: Build a gateway-devel image with the branch** + +Make sure `local.env` in `/home/jzac/projects/omnect-os` points kas at the local meta-omnect checkout (branch `jz-2026-08-03-aziot-identity-recovery`), then: + +```bash +cd /home/jzac/projects/omnect-os +./dobi.sh build-omnect-gateway-devel-rpi4-64 +``` + +- [ ] **Step 2: Inspect the generated units in the build tree** + +```bash +cd /home/jzac/projects/omnect-os +find build/build/tmp*/work -path "*aziot*" -name "*.socket" -exec grep -l TriggerLimitIntervalSec {} \; +find build/build/tmp*/work -name "aziot-identityd-precondition.service" -path "*image*" -exec grep ExecStartPost {} \; +find build/build/tmp*/work -name "deviceupdate-agent.service" -path "*image*" -exec grep -E "StartLimit|RestartSteps" {} \; +``` + +Expected: all aziot sockets contain `TriggerLimitIntervalSec=0`; the precondition unit contains the substituted `ExecStartPost` (no `@@` left); the agent unit has `StartLimitIntervalSec=0`, `RestartSteps=5`, `RestartMaxDelaySec=60`. + +- [ ] **Step 3: Manual repro on an rpi4 (gateway-devel)** + +Flash the image, then run the repro from the issue doc: + +```bash +sudo systemctl stop aziot-identityd.service +sudo rm /etc/aziot/identityd/config.d/00-super.toml +sudo systemctl restart deviceupdate-agent.service +sleep 60 +journalctl -b0 | grep -c 'Trigger limit' # expected: 0 +systemctl is-active aziot-identityd.socket # expected: active +systemctl show -p ActiveState,SubState,NRestarts deviceupdate-agent.service +# expected: activating / auto-restart, NRestarts growing, never 'failed' +``` + +Then restore and watch recovery: + +```bash +sudo systemctl start aziot-identityd-precondition.service # regenerates the super config via config apply +systemctl is-active aziot-identityd.service # expected: active right after apply (fix b) +sleep 90 +systemctl is-active deviceupdate-agent.service # expected: active within RestartMaxDelaySec +systemctl is-system-running # expected: running +``` + +Also confirm the load-bearing assumption of fix (a): while the super config is missing, `journalctl -f` must not show a skip-loop storm faster than the poll limit allows, and systemd's CPU usage stays normal. + +- [ ] **Step 4: Report results** + +No commit. Record the device observations in the PR conversation (not the description). diff --git a/docs/superpowers/specs/2026-08-03-aziot-identity-recovery-design.md b/docs/superpowers/specs/2026-08-03-aziot-identity-recovery-design.md new file mode 100644 index 00000000..7d604a29 --- /dev/null +++ b/docs/superpowers/specs/2026-08-03-aziot-identity-recovery-design.md @@ -0,0 +1,88 @@ +# aziot identity recovery design + +Fixes the dead end described in `aziot-identity-recovery-issue.md`: a missing +`/etc/aziot/identityd/config.d/00-super.toml` fails `aziot-identityd.socket` +via its trigger limit, a late successful `config apply` starts nothing, and +the ADU agent exhausts its start limit — the device stays `degraded` with no +identity until `deviceupdate-agent.timer` fires. + +Scope: unit-file changes in meta-omnect only, no source patches. The +companion test change (remove the transitional `repair_identity_units` step +from `ci/tests/precondition_recovery_test.sh`) is a separate omnect-os PR. + +Each of the three fixes removes the dead end on its own; together they make +recovery fast and independent of which client happens to run. + +## 1. Sockets never fail from skipped activations + +Add `TriggerLimitIntervalSec=0` under `[Socket]` for: + +- `aziot-identityd.socket`, `aziot-keyd.socket`, `aziot-certd.socket`, + `aziot-tpmd.socket` — in `aziot-identityd.inc`, same sed style as the + service edits +- `aziot-edged.mgmt.socket`, `aziot-edged.workload.socket` — in + `aziot-edged.inc`; `aziot-edged.service` carries the same + `ConditionPathExists` lines, so its sockets share the dead end + +While the service is skipped by its condition the socket stays listening; the +first client connect after the super config is back activates the service. + +Load-bearing assumption, verify on a device: a pending connection to a +skipped service must not busy-loop systemd. On systemd 255 the poll limit +(`PollLimitIntervalSec`, active by default) pauses polling instead of failing +the unit. meta-omnect main is scarthgap-only (systemd 255.21). + +## 2. Late `config apply` starts the stack + +`aziot-identityd-precondition.service` gets + + ExecStartPost=-/bin/systemctl start --no-block aziot-identityd.service + +Starting identityd is enough for the rest of the identity stack: identityd's +connects to the keyd/certd/tpmd sockets activate those services. On +iotedge distros the recipe's existing `DISTRO_FEATURES` branch (the +`@@AZIOTCLI@@` substitution) adds `aziot-edged.service` to the started units; +edged needs its own start. + +- `--no-block` is mandatory: the precondition is ordered `Before=` these + units, a blocking start inside `ExecStartPost` deadlocks. +- On a normal boot the start merges with the already-queued jobs (no-op). +- The `-` prefix keeps a failed `systemctl` call from failing an otherwise + successful apply. + +## 3. ADU agent retries forever with backoff + +`deviceupdate-agent.service`: drop `StartLimitBurst`, set +`StartLimitIntervalSec=0`, keep `Restart=always`, add `RestartSteps=5` and +`RestartMaxDelaySec=60` on top of `RestartSec=5` (delay grows from 5 s to +60 s). Needs systemd >= 254; scarthgap has 255. + +- Each auto-restart re-queues `Wants=aziot-identityd.service`, so the agent + is a recovery path for identity too. +- The unit never reaches `failed`; a persistent loop is visible through the + crash-loop checker. Planning-time check: does the checker's threshold still + flag a loop that has slowed to one restart per 60 s? +- `deviceupdate-agent.timer` stays unchanged — its job is the + skipped-at-boot case (condition blocked the start, `Restart=` never + engaged), not start-limit rescue. + +## Verification + +- Manual repro from the issue doc on an rpi4: socket survives the + config-less window, identityd comes up right after a successful apply, the + agent within `RestartMaxDelaySec`. +- omnect-os follow-up: `precondition_recovery_test.sh` loses + `repair_identity_units` and its TODO; `RECOVER_DEADLINE` grows to cover one + `RestartMaxDelaySec` (~90 s). +- Whitelist check: the failure window now produces repeated "skipped because + of an unmet condition" lines instead of a one-time trigger-limit failure; + base_test's syslog check may need a whitelist entry. + +## Commits + +One commit per fix, the socket fix split by recipe: + +1. `fix(aziot-identityd): keep sockets active while the service is skipped` +2. `fix(aziot-edged): keep sockets active while the service is skipped` +3. `fix(aziot-identityd): start the identity stack after a late config apply` +4. `fix(adu): retry forever with backoff instead of reaching the start limit` From 5c7b665f7f83920dde4aa979d790a52b8784665e Mon Sep 17 00:00:00 2001 From: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com> Date: Mon, 3 Aug 2026 19:23:41 +0200 Subject: [PATCH 2/6] fix(aziot-identityd): keep sockets active while the service is skipped While the aziot super config is missing, aziot-identityd.service (and keyd/certd/tpmd) are skipped by their ConditionPathExists. Each skipped activation counts against the socket's default trigger limit (20 in 2s), so one reconnecting client fails the socket, and a failed socket is never activated again - the device keeps no path back to a working identity. Disable the trigger limit on the aziot sockets. The socket then stays listening while the service is skipped, and the first client connect after the config is back activates the service. Rapid re-triggering is throttled by systemd's poll limit instead of failing the unit. Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com> --- classes/aziot.bbclass | 7 +++++++ recipes-azure-iot/azure-identityd/aziot-identityd.inc | 4 ++++ 2 files changed, 11 insertions(+) diff --git a/classes/aziot.bbclass b/classes/aziot.bbclass index 085f2e13..8efabf2c 100644 --- a/classes/aziot.bbclass +++ b/classes/aziot.bbclass @@ -29,3 +29,10 @@ do_install:prepend() { install -d -m 0750 -g aziotks ${D}${sysconfdir}/aziot/keyd install -d -m 0700 -o aziotks -g aziotks ${D}${sysconfdir}/aziot/keyd/config.d } + +# a service skipped by its condition must not fail its socket through the +# trigger limit: without the limit the socket stays listening, and the first +# connect after the condition turns true activates the service +aziot_disable_socket_trigger_limit() { + sed -i 's/^\[Socket\]$/[Socket]\nTriggerLimitIntervalSec=0/' "$1" +} diff --git a/recipes-azure-iot/azure-identityd/aziot-identityd.inc b/recipes-azure-iot/azure-identityd/aziot-identityd.inc index 93869f47..66b6be41 100644 --- a/recipes-azure-iot/azure-identityd/aziot-identityd.inc +++ b/recipes-azure-iot/azure-identityd/aziot-identityd.inc @@ -131,6 +131,7 @@ do_install() { install -m 0644 ${S}/cert/aziot-certd/aziot-certd.socket.in ${D}${systemd_system_unitdir}/aziot-certd.socket fill_placeholders ${D}${systemd_system_unitdir}/aziot-certd.service fill_placeholders ${D}${systemd_system_unitdir}/aziot-certd.socket + aziot_disable_socket_trigger_limit ${D}${systemd_system_unitdir}/aziot-certd.socket # enable identity service to create cert "device-id" (e.g. for x509 dps provisioning) install -m 0600 -o aziotcs -g aziotcs ${WORKDIR}/iot-identity-service-certd.template.toml ${D}${sysconfdir}/aziot/certd/config.d/aziotid.toml @@ -147,6 +148,7 @@ do_install() { install -m 0644 ${S}/identity/aziot-identityd/aziot-identityd.socket.in ${D}${systemd_system_unitdir}/aziot-identityd.socket fill_placeholders ${D}${systemd_system_unitdir}/aziot-identityd.service fill_placeholders ${D}${systemd_system_unitdir}/aziot-identityd.socket + aziot_disable_socket_trigger_limit ${D}${systemd_system_unitdir}/aziot-identityd.socket install -m 0644 ${S}/key/aziot-keyd/aziot-keyd.service.in ${D}${systemd_system_unitdir}/aziot-keyd.service sed -i \ @@ -159,6 +161,7 @@ do_install() { install -m 0644 ${S}/key/aziot-keyd/aziot-keyd.socket.in ${D}${systemd_system_unitdir}/aziot-keyd.socket fill_placeholders ${D}${systemd_system_unitdir}/aziot-keyd.service fill_placeholders ${D}${systemd_system_unitdir}/aziot-keyd.socket + aziot_disable_socket_trigger_limit ${D}${systemd_system_unitdir}/aziot-keyd.socket if ${@bb.utils.contains('MACHINE_FEATURES', 'tpm2', 'true', 'false', d)}; then install -m 0644 ${S}/tpm/aziot-tpmd/aziot-tpmd.service.in ${D}${systemd_system_unitdir}/aziot-tpmd.service @@ -172,6 +175,7 @@ do_install() { install -m 0644 ${S}/tpm/aziot-tpmd/aziot-tpmd.socket.in ${D}${systemd_system_unitdir}/aziot-tpmd.socket fill_placeholders ${D}${systemd_system_unitdir}/aziot-tpmd.service fill_placeholders ${D}${systemd_system_unitdir}/aziot-tpmd.socket + aziot_disable_socket_trigger_limit ${D}${systemd_system_unitdir}/aziot-tpmd.socket fi # libaziot-key-openssl-engine-shared From e972cf668f928821deaefaa927726b8f56f755cf Mon Sep 17 00:00:00 2001 From: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com> Date: Mon, 3 Aug 2026 19:25:43 +0200 Subject: [PATCH 3/6] fix(aziot-edged): keep sockets active while the service is skipped aziot-edged.service carries the same ConditionPathExists on the aziot super config as the identity services, so its mgmt and workload sockets can fail through the trigger limit the same way. Disable the limit so the sockets stay listening and the first connect after the config is back activates edged. Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com> --- .../recipes-azure-iot/iotedge/aziot-edged/aziot-edged.inc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/dynamic-layers/virtualization/recipes-azure-iot/iotedge/aziot-edged/aziot-edged.inc b/dynamic-layers/virtualization/recipes-azure-iot/iotedge/aziot-edged/aziot-edged.inc index 661bfaf0..a756b22a 100644 --- a/dynamic-layers/virtualization/recipes-azure-iot/iotedge/aziot-edged/aziot-edged.inc +++ b/dynamic-layers/virtualization/recipes-azure-iot/iotedge/aziot-edged/aziot-edged.inc @@ -60,6 +60,8 @@ do_install () { install -m 0644 ${S}/edgelet/contrib/systemd/debian/aziot-edged.workload.socket ${D}${systemd_system_unitdir}/aziot-edged.workload.socket install -m 0644 ${S}/edgelet/contrib/systemd/debian/aziot-edged.mgmt.socket ${D}${systemd_system_unitdir}/aziot-edged.mgmt.socket + aziot_disable_socket_trigger_limit ${D}${systemd_system_unitdir}/aziot-edged.workload.socket + aziot_disable_socket_trigger_limit ${D}${systemd_system_unitdir}/aziot-edged.mgmt.socket } USERADD_PARAM:${PN} += " \ From da7f2b9a5cbe9d67cde13614ddb135669b6d10eb Mon Sep 17 00:00:00 2001 From: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com> Date: Mon, 3 Aug 2026 19:27:46 +0200 Subject: [PATCH 4/6] fix(aziot-identityd): start the identity stack after a late config apply When the precondition unit runs after boot (timer, update-validation flags), a successful 'config apply' only writes the configuration and nothing starts the identity services. Start them from ExecStartPost so the after-boot path behaves like the boot path. identityd's connects to the keyd/certd/tpmd sockets activate those services; edged needs its own start on iotedge distros. Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com> --- recipes-azure-iot/azure-identityd/aziot-identityd.inc | 2 ++ .../aziot-identityd/aziot-identityd-precondition.service | 5 +++++ 2 files changed, 7 insertions(+) diff --git a/recipes-azure-iot/azure-identityd/aziot-identityd.inc b/recipes-azure-iot/azure-identityd/aziot-identityd.inc index 66b6be41..082bc43b 100644 --- a/recipes-azure-iot/azure-identityd/aziot-identityd.inc +++ b/recipes-azure-iot/azure-identityd/aziot-identityd.inc @@ -191,8 +191,10 @@ do_install() { install -m 0644 ${WORKDIR}/aziot-identityd-precondition.service ${D}${systemd_system_unitdir}/aziot-identityd-precondition.service if ${@bb.utils.contains('DISTRO_FEATURES', 'iotedge', 'true', 'false', d)}; then sed -i "s/@@AZIOTCLI@@/iotedge/" ${D}${systemd_system_unitdir}/aziot-identityd-precondition.service + sed -i "s/@@STACK_UNITS@@/aziot-identityd.service aziot-edged.service/" ${D}${systemd_system_unitdir}/aziot-identityd-precondition.service else sed -i "s/@@AZIOTCLI@@/aziotctl/" ${D}${systemd_system_unitdir}/aziot-identityd-precondition.service + sed -i "s/@@STACK_UNITS@@/aziot-identityd.service/" ${D}${systemd_system_unitdir}/aziot-identityd-precondition.service fi install -m 0644 ${WORKDIR}/aziot-identityd-precondition.timer ${D}${systemd_system_unitdir}/aziot-identityd-precondition.timer } diff --git a/recipes-azure-iot/azure-identityd/aziot-identityd/aziot-identityd-precondition.service b/recipes-azure-iot/azure-identityd/aziot-identityd/aziot-identityd-precondition.service index 71f33e96..39e48d78 100644 --- a/recipes-azure-iot/azure-identityd/aziot-identityd/aziot-identityd-precondition.service +++ b/recipes-azure-iot/azure-identityd/aziot-identityd/aziot-identityd-precondition.service @@ -26,6 +26,11 @@ StartLimitIntervalSec=120 [Service] Type=oneshot ExecStart=@@AZIOTCLI@@ config apply +# a run after boot only writes the config; bring the services up like the +# boot transaction would. --no-block is required: this unit is ordered +# Before= the started units, a blocking start would deadlock. On a normal +# boot the request merges with the already queued jobs. +ExecStartPost=-/bin/systemctl start --no-block @@STACK_UNITS@@ Restart=on-failure RestartSec=5 RemainAfterExit=true From 429110b65fc13d45d3544b6408eef2d50371b38c Mon Sep 17 00:00:00 2001 From: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com> Date: Mon, 3 Aug 2026 19:30:02 +0200 Subject: [PATCH 5/6] fix(adu): retry forever with backoff instead of reaching the start limit The agent exits when its startup health check cannot reach identityd, and ~50 s of identity outage used to park the unit in 'failed' until the retry timer fired minutes later. Drop the start limit and back the restart delay off from 5 s to 60 s: the agent recovers within one delay once identity is back, and each restart pulls identityd up again through its Wants=. A persistent loop is still flagged by the crash-loop check, which keys on the auto-restart sub-state. Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com> --- .../iot-hub-device-update/deviceupdate-agent.service | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/recipes-azure-iot/iot-hub-device-update/iot-hub-device-update/deviceupdate-agent.service b/recipes-azure-iot/iot-hub-device-update/iot-hub-device-update/deviceupdate-agent.service index 2664e7ec..626a248d 100644 --- a/recipes-azure-iot/iot-hub-device-update/iot-hub-device-update/deviceupdate-agent.service +++ b/recipes-azure-iot/iot-hub-device-update/iot-hub-device-update/deviceupdate-agent.service @@ -5,14 +5,17 @@ Wants=network-online.target aziot-identityd.service ConditionPathExists=!/run/omnect-device-service/omnect_validate_update -StartLimitBurst=10 -StartLimitIntervalSec=120 +# never park in 'failed': each restart also pulls identity up again via +# Wants=, and a persistent loop is caught by the crash-loop check +StartLimitIntervalSec=0 [Service] Type=notify Restart=always RestartSec=5 +RestartSteps=5 +RestartMaxDelaySec=60 User=adu Group=adu From 134d62e1d4aa2dbfda12f300df7384f26fd13488 Mon Sep 17 00:00:00 2001 From: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com> Date: Mon, 3 Aug 2026 20:07:06 +0200 Subject: [PATCH 6/6] chore: ignore the local docs directory Working documents (design specs, implementation plans) stay local; the repo's tracked documentation lives in doc/. Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com> --- .gitignore | 1 + .../2026-08-03-aziot-identity-recovery.md | 412 ------------------ ...26-08-03-aziot-identity-recovery-design.md | 88 ---- 3 files changed, 1 insertion(+), 500 deletions(-) delete mode 100644 docs/superpowers/plans/2026-08-03-aziot-identity-recovery.md delete mode 100644 docs/superpowers/specs/2026-08-03-aziot-identity-recovery-design.md diff --git a/.gitignore b/.gitignore index d5768ae3..51824afb 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ /tmp *~ *.sw? +/docs diff --git a/docs/superpowers/plans/2026-08-03-aziot-identity-recovery.md b/docs/superpowers/plans/2026-08-03-aziot-identity-recovery.md deleted file mode 100644 index 1a891fc6..00000000 --- a/docs/superpowers/plans/2026-08-03-aziot-identity-recovery.md +++ /dev/null @@ -1,412 +0,0 @@ -# aziot identity recovery implementation plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Remove the identity dead end: sockets survive skipped activations, a late `config apply` starts the stack, the ADU agent retries forever with backoff. - -**Architecture:** Unit-file changes only, applied through the Yocto recipes in meta-omnect. No source patches. Spec: `docs/superpowers/specs/2026-08-03-aziot-identity-recovery-design.md`. - -**Tech Stack:** BitBake recipes (`.inc`, `.bbclass`), systemd unit files, bash. - -## Global Constraints - -- Branch: `jz-2026-08-03-aziot-identity-recovery` (off `upstream/main`), repo `/home/jzac/projects/meta-omnect`. -- Target systemd is 255.21 (scarthgap); `RestartSteps`/`RestartMaxDelaySec` (need >= 254) are available. -- `systemctl` lives at `/bin/systemctl` on the target (no `usrmerge` in `DISTRO_FEATURES`). -- Every commit message ends with `Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>`. -- Comments in unit files/recipes: short, explain only the why, no PR/issue references, no version numbers. -- There is no unit-test suite in this repo. Each task verifies its sed/edit against a fixture in the scratchpad (`/tmp/claude-1000/-home-jzac-projects-omnect-os/1d9ec82f-888d-46f1-9563-aeaf209297ae/scratchpad`), the final task builds an image and verifies on a device. - -**Out of scope:** the omnect-os test change (drop `repair_identity_units` from `ci/tests/precondition_recovery_test.sh`, raise `RECOVER_DEADLINE`, whitelist check) — separate PR in omnect-os after this is merged. - ---- - -### Task 1: identity sockets survive skipped activations - -**Files:** -- Modify: `classes/aziot.bbclass` (add helper at end of file) -- Modify: `recipes-azure-iot/azure-identityd/aziot-identityd.inc` (4 socket install sites) - -**Interfaces:** -- Produces: shell function `aziot_disable_socket_trigger_limit `, defined in `classes/aziot.bbclass`, usable from `do_install` of every recipe that inherits `aziot`. Task 2 calls it. - -Background: `aziot-identityd.service` (and keyd/certd/tpmd) carry `ConditionPathExists=` on the aziot super config. While the config is missing, every socket activation is skipped instantly and counts against the socket's default trigger limit (20 in 2 s); when hit, the socket fails and is never activated again. `TriggerLimitIntervalSec=0` disables the limit, so the socket stays listening and the first connect after the config is back activates the service. On systemd 255 the poll limit (on by default) throttles rapid re-triggering by pausing polling, so no busy loop. - -- [ ] **Step 1: Add the helper to `classes/aziot.bbclass`** - -Append at the end of the file: - -```sh -# a service skipped by its condition must not fail its socket through the -# trigger limit: without the limit the socket stays listening, and the first -# connect after the condition turns true activates the service -aziot_disable_socket_trigger_limit() { - sed -i 's/^\[Socket\]$/[Socket]\nTriggerLimitIntervalSec=0/' "$1" -} -``` - -- [ ] **Step 2: Call it for the four sockets in `aziot-identityd.inc`** - -After each socket's `fill_placeholders` line, add the helper call. The four sites (line numbers from current `upstream/main`): - -`aziot-identityd.inc:133` (certd): -```sh - fill_placeholders ${D}${systemd_system_unitdir}/aziot-certd.socket - aziot_disable_socket_trigger_limit ${D}${systemd_system_unitdir}/aziot-certd.socket -``` - -`aziot-identityd.inc:149` (identityd): -```sh - fill_placeholders ${D}${systemd_system_unitdir}/aziot-identityd.socket - aziot_disable_socket_trigger_limit ${D}${systemd_system_unitdir}/aziot-identityd.socket -``` - -`aziot-identityd.inc:161` (keyd): -```sh - fill_placeholders ${D}${systemd_system_unitdir}/aziot-keyd.socket - aziot_disable_socket_trigger_limit ${D}${systemd_system_unitdir}/aziot-keyd.socket -``` - -`aziot-identityd.inc:174` (tpmd, inside the `tpm2` branch): -```sh - fill_placeholders ${D}${systemd_system_unitdir}/aziot-tpmd.socket - aziot_disable_socket_trigger_limit ${D}${systemd_system_unitdir}/aziot-tpmd.socket -``` - -- [ ] **Step 3: Verify the sed against a fixture** - -The upstream socket files all have a plain `[Socket]` line (verified at SRCREV `833381a` for identityd/keyd/certd/tpmd). Reproduce one and run the exact sed: - -```bash -SCRATCH=/tmp/claude-1000/-home-jzac-projects-omnect-os/1d9ec82f-888d-46f1-9563-aeaf209297ae/scratchpad -cat > ${SCRATCH}/fixture.socket <<'EOF' -[Unit] -Description=Azure IoT Identity Service API socket -PartOf=aziot-identityd.service - -[Socket] -ListenStream=@socket_dir@/identityd.sock -SocketMode=0660 -DirectoryMode=0755 -SocketUser=@user_aziotid@ -SocketGroup=@user_aziotid@ - -[Install] -WantedBy=sockets.target -EOF -sed -i 's/^\[Socket\]$/[Socket]\nTriggerLimitIntervalSec=0/' ${SCRATCH}/fixture.socket -grep -A1 '^\[Socket\]$' ${SCRATCH}/fixture.socket -``` - -Expected output: -``` -[Socket] -TriggerLimitIntervalSec=0 -``` - -- [ ] **Step 4: Commit** - -```bash -cd /home/jzac/projects/meta-omnect -git add classes/aziot.bbclass recipes-azure-iot/azure-identityd/aziot-identityd.inc -git commit -m "fix(aziot-identityd): keep sockets active while the service is skipped - -While the aziot super config is missing, aziot-identityd.service (and -keyd/certd/tpmd) are skipped by their ConditionPathExists. Each skipped -activation counts against the socket's default trigger limit (20 in 2s), -so one reconnecting client fails the socket, and a failed socket is never -activated again - the device keeps no path back to a working identity. - -Disable the trigger limit on the aziot sockets. The socket then stays -listening while the service is skipped, and the first client connect after -the config is back activates the service. Rapid re-triggering is throttled -by systemd's poll limit instead of failing the unit. - -Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>" -``` - ---- - -### Task 2: edged sockets survive skipped activations - -**Files:** -- Modify: `dynamic-layers/virtualization/recipes-azure-iot/iotedge/aziot-edged/aziot-edged.inc` (2 socket install sites in `do_install`) - -**Interfaces:** -- Consumes: `aziot_disable_socket_trigger_limit ` from `classes/aziot.bbclass` (Task 1). The recipe already inherits `aziot`. - -`aziot-edged.service` carries the same two `ConditionPathExists` lines (added by this recipe's `do_install` sed), so its two sockets have the same dead end. - -- [ ] **Step 1: Add the helper calls** - -In `do_install`, directly after the two socket installs: - -```sh - install -m 0644 ${S}/edgelet/contrib/systemd/debian/aziot-edged.workload.socket ${D}${systemd_system_unitdir}/aziot-edged.workload.socket - install -m 0644 ${S}/edgelet/contrib/systemd/debian/aziot-edged.mgmt.socket ${D}${systemd_system_unitdir}/aziot-edged.mgmt.socket - aziot_disable_socket_trigger_limit ${D}${systemd_system_unitdir}/aziot-edged.workload.socket - aziot_disable_socket_trigger_limit ${D}${systemd_system_unitdir}/aziot-edged.mgmt.socket -``` - -- [ ] **Step 2: Verify the sed against the edged fixture** - -The upstream edged sockets also have a plain `[Socket]` line (verified at SRCREV `306856c`): - -```bash -SCRATCH=/tmp/claude-1000/-home-jzac-projects-omnect-os/1d9ec82f-888d-46f1-9563-aeaf209297ae/scratchpad -cat > ${SCRATCH}/edged-fixture.socket <<'EOF' -[Unit] -Description=Azure IoT Edge daemon management socket -Documentation=man:aziot-edged(8) -PartOf=aziot-edged.service - -[Socket] -ListenStream=/var/run/iotedge/mgmt.sock -SocketMode=0660 -DirectoryMode=0755 -SocketUser=edgeagentuser -SocketGroup=iotedge -Service=aziot-edged.service - -[Install] -WantedBy=sockets.target -EOF -sed -i 's/^\[Socket\]$/[Socket]\nTriggerLimitIntervalSec=0/' ${SCRATCH}/edged-fixture.socket -grep -A1 '^\[Socket\]$' ${SCRATCH}/edged-fixture.socket -``` - -Expected output: -``` -[Socket] -TriggerLimitIntervalSec=0 -``` - -- [ ] **Step 3: Commit** - -```bash -cd /home/jzac/projects/meta-omnect -git add dynamic-layers/virtualization/recipes-azure-iot/iotedge/aziot-edged/aziot-edged.inc -git commit -m "fix(aziot-edged): keep sockets active while the service is skipped - -aziot-edged.service carries the same ConditionPathExists on the aziot -super config as the identity services, so its mgmt and workload sockets -can fail through the trigger limit the same way. Disable the limit so the -sockets stay listening and the first connect after the config is back -activates edged. - -Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>" -``` - ---- - -### Task 3: late `config apply` starts the stack - -**Files:** -- Modify: `recipes-azure-iot/azure-identityd/aziot-identityd/aziot-identityd-precondition.service` -- Modify: `recipes-azure-iot/azure-identityd/aziot-identityd.inc` (the `@@AZIOTCLI@@` substitution block, lines 188-192) - -**Interfaces:** -- Produces: placeholder `@@STACK_UNITS@@` in the precondition unit, substituted by `aziot-identityd.inc`'s `do_install`. Internal to this task. - -Background: on a normal boot the precondition is ordered `Before=` the aziot services and they start in the same transaction. When the unit runs after boot (its timer, the update-validation flags), a successful `config apply` only writes the config - nothing starts the services, and a client connect is the only trigger left. Starting identityd from `ExecStartPost` closes that gap; identityd's connects to the keyd/certd/tpmd sockets activate those services. edged is not pulled by identityd and needs its own start on iotedge distros. - -- [ ] **Step 1: Add `ExecStartPost` to the precondition unit** - -In `aziot-identityd-precondition.service`, `[Service]` section, after the `ExecStart` line: - -```ini -[Service] -Type=oneshot -ExecStart=@@AZIOTCLI@@ config apply -# a run after boot only writes the config; bring the services up like the -# boot transaction would. --no-block is required: this unit is ordered -# Before= the started units, a blocking start would deadlock. On a normal -# boot the request merges with the already queued jobs. -ExecStartPost=-/bin/systemctl start --no-block @@STACK_UNITS@@ -Restart=on-failure -RestartSec=5 -RemainAfterExit=true -``` - -- [ ] **Step 2: Substitute `@@STACK_UNITS@@` in `aziot-identityd.inc`** - -Extend the existing `DISTRO_FEATURES` branch (currently only substituting `@@AZIOTCLI@@`): - -```sh - if ${@bb.utils.contains('DISTRO_FEATURES', 'iotedge', 'true', 'false', d)}; then - sed -i "s/@@AZIOTCLI@@/iotedge/" ${D}${systemd_system_unitdir}/aziot-identityd-precondition.service - sed -i "s/@@STACK_UNITS@@/aziot-identityd.service aziot-edged.service/" ${D}${systemd_system_unitdir}/aziot-identityd-precondition.service - else - sed -i "s/@@AZIOTCLI@@/aziotctl/" ${D}${systemd_system_unitdir}/aziot-identityd-precondition.service - sed -i "s/@@STACK_UNITS@@/aziot-identityd.service/" ${D}${systemd_system_unitdir}/aziot-identityd-precondition.service - fi -``` - -- [ ] **Step 3: Verify both substitution variants** - -```bash -SCRATCH=/tmp/claude-1000/-home-jzac-projects-omnect-os/1d9ec82f-888d-46f1-9563-aeaf209297ae/scratchpad -src=/home/jzac/projects/meta-omnect/recipes-azure-iot/azure-identityd/aziot-identityd/aziot-identityd-precondition.service -sed "s/@@STACK_UNITS@@/aziot-identityd.service aziot-edged.service/" $src | grep ExecStartPost -sed "s/@@STACK_UNITS@@/aziot-identityd.service/" $src | grep ExecStartPost -grep -c "@@" $src -``` - -Expected: -``` -ExecStartPost=-/bin/systemctl start --no-block aziot-identityd.service aziot-edged.service -ExecStartPost=-/bin/systemctl start --no-block aziot-identityd.service -2 -``` -(the `2` = the two remaining placeholders in the source file: `@@AZIOTCLI@@` and `@@STACK_UNITS@@`) - -- [ ] **Step 4: Commit** - -```bash -cd /home/jzac/projects/meta-omnect -git add recipes-azure-iot/azure-identityd/aziot-identityd/aziot-identityd-precondition.service recipes-azure-iot/azure-identityd/aziot-identityd.inc -git commit -m "fix(aziot-identityd): start the identity stack after a late config apply - -When the precondition unit runs after boot (timer, update-validation -flags), a successful 'config apply' only writes the configuration and -nothing starts the identity services. Start them from ExecStartPost so -the after-boot path behaves like the boot path. identityd's connects to -the keyd/certd/tpmd sockets activate those services; edged needs its own -start on iotedge distros. - -Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>" -``` - ---- - -### Task 4: ADU agent retries forever with backoff - -**Files:** -- Modify: `recipes-azure-iot/iot-hub-device-update/iot-hub-device-update/deviceupdate-agent.service` - -**Interfaces:** none (self-contained unit change). - -Background: the agent exits with code 1 when its startup health check cannot reach identityd. With `StartLimitBurst=10`/`StartLimitIntervalSec=120` an identity outage of ~50 s parks the unit in `failed`, and only `deviceupdate-agent.timer` (minutes) starts it again. Retry forever instead: each auto-restart also re-queues `Wants=aziot-identityd.service`, so the agent is itself a recovery path for identity. A persistent loop stays visible: the crash-loop check keys on the `activating (auto-restart)` sub-state, which the unit occupies during every backoff delay. - -- [ ] **Step 1: Edit the unit** - -In the `[Unit]` section, replace - -```ini -StartLimitBurst=10 -StartLimitIntervalSec=120 -``` - -with - -```ini -# never park in 'failed': each restart also pulls identity up again via -# Wants=, and a persistent loop is caught by the crash-loop check -StartLimitIntervalSec=0 -``` - -In the `[Service]` section, replace - -```ini -Restart=always -RestartSec=5 -``` - -with - -```ini -Restart=always -RestartSec=5 -RestartSteps=5 -RestartMaxDelaySec=60 -``` - -- [ ] **Step 2: Verify the resulting unit** - -```bash -grep -E "StartLimit|Restart" /home/jzac/projects/meta-omnect/recipes-azure-iot/iot-hub-device-update/iot-hub-device-update/deviceupdate-agent.service -``` - -Expected (exactly these, no `StartLimitBurst` left): -``` -StartLimitIntervalSec=0 -Restart=always -RestartSec=5 -RestartSteps=5 -RestartMaxDelaySec=60 -``` - -- [ ] **Step 3: Commit** - -```bash -cd /home/jzac/projects/meta-omnect -git add recipes-azure-iot/iot-hub-device-update/iot-hub-device-update/deviceupdate-agent.service -git commit -m "fix(adu): retry forever with backoff instead of reaching the start limit - -The agent exits when its startup health check cannot reach identityd, -and ~50 s of identity outage used to park the unit in 'failed' until the -retry timer fired minutes later. Drop the start limit and back the -restart delay off from 5 s to 60 s: the agent recovers within one delay -once identity is back, and each restart pulls identityd up again through -its Wants=. A persistent loop is still flagged by the crash-loop check, -which keys on the auto-restart sub-state. - -Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>" -``` - ---- - -### Task 5: build and device verification - -**Files:** none (verification only, no commit). - -- [ ] **Step 1: Build a gateway-devel image with the branch** - -Make sure `local.env` in `/home/jzac/projects/omnect-os` points kas at the local meta-omnect checkout (branch `jz-2026-08-03-aziot-identity-recovery`), then: - -```bash -cd /home/jzac/projects/omnect-os -./dobi.sh build-omnect-gateway-devel-rpi4-64 -``` - -- [ ] **Step 2: Inspect the generated units in the build tree** - -```bash -cd /home/jzac/projects/omnect-os -find build/build/tmp*/work -path "*aziot*" -name "*.socket" -exec grep -l TriggerLimitIntervalSec {} \; -find build/build/tmp*/work -name "aziot-identityd-precondition.service" -path "*image*" -exec grep ExecStartPost {} \; -find build/build/tmp*/work -name "deviceupdate-agent.service" -path "*image*" -exec grep -E "StartLimit|RestartSteps" {} \; -``` - -Expected: all aziot sockets contain `TriggerLimitIntervalSec=0`; the precondition unit contains the substituted `ExecStartPost` (no `@@` left); the agent unit has `StartLimitIntervalSec=0`, `RestartSteps=5`, `RestartMaxDelaySec=60`. - -- [ ] **Step 3: Manual repro on an rpi4 (gateway-devel)** - -Flash the image, then run the repro from the issue doc: - -```bash -sudo systemctl stop aziot-identityd.service -sudo rm /etc/aziot/identityd/config.d/00-super.toml -sudo systemctl restart deviceupdate-agent.service -sleep 60 -journalctl -b0 | grep -c 'Trigger limit' # expected: 0 -systemctl is-active aziot-identityd.socket # expected: active -systemctl show -p ActiveState,SubState,NRestarts deviceupdate-agent.service -# expected: activating / auto-restart, NRestarts growing, never 'failed' -``` - -Then restore and watch recovery: - -```bash -sudo systemctl start aziot-identityd-precondition.service # regenerates the super config via config apply -systemctl is-active aziot-identityd.service # expected: active right after apply (fix b) -sleep 90 -systemctl is-active deviceupdate-agent.service # expected: active within RestartMaxDelaySec -systemctl is-system-running # expected: running -``` - -Also confirm the load-bearing assumption of fix (a): while the super config is missing, `journalctl -f` must not show a skip-loop storm faster than the poll limit allows, and systemd's CPU usage stays normal. - -- [ ] **Step 4: Report results** - -No commit. Record the device observations in the PR conversation (not the description). diff --git a/docs/superpowers/specs/2026-08-03-aziot-identity-recovery-design.md b/docs/superpowers/specs/2026-08-03-aziot-identity-recovery-design.md deleted file mode 100644 index 7d604a29..00000000 --- a/docs/superpowers/specs/2026-08-03-aziot-identity-recovery-design.md +++ /dev/null @@ -1,88 +0,0 @@ -# aziot identity recovery design - -Fixes the dead end described in `aziot-identity-recovery-issue.md`: a missing -`/etc/aziot/identityd/config.d/00-super.toml` fails `aziot-identityd.socket` -via its trigger limit, a late successful `config apply` starts nothing, and -the ADU agent exhausts its start limit — the device stays `degraded` with no -identity until `deviceupdate-agent.timer` fires. - -Scope: unit-file changes in meta-omnect only, no source patches. The -companion test change (remove the transitional `repair_identity_units` step -from `ci/tests/precondition_recovery_test.sh`) is a separate omnect-os PR. - -Each of the three fixes removes the dead end on its own; together they make -recovery fast and independent of which client happens to run. - -## 1. Sockets never fail from skipped activations - -Add `TriggerLimitIntervalSec=0` under `[Socket]` for: - -- `aziot-identityd.socket`, `aziot-keyd.socket`, `aziot-certd.socket`, - `aziot-tpmd.socket` — in `aziot-identityd.inc`, same sed style as the - service edits -- `aziot-edged.mgmt.socket`, `aziot-edged.workload.socket` — in - `aziot-edged.inc`; `aziot-edged.service` carries the same - `ConditionPathExists` lines, so its sockets share the dead end - -While the service is skipped by its condition the socket stays listening; the -first client connect after the super config is back activates the service. - -Load-bearing assumption, verify on a device: a pending connection to a -skipped service must not busy-loop systemd. On systemd 255 the poll limit -(`PollLimitIntervalSec`, active by default) pauses polling instead of failing -the unit. meta-omnect main is scarthgap-only (systemd 255.21). - -## 2. Late `config apply` starts the stack - -`aziot-identityd-precondition.service` gets - - ExecStartPost=-/bin/systemctl start --no-block aziot-identityd.service - -Starting identityd is enough for the rest of the identity stack: identityd's -connects to the keyd/certd/tpmd sockets activate those services. On -iotedge distros the recipe's existing `DISTRO_FEATURES` branch (the -`@@AZIOTCLI@@` substitution) adds `aziot-edged.service` to the started units; -edged needs its own start. - -- `--no-block` is mandatory: the precondition is ordered `Before=` these - units, a blocking start inside `ExecStartPost` deadlocks. -- On a normal boot the start merges with the already-queued jobs (no-op). -- The `-` prefix keeps a failed `systemctl` call from failing an otherwise - successful apply. - -## 3. ADU agent retries forever with backoff - -`deviceupdate-agent.service`: drop `StartLimitBurst`, set -`StartLimitIntervalSec=0`, keep `Restart=always`, add `RestartSteps=5` and -`RestartMaxDelaySec=60` on top of `RestartSec=5` (delay grows from 5 s to -60 s). Needs systemd >= 254; scarthgap has 255. - -- Each auto-restart re-queues `Wants=aziot-identityd.service`, so the agent - is a recovery path for identity too. -- The unit never reaches `failed`; a persistent loop is visible through the - crash-loop checker. Planning-time check: does the checker's threshold still - flag a loop that has slowed to one restart per 60 s? -- `deviceupdate-agent.timer` stays unchanged — its job is the - skipped-at-boot case (condition blocked the start, `Restart=` never - engaged), not start-limit rescue. - -## Verification - -- Manual repro from the issue doc on an rpi4: socket survives the - config-less window, identityd comes up right after a successful apply, the - agent within `RestartMaxDelaySec`. -- omnect-os follow-up: `precondition_recovery_test.sh` loses - `repair_identity_units` and its TODO; `RECOVER_DEADLINE` grows to cover one - `RestartMaxDelaySec` (~90 s). -- Whitelist check: the failure window now produces repeated "skipped because - of an unmet condition" lines instead of a one-time trigger-limit failure; - base_test's syslog check may need a whitelist entry. - -## Commits - -One commit per fix, the socket fix split by recipe: - -1. `fix(aziot-identityd): keep sockets active while the service is skipped` -2. `fix(aziot-edged): keep sockets active while the service is skipped` -3. `fix(aziot-identityd): start the identity stack after a late config apply` -4. `fix(adu): retry forever with backoff instead of reaching the start limit`