fix(ansible): render STATE_LEVEL_TENANT_ID from inventory (production login regression)

[KDwevedi]

What happened

On 2026-05-14 ~06:23 UTC a deploy of this branch onto naipepea rewrote /opt/digit/docker-compose.egov-digit.yaml with the upstream defaults (pg / pg.citya / mz for STATE_LEVEL_TENANT_ID and EGOV_STATE_LEVEL_TENANT_ID). Login was broken for all ke.nairobi users — and most ke users — until manually patched ~25 hours later.

The trigger was the playbook's Copy registry-prefixed Docker Compose file step doing a verbatim copy: of the source compose, with no follow-up replace: task for these env vars. Only Grafana's domain gets rewritten today (api.egov.theflywheel.in → {{ domain }}).

How it breaks login

DIGIT services that need a "state-level tenant" read it from env. Most critical: egov-user. At login, the user-service does this dance:

1. Receives username=ADMIN, tenantId=ke (or ke.nairobi).
2. To look up the user row, it has to encrypt the supplied username — because eg_user.username is stored encrypted (Kenya DPA). The encrypt is done with the tenant from STATE_LEVEL_TENANT_ID, not the request's tenantId.
3. With STATE_LEVEL_TENANT_ID=pg, egov-enc-service uses pg's symmetric key (id 177813) → resulting ciphertext is 177813|<base64>.
4. egov-user looks up eg_user WHERE username='177813|<base64>' → no match (the row is stored as 74493|<base64> because it was created when STATE_LEVEL=ke).
5. UserNotFoundException → OAuth2Exception: Invalid login credentials → HTTP 400.

Real trace pulled from digit-egov-user-1 on naipepea before the fix:

TENANTID=ke.nairobi, CORRELATION_ID=6cf9ff6e-... : Sending request to
  http://egov-enc-service:1234/egov-enc-service/crypto/v1/_encrypt
TENANTID=ke.nairobi, CORRELATION_ID=6cf9ff6e-... : Received response
TENANTID=ke.nairobi, CORRELATION_ID=6cf9ff6e-... ERROR:
  CustomAuthenticationProvider : User not found
org.egov.user.domain.exception.UserNotFoundException:
  User not found for given criteria:
  UserSearchCriteria(userName=177813|fMQdWYTF1770IFtDHDc7HIovpJrnxsf+xIqwCMI=,
                     tenantId=ke.nairobi, ...)

Key-id 177813 is pg's key, not ke.nairobi's.

Direct evidence the encrypt service itself was healthy — same machine, two different tenantIds:

$ curl -X POST egov-enc-service/_encrypt -d '{"encryptionRequests":[{"tenantId":"pg","type":"Normal","value":"TEST"}]}'
["177813|..."]
$ curl -X POST egov-enc-service/_encrypt -d '{"encryptionRequests":[{"tenantId":"ke","type":"Normal","value":"TEST"}]}'
["74493|..."]

So the per-tenant key dispatch on egov-enc-service works correctly; the bug is upstream: egov-user is asking the wrong tenant.

Why only some accounts kept working

Pre-fix eg_user.username key distribution on naipepea:

key prefix	tenant	rows
`74493	`	ke
`74493	`	ke.nairobi
`177813	`	ke
`177813	`	ke.nairobi
`177813	`	ke.care
`618647	`	ke.nairobi

ADMIN was already stored under 177813| (probably from an earlier deploy when STATE_LEVEL=pg was also active — this is a recurring regression). So with STATE_LEVEL=pg on 2026-05-14, ADMIN happened to work by accidental matching, masking the breakage long enough that the user-facing 400s only started getting reported when a non-ADMIN tried.

Manual fix applied on naipepea (pre-this-PR)

Two phases:

1. Stop the bleeding — sed the 13 STATE_LEVEL_TENANT_ID / EGOV_STATE_LEVEL_TENANT_ID lines in /opt/digit/docker-compose.egov-digit.yaml from pg / pg.citya / mz back to ke / ke.nairobi, then docker compose up -d to recreate egov-user, egov-enc-service, egov-workflow-v2, boundary-service, default-data-handler, egov-hrms, egov-bndry-mgmnt, pgr-services.
2. Migrate the 18 rows that were sitting under 177813| (ADMIN + 14 ke.nairobi + 3 ke.care) to 74493| so they survive the new lookup tenant: _decrypt → _encrypt with tenantId=ke → UPDATE eg_user. Two ke CITIZEN duplicates of registered mobile numbers couldn't be migrated (unique key collision); harmless — the live mobile-number row already exists under 74493|.

This PR is the fix-at-source so the next ./deploy.sh nairobi (or any tenant) doesn't reintroduce the regression.

What's in this PR

Change	File
Add state_root (state-tier tenant — ke, pb, mz) to the host_vars contract, distinguished from state_tenant_id (the city-tier tenant the SPA binds to)	host_vars/_example.yml
Document the state_root default + the failure mode	group_vars/digit.yml
Four replace: tasks rewriting the compose env values after the copy: step	playbook-deploy.yml

The four substitutions:

STATE_LEVEL_TENANT_ID: pg       -> {{ state_root }}      # 7 sites
EGOV_STATE_LEVEL_TENANT_ID: pg  -> {{ state_root }}      # 4 sites
STATE_LEVEL_TENANT_ID: pg.citya -> {{ tenant_id }}       # 2 sites (enc + workflow)
STATE_LEVEL_TENANT_ID: mz       -> {{ state_root }}      # 1 site  (inbox)

Three other pg env occurrences in the compose are intentionally left alone (EGOV_STATELEVEL_TENANTID on egov-indexer, EGOV_STATELEVEL_TENANT on egov-hrms, PGR_STATELEVEL_TENANTID on pgr-services). Each is either typo'd or differently-named relative to what the services actually consume, and each was pg on the working pre-regression compose too. Keeps this PR scoped to the variables that demonstrably broke auth.

Verification

Locally rendered the playbook against a synthetic state_root: ke / tenant_id: ke.nairobi. grep confirms all 13 affected lines flip to the inventory values. The 3 leave-alone pg lines stay pg. No other compose lines change.

Test plan for whoever runs the next deploy

• ./deploy.sh nairobi → docker exec digit-egov-user-1 env | grep STATE_LEVEL shows ke (not pg).
• curl POST /user/oauth/token with a ke.nairobi-tenant employee returns a 200 + access_token (not 400 invalid_credentials).
• ./deploy.sh bomet → same, but ke.bomet user works (assumes state_root: ke and tenant_id: ke.bomet in bomet host_vars).
• Regression: ADMIN at tenantId=ke still works.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 4163d551-7f81-4dd3-80e8-e847f67c4d20

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[KDwevedi]

Superseded by #52 (consolidated Mac bring-up — 16 commits absorbing this PR + 5 others into one canonical path on nairobi). Closing.