Skip to content

feat(remediation): outcome vocabulary so a changed host is never reported as unchanged - #761

Open
remyluslosius wants to merge 2 commits into
mainfrom
feat/remediation-outcome-vocabulary
Open

feat(remediation): outcome vocabulary so a changed host is never reported as unchanged#761
remyluslosius wants to merge 2 commits into
mainfrom
feat/remediation-outcome-vocabulary

Conversation

@remyluslosius

Copy link
Copy Markdown
Contributor

What

Remediation reported two terminal outcomes, executed and failed, and "failed" meant five different things carrying one message:

Remediation did not complete. No host change was committed.

That message is false for Kensa v0.8.0's new staged status, and this PR is the gate on the v0.8.0 bump (v0.7 exit criteria 4 and 5).

The defect

On a host whose audit config is immutable (auditctl -s reports enabled 2) an audit_rule_set apply writes the reboot-deferred persist layer and terminates staged. The host is mutated. OpenWatch absorbed the unknown status through a default: branch and:

  1. marked the request failed,
  2. told the operator nothing had changed,
  3. journalled it as skipped, and
  4. refused rollback, because rollback required executed.

The change sat on the host, invisible, with no route back through the UI. Recovery meant SSH plus kensa rollback, by someone who had been told there was nothing to reverse.

Not theoretical. Probed the dev fleet with a read-only auditctl -s: 2 of 6 reachable hosts are immutable (owas-tst01 RHEL 8.10, owas-tst02 RHEL 9.6), both hardened RHEL with immutable.rules. Kensa ships 103 audit_rule_set rule files. owas-tst02's Remediation tab already shows audit-cmd-chsh and audit-sudoers as Failed for this reason; under v0.7.6 that verdict is honest (the engine rolls back to nothing), under v0.8.0 those exact rows become silent host mutations.

The vocabulary

Outcome Host state Operator action
executed converged, protected none
staged changed, not converged reboot, or roll back
reverted untouched, Kensa restored it investigate the rule
not_applied untouched, engine declined reconcile
partially_applied unknown, steps stranded inspect now
failed untouched retry

Three behaviours follow, and they are the point:

  • staged does NOT flip host_rule_state to pass. The kernel has not loaded the change and a re-scan correctly still fails the rule. Flipping it would claim a protection the host does not have.
  • staged IS rollback-eligible. Both the precondition and the rollback-handle lookup widened; either one alone still stranded the change.
  • reverted is not red and does not page. Validation failing and the host being restored is the atomic model working. Paging on it teaches operators to ignore the alert.

The guard that was missing

remediation.OutcomeOf is now the single place a Kensa status is interpreted, and it returns (Status, known bool). An unknown status logs at WARN naming the value and fails closed, never to a success-shaped outcome. That is AC-11, and its absence is precisely what caused this bug.

Deliberately not done

Detecting Kensa's "engine refused without mutating" case (the v0.8.0 duplicate-audit-action guard). Kensa surfaces it as StepResult.Success=false plus human-readable Detail, with no distinguishable TransactionStatus. Telling it from a real failure would mean pattern-matching step text on a path that runs as root, where an upstream wording change would silently reclassify real failures. Refusals map by status to reverted (host untouched), which is truthful if less specific. StatusNotApplied and the DB enum already accept the value, so wiring it later is one line.

Same reasoning for already-compliant runs, which Kensa reports as committed with a synthetic check step (its own docs flag the overload as a known follow-up). Today that shows a green "Fixed" chip and a Roll back button for a change that never happened. Both are filed upstream rather than worked around here.

Also fixed

RequestRemediationModal told operators "Applying the fix on the host is an OpenWatch Enterprise feature." False since single-rule remediation shipped free in v0.2.0-rc.11. Verified against the registry: audit_export is the only license_gated permission. Separate commit.

SDD

  • api-remediation 1.1.0 -> 1.2.0 (C-09, AC-09/10/11, AC-11 is the negative path)
  • frontend-remediation-tab 1.1.0 -> 1.2.0 (AC-08)
  • 116 specs, 116 passing.

Migration 0054

Widens both CHECK constraints (remediation_requests.status and remediation_transactions.phase_result). The one-open-request partial unique index is deliberately untouched: every new value is terminal, so a staged request must not block a fresh request for the same host and rule. Down-migration folds the new values back before restoring the narrower constraint.

Verification

gofmt, go vet, go build ./..., full go test ./internal/..., tsc, prettier, vitest 361/361, OpenAPI drift gate green after regeneration.

Note on .secrets.baseline

Refreshed for two reasons, neither a new finding: line drift in the generated server.gen.go from the enum addition, and one pre-existing false positive at api/openapi.yaml:127 (the word "password" in prose about editable profile fields). That finding fires on unmodified main too; the baseline predates it.

Follow-on for v0.7

This unblocks the Kensa v0.8.0 bump (exit criterion 6), which is a separate PR: KensaModuleVersion, two spec refs, and the 748 -> 769 corpus counts in the tracked guides.

… feature

The Request Remediation modal told the operator "Applying the fix on the host
is an OpenWatch Enterprise feature: the free tier governs the request and
approval only." That is false, and has been since single-rule remediation
shipped free in v0.2.0-rc.11.

Verified against the registry rather than the copy: audit_export is the ONLY
license_gated permission in auth/permissions.yaml. remediation:execute and
remediation:rollback carry no license gate, and api-remediation C-06 states it
outright ("Execute/rollback are FREE core (Tier A), NOT license-gated ... No
remediation act endpoint returns 402"). frontend-remediation-tab AC-07 already
required this upsell copy to be gone; it was removed from the tab but not from
this modal.

The practical effect is a Community user being told to buy something they
already have, on the screen where they decide whether to act.

Also reformats the paragraph: PR #755's "OpenWatch+" -> "OpenWatch Enterprise"
rename pushed the line past the print width and merged unformatted, because CI
does not run prettier (only the local pre-commit hook does). Worth noting as a
gap rather than a one-off.
…rted as unchanged

Remediation reported two terminal outcomes, executed and failed, and "failed"
meant five different things carrying one message: "Remediation did not
complete. No host change was committed."

That message is false for Kensa v0.8.0's new `staged` status. On a host whose
audit config is immutable (auditctl -s reports `enabled 2`) an audit_rule_set
apply writes the reboot-deferred persist layer and terminates staged. The host
IS mutated. OpenWatch absorbed the unknown status through a default branch,
marked the request failed, told the operator nothing had changed, journalled it
as 'skipped', then refused rollback because rollback required 'executed'. The
change sat on the host, invisible, with no route back through the UI. Verified
reachable on 2 of 6 reachable dev-fleet hosts (owas-tst01 RHEL 8.10, owas-tst02
RHEL 9.6, both enabled 2), across 103 audit_rule_set rules.

Outcomes now map one-to-one onto operator actions:

  executed           runtime converged, host protected
  staged             persist written, NOT converged, host changed, needs reboot
  reverted           validation failed, Kensa restored pre-state, host clean
  not_applied        engine declined, host untouched
  partially_applied  stranded steps, host state unknown
  failed             errored or unreachable

Three behaviours follow, and they are the point of the change:

  - staged does NOT flip host_rule_state to pass. The kernel has not loaded
    the change, a re-scan correctly still fails the rule, and claiming a pass
    would assert a protection the host does not have.
  - staged IS rollback-eligible. Kensa captured pre-state and reverses a
    staged drop-in byte-perfect; the precondition and the rollback-handle
    lookup both widened, since either alone still stranded the change.
  - reverted is not red and does not page. Validation failing and the host
    being restored is the atomic model working. Alerting on it teaches
    operators to ignore the alert.

remediation.OutcomeOf is now the single place a Kensa status is interpreted,
returning (Status, known bool). An unknown status logs at WARN naming the value
and fails closed to failed, never to a success-shaped outcome. That guard is
AC-11, and its absence is what caused this bug: a default branch silently
swallowed a new terminal state.

Deliberately NOT done: detecting Kensa's "engine refused without mutating"
case (the v0.8.0 duplicate-audit-action guard). Kensa surfaces it as
StepResult.Success=false plus human-readable Detail with no distinguishable
TransactionStatus, so telling it from a real failure would mean matching step
text on a path that runs as root. Refusals map by status to reverted, which is
truthful if less specific. StatusNotApplied and the DB enum already accept the
value, so wiring it is one line once Kensa exposes a signal. Same reasoning for
already-compliant runs, which Kensa reports as committed with a synthetic check
step. Both filed upstream.

Migration 0054 widens both CHECK constraints. The one-open-request partial
unique index is deliberately untouched: every new value is terminal, so a
staged request must not block a fresh request for the same host and rule.

.secrets.baseline is refreshed for two reasons, neither a new finding: line
drift in the generated server.gen.go from the enum addition, and one
pre-existing false positive at api/openapi.yaml:127 (the word "password" in
prose about which profile fields are editable). That finding fires on
unmodified main too; the baseline predates it.

Specs: api-remediation 1.1.0 -> 1.2.0 (C-09, AC-09/10/11),
frontend-remediation-tab 1.1.0 -> 1.2.0 (AC-08). 116 specs, 116 passing.
Verified: gofmt, go vet, go build, full go test ./internal/..., tsc, prettier,
vitest 361/361, OpenAPI drift gate green after regeneration.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant