Skip to content

operator-runbook.md

Latest commit

 

History

History
371 lines (270 loc) · 11.8 KB

operator-runbook.md

File metadata and controls

371 lines (270 loc) · 11.8 KB
Raw

Operator runbook

This runbook covers day-2 operational workflows for the OpenBao Secret Sync plugin: blocked sync, restore guard, queue recovery, reconcile, drift, and incident evidence. It assumes the plugin is mounted at secret-sync/; adjust paths if the mount name differs.

Use the user guide for the normal first-sync workflow. Use the sync model for the shared model behind source paths, associations, and provider objects. Use Convergence for queue, status, manual sync, retry, cancel, and drain behavior. Use Reconcile and drift for provider read-state and background drift behavior. Use Ownership and safety before repairing stale, drifted, missing, or ownership-lost remote objects.

First checks

Confirm the mount responds:

bao read secret-sync/config

If restore_guard=true, remote mutation is blocked until restore or clone review is complete and the guard is acknowledged:

bao write -force secret-sync/config/restore-guard/acknowledge

Check mount-wide pause and queue capacity:

bao read secret-sync/config

Pause or resume remote mutation:

bao write secret-sync/config disabled=true
bao write secret-sync/config disabled=false

Response diagnostics

Blocked or terminal responses can include a human-readable hint and structured next_actions. Treat next_actions as the first recovery path. They name the operation, plugin-relative path, parameters, whether the action can mutate remote state, and an example bao command.

On successful responses, hint and next_actions are top-level response data fields. On OpenBao error responses, the structured copy is nested under the error data field so clients still recognize the response as an error. The human-readable error text also includes a compact Hint: line and the first Next action: command so bao CLI users see the recovery path without parsing JSON.

Examples:

  • manual_sync: run after resolving REMOTE_MISSING, DRIFTED, or REMOTE_OWNERSHIP_LOST remote state.
  • enable_source: run when hardened posture blocks association activation or dispatch because source sync is not enabled for the source path.
  • acknowledge_restore_guard: run only after restore or clone review is complete.
  • read_queue, drain_queue, or retry_operation: use when queue or provider capacity blocks sync.

Destination checks

Read destination config. Sensitive fields must be redacted:

bao read secret-sync/destinations/<type>/<name>

Check destination readiness:

bao read secret-sync/destinations/<type>/<name>/check

Validate static destination configuration:

bao read secret-sync/destinations/<type>/<name>/validate

Check destination reachability and authorization:

bao read secret-sync/destinations/<type>/<name>/health

Use validation for configuration mistakes and health for runtime dependency state. A destination can validate correctly but still be unhealthy because of network, IAM, token, RBAC, or provider availability issues.

Source and association checks

In hardened posture, confirm source sync is explicitly enabled for the path:

bao write -force secret-sync/sources/app/db/enable

This is required only when security_posture=hardened. Mounts default to security_posture=standard.

In hardened posture, destination checks also report destination_unconstrained until the destination has both source-path and resolved-name prefix constraints.

To inspect the underlying metadata:

bao read secret-sync/metadata/app/db

Read the current source version:

bao read secret-sync/data/app/db

Check source readiness:

bao read secret-sync/sources/app/db/check

Plan the association before creating or changing it:

bao write secret-sync/associations/app/db/plan \
  destination=<type>/<name> \
  resolved_name=<remote-name>

secret-path, json, retain, and enabled=true are the defaults. Read secret-sync/info to inspect static association defaults and registered provider capability flags.

When updating an existing association, omitted optional fields keep the stored association values when the source path and destination identify a single existing association. This prevents partial updates from changing granularity, name template, delete mode, or enabled state by accident. Use the read output above when you need to make the update shape explicit.

Use destination=<type>/<name> for normal association disable, enable, and manual sync operations. Association IDs remain available for exact reads, deletes, and ambiguity escape hatches, but they are not the default lifecycle selector.

Current-version source lifecycle endpoints participate in sync. DELETE data/<path>, delete/<path>, and destroy/<path> cancel stale queued upserts and enqueue remote deletes for associations with delete_mode=delete. undelete/<path> on the current version queues replacement upserts for enabled associations.

Read existing associations:

bao read secret-sync/associations/app/db

Disable, enable, or manually sync one association:

bao write secret-sync/associations/app/db/disable destination=<type>/<name>
bao write secret-sync/associations/app/db/enable destination=<type>/<name>
bao write secret-sync/associations/app/db/sync destination=<type>/<name>

Queue operations

Read the queue summary:

bao read secret-sync/queue

Queue summaries include capacity and utilization. Treat sustained high utilization as backpressure: increase drain frequency, reduce producer rate, or raise queue_capacity after checking storage and provider limits. Set queue_capacity=0 only for a deliberate enqueue freeze; existing queued work can still drain. Successful operations are removed from the queue after object status is persisted; use status/<path> rather than queue/<operation-id> to confirm completed sync.

Event-triggered dispatch is enabled by default and wakes a bounded drain after enqueue-producing requests. Use queue/drain when you need deterministic local testing, a controlled catch-up batch, or an explicit operator action after temporarily disabling event dispatch. When the request reaches a read-enabled HA standby, OpenBao forwards it to the active node before the plugin inspects or claims queue work.

For deterministic local testing or controlled catch-up, drain due operations:

bao write secret-sync/queue/drain max_operations=10

Read one operation:

bao read secret-sync/queue/<operation-id>

Retry or cancel one operation:

bao write -force secret-sync/queue/<operation-id>/retry
bao write -force secret-sync/queue/<operation-id>/cancel

Cancel discards pending, retry-wait, or terminal failed work; it is not retained in the queue summary. Terminal failures are otherwise retained for at most seven days, with the newest 1,000 records taking precedence when the limit is reached.

queue/drain can execute remote mutations. Keep it operator-scoped. Forwarding is limited to an HA standby within the active cluster; Secret Sync does not forward drains from performance secondaries or enable cross-cluster forwarded storage.

Operational signals

The plugin emits OpenTelemetry metric API calls only. Exporter setup remains an OpenBao deployment concern.

Useful alert inputs:

  • openbao.secret_sync.restore_guard.active stays 1 after an expected restore or deployment review window;
  • openbao.secret_sync.queue.utilization remains high or increases while openbao.secret_sync.queue.depth{state="pending"} is not draining;
  • openbao.secret_sync.remote_mutation.blocked increases with reason disabled, restore_guard, or replication_state;
  • openbao.secret_sync.provider.requests failures increase by provider, operation, or error class;
  • openbao.secret_sync.drift.repairs failures or retries increase by destination type or granularity;
  • openbao.secret_sync.readiness.checks failures identify onboarding blockers without exposing source paths or destination names.

Status and reconcile

Read per-source status:

bao read secret-sync/status/app/db

Use JSON output when copying identifiers into commands:

bao read -format=json secret-sync/status/app/db | jq .data

Plan reconcile without changing local status or remote state:

bao read secret-sync/reconcile/app/db/plan

Apply reconcile to refresh local status from provider read-state:

bao write -force secret-sync/reconcile/app/db

Reconcile reads remote state. It does not write destination secrets and is safe to use while the restore guard is active.

Common failure classes

DESTINATION_AUTH_ERROR or provider error class authn:

  • check destination credential material or workload identity;
  • rerun destination readiness, validation, and health;
  • rotate the destination credential if compromise is suspected.

DESTINATION_POLICY_ERROR or provider error class authz:

  • check IAM, RBAC, token scopes, project permissions, or namespace access;
  • verify the destination can perform the requested create, update, read-state, and delete operations.

DESTINATION_RATE_LIMITED or provider error class rate_limit:

  • inspect queue retry state;
  • allow automatic retry to progress if attempts remain;
  • reduce drain batch size during manual catch-up.

DESTINATION_UNAVAILABLE or provider error class unavailable:

  • verify provider endpoint reachability;
  • check proxy, DNS, private endpoint, or local test stack health;
  • retry after the destination recovers.

REMOTE_OWNERSHIP_LOST or provider error class ownership:

  • inspect the remote object metadata before retrying;
  • decide whether the remote object was intentionally taken over;
  • create a new association or remote name instead of forcing overwrite unless an operator explicitly accepts that risk.
  • if the remote object was deleted or ownership was otherwise resolved, run the manual_sync action returned by status/<path> or reconcile/<path>.

VALIDATION_ERROR:

  • check destination config fields;
  • check provider name rules for the rendered remote object name;
  • check payload size and granularity support.
  • use the returned hint and next_actions; source sync failures point to sources/<path>/enable, while generic validation failures point to the association plan.

QUEUE_BLOCKED:

  • read secret-sync/config for mount-wide pause or restore guard;
  • check queue capacity;
  • verify the association and destination are enabled.
  • use the returned next_actions to inspect queue, drain due work where safe, or retry the failed queue operation.

Restore or clone review

Use Restore and clone review for the full review workflow.

After restore or clone, keep mutation blocked until remote ownership has been reviewed:

bao read secret-sync/config
bao read secret-sync/reconcile/app/db/plan

For each important source path:

  1. Review local source version and association state.
  2. Run reconcile plan.
  3. Inspect remote ownership and payload hash status.
  4. Cancel, retry, or re-plan queued operations as needed.
  5. Acknowledge restore guard only after the destination state is understood.

Resume remote mutation:

bao write -force secret-sync/config/restore-guard/acknowledge

Evidence to capture

For troubleshooting or issue reports, capture:

  • mount path and plugin version;
  • destination type and redacted destination config;
  • association ID and source path;
  • source version, not secret value;
  • operation ID and queue state;
  • status output with payload values removed if present in source reads;
  • provider-side object metadata, not secret value;
  • exact error class and message.

Do not paste secret payloads, provider tokens, static credentials, GitLab tokens, AWS external IDs, kubeconfigs, or raw audit records containing secrets.