Skip to content

OpenBao's self-signed leaf is its own trust anchor, so every renewal breaks ESO and the in-pod CLI #879

Closed
dmfdeploy/dmf-infra
#145

Description

@znerol2

Problem

OpenBao serves a certificate from a namespaced self-signed issuer (openbao-selfsigned, roles/base/cert-manager/tasks/main.yml). The leaf certificate is also the trust anchor. The ESO role copies tls.crt into the ClusterSecretStore caBundle (roles/base/external-secrets/tasks/main.yml), and the OpenBao pods use the same mounted file to verify themselves.

So whenever cert-manager reissues that certificate, trust breaks until someone runs manual steps:

  • ESO trusts only the new certificate while the running pods still serve the old one. The store goes InvalidProviderConfig ("unable to create client") until every pod restarts.
  • The in-pod bao CLI fails TLS with "x509: certificate signed by unknown authority" until the pod restarts. That breaks bin/unseal-openbao.sh and every playbook that runs bao in the pod.

A reissue happens on every renewal (the default 90-day duration), and also whenever the DNS names change.

Evidence

Seen on the sandbox during the #864 WP3 rehearsal on 2026-11-08, after the cert-manager play added the Raft peer names. The store stayed InvalidProviderConfig until pod 0 restarted, and the in-pod CLI refused the server's certificate.

Direction

Issue the OpenBao leaf from a CA (for example the existing dmf-local-ca ClusterIssuer, or a dedicated OpenBao CA Issuer) and distribute the CA, not the leaf, to ESO and to the pods. A renewal then changes nothing that a client trusts. Check the other consumers that read tls.crt as their trust anchor first.

Refs #864

Activity

  1. znerol2 commented on Nov 12, 2026

    @znerol2
    ContributorAuthor

    Done by dmfdeploy/dmf-infra#145 (OpenBao certificate issued by a CA, with a reload after each issuance), with the runbook in #904 and the unseal helper text in dmfdeploy/dmf-env#39. Closed by hand because a cross-repo closing reference does not close the issue on merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    component:dmf-infradmf-infra component repoplatform-debtReal work, orthogonal to the product surface. Unscheduled by design: no milestoneworkstream:identity-secretsAuth, OpenBao, secrets work

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions