Support externally managed TLS certificates for all components

[lhotari]

Fixes #721

Motivation

#721 asks how to use publicly trusted certificates (issued by an ACME issuer through cert-manager) with the chart, and reports several problems when trying to provide the certificates externally:

• The chart always adds the Kubernetes service names to the certificates it creates, and a public CA can't issue certificates for them.
• Only the proxy and function worker certificates can be provided externally (createCert: false). The broker, bookie, ZooKeeper, autorecovery, toolset and standalone certificates are always created by the chart.
• The CA Secret (certs.issuers.ca.secretName) is required even when no certificates are created by the chart, and its purpose isn't documented.
• ZooKeeper fails with InvalidKeySpecException: Inappropriate key specification: Unable to decode key with an externally created tls-combined.pem.

The Kubernetes service names can't be removed from the certificates: the components connect to each other using these names and verify that the certificate matches. In addition, the components present their certificates as TLS client certificates (the brokers to the bookies, and all ZooKeeper clients to ZooKeeper), and public CAs are removing the clientAuth extended key usage from the certificates they issue (see the ASF advisory and the users@/dev@ thread). Publicly trusted certificates therefore only work for the proxy, which uses its certificate as a server certificate.

Modifications

• Add createCert (default true) to tls.broker, tls.bookie, tls.zookeeper, tls.autorecovery, tls.toolset and tls.standalone, as tls.proxy and tls.function_worker already have, so that all certificate Secrets can be provided externally. Only an explicit createCert: false (bool or string) disables a certificate, so values without the key, for example reused values from an earlier release with helm upgrade --reuse-values, keep creating it.
• Don't mount the unused tls-combined.pem key from the proxy Secret. Nothing reads /pulsar/certs/proxy/tls-combined.pem, and the requirement prevented using a plain publicly trusted certificate Secret for the proxy when ZooKeeper TLS is enabled.
• Document in the ## TLS comments of values.yaml:
  • why publicly trusted certificates can't be used for the connections between the components, and the recommended setup: a public certificate for the proxy only (tls.proxy.createCert: false with your own cert-manager Certificate), and a private CA for the other components (certs.internal_issuer, or a cert-manager issuer backed by a private CA such as OpenBao, Vault or step-ca through certs.issuers.ca);
  • that direct client access to the brokers without the proxy requires advertisedListeners, which the chart doesn't support yet;
  • the Secret keys required for externally managed certificates (tls.crt, tls.key and, with ZooKeeper TLS, tls-combined.pem), that the private key must be PKCS#8 (the cause of the ZooKeeper error in TLS Configuration with Public Certificates #721), and the matching cert-manager settings;
  • which Kubernetes names the certificates must include, and the purpose of certs.issuers.ca.secretName.
• Explain in the certs.issuers.ca.secretName error message what the Secret must contain.
• Add the change to the 4.8.0 upgrade notes in README.md.

Verifying this change

• helm lint passes, and the chart renders with all .ci/clusters/*.yaml and .ci/templates-all-values*.yaml values files. .ci/test-certificate-san-modes.sh passes.

• With TLS enabled for all components (default values otherwise), the rendered output is unchanged apart from the removed tls-combined.pem item of the proxy certificate volume, both for the regular and the standalone deployment.

• With createCert: false for all components, certs.internal_issuer.enabled: false and only certs.issuers.ca.secretName set, the chart renders without any Certificate or Issuer resources.

• --set-string tls.broker.createCert=false skips only the broker certificate, and values without the createCert key still create the certificate.

• Make sure that the change passes the CI checks.