-
Notifications
You must be signed in to change notification settings - Fork 21
ESO-447: Expand Vault e2e for TLS trustedCABundle and ExternalSecret templating #169
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,7 +19,7 @@ make test-e2e E2E_GINKGO_LABEL_FILTER="" | |
| | Key | Values | Meaning | | ||
| |-----|--------|---------| | ||
| | `Platform` | `AWS`, `GCP`, `Generic` | Cluster or portability requirement | | ||
| | `Provider` | `AWS`, `Bitwarden` | External secret backend integration | | ||
| | `Provider` | `AWS`, `Bitwarden`, `Vault` | External secret backend integration | | ||
| | `API` | `Bitwarden` | Direct HTTP tests against bitwarden-sdk-server | | ||
| | `Feature` | see below | Optional capability or functional area | | ||
|
|
||
|
|
@@ -34,7 +34,8 @@ make test-e2e E2E_GINKGO_LABEL_FILTER="" | |
| | `CustomLabels` | Custom and managed label lifecycle | | ||
| | `NetworkPolicy` | Static and custom network policy naming | | ||
| | `Proxy` | Proxy egress network policy (requires cluster-wide OpenShift proxy) | | ||
| | `TrustedCABundle` | trustedCABundle ConfigMap mounting and validation | | ||
| | `TrustedCABundle` | trustedCABundle ConfigMap mounting/validation and Vault TLS failure→recovery | | ||
| | `ExternalSecretsTemplating` | ExternalSecret template merge (Kubernetes + Vault → dockerconfigjson) | | ||
| | `Upgrade` | Post-upgrade migration checks (temporary) | | ||
|
|
||
| ## Default filter | ||
|
|
@@ -51,10 +52,6 @@ This runs portable tests plus AWS provider tests, and **API:Bitwarden** health/a | |
|
|
||
| ### Secrets you provision | ||
|
|
||
| ```bash | ||
| hack/e2e-setup-secrets.sh setup | ||
| ``` | ||
|
|
||
| | Secret | Namespace | Keys | Required when filter includes | | ||
| |--------|-----------|------|-------------------------------| | ||
| | `aws-creds` | `kube-system` | `aws_access_key_id`, `aws_secret_access_key` | `Provider:AWS` (`Platform:AWS` or `Platform:GCP`) | | ||
|
|
@@ -73,6 +70,7 @@ The bitwarden-sdk-server plugin uses **`bitwarden-tls-certs`** (TLS materials fo | |
| | Requirement | Required when filter includes | | ||
| |-------------|-------------------------------| | ||
| | Cluster-wide OpenShift proxy (`proxy.config.openshift.io/cluster`) | `Feature:Proxy` | | ||
| | OpenShift with `redhat-operators` catalog (`openshift-marketplace`) | Main e2e Describe (`e2e_test.go`) — root `BeforeAll` installs Red Hat cert-manager Operator via OLM if not already present | | ||
|
|
||
| If a prerequisite is missing, the affected spec **fails** with a message pointing here — it does not skip. | ||
|
|
||
|
|
@@ -86,9 +84,11 @@ If a prerequisite is missing, the affected spec **fails** with a message pointin | |
| | `Platform:GCP && Provider:AWS` | GCP cluster using AWS Secrets Manager | | ||
| | `Provider:AWS` | Any AWS Secrets Manager integration (`Platform:AWS` or `Platform:GCP`) | | ||
| | `Provider:Bitwarden` | Bitwarden provider sync and API Secrets API | | ||
| | `Provider:Vault` | Vault HTTPS + trustedCABundle failure/recovery (uses suite-installed Red Hat cert-manager Operator) | | ||
| | `API:Bitwarden` | bitwarden-sdk-server HTTP API (deploys plugin + `bitwarden-tls-certs` automatically) | | ||
| | `API:Bitwarden \|\| Provider:Bitwarden` | All Bitwarden HTTP and provider tests (requires `bitwarden-creds` for Secrets API / provider sync) | | ||
| | `Feature:TrustedCABundle` | Trusted CA bundle suite | | ||
| | `Feature:TrustedCABundle` | Trusted CA bundle suite (mount/validation + Vault TLS path when combined with `Provider:Vault`) | | ||
| | `Feature:ExternalSecretsTemplating` | ExternalSecret templating merge (Kubernetes + Vault dockerconfig) | | ||
| | `Feature:Proxy` | Proxy egress network policy | | ||
| | `Feature:Upgrade` | Post-upgrade network policy migration check | | ||
| | `Feature:NetworkPolicy` | Static and custom network policy naming | | ||
|
|
@@ -110,6 +110,9 @@ make test-e2e E2E_GINKGO_LABEL_FILTER="Feature:NetworkPolicy || Feature:Proxy" | |
|
|
||
| # AWS integration only (any platform label that uses AWS SM) | ||
| make test-e2e E2E_GINKGO_LABEL_FILTER="Provider:AWS" | ||
|
|
||
| # Vault TLS + trustedCABundle (uses cert-manager from root BeforeAll) | ||
| make test-e2e E2E_GINKGO_LABEL_FILTER="Provider:Vault" | ||
| ``` | ||
|
|
||
| ## Specs by label | ||
|
|
@@ -147,10 +150,29 @@ File: `trusted_ca_bundle_test.go` | |
|
|
||
| | Feature | Describe | | ||
| |---------|----------| | ||
| | `TrustedCABundle` | Trusted CA Bundle | | ||
| | `TrustedCABundle` | Trusted CA Bundle (mount / `SSL_CERT_DIR` / Degraded / watch-label restore) | | ||
|
|
||
| The **Custom Network Policy Naming** spec adds a dummy egress port to `ExternalSecretsConfig` (if not already present — entries cannot be removed due to CEL immutability), verifies the operator creates `eso-user-e2e-test-custom-np` in the operand namespace (`external-secrets`), and leaves the CR entry in place. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win Move the Custom Network Policy paragraph to its owning section. This paragraph appears immediately under 🤖 Prompt for AI Agents |
||
|
|
||
| ### `Provider:Vault` + `Feature:TrustedCABundle` | ||
|
|
||
| File: `e2e_test.go` — **Vault Secret Manager** | ||
|
|
||
| - **BeforeAll:** cert-manager CA (`isCA`) + server Certificate for Vault HTTPS; Vault deploy/init | ||
| - Uses Red Hat cert-manager Operator installed by the root `BeforeAll` of the main e2e Describe (`ensureCertManagerOperatorReady`) | ||
| - Asserts SecretStore `Ready=False` / `InvalidProviderConfig` when `trustedCABundle` points at a valid but non-matching CA (`vault-e2e-sample-ca`), then Ready after switching to the Vault CA ConfigMap (`vault-server-ca` from the isCA Certificate) | ||
| - After trustedCABundle switch: SecretStore Ready → PushSecret → ExternalSecret → verify synced Secret | ||
| - Excluded from the default label filter (longer Vault HTTPS setup) | ||
| - Does not uninstall cert-manager on teardown (shared for other suites) | ||
|
|
||
| ### `Provider:Vault` + `Feature:ExternalSecretsTemplating` | ||
|
|
||
| File: `e2e_test.go` — **Vault Secret Manager** (same Context) | ||
|
|
||
| - Creates a namespaced Kubernetes SecretStore (`kubernetes-backend`) with dedicated `eso-secret-reader` SA/Role/RoleBinding (no existing e2e Kubernetes-provider fixture) | ||
| - Pushes a dockerconfigjson Secret to Vault, reads another from Kubernetes, merges both via ExternalSecret `target.template` into `merged-registry-pull-secret` | ||
| - Asserts the target Secret is `kubernetes.io/dockerconfigjson` and contains both registry auth hosts | ||
|
|
||
| ### `Provider:Bitwarden` | ||
|
|
||
| File: `bitwarden_es_test.go` — **Bitwarden Provider** | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,184 @@ | ||
| //go:build e2e | ||
| // +build e2e | ||
|
|
||
| /* | ||
| Copyright 2026. | ||
| Licensed under the Apache License, Version 2.0 (the "License"); | ||
| you may not use this file except in compliance with the License. | ||
| You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, software | ||
| distributed under the License is distributed on an "AS IS" BASIS, | ||
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
| */ | ||
|
|
||
| package e2e | ||
|
|
||
| import ( | ||
| "context" | ||
| "fmt" | ||
| "strings" | ||
| "time" | ||
|
|
||
| corev1 "k8s.io/api/core/v1" | ||
| metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" | ||
| "k8s.io/apimachinery/pkg/runtime/schema" | ||
| "k8s.io/apimachinery/pkg/util/wait" | ||
| "k8s.io/client-go/dynamic" | ||
| "k8s.io/client-go/kubernetes" | ||
|
|
||
| . "github.com/onsi/ginkgo/v2" | ||
|
|
||
| "github.com/openshift/external-secrets-operator/test/utils" | ||
| ) | ||
|
|
||
| const ( | ||
| certManagerOperatorNamespace = "cert-manager-operator" | ||
| certManagerOperandNamespace = "cert-manager" | ||
| certManagerOperatorManifest = "testdata/cert-manager/operator.yaml" | ||
|
|
||
| certManagerOperatorPodPrefix = "cert-manager-operator-controller-manager-" | ||
| certManagerCAInjectorPodPrefix = "cert-manager-cainjector-" | ||
| certManagerWebhookPodPrefix = "cert-manager-webhook-" | ||
|
|
||
| certManagerInstallTimeout = 10 * time.Minute | ||
| ) | ||
|
|
||
| var certificateGVR = schema.GroupVersionResource{ | ||
| Group: "cert-manager.io", | ||
| Version: "v1", | ||
| Resource: "certificates", | ||
| } | ||
|
|
||
| // ensureCertManagerOperatorReady installs the Red Hat cert-manager Operator via OLM if needed, | ||
| // then waits until the operator, operand pods, and Certificate API are ready. | ||
| // Idempotent: skips install when the Certificate API is already usable and operands are Ready. | ||
| func ensureCertManagerOperatorReady(ctx context.Context, clientset *kubernetes.Clientset, dynamicClient dynamic.Interface) error { | ||
| By("Ensuring Red Hat cert-manager Operator is ready") | ||
|
|
||
| if err := certManagerCertificateCRDInstalled(ctx, dynamicClient); err == nil { | ||
| if err := waitForCertManagerOperandPods(ctx, clientset, 2*time.Minute); err == nil { | ||
| By("cert-manager Certificate API and operand pods already ready") | ||
| return nil | ||
| } | ||
| } | ||
|
|
||
| By("Applying cert-manager Operator OLM manifests (Namespace, OperatorGroup, Subscription)") | ||
| if err := utils.ApplyManifestFromReader(ctx, dynamicClient, testassets.ReadFile, certManagerOperatorManifest); err != nil { | ||
| return fmt.Errorf("apply cert-manager operator manifests: %w", err) | ||
| } | ||
|
|
||
| By(fmt.Sprintf("Waiting for cert-manager operator pod in namespace %s", certManagerOperatorNamespace)) | ||
| if err := waitForReadyPodsByNamePrefixes(ctx, clientset, certManagerOperatorNamespace, []string{certManagerOperatorPodPrefix}, certManagerInstallTimeout); err != nil { | ||
| return fmt.Errorf("wait for cert-manager operator pod: %w", err) | ||
| } | ||
|
|
||
| By(fmt.Sprintf("Waiting for cert-manager operand pods in namespace %s", certManagerOperandNamespace)) | ||
| if err := waitForCertManagerOperandPods(ctx, clientset, certManagerInstallTimeout); err != nil { | ||
| return fmt.Errorf("wait for cert-manager operand pods: %w", err) | ||
| } | ||
|
|
||
| By("Waiting for cert-manager Certificate API") | ||
| if err := wait.PollUntilContextTimeout(ctx, 5*time.Second, 2*time.Minute, true, func(ctx context.Context) (bool, error) { | ||
| if err := certManagerCertificateCRDInstalled(ctx, dynamicClient); err != nil { | ||
| return false, nil | ||
| } | ||
| return true, nil | ||
| }); err != nil { | ||
| return fmt.Errorf("cert-manager Certificate API not available after install: %w", err) | ||
| } | ||
|
|
||
| By("Red Hat cert-manager Operator is ready") | ||
| return nil | ||
| } | ||
|
|
||
| // certManagerCertificateCRDInstalled returns nil when the cert-manager Certificate API is available. | ||
| func certManagerCertificateCRDInstalled(ctx context.Context, dynamicClient dynamic.Interface) error { | ||
| _, err := dynamicClient.Resource(certificateGVR).Namespace("default").List(ctx, metav1.ListOptions{Limit: 1}) | ||
| if err != nil { | ||
| return fmt.Errorf("cert-manager Certificate API unavailable: %w", err) | ||
| } | ||
| return nil | ||
| } | ||
|
|
||
| // waitForCertManagerOperandPods waits until controller, cainjector, and webhook pods are Ready. | ||
| func waitForCertManagerOperandPods(ctx context.Context, clientset kubernetes.Interface, timeout time.Duration) error { | ||
| return wait.PollUntilContextTimeout(ctx, 5*time.Second, timeout, true, func(ctx context.Context) (bool, error) { | ||
| pods, err := clientset.CoreV1().Pods(certManagerOperandNamespace).List(ctx, metav1.ListOptions{}) | ||
| if err != nil { | ||
| if strings.Contains(err.Error(), "not found") { | ||
| return false, nil | ||
| } | ||
| return false, err | ||
| } | ||
|
|
||
| var hasController, hasCAInjector, hasWebhook bool | ||
| for i := range pods.Items { | ||
| pod := &pods.Items[i] | ||
| if pod.Status.Phase != corev1.PodRunning || !isPodReadyForCertManager(pod) { | ||
| continue | ||
| } | ||
| name := pod.Name | ||
| switch { | ||
| case strings.HasPrefix(name, certManagerCAInjectorPodPrefix): | ||
| hasCAInjector = true | ||
| case strings.HasPrefix(name, certManagerWebhookPodPrefix): | ||
| hasWebhook = true | ||
| case strings.HasPrefix(name, "cert-manager-") && | ||
| !strings.HasPrefix(name, certManagerCAInjectorPodPrefix) && | ||
| !strings.HasPrefix(name, certManagerWebhookPodPrefix) && | ||
| !strings.Contains(name, "startupapicheck"): | ||
| // Main controller Deployment pods: cert-manager-<replicaset>-<pod> | ||
| hasController = true | ||
| } | ||
| } | ||
| return hasController && hasCAInjector && hasWebhook, nil | ||
| }) | ||
| } | ||
|
|
||
| func waitForReadyPodsByNamePrefixes(ctx context.Context, clientset kubernetes.Interface, namespace string, prefixes []string, timeout time.Duration) error { | ||
| return wait.PollUntilContextTimeout(ctx, 5*time.Second, timeout, true, func(ctx context.Context) (bool, error) { | ||
| pods, err := clientset.CoreV1().Pods(namespace).List(ctx, metav1.ListOptions{}) | ||
| if err != nil { | ||
| return false, err | ||
| } | ||
| matched := make(map[string]bool, len(prefixes)) | ||
| for _, prefix := range prefixes { | ||
| matched[prefix] = false | ||
| } | ||
| for i := range pods.Items { | ||
| pod := &pods.Items[i] | ||
| if pod.Status.Phase != corev1.PodRunning || !isPodReadyForCertManager(pod) { | ||
| continue | ||
| } | ||
| for _, prefix := range prefixes { | ||
| if strings.HasPrefix(pod.Name, prefix) { | ||
| matched[prefix] = true | ||
| } | ||
| } | ||
| } | ||
| for _, ok := range matched { | ||
| if !ok { | ||
| return false, nil | ||
| } | ||
| } | ||
| return true, nil | ||
| }) | ||
| } | ||
|
|
||
| func isPodReadyForCertManager(pod *corev1.Pod) bool { | ||
| ready := map[string]bool{ | ||
| "Ready": false, | ||
| "ContainersReady": false, | ||
| } | ||
| for _, cond := range pod.Status.Conditions { | ||
| if _, ok := ready[string(cond.Type)]; ok && cond.Status == corev1.ConditionTrue { | ||
| ready[string(cond.Type)] = true | ||
| } | ||
| } | ||
| return ready["Ready"] && ready["ContainersReady"] | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Keep the executable default filter aligned with the Vault documentation.
The README and PR objective say
Provider:Vaultis excluded by default, butMakefileLine 217 excludes onlyBitwarden. Vault specs that satisfy the platform predicate can therefore run duringmake test-e2e, unexpectedly requiring cert-manager/OLM prerequisites.Update the Makefile filter to exclude
{Bitwarden, Vault}, or correct this documentation if Vault is intended to run by default.🤖 Prompt for AI Agents