Skip to content

Commit e509682

Browse files
yehloyehlo
authored
Basee2e (#122)
* feat: add chainsaw e2e test infrastructure Add chainsaw test framework with git clean/smudge filters for values sanitization, interactive setup script, and Makefile targets for running tests against fresh and pre-existing StorageGrid environments. Includes lifecycle, secret-rollover, quota-update, class-change, deletion-protection, and invalid-class test scenarios. * feat: add BoundTenant display field to S3TenantAccount status Add status.boundTenant (namespace/name) and update the kubectl printcolumn to source from status instead of spec.s3TenantRef.name. * refactor: extract bindTenant/unbindTenant helpers, fix deletion policy bugs - Extract bindTenant() and unbindTenant() from inline logic - Fix stuck-Deleting accounts: keep DeletionTimestamp set on delete failure and add recovery path for accounts with DeletionTimestampReached condition but no k8s finalizer deletion - Fix reconcileS3TenantClass error handling: return error and set condition on failure instead of silently continuing - Remove stale owner-reference logic from doReconcile * fix: guard against empty tenant ID from StorageGrid API Return an error if backend returns an empty tenant ID after creation. * fix: use Status().Patch instead of Status().Update in all controllers Replace all deferred Status().Update() calls with Status().Patch() using client.MergeFrom(). Patch applies a JSON merge diff and does not use optimistic concurrency, so it cannot conflict when another controller modifies metadata (annotations, labels) on the same object while reconciliation is in-flight. In S3TenantAccount reconcileCreate(), add an immediate Status().Patch() right after tenant creation to persist the Created condition and TenantID before returning. This prevents a race where the S3Tenant controller bumps resourceVersion during CreateTenant(), causing the deferred status write to conflict and the Created gate to never persist — leading to duplicate tenant creation and permanent 401s. * feat: block StorageGrid deletion while S3TenantAccounts reference it Add DELETE verb to the StorageGrid validating webhook. ValidateDelete now lists all S3TenantAccounts and rejects deletion if any still reference the StorageGrid being deleted. * docs: add CONTRIBUTING.md * chore: remove unused golang.org/x/crypto dependency
1 parent f0f03a9 commit e509682

30 files changed

Lines changed: 1320 additions & 77 deletions

‎.gitattributes‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Chainsaw values files: sanitize on commit via clean filter.
2+
# Run `make setup-git-filters` to activate.
3+
test/e2e/chainsaw/values.yaml filter=chainsaw-values
4+
test/e2e/chainsaw/values-existing.yaml filter=chainsaw-values

‎CONTRIBUTING.md‎

Lines changed: 271 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,271 @@
1+
# Contributing to StorageGrid Operator
2+
3+
This guide covers the development workflow, tooling, and testing setup for the StorageGrid Operator.
4+
5+
## Prerequisites
6+
7+
| Tool | Version | Purpose |
8+
| ------- | ------- | ---------------------------- |
9+
| Go | 1.25+ | Build and test |
10+
| Docker | — | Container image builds |
11+
| kubectl | — | Cluster interaction |
12+
| yq | 4.x | Chainsaw values sanitization |
13+
| Make | — | Build automation |
14+
15+
Most development tools (controller-gen, kustomize, golangci-lint, chainsaw, etc.) are downloaded automatically into `bin/` by Make targets.
16+
17+
## Project Structure
18+
19+
```text
20+
api/v1alpha1/ # CRD type definitions (StorageGrid, S3Tenant, S3Bucket, etc.)
21+
cmd/ # Operator entrypoint
22+
internal/
23+
controller/ # Reconciliation logic for all CRDs
24+
webhook/ # Admission webhooks
25+
pkg/
26+
grid/ # StorageGrid API client and business logic
27+
kube/ # Reusable Kubernetes utility functions
28+
s3/ # S3 client operations
29+
config/ # Kustomize manifests (CRDs, RBAC, deployment, webhooks)
30+
docs/architecture/ # Architecture docs (controller patterns, separation of concerns, etc.)
31+
test/e2e/ # End-to-end tests (Kind-based and Chainsaw)
32+
hack/ # Development scripts
33+
```
34+
35+
See [docs/architecture/](docs/architecture/) for detailed design documentation.
36+
37+
## Quick Start
38+
39+
```bash
40+
# Install all tool dependencies
41+
make kustomize controller-gen envtest golangci-lint chainsaw
42+
43+
# Generate CRDs and deepcopy methods
44+
make manifests generate
45+
46+
# Run unit tests
47+
make test
48+
49+
# Build the operator binary
50+
make build
51+
52+
# Build and push the Docker image
53+
make docker-build-and-push
54+
```
55+
56+
## Development Workflow
57+
58+
### Code Generation
59+
60+
After modifying any `*_types.go` file under `api/v1alpha1/`:
61+
62+
```bash
63+
make manifests generate
64+
```
65+
66+
This regenerates:
67+
68+
- CRD manifests in `config/crd/bases/`
69+
- RBAC role in `config/rbac/role.yaml`
70+
- DeepCopy methods in `zz_generated.deepcopy.go`
71+
72+
### Running Locally
73+
74+
```bash
75+
# Install CRDs into the current cluster
76+
make install
77+
78+
# Run the operator against the current kubeconfig
79+
make run
80+
```
81+
82+
### Linting
83+
84+
```bash
85+
make lint # Check for issues
86+
make lint-fix # Auto-fix where possible
87+
make lint-config # Verify linter configuration
88+
```
89+
90+
### Building Container Images
91+
92+
The default registry is `bedag/storagegrid-operator`. Most developers won't have push access to it. Override `IMAGE_REGISTRY` to use your own:
93+
94+
```bash
95+
# Build and push to your own registry
96+
make docker-build-and-push IMAGE_REGISTRY=my-registry.example.com/storagegrid-operator
97+
98+
# Build only (no push)
99+
make docker-build IMAGE_REGISTRY=my-registry.example.com/storagegrid-operator
100+
```
101+
102+
The image is automatically tagged with `:latest`, `:$GIT_COMMIT`, and `:$GIT_BRANCH` or `:$GIT_TAG`.
103+
104+
When deploying to a cluster, set `IMG` to match:
105+
106+
```bash
107+
make deploy IMG=my-registry.example.com/storagegrid-operator:latest
108+
```
109+
110+
## Testing
111+
112+
### Unit Tests
113+
114+
```bash
115+
make test
116+
```
117+
118+
Uses `envtest` for controller tests with a real API server but no real cluster.
119+
120+
### End-to-End Tests (Chainsaw)
121+
122+
The project uses [Kyverno Chainsaw](https://kyverno.github.io/chainsaw/) for declarative e2e tests. Tests are located under `test/e2e/chainsaw/`.
123+
124+
#### Initial Setup
125+
126+
1. **Configure git filters** (one-time, prevents committing real cluster values):
127+
128+
```bash
129+
make setup-git-filters
130+
```
131+
132+
This registers a git clean filter that automatically sanitizes `values.yaml` and `values-existing.yaml` when staging. Your local copies keep real values; committed versions contain `REPLACE_ME` placeholders. Requires `yq`.
133+
134+
2. **Configure test values** — values are prompted interactively the first time you run a test target. You can also edit directly:
135+
136+
- `test/e2e/chainsaw/values.yaml` — for fresh infrastructure
137+
- `test/e2e/chainsaw/values-existing.yaml` — for pre-existing StorageGrid with namespace prefix
138+
139+
Key fields:
140+
141+
| Field | Description |
142+
| --------------------------- | ------------------------------------------------------ |
143+
| `namespacePrefix` | Prefix for test namespaces (empty for no prefix) |
144+
| `storageGrid.name` | Name of the StorageGrid CR in the cluster |
145+
| `tenantClass.name` | Name of the S3TenantClass CR |
146+
| `alternateTenantClass.name` | Second S3TenantClass for class-change tests (optional) |
147+
148+
#### Running Tests
149+
150+
```bash
151+
# All chainsaw tests — fresh infrastructure
152+
make test-chainsaw
153+
154+
# All chainsaw tests — existing StorageGrid
155+
make test-chainsaw-existing
156+
157+
# S3Tenant tests only — fresh infrastructure
158+
make e2e-s3tnt
159+
160+
# S3Tenant tests only — existing StorageGrid
161+
make e2e-s3tnt-existing
162+
```
163+
164+
If your values file still contains `REPLACE_ME` placeholders, the Make target will launch an interactive prompt to configure them before running tests.
165+
166+
Pass extra flags to Chainsaw via `CHAINSAW_ARGS`:
167+
168+
```bash
169+
# Pause on failure for interactive debugging
170+
make e2e-s3tnt-existing CHAINSAW_ARGS="--pause-on-failure"
171+
172+
# Or export for the whole session
173+
export CHAINSAW_ARGS="--pause-on-failure"
174+
make e2e-s3tnt-existing
175+
```
176+
177+
#### Test Structure
178+
179+
```text
180+
test/e2e/chainsaw/
181+
.chainsaw.yaml # Config for fresh infrastructure
182+
.chainsaw-existing.yaml # Config with namespace prefix support
183+
values.yaml # Test values (git-sanitized)
184+
values-existing.yaml # Test values for existing infra (git-sanitized)
185+
s3tenant/
186+
_step-templates/ # Reusable step templates
187+
verify-account-set-delete-policy.yaml
188+
lifecycle/ # Full create → update → delete cycle
189+
deletion-protection/ # Annotation-based deletion protection
190+
... other test scenarios
191+
```
192+
193+
#### Writing Tests
194+
195+
Each test is a `chainsaw-test.yaml` in its own directory. Key conventions:
196+
197+
- **Top-level bindings**: Define `tenantName` (and any other test-scoped names) in `spec.bindings` and reference with `($tenantName)` in YAML resources or `$tenantName` in scripts.
198+
- **Values references**: Use `($values.storageGrid.name)`, `($values.tenantClass.name)`, etc. for cluster-specific values.
199+
- **Step templates**: Reuse shared logic via `use.template` referencing files in `_step-templates/`.
200+
- **Cleanup**: If a test creates resources with deletion protection or other guards, add a `cleanup` block to remove the guard before Chainsaw's auto-cleanup runs.
201+
- **Scripts vs native operations**: Prefer native Chainsaw operations (`assert`, `patch`, `delete` with `expect`) over scripts. Use scripts only when there's no native equivalent (e.g., capturing secret values, polling for deletion).
202+
203+
Example binding pattern:
204+
205+
```yaml
206+
spec:
207+
bindings:
208+
- name: tenantName
209+
value: my-test
210+
steps:
211+
- name: create
212+
try:
213+
- create:
214+
resource:
215+
apiVersion: s3.bedag.ch/v1alpha1
216+
kind: S3Tenant
217+
metadata:
218+
name: ($tenantName) # JMESPath — for YAML resources
219+
- name: check-secret
220+
try:
221+
- script:
222+
content: |
223+
kubectl get secret $tenantName-admin-credentials -n $NAMESPACE # env var — for scripts
224+
```
225+
226+
#### Chainsaw Configuration
227+
228+
Two configuration files support different environments:
229+
230+
| File | Use Case |
231+
| ------------------------- | ------------------------------------------------------------------------------------------- |
232+
| `.chainsaw.yaml` | Fresh infrastructure, no namespace prefix |
233+
| `.chainsaw-existing.yaml` | Existing cluster with namespace prefix (`join('-', [$values.namespacePrefix, $namespace])`) |
234+
235+
Timeouts: apply 30s, assert 2m, delete 2m, cleanup 2m. Tests use `failFast` mode.
236+
237+
## Git Filters for Values Sanitization
238+
239+
The chainsaw values files contain environment-specific names that should not be committed. A git clean filter handles this automatically:
240+
241+
```test
242+
Working copy (real values) ──git add──▶ Clean filter (yq) ──▶ Staged with REPLACE_ME
243+
```
244+
245+
**How it works:**
246+
247+
1. `.gitattributes` assigns the `chainsaw-values` filter to both values files
248+
2. `make setup-git-filters` registers the filter in your local `.git/config`
249+
3. On `git add`, `hack/sanitize-chainsaw-values.sh` replaces all values with `REPLACE_ME`
250+
4. Your local files are never modified — only the staged version is sanitized
251+
252+
**Setup:** `make setup-git-filters` (required once per clone)
253+
254+
**Requires:** `yq` — if not installed, the filter exits with an error and the commit is blocked.
255+
256+
## Make Reference
257+
258+
Run `make help` for the full list. Key targets:
259+
260+
| Target | Description |
261+
| ---------------------------- | ------------------------------------------------------ |
262+
| `make manifests generate` | Regenerate CRDs, RBAC, and DeepCopy after type changes |
263+
| `make test` | Unit tests with envtest |
264+
| `make lint` | Lint with golangci-lint |
265+
| `make build` | Build operator binary |
266+
| `make docker-build-and-push` | Build and push container image |
267+
| `make install` | Install CRDs into cluster |
268+
| `make run` | Run operator locally against current kubeconfig |
269+
| `make deploy` / `undeploy` | Deploy/remove operator in cluster |
270+
| `make setup-git-filters` | Configure git clean filter for values sanitization |
271+
| `make e2e-s3tnt-existing` | Run S3Tenant e2e tests against existing StorageGrid |

‎Makefile‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,48 @@ test-e2e: setup-test-e2e manifests generate fmt vet ## Run the e2e tests. Expect
104104
cleanup-test-e2e: ## Tear down the Kind cluster used for e2e tests
105105
@$(KIND) delete cluster --name $(KIND_CLUSTER)
106106

107+
CHAINSAW_VALUES ?= test/e2e/chainsaw/values.yaml
108+
CHAINSAW_VALUES_EXISTING ?= test/e2e/chainsaw/values-existing.yaml
109+
CHAINSAW_CONFIG ?= test/e2e/chainsaw/.chainsaw.yaml
110+
CHAINSAW_CONFIG_EXISTING ?= test/e2e/chainsaw/.chainsaw-existing.yaml
111+
CHAINSAW_ARGS ?=
112+
113+
.PHONY: setup-git-filters
114+
setup-git-filters: ## Configure git clean/smudge filters for chainsaw values sanitization.
115+
@git config filter.chainsaw-values.clean hack/sanitize-chainsaw-values.sh
116+
@git config filter.chainsaw-values.smudge cat
117+
@echo "Git filter 'chainsaw-values' configured. Staged values files will be sanitized automatically."
118+
119+
# check-chainsaw-values checks a values file for REPLACE_ME placeholders and
120+
# launches the interactive setup script if any are found.
121+
# Usage: $(call check-chainsaw-values,<values-file>)
122+
define check-chainsaw-values
123+
@if command -v yq &>/dev/null && yq eval '.. | select(. == "REPLACE_ME") | path | join(".")' $(1) 2>/dev/null | grep -q .; then \
124+
echo "Values file $(1) contains REPLACE_ME placeholders."; \
125+
hack/setup-chainsaw-values.sh $(1); \
126+
fi
127+
endef
128+
129+
.PHONY: test-chainsaw
130+
test-chainsaw: chainsaw ## Run Chainsaw e2e tests with fresh infrastructure.
131+
$(call check-chainsaw-values,$(CHAINSAW_VALUES))
132+
$(CHAINSAW) test test/e2e/chainsaw/ --config $(CHAINSAW_CONFIG) --values $(CHAINSAW_VALUES) $(CHAINSAW_ARGS)
133+
134+
.PHONY: test-chainsaw-existing
135+
test-chainsaw-existing: chainsaw ## Run Chainsaw e2e tests against pre-existing StorageGrid.
136+
$(call check-chainsaw-values,$(CHAINSAW_VALUES_EXISTING))
137+
$(CHAINSAW) test test/e2e/chainsaw/ --config $(CHAINSAW_CONFIG_EXISTING) --values $(CHAINSAW_VALUES_EXISTING) $(CHAINSAW_ARGS)
138+
139+
.PHONY: e2e-s3tnt
140+
e2e-s3tnt: chainsaw ## Run S3Tenant e2e tests.
141+
$(call check-chainsaw-values,$(CHAINSAW_VALUES))
142+
$(CHAINSAW) test test/e2e/chainsaw/s3tenant/ --config $(CHAINSAW_CONFIG) --values $(CHAINSAW_VALUES) $(CHAINSAW_ARGS)
143+
144+
.PHONY: e2e-s3tnt-existing
145+
e2e-s3tnt-existing: chainsaw ## Run S3Tenant e2e tests against pre-existing StorageGrid.
146+
$(call check-chainsaw-values,$(CHAINSAW_VALUES_EXISTING))
147+
$(CHAINSAW) test test/e2e/chainsaw/s3tenant/ --config $(CHAINSAW_CONFIG_EXISTING) --values $(CHAINSAW_VALUES_EXISTING) $(CHAINSAW_ARGS)
148+
107149
.PHONY: lint
108150
lint: golangci-lint ## Run golangci-lint linter
109151
$(GOLANGCI_LINT) run
@@ -211,6 +253,7 @@ KUSTOMIZE ?= $(LOCALBIN)/kustomize
211253
CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen
212254
ENVTEST ?= $(LOCALBIN)/setup-envtest
213255
GOLANGCI_LINT = $(LOCALBIN)/golangci-lint
256+
CHAINSAW ?= $(LOCALBIN)/chainsaw
214257

215258
## Tool Versions
216259
KUSTOMIZE_VERSION ?= v5.7.1
@@ -220,6 +263,7 @@ ENVTEST_VERSION ?= $(shell go list -m -f "{{ .Version }}" sigs.k8s.io/controller
220263
#ENVTEST_K8S_VERSION is the version of Kubernetes to use for setting up ENVTEST binaries (i.e. 1.31)
221264
ENVTEST_K8S_VERSION ?= $(shell go list -m -f "{{ .Version }}" k8s.io/api | awk -F'[v.]' '{printf "1.%d", $$3}')
222265
GOLANGCI_LINT_VERSION ?= v2.11.3
266+
CHAINSAW_VERSION ?= v0.2.14
223267

224268
.PHONY: kustomize
225269
kustomize: $(KUSTOMIZE) ## Download kustomize locally if necessary.
@@ -249,6 +293,19 @@ golangci-lint: $(GOLANGCI_LINT) ## Download golangci-lint locally if necessary.
249293
$(GOLANGCI_LINT): $(LOCALBIN)
250294
$(call go-install-tool,$(GOLANGCI_LINT),github.com/golangci/golangci-lint/v2/cmd/golangci-lint,$(GOLANGCI_LINT_VERSION))
251295

296+
.PHONY: chainsaw
297+
chainsaw: $(CHAINSAW) ## Download chainsaw locally if necessary.
298+
$(CHAINSAW): $(LOCALBIN)
299+
@[ -f "$(CHAINSAW)-$(CHAINSAW_VERSION)" ] || { \
300+
set -e; \
301+
OS=$$(uname -s | tr '[:upper:]' '[:lower:]') ;\
302+
ARCH=$$(uname -m | sed 's/x86_64/amd64/' | sed 's/aarch64/arm64/') ;\
303+
echo "Downloading chainsaw $(CHAINSAW_VERSION) for $${OS}/$${ARCH}" ;\
304+
curl -fsSL "https://github.com/kyverno/chainsaw/releases/download/$(CHAINSAW_VERSION)/chainsaw_$${OS}_$${ARCH}.tar.gz" | tar xz -C $(LOCALBIN) chainsaw ;\
305+
mv $(LOCALBIN)/chainsaw $(CHAINSAW)-$(CHAINSAW_VERSION) ;\
306+
}
307+
@ln -sf $(CHAINSAW)-$(CHAINSAW_VERSION) $(CHAINSAW)
308+
252309
# go-install-tool will 'go install' any package with custom target and name of binary, if it doesn't exist
253310
# $1 - target path with name of binary
254311
# $2 - package url which can be installed

‎api/v1alpha1/s3tenantaccount_types.go‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,10 @@ type S3TenantAccountStatus struct {
9797
// +optional
9898
S3TenantRef *corev1.ObjectReference `json:"s3TenantRef,omitempty"`
9999

100+
// BoundTenant is the namespace/name of the S3Tenant bound to this account (display field).
101+
// +optional
102+
BoundTenant string `json:"boundTenant,omitempty"`
103+
100104
// ObservedTenantBackendName is the actual name in the backend.
101105
// +optional
102106
// +kubebuilder:default=""
@@ -145,7 +149,7 @@ type S3TenantAccountStatus struct {
145149

146150
// +kubebuilder:object:root=true
147151
// +kubebuilder:subresource:status
148-
// +kubebuilder:printcolumn:name="S3Tenant",type="string",JSONPath=".spec.s3TenantRef.name",description="The s3 Tenant this account is bound to"
152+
// +kubebuilder:printcolumn:name="S3Tenant",type="string",JSONPath=".status.boundTenant",description="The s3 Tenant this account is bound to"
149153
// +kubebuilder:printcolumn:name="TenantBackendName",type="string",JSONPath=".status.observedTenantBackendName",description="The name of the tenant in the backend"
150154
// +kubebuilder:printcolumn:name="StorageGrid",type="string",JSONPath=".spec.storageGridRef.name",description="The StorageGrid this tenant account belongs to"
151155
// +kubebuilder:printcolumn:name="Capacity",type="string",JSONPath=".status.quota.limit",description="Configured capacity of the tenant"

0 commit comments

Comments
 (0)