diff --git a/vault/development/skills/architecture/SKILL.md b/vault/development/skills/architecture/SKILL.md new file mode 100644 index 0000000..75040ce --- /dev/null +++ b/vault/development/skills/architecture/SKILL.md @@ -0,0 +1,256 @@ +--- +name: vault-architecture +description: Design and organize code in HashiCorp Vault. Use when designing new features, refactoring code, working with CE/EE splits, making API design decisions, understanding Vault's plugin architecture, or deciding where code should live (api/ vs sdk/ vs vault/). +compatibility: Requires Go 1.22+, access to Vault repository +--- + +# Vault Architecture + +## Repository Structure + +### Public vs Internal + +Only these packages are public (importable externally): +``` +api/ # Vault API client +sdk/ # Plugin SDK +``` + +Everything in `vault/` is internal - never import directly. + +### Core Organization + +``` +vault/ # Core server (INTERNAL) +├── logical/ # Backend interfaces +├── physical/ # Storage backends +└── audit/ # Audit backends + +builtin/ # Built-in plugins +├── logical/ # Secret engines +└── credential/ # Auth methods + +command/ # CLI commands +http/ # HTTP API handlers +``` + +## CE/EE Code Separation + +Use build tags for compile-time separation: + +### File Naming + +``` +feature.go # Shared (CE + EE) +feature_oss.go # Community Edition / Open Source only +feature_ent.go # Enterprise only +feature_test.go # Shared tests +feature_ent_test.go # Enterprise tests only +``` + +### Build Tags + +```go +//go:build !enterprise +// CE-only code + +//go:build enterprise +// EE-only code +``` + +### Pattern: Interface-Based Separation + +```go +// feature.go - shared interface +type FeatureManager interface { + Process(ctx context.Context) error + GetCapabilities() []string +} + +// feature_oss.go +//go:build !enterprise + +func NewFeatureManager() FeatureManager { + return &ossManager{} // Basic implementation +} + +// feature_ent.go +//go:build enterprise + +func NewFeatureManager() FeatureManager { + return &entManager{ // Advanced implementation + replicator: NewReplicator(), + } +} +``` + +**Key rules**: +- Define interface in shared file +- Same function signatures in both files +- Use build tags, not runtime checks +- Test both editions: `make subtest` (CE), `make test` (EE) + +## API Design + +### RESTful Endpoints + +``` +Create: POST /v1/resource +Read: GET /v1/resource/:id +Update: POST /v1/resource/:id +Delete: DELETE /v1/resource/:id +List: LIST /v1/resource # Note: LIST not GET +``` + +### Request/Response Pattern + +```go +func (b *backend) pathEntityCreate( + ctx context.Context, + req *logical.Request, + data *framework.FieldData, +) (*logical.Response, error) { + // Parse + name := data.Get("name").(string) + + // Validate + if err := validateName(name); err != nil { + return logical.ErrorResponse(err.Error()), nil + } + + // Process + entity, err := b.createEntity(ctx, name) + if err != nil { + return nil, err // Internal error + } + + // Return + return &logical.Response{ + Data: map[string]interface{}{ + "id": entity.ID, + }, + }, nil +} +``` + +**Error handling**: +- `logical.ErrorResponse()` for validation errors (user-facing) +- `return nil, err` for internal errors (logged, user sees generic message) + +## Plugin Architecture + +### Backend Interface + +```go +func Factory(ctx context.Context, conf *logical.BackendConfig) (logical.Backend, error) { + b := &backend{} + + b.Backend = &framework.Backend{ + BackendType: logical.TypeLogical, + Paths: []*framework.Path{ + b.pathCreate(), + b.pathRead(), + }, + } + + if err := b.Setup(ctx, conf); err != nil { + return nil, err + } + + return b, nil +} +``` + +## Storage Patterns + +```go +// Write +entry := &logical.StorageEntry{ + Key: "entity/" + id, + Value: marshaledData, +} +req.Storage.Put(ctx, entry) + +// Read +entry, err := req.Storage.Get(ctx, "entity/"+id) + +// Delete +req.Storage.Delete(ctx, "entity/"+id) + +// List +keys, err := req.Storage.List(ctx, "entity/") +``` + +**Key naming**: +``` +entity/ # Single entity +entity//alias/ # Nested +config/ # Configuration +``` + +## Decision Trees + +### Where Does Code Go? + +``` +Is it a public API? +├─ YES → api/ or sdk/ +│ ├─ External consumers need it → api/ +│ └─ Plugin developers need it → sdk/ +└─ NO → CE, EE, or both? + ├─ Both → .go file + ├─ CE only → _oss.go file + └─ EE only → _ent.go file +``` + +**Default rule**: If external projects might need it, put in `api/` or `sdk/`. Otherwise, use `vault/`. + +### When to Split CE/EE? + +Use CE/EE split when: +- Feature exists in both with different implementations +- EE adds significant capabilities +- Single codebase needed + +Don't split when: +- Feature identical in both +- Feature 100% EE-only (just use `_ent.go`) +- Difference is trivial + +## Adding New Components + +### New Secret Engine + +1. Create package in `builtin/logical//` +2. Implement `logical.Backend` interface +3. Define paths using `framework.PathAppend` +4. Add CRUD operations +5. Implement secret revocation +6. Write tests + acceptance tests + +### New Auth Method + +1. Create package in `builtin/credential//` +2. Implement `logical.Backend` interface +3. Define authentication paths +4. Implement token generation +5. Add credential validation +6. Write tests + +## Dependency Management + +```bash +# Add dependency +go get github.com/example/package@latest + +# Update go.mod +make go-mod-tidy + +# Verify tests +make test TEST=./path/to/package +``` + +**Rules**: +- Pin exact versions +- Minimize dependencies (smaller attack surface) +- Run full tests after any dep change diff --git a/vault/development/skills/debugging/SKILL.md b/vault/development/skills/debugging/SKILL.md new file mode 100644 index 0000000..1534126 --- /dev/null +++ b/vault/development/skills/debugging/SKILL.md @@ -0,0 +1,233 @@ +--- +name: vault-debugging +description: Debug build failures, test failures, race conditions, and runtime issues in HashiCorp Vault. Use when troubleshooting compilation errors, test timeouts, nil pointer errors, investigating crashes, or when tests hang with no output. +compatibility: Requires Go 1.22+, make, access to Vault repository +--- + +# Vault Debugging + +## Systematic Workflow + +### 1. Reproduce Consistently + +```bash +# Run failing test multiple times +for i in {1..10}; do + make test TEST=./vault/identity TESTARGS="-run TestFailingTest" +done + +# Check for race conditions +make testrace TEST=./vault/identity TESTARGS="-run TestFailingTest" +``` + +### 2. Isolate the Problem + +```bash +# Run with verbose output +make test TEST=./vault/identity TESTARGS="-v -run TestSpecificTest" + +# Enable debug logging +VAULT_LOG_LEVEL=debug make test TEST=./vault/identity +``` + +### 3. Gather Information + +```bash +# Check build tags +go list -f '{{.GoFiles}}' ./vault/identity +go list -tags enterprise -f '{{.GoFiles}}' ./vault/identity + +# Check dependencies +go list -m all | grep hashicorp +``` + +## Common Error Patterns + +### "undefined: ConstantName" + +**Symptom**: `undefined: EnterpriseThing` + +**Cause**: Build tags not applied, or CE code referencing EE code + +**Fix**: +1. Use `make test` not `go test` +2. Add `//go:build enterprise` to file +3. Move EE code to `_ent.go` file + +**Debug**: +```bash +# Verify which files are compiled +go list -f '{{.GoFiles}}' ./vault/identity +go list -tags enterprise -f '{{.GoFiles}}' ./vault/identity +``` + +### Nil Pointer Dereference + +**Symptom**: `panic: runtime error: invalid memory address` + +**Common causes**: +- Uninitialized maps: `var m map[string]string` (nil map) +- Error interface with nil value +- Missing initialization + +**Fix**: +```go +// Initialize maps +m := make(map[string]string) + +// Check before dereferencing +if obj != nil && obj.Field != nil { + value = obj.Field.Value +} + +// Return concrete errors, not interface variables +if err != nil { + return nil, err // Not: return nil, someInterfaceVar +} +``` + +### Race Conditions + +**Symptom**: `WARNING: DATA RACE` + +**Debug**: +```bash +# Always use race detector +make testrace TEST=./vault/identity + +# Increase iterations +make testrace TEST=./vault/identity TESTARGS="-count=100" +``` + +**Fix**: Add synchronization +```go +type SafeMap struct { + mu sync.RWMutex + data map[string]string +} + +func (m *SafeMap) Get(key string) string { + m.mu.RLock() + defer m.mu.RUnlock() + return m.data[key] +} + +func (m *SafeMap) Set(key, value string) { + m.mu.Lock() + defer m.mu.Unlock() + m.data[key] = value +} +``` + +### Context Deadline Exceeded + +**Symptom**: `context deadline exceeded` in tests + +**Common causes**: +- Test cluster not ready +- Missing `vault.TestWaitActive()` +- Actual timeout + +**Fix**: +```go +cluster := vault.NewTestCluster(t, coreConfig, clusterOptions) +defer cluster.Cleanup() + +core := cluster.Cores[0].Core +vault.TestWaitActive(t, core) // CRITICAL: Add this + +// Now safe to proceed +client := cluster.Cores[0].Client +``` + +## Build Failures + +```bash +# Clean caches +go clean -cache -modcache -testcache + +# Update dependencies +go mod tidy +go mod download + +# Verify version +go version # Should be 1.22+ +``` + +## Debugging Tools + +### Printf Debugging + +```go +log.Printf("DEBUG: Entering function with args: %+v", args) +log.Printf("DEBUG: Variable state: %v", variable) +``` + +### Delve Debugger + +```bash +# Install +go install github.com/go-delve/delve/cmd/dlv@latest + +# Debug test +dlv test ./vault/identity -- -test.run TestSpecificTest + +# Commands +(dlv) break vault/identity.CreateEntity +(dlv) continue +(dlv) print entity +(dlv) next +``` + +### Stack Traces + +```go +import "runtime/debug" + +debug.PrintStack() // Print current stack + +stack := debug.Stack() +log.Printf("Stack:\n%s", stack) +``` + +## Troubleshooting Checklist + +When stuck: + +- [ ] Can reproduce consistently? +- [ ] Ran with `-v` for verbose output? +- [ ] Ran with race detector? +- [ ] Checked build tags? +- [ ] Verified dependencies? +- [ ] Checked for nil pointers? +- [ ] Added debug logging? +- [ ] Isolated minimal reproduction? +- [ ] Checked git diff for recent changes? + +## Long-Running Builds: Don't Assume Hangs + +Building Vault and running tests can show **no output for 1-2 minutes**. This is normal. + +**Observed timings** (macOS Apple Silicon): +- First `make test` prints nothing for ~50-70 seconds during "Cleaning..." and "go generate" +- Focused subset: `make test TESTARGS="-run TestCore_ -v"` completes in ~60 seconds after prep + +**Progress signals**: +``` +"Checking that build is using go version..." +"Using go version ..." +"Cleaning..." ← Can be quiet for 30-90s +"Running go generate..." ← Can be quiet for 30-90s +[burst of test output] +``` + +**Tips**: +```bash +# Wrap with timestamps +date; make test TEST=./vault TESTARGS="-run TestJWT -v"; date + +# Always use -v for steady output stream +make test TEST=./vault TESTARGS="-run TestSpecificTest -v -count=1" +``` + +**Timeouts**: `TEST_TIMEOUT=45m`, `INTEG_TEST_TIMEOUT=120m` — lack of output for several minutes is normal. diff --git a/vault/development/skills/performance/SKILL.md b/vault/development/skills/performance/SKILL.md new file mode 100644 index 0000000..a75676a --- /dev/null +++ b/vault/development/skills/performance/SKILL.md @@ -0,0 +1,237 @@ +--- +name: vault-performance +description: Optimize performance and write benchmarks for HashiCorp Vault. Use when investigating slow operations, reducing allocations, writing benchmarks, profiling CPU/memory usage, or working on scalability improvements. +compatibility: Requires Go 1.22+, go tool pprof +--- + +# Vault Performance + +## Philosophy + +**Measure first**: Don't optimize without data. Profile before optimizing. + +**Optimize what matters**: Focus on hot paths and bottlenecks, not micro-optimizations. + +**Maintain readability**: Clear code > slightly faster code. + +## Benchmarking + +### Writing Benchmarks + +```go +func BenchmarkCreateEntity(b *testing.B) { + // Setup (not timed) + store := setupStore(b) + + b.ResetTimer() // Exclude setup time + + for i := 0; i < b.N; i++ { + entity := &Entity{Name: fmt.Sprintf("entity-%d", i)} + _ = store.CreateEntity(context.Background(), entity) + } +} +``` + +### Running Benchmarks + +```bash +# Run all benchmarks +go test -bench=. ./vault/identity + +# Specific benchmark +go test -bench=BenchmarkCreateEntity ./vault/identity + +# With memory stats +go test -bench=. -benchmem ./vault/identity + +# Compare before/after +go test -bench=. ./vault/identity > old.txt +# Make changes +go test -bench=. ./vault/identity > new.txt +benchcmp old.txt new.txt +``` + +### Interpreting Output + +``` +BenchmarkCreateEntity-8 50000 35420 ns/op 4832 B/op 102 allocs/op + | | | | | + cores iterations ns/op bytes/op allocs/op +``` + +## Profiling + +### CPU Profile + +```bash +# Generate profile +go test -cpuprofile=cpu.prof -bench=. ./vault/identity + +# Analyze +go tool pprof cpu.prof + +# Commands +(pprof) top10 # Top 10 functions +(pprof) list FuncName # Source with annotations +(pprof) web # Visual graph +``` + +### Memory Profile + +```bash +# Generate profile +go test -memprofile=mem.prof -bench=. ./vault/identity + +# Analyze +go tool pprof mem.prof +(pprof) top10 +(pprof) alloc_space # Total allocations +``` + +## Common Optimizations + +### 1. Reduce Allocations + +```go +// ❌ BEFORE - many allocations +func ProcessEntities(entities []Entity) []string { + var names []string + for _, e := range entities { + names = append(names, e.Name) + } + return names +} + +// ✅ AFTER - preallocate +func ProcessEntities(entities []Entity) []string { + names := make([]string, 0, len(entities)) // Preallocate capacity + for _, e := range entities { + names = append(names, e.Name) + } + return names +} +``` + +### 2. String Concatenation + +```go +// ❌ SLOW +var result string +for _, s := range strings { + result += s // New string each iteration +} + +// ✅ FAST +var builder strings.Builder +builder.Grow(estimatedSize) // Preallocate if known +for _, s := range strings { + builder.WriteString(s) +} +result := builder.String() +``` + +### 3. Map Preallocation + +```go +// ❌ BEFORE +m := make(map[string]*Entity) + +// ✅ AFTER +m := make(map[string]*Entity, len(entities)) // Preallocate +``` + +### 4. Avoid Unnecessary Copies + +```go +// ❌ BEFORE - copies struct +func ProcessEntity(e Entity) {} + +// ✅ AFTER - pass pointer +func ProcessEntity(e *Entity) {} +``` + +### 5. Sync.Pool for Temporary Objects + +```go +var bufferPool = sync.Pool{ + New: func() interface{} { + return new(bytes.Buffer) + }, +} + +func ProcessData(data []byte) ([]byte, error) { + buf := bufferPool.Get().(*bytes.Buffer) + buf.Reset() + defer bufferPool.Put(buf) + + buf.Write(data) + // Process + + return buf.Bytes(), nil +} +``` + +## Optimization Strategies + +### Batching + +```go +// ❌ BEFORE - many small operations +for _, entity := range entities { + store.Save(entity) +} + +// ✅ AFTER - batch operation +store.SaveBatch(entities) +``` + +### Parallel Processing + +```go +func ProcessParallel(entities []*Entity) error { + var wg sync.WaitGroup + errCh := make(chan error, len(entities)) + + for _, entity := range entities { + wg.Add(1) + go func(e *Entity) { + defer wg.Done() + if err := process(e); err != nil { + errCh <- err + } + }(entity) + } + + wg.Wait() + close(errCh) + + for err := range errCh { + if err != nil { + return err + } + } + return nil +} +``` + +## Performance Checklist + +Before deploying optimizations: + +- [ ] Benchmarked before and after +- [ ] Profiled CPU and memory +- [ ] Checked for memory leaks +- [ ] Verified concurrent access is safe +- [ ] Load tested with realistic workload +- [ ] Confirmed correctness with tests + +## Test Timeout Considerations + +**Symptom**: Tests hang or timeout + +**Solutions**: +1. Check for goroutine leaks - use `go tool trace` +2. Reduce complexity in test setup +3. Use Docker-based tests for isolation: `make integ` +4. Set explicit timeouts in integration tests +5. Note: `TEST_TIMEOUT=45m`, `INTEG_TEST_TIMEOUT=120m` are defaults diff --git a/vault/development/skills/security/SKILL.md b/vault/development/skills/security/SKILL.md new file mode 100644 index 0000000..de6f406 --- /dev/null +++ b/vault/development/skills/security/SKILL.md @@ -0,0 +1,224 @@ +--- +name: vault-security +description: Implement security-critical features in HashiCorp Vault. Use when working with authentication, authorization, secrets handling, cryptography, input validation, audit logging, or error message sanitization. Ensures security best practices are followed. +compatibility: Requires Go 1.22+, crypto/subtle for comparisons +--- + +# Vault Security + +## Security-First Mindset + +Vault is a **security product**. Vulnerabilities expose customer secrets in production. + +**Core principles**: +1. Assume all input is malicious until validated +2. Never log secrets or sensitive data +3. Validate at all boundaries (API, storage, cache) +4. Fail securely (deny by default) +5. Use constant-time comparisons for secrets +6. Sanitize error messages + +## Critical Rules + +### 1. Logging Rules + +#### 1.1 Use `github.com/hashicorp/go-hclog` for logging + +```go +type JwtAuthManager struct { + // logger for operations + logger hclog.Logger +} +``` + +#### 1.2 Never log secrets + +```go +// ❌ WRONG +log.Printf("Token: %s", token) +log.Printf("Processing: %+v", req) // req may contain secrets + +// ✅ CORRECT +log.Printf("Token received") +log.Printf("Processing request for entity %s", entityID) +``` + +#### 1.3 Never log information on user requests + +If the code path is directly encountered on a user request, +do not log info statements. Only log errors and debug statements if required. + +### 2. Constant-Time Comparisons + +```go +// ❌ WRONG - timing attack vulnerable +if token == expectedToken { + // grant access +} + +// ✅ CORRECT +import "crypto/subtle" + +if subtle.ConstantTimeCompare([]byte(token), []byte(expectedToken)) == 1 { + // grant access +} +``` + +### 3. Input Validation + +Always validate at API boundaries: + +```go +func CreateEntity(ctx context.Context, name string) error { + // Validate + if name == "" { + return fmt.Errorf("entity name cannot be empty") + } + if len(name) > 512 { + return fmt.Errorf("entity name too long") + } + if !validNamePattern.MatchString(name) { + return fmt.Errorf("entity name contains invalid characters") + } + + // Process validated input +} +``` + +### 4. Sanitize Error Messages + +```go +// ❌ WRONG - leaks sensitive info +return fmt.Errorf("authentication failed: invalid token %s", token) + +// ✅ CORRECT +return fmt.Errorf("authentication failed") +``` + +## Authentication Patterns + +```go +func ValidateToken(ctx context.Context, token string) (*TokenEntry, error) { + // Never log token + if token == "" { + return nil, fmt.Errorf("empty token") + } + + te, err := c.tokenStore.Lookup(ctx, token) + if err != nil { + return nil, fmt.Errorf("token lookup failed") // Generic error + } + if te == nil { + return nil, fmt.Errorf("invalid token") + } + + if te.IsExpired() { + return nil, fmt.Errorf("token expired") + } + + return te, nil +} +``` + +## Cryptographic Operations + +```go +import ( + "crypto/aes" + "crypto/cipher" + "crypto/rand" // Use crypto/rand, NEVER math/rand +) + +func Encrypt(plaintext, key []byte) ([]byte, error) { + block, err := aes.NewCipher(key) + if err != nil { + return nil, err + } + + // Use GCM for authenticated encryption + gcm, err := cipher.NewGCM(block) + if err != nil { + return nil, err + } + + // Generate random nonce + nonce := make([]byte, gcm.NonceSize()) + if _, err := rand.Read(nonce); err != nil { + return nil, err + } + + return gcm.Seal(nonce, nonce, plaintext, nil), nil +} +``` + +## Security Checklist + +Before submitting code: + +- [ ] No secrets in logs or error messages +- [ ] Input validated at all boundaries +- [ ] Constant-time comparisons for secrets +- [ ] Error messages don't leak info +- [ ] Using `crypto/rand` not `math/rand` +- [ ] Sensitive data cleared from memory +- [ ] Audit logging for sensitive operations +- [ ] Rate limiting considered for endpoints +- [ ] Authentication requirements enforced +- [ ] ACL checks before privileged operations + +## Input Validation Patterns + +**Validate at write-time** (config creation) to prevent runtime failures: + +```go +// Separate validation functions for testability +func validateJWKSUri(jwksUri string, warnings *[]string) error { + if jwksUri == "" { + return fmt.Errorf("JWKS URI is empty") + } + + parsed, err := url.Parse(jwksUri) + if err != nil { + return fmt.Errorf("invalid JWKS URI: %w", err) + } + + if !parsed.IsAbs() { + return fmt.Errorf("JWKS URI must be absolute URL") + } + + // Warning (non-blocking) vs Error (blocking) + if parsed.Scheme == "http" { + *warnings = append(*warnings, "JWKS URI uses http:// - insecure") + } + + return nil +} + +// Integration in handler +func (b *Backend) handleConfigWrite(ctx context.Context, req *logical.Request, data *framework.FieldData) (*logical.Response, error) { + var warnings []string + + // Blocking validation + if err := validateJWKSUri(uri, &warnings); err != nil { + return logical.ErrorResponse(err.Error()), nil + } + + // Store config... + + // Return with warnings if any + if len(warnings) > 0 { + return &logical.Response{Warnings: warnings}, nil + } + return nil, nil +} +``` + +## Responsible Disclosure + +**If you discover a security vulnerability**: + +1. **DO NOT** create public GitHub issue +2. **DO NOT** discuss in public channels +3. **DO** email +4. **DO** include detailed reproduction steps +5. **DO** wait for security team response diff --git a/vault/development/skills/test-authoring/SKILL.md b/vault/development/skills/test-authoring/SKILL.md new file mode 100644 index 0000000..d90eae5 --- /dev/null +++ b/vault/development/skills/test-authoring/SKILL.md @@ -0,0 +1,297 @@ +--- +name: vault-test-authoring +description: Design and write tests for HashiCorp Vault following best practices. Use when deciding what type of test to write (unit vs core), choosing cluster implementations (NewTestCluster vs NewCore), implementing API-based tests, or applying the DoTest pattern. Covers test structure, package organization, and blackbox testing guidelines. +compatibility: Requires Go 1.22+, access to Vault repository +--- + +# Vault Test Authoring + +## Overview + +This skill provides guidance on designing and structuring tests for Vault, helping you choose the right testing approach for different scenarios. + +## Test Categories + +### Unit Tests vs Core Tests + +| Type | Description | When to Use | +|------|-------------|-------------| +| **Unit tests** | Tests that don't require a `vault.Core` | Pure logic, parsers, validators, utilities | +| **Core tests** | Tests using `vault.Core` (via clusters or direct) | Integration behavior, API endpoints, storage | + +**Rule**: Prefer unit tests when possible. If you need a Core, consider refactoring the code out of package `vault/`. + +### Core Test Types + +| Implementation | Package | Use Case | +|----------------|---------|----------| +| `NewTestCluster` | `vault.NewTestCluster` | Default choice for core tests | +| `NewTestDockerCluster` | Docker containers | Cross-binary testing | +| `NewTestExecDevCluster` | Subprocesses | External process testing | + +## Key Principles + +### 1. Prefer Unit Tests + +```go +// GOOD: Unit test with no Core dependency +func TestParseConfig(t *testing.T) { + cfg, err := ParseConfig([]byte(`{"key": "value"}`)) + require.NoError(t, err) + require.Equal(t, "value", cfg.Key) +} +``` + +**Benefits**: + +- Fast compilation +- No non-determinism from Core internals +- Lower CI costs + +### 2. Consider Both Unit AND Core Tests + +Even with comprehensive unit tests, write a core test for integration assurance: + +```go +// Unit test for quick iteration +func TestEntityValidation(t *testing.T) { /* ... */ } + +// Core test for integration coverage +func TestEntityViaAPI(t *testing.T) { /* ... */ } +``` + +### 3. Use the API Exclusively + +**Never access Core directly in core tests**. Use HTTP API only. + +```go +// BAD: Direct Core access (whitebox testing) +func TestBad(t *testing.T) { + cluster := vault.NewTestCluster(t, &conf, &opts) + core := cluster.Cores[0].Core + core.HandleRequest(ctx, req) // ❌ Avoid +} + +// GOOD: API-based testing (blackbox testing) +func TestGood(t *testing.T) { + cluster := vault.NewTestCluster(t, &conf, &opts) + client := cluster.Cores[0].Client + _, err := client.Logical().Write("path/to/endpoint", data) // ✓ Use API +} +``` + +**Why API-only?** + +- Blackbox testing catches real integration issues +- Tests can run against Docker/exec clusters +- Dogfoods the API, revealing UX issues +- Enables future cloud testing + +### 4. Avoid NewCore and Its Wrappers + +**Do not use** these in `package vault`: + +- `NewCore` +- `TestCoreWithSealAndUI` +- `TestCoreWithSeal` +- `TestCoreWithConfig` +- `TestCoreUnsealedWithConfig` + +**Problems**: + +- Only marginally faster than `NewTestCluster` with `NumCores=1` +- Less coverage (no auditing, networking, etc.) +- Creates duplication and wrapper proliferation +- Keeps tests in `package vault`, which should shrink + +### 5. Use NewTestCluster as Default + +Standard pattern: + +```go +func TestFeature(t *testing.T) { + conf, opts := teststorage.ClusterSetup(nil, nil, nil) + cluster := vault.NewTestCluster(t, &conf, &opts) + t.Cleanup(cluster.Cleanup) + + client := cluster.Cores[0].Client + // Test using client API +} +``` + +**Note**: `cluster.Start()` is now a no-op and can be omitted. + +### 6. Use Single-Node Clusters When Possible + +If your test doesn't exercise performance standbys: + +```go +// Explicit about single node +opts.NumCores = 1 + +// Or use the helper +cluster := minimal.NewTestSoloCluster(t) +``` + +This saves resources and makes test intent clear. + +### 7. Avoid logs in tests that don't add value + +The following log line do not add any extra information about a test and it's execution path or the state of the system under test. + +```go +func DoTestJWTToken_CubbyholeIsolation(t *testing.T, cluster testcluster.VaultCluster) { + // code here to test JWT Cubbyhole isolation... + t.Log("JWT Token Cubbyhole Isolation passed successfully") +``` + +If required log important checkpoints and system state, errors of system in a test that aid in further debugging. Do not log errors that would anyway be logged in the test execution summary. + +## The DoTest Pattern + +Factor out cluster creation for maximum flexibility and compilation speed. + +### Structure + +``` +vault/external_tests/$pkgname/ +├── tests/ +│ └── do_test_x.go # DoTestX function (NOT _test.go) +├── cluster_test.go # TestX using NewTestCluster +└── binary/ + └── docker_test.go # TestX using NewTestDockerCluster +``` + +### Implementation + +**Step 1**: Define the test logic (not in a `_test.go` file): + +```go +// vault/external_tests/identity/tests/do_test_entity.go +package tests + +import ( + "testing" + "github.com/hashicorp/vault/sdk/helper/testcluster" +) + +func DoTestEntityCreate(t *testing.T, cluster testcluster.VaultCluster) { + client := cluster.Nodes()[0].APIClient() + + // Actual test using only API calls + _, err := client.Logical().Write("identity/entity", map[string]interface{}{ + "name": "test-entity", + }) + require.NoError(t, err) +} +``` + +**Step 2**: Call from cluster-specific test: + +```go +// vault/external_tests/identity/cluster_test.go +package identity_test + +func TestEntityCreate(t *testing.T) { + conf, opts := teststorage.ClusterSetup(nil, nil, nil) + cluster := vault.NewTestCluster(t, &conf, &opts) + t.Cleanup(cluster.Cleanup) + + tests.DoTestEntityCreate(t, cluster) +} +``` + +**Step 3**: Optionally test with Docker: + +```go +// vault/external_tests/identity/binary/docker_test.go +package binary + +func TestEntityCreate(t *testing.T) { + if os.Getenv("RUN_DOCKER_TESTS") == "" { + t.Skip("Set RUN_DOCKER_TESTS to run") + } + + cluster := testcluster.NewDockerCluster(t, opts) + t.Cleanup(cluster.Cleanup) + + tests.DoTestEntityCreate(t, cluster) +} +``` + +### Benefits + +- `DoTestX` compiles fast (no `vault/` import) +- Same test logic runs against multiple cluster types +- Tests live outside `package vault` + +## Package Organization + +### Avoid `package vault_test` + +Using `vault_test` in files within `vault/` creates hidden complexity. Instead: + +**Move tests to `external_tests/`**: + +``` +vault/external_tests/ +├── identity/ +├── quotas/ +└── replication/ +``` + +### Why External Tests? + +1. Shrinks `package vault` (compilation speed) +2. Forces API-only testing +3. Clear separation of concerns + +## Decision Tree + +``` +Need to test new code? +│ +├─ Can test without Core? +│ └─ YES → Write unit test +│ +├─ Need Core for integration? +│ │ +│ ├─ Code lives in package vault? +│ │ └─ Consider refactoring out first +│ │ +│ └─ Write core test: +│ ├─ Use NewTestCluster (not NewCore) +│ ├─ Use API only (not Core methods) +│ ├─ Place in external_tests/ +│ └─ Consider DoTest pattern +│ +└─ Testing both unit AND integration? + └─ BEST APPROACH: Write both! +``` + +## Quick Reference + +| Goal | Approach | +|------|----------| +| Test pure logic | Unit test, no Core | +| Test API behavior | `NewTestCluster` + API client | +| Test with real binary | `NewTestDockerCluster` | +| Reusable across clusters | DoTest pattern | +| Reduce `vault/` size | `external_tests/` + API-only | +| Single-node test | `NumCores: 1` or `NewTestSoloCluster` | + +## Final Step: Format Code + +**Always run `make fmt` as the last step** in any development or test authoring cycle: + +```bash +make fmt +``` + +This ensures all Go files are properly formatted before committing or submitting a PR. + +## Next Steps + +- See [references/DOTEST_EXAMPLES.md](references/DOTEST_EXAMPLES.md) for complete DoTest examples +- See [references/CLUSTER_SETUP.md](references/CLUSTER_SETUP.md) for cluster configuration options +- See [references/REPLICATION_TESTING.md](references/REPLICATION_TESTING.md) for replication test setup (Enterprise) diff --git a/vault/development/skills/test-authoring/references/CLUSTER_SETUP.md b/vault/development/skills/test-authoring/references/CLUSTER_SETUP.md new file mode 100644 index 0000000..65efbe4 --- /dev/null +++ b/vault/development/skills/test-authoring/references/CLUSTER_SETUP.md @@ -0,0 +1,228 @@ +# Cluster Setup Reference + +Options and configurations for Vault test clusters. + +## teststorage.ClusterSetup + +The standard way to get cluster configuration: + +```go +import "github.com/hashicorp/vault/helper/teststorage" + +func TestExample(t *testing.T) { + conf, opts := teststorage.ClusterSetup(nil, nil, nil) + cluster := vault.NewTestCluster(t, &conf, &opts) + t.Cleanup(cluster.Cleanup) +} +``` + +### Parameters + +```go +func ClusterSetup( + conf *vault.CoreConfig, // nil for defaults + opts *vault.TestClusterOptions, // nil for defaults + setup *teststorage.SetupOpts, // nil for defaults +) (vault.CoreConfig, vault.TestClusterOptions) +``` + +## CoreConfig Options + +```go +conf := vault.CoreConfig{ + // Logger configuration + Logger: logging.NewVaultLogger(hclog.Debug), + + // Disable default policies + DisableMlock: true, + + // Custom seal (usually not needed) + Seal: seal, + + // Physical storage backend + Physical: physicalBackend, + + // Enable specific features + EnableUI: true, + + // License (EE only) + LicensePath: "/path/to/license", +} +``` + +## TestClusterOptions + +```go +opts := vault.TestClusterOptions{ + // Number of nodes (default: 3) + NumCores: 1, + + // HTTP handler (REQUIRED for API access) + HandlerFunc: vaulthttp.Handler, + + // Skip initialization + SkipInit: false, + + // Keep standbys sealed + KeepStandbysSealed: false, + + // Custom ports + BaseListenAddress: "127.0.0.1", + BaseClusterPort: 0, // random + + // TLS configuration + TLSDisable: false, + + // Temp directory for data + TempDir: t.TempDir(), + + // Plugins + PluginDirectory: "/path/to/plugins", +} +``` + +## Common Configurations + +### Single Node (Simplest) + +```go +conf, opts := teststorage.ClusterSetup(nil, nil, nil) +opts.NumCores = 1 + +cluster := vault.NewTestCluster(t, &conf, &opts) +``` + +### With Custom Logger + +```go +import "github.com/hashicorp/go-hclog" + +logger := hclog.New(&hclog.LoggerOptions{ + Name: "test", + Level: hclog.Debug, +}) + +conf := vault.CoreConfig{ + Logger: logger, +} + +_, opts := teststorage.ClusterSetup(&conf, nil, nil) +cluster := vault.NewTestCluster(t, &conf, &opts) +``` + +### With Plugins + +```go +opts := vault.TestClusterOptions{ + NumCores: 1, + HandlerFunc: vaulthttp.Handler, + PluginDirectory: "/path/to/plugins", +} + +conf, _ := teststorage.ClusterSetup(nil, &opts, nil) +cluster := vault.NewTestCluster(t, &conf, &opts) +``` + +### Replication Setup + +For replication testing (perf replication, DR replication, or full 4-group topology), see [REPLICATION_TESTING.md](REPLICATION_TESTING.md). + +## minimal.NewTestSoloCluster + +Convenience function for single-node clusters: + +```go +import "github.com/hashicorp/vault/vault/minimal" + +func TestSimple(t *testing.T) { + cluster := minimal.NewTestSoloCluster(t) + t.Cleanup(cluster.Cleanup) + + client := cluster.Cores[0].Client + // Test with client +} +``` + +## Waiting for Cluster Ready + +After creating a cluster, wait for it to be active: + +```go +cluster := vault.NewTestCluster(t, &conf, &opts) +t.Cleanup(cluster.Cleanup) + +// Get the active core +core := cluster.Cores[0].Core +vault.TestWaitActive(t, core) + +// Now safe to use +client := cluster.Cores[0].Client +``` + +## Storage Backends + +### In-Memory (Default) + +```go +conf, opts := teststorage.ClusterSetup(nil, nil, nil) +// Uses inmem by default +``` + +### Consul + +```go +setup := &teststorage.SetupOpts{ + StorageBackend: teststorage.ConsulBackend, +} + +conf, opts := teststorage.ClusterSetup(nil, nil, setup) +``` + +### Raft + +```go +setup := &teststorage.SetupOpts{ + StorageBackend: teststorage.RaftBackend, +} + +conf, opts := teststorage.ClusterSetup(nil, nil, setup) +``` + +## Docker Cluster Options + +```go +import "github.com/hashicorp/vault/sdk/helper/testcluster/docker" + +cluster := docker.NewDockerCluster(t, &docker.DockerClusterOptions{ + ImageRepo: "hashicorp/vault", + ImageTag: "latest", + NumCores: 1, + + // Network configuration + NetworkName: "vault-test", + + // Volume mounts + VolumeBinds: []string{ + "/host/path:/container/path", + }, + + // Environment variables + Env: []string{ + "VAULT_LOG_LEVEL=debug", + }, +}) +``` + +## Exec Cluster Options + +```go +import "github.com/hashicorp/vault/sdk/helper/testcluster/exec" + +cluster := exec.NewTestExecDevCluster(t, &exec.ExecDevClusterOptions{ + NumCores: 1, + BinaryPath: "/path/to/vault", + + // Additional CLI args + Args: []string{"-dev-root-token-id=root"}, +}) +``` diff --git a/vault/development/skills/test-authoring/references/DOTEST_EXAMPLES.md b/vault/development/skills/test-authoring/references/DOTEST_EXAMPLES.md new file mode 100644 index 0000000..ad78d5b --- /dev/null +++ b/vault/development/skills/test-authoring/references/DOTEST_EXAMPLES.md @@ -0,0 +1,208 @@ +# DoTest Pattern Examples + +Complete examples of the DoTest pattern for Vault tests. + +## Basic Example: Identity Entity Test + +### Test Logic (non-test file) + +```go +// vault/external_tests/identity/tests/do_test_entity.go +package tests + +import ( + "testing" + + "github.com/hashicorp/vault/sdk/helper/testcluster" + "github.com/stretchr/testify/require" +) + +// DoTestEntityCreate tests entity creation via API +// This file does NOT end in _test.go so it compiles quickly +func DoTestEntityCreate(t *testing.T, cluster testcluster.VaultCluster) { + t.Helper() + + client := cluster.Nodes()[0].APIClient() + + // Create entity + resp, err := client.Logical().Write("identity/entity", map[string]interface{}{ + "name": "test-entity", + "metadata": map[string]string{ + "team": "foundations", + }, + }) + require.NoError(t, err) + require.NotNil(t, resp) + + entityID := resp.Data["id"].(string) + require.NotEmpty(t, entityID) + + // Read entity back + resp, err = client.Logical().Read("identity/entity/id/" + entityID) + require.NoError(t, err) + require.Equal(t, "test-entity", resp.Data["name"]) +} + +// DoTestEntityAlias tests alias creation +func DoTestEntityAlias(t *testing.T, cluster testcluster.VaultCluster) { + t.Helper() + + client := cluster.Nodes()[0].APIClient() + + // First create entity + entityResp, err := client.Logical().Write("identity/entity", map[string]interface{}{ + "name": "alias-test-entity", + }) + require.NoError(t, err) + entityID := entityResp.Data["id"].(string) + + // Enable userpass for alias mount + err = client.Sys().EnableAuthWithOptions("userpass", &api.EnableAuthOptions{ + Type: "userpass", + }) + require.NoError(t, err) + + // Get mount accessor + mounts, err := client.Sys().ListAuth() + require.NoError(t, err) + accessor := mounts["userpass/"].Accessor + + // Create alias + _, err = client.Logical().Write("identity/entity-alias", map[string]interface{}{ + "name": "testuser", + "canonical_id": entityID, + "mount_accessor": accessor, + }) + require.NoError(t, err) +} +``` + +### NewTestCluster Test + +```go +// vault/external_tests/identity/cluster_test.go +package identity_test + +import ( + "testing" + + "github.com/hashicorp/vault/helper/teststorage" + "github.com/hashicorp/vault/vault" + "github.com/hashicorp/vault/vault/external_tests/identity/tests" +) + +func TestEntityCreate(t *testing.T) { + t.Parallel() + + conf, opts := teststorage.ClusterSetup(nil, nil, nil) + opts.NumCores = 1 // Single node is sufficient + + cluster := vault.NewTestCluster(t, &conf, &opts) + t.Cleanup(cluster.Cleanup) + + tests.DoTestEntityCreate(t, cluster) +} + +func TestEntityAlias(t *testing.T) { + t.Parallel() + + conf, opts := teststorage.ClusterSetup(nil, nil, nil) + opts.NumCores = 1 + + cluster := vault.NewTestCluster(t, &conf, &opts) + t.Cleanup(cluster.Cleanup) + + tests.DoTestEntityAlias(t, cluster) +} +``` + +### Docker Cluster Test + +```go +// vault/external_tests/identity/binary/docker_test.go +package binary + +import ( + "os" + "testing" + + "github.com/hashicorp/vault/sdk/helper/testcluster/docker" + "github.com/hashicorp/vault/vault/external_tests/identity/tests" +) + +func TestEntityCreate(t *testing.T) { + if os.Getenv("RUN_DOCKER_TESTS") == "" { + t.Skip("Set RUN_DOCKER_TESTS=1 to run docker tests") + } + + cluster := docker.NewDockerCluster(t, &docker.DockerClusterOptions{ + ImageTag: "latest", + NumCores: 1, + }) + t.Cleanup(cluster.Cleanup) + + tests.DoTestEntityCreate(t, cluster) +} +``` + +## Advanced Example: Replication Test + +```go +// vault/external_tests/replication/tests/do_test_replication.go +package tests + +import ( + "testing" + "time" + + "github.com/hashicorp/vault/sdk/helper/testcluster" + "github.com/stretchr/testify/require" +) + +// DoTestReplicationBasic verifies data replicates between clusters +func DoTestReplicationBasic(t *testing.T, primary, secondary testcluster.VaultCluster) { + t.Helper() + + primaryClient := primary.Nodes()[0].APIClient() + secondaryClient := secondary.Nodes()[0].APIClient() + + // Write secret to primary + _, err := primaryClient.Logical().Write("secret/data/test", map[string]interface{}{ + "data": map[string]string{ + "key": "value", + }, + }) + require.NoError(t, err) + + // Wait for replication + time.Sleep(2 * time.Second) + + // Read from secondary + resp, err := secondaryClient.Logical().Read("secret/data/test") + require.NoError(t, err) + require.NotNil(t, resp) + + data := resp.Data["data"].(map[string]interface{}) + require.Equal(t, "value", data["key"]) +} +``` + +## File Organization Summary + +``` +vault/external_tests/identity/ +├── tests/ +│ ├── do_test_entity.go # DoTestEntityCreate, DoTestEntityAlias +│ └── do_test_group.go # DoTestGroupCreate, etc. +├── cluster_test.go # TestEntityCreate, TestGroupCreate (NewTestCluster) +└── binary/ + └── docker_test.go # TestEntityCreate (Docker) +``` + +## Key Points + +1. **DoTestX files don't end in `_test.go`** - They compile separately from test dependencies +2. **Accept `testcluster.VaultCluster` interface** - Works with any cluster implementation +3. **Use only API client** - Never access Core or internal state +4. **TestX files wrap DoTestX** - Handle cluster creation/cleanup +5. **Skip docker tests by default** - Use env var to opt-in diff --git a/vault/development/skills/test-authoring/references/REPLICATION_TESTING.md b/vault/development/skills/test-authoring/references/REPLICATION_TESTING.md new file mode 100644 index 0000000..e9b5b6e --- /dev/null +++ b/vault/development/skills/test-authoring/references/REPLICATION_TESTING.md @@ -0,0 +1,267 @@ +# Replication Testing Reference + +Testing Vault replication requires Enterprise features. Use the provided helpers to spin up replicated clusters easily. + +## Quick Start + +Pick your topology, call one function, use the accessors. Done. + +## Topologies + +### Perf Primary + Perf Secondary (2 replication groups) + +```go +conf, opts := teststorage.ClusterSetup(nil, nil, teststorage.InmemBackendSetup) +clusters := testhelpers.GetPerfReplicatedClusters(t, conf, opts) +defer clusters.Cleanup() + +_, _, primaryClient := clusters.Primary() +_, _, secondaryClient := clusters.Secondary() +``` + +### Perf Primary + DR Secondary (2 replication groups) + +```go +conf, opts := teststorage.ClusterSetup(nil, nil, teststorage.InmemBackendSetup) +clusters := testhelpers.GetDRReplicatedClusters(t, conf, opts) +defer clusters.Cleanup() + +_, _, primaryClient := clusters.Primary() +_, _, drClient := clusters.PrimaryDR() +``` + +### Full 4-Group Topology (perf + both DRs) + +```go +conf, opts := teststorage.ClusterSetup(&vault.CoreConfig{ + DisableAutopilot: true, +}, &vault.TestClusterOptions{ + HandlerFunc: vaulthttp.Handler, +}, teststorage.InmemBackendSetup) + +clusters := testhelpers.GetFourReplicatedClustersWithConf(t, conf, opts) +defer clusters.Cleanup() + +_, _, primaryClient := clusters.Primary() +_, _, secondaryClient := clusters.Secondary() +_, _, primaryDRClient := clusters.PrimaryDR() +_, _, secondaryDRClient := clusters.SecondaryDR() +``` + +### Multiple HA Nodes per Replication Group + +Set `NumCores` before passing `opts`: + +```go +opts.NumCores = 3 // active + 2 standbys in each group +``` + +## Rules + +1. **Only use the `*api.Client` from the accessors.** Never touch `.Core` directly. +2. **Always `defer clusters.Cleanup()` immediately after creation.** +3. **Use `testhelpers.EnsureCoresUnsealed(t, clusters.PerfPrimaryCluster)` if you unseal/reseal in the test.** +4. **Add `//go:build ent` at the top of any file using these helpers.** + +--- + +## Common Replication Test Operations + +### Wait for Sync / Convergence + +| Operation | Helper | How to Assert Correctly | Gaps | +|---|---|---|---| +| Wait for WAL index | `testhelpers.WaitForWAL(t, core, walIndex)` | Blocks until `core.EntLastWAL() >= walIndex` | After DR promotion, re-verify threshold WAL was reached on new primary before reading data | +| Perf/DR WAL drain | `WaitForPerformanceWAL(t, clusters)` / `WaitForDRWAL(t, clusters)` | Poll until secondary WAL ≥ primary WAL | No assertion that data written before the wait is actually readable on secondary after | +| Merkle roots match | `WaitForMatchingMerkleRoots(t, endpoint, pri, sec)` / `WaitForMatchingMerkleRootsCore` | Compare `status.Data["dr"]["merkle_root"] == drStatus.Data["merkle_root"]` directly | Not just "equal eventually" — assert exact root values from both sides in the same snapshot | +| Perf replication working | `WaitForPerfReplicationWorking(t, pri, sec)` | Writes a probe KV key on primary, reads non-nil on secondary | Assert the read value matches the written value; probe key is deleted but not value-checked | +| Perf replication working (by path) | `WaitForPerfReplicationWorkingWithMount(t, pri, sec, mount)` | Same as above, scoped to a specific mount | Same value-check gap as above | +| Perf replication working (clients) | `WaitForPerfReplicationWorkingClients(ctx, priClient, secClient, mount)` | Polls `secClient.Logical().Read(path)` until non-nil | Does not assert response value matches what was written | +| DR replication working | `WaitForDRReplicationWorking(t, pri, sec)` | Waits for `state == StreamWALs` and `last_remote_wal > 0` | No assertion that actual data (beyond token events) is visible on DR secondary | +| Replication connection status | `WaitForPerfReplicationConnectionStatus(t, secClient)` | Polls until `connection_status == "connected"` | Also assert `last_heartbeat` is recent (non-zero, within 30s) | +| Active node + perf standbys | `WaitForActiveNodeAndPerfStandbys(t, cluster)` | Waits until active and standbys are ready | Assert each standby returns `health.PerformanceStandby == true` via `EnsureCoreIsPerfStandby` | +| Replication state enum | `WaitForReplicationState(t, core, state)` | Polls `core.ReplicationState().HasState(state)` | No timeout surfacing — failure message does not include observed state | +| Wait for arbitrary status predicate | `WaitForReplicationStatus(t, client, isDR, pred)` | Predicate receives `status.Data["dr"/"performance"]` map | Tests often check only `mode`; `last_wal`, `last_remote_wal`, `state` frequently unchecked | + +--- + +### Simulate Node/Primary Failure + +| Operation | Helper | How to Assert Correctly | Gaps | +|---|---|---|---| +| Stop a node | `cluster.StopCore(t, nodeIndex)` | Call `WaitForActiveNode(t, cluster)` to verify another node activated | Assert stopped node's client returns errors (unhealthy/sealed), not just that a new active exists | +| Seal all nodes | `testhelpers.SealCores(t, cluster)` | Before DR promotion: assert `health.Sealed == true` on all old-primary nodes | No assertion that writes to the old primary during sealed state return appropriate errors | +| Unseal all nodes | `testhelpers.EnsureCoresUnsealed(t, cluster)` | Assert `health.Sealed == false` and `health.Initialized == true` on each node | No re-check of replication state post-unseal | +| Inject replication failure mode | `vault.SetReplicationFailureMode(core, vault.ReplicationFailureModeReindexNeeded)` | Assert `sys/replication/status` surfaces a warning; then assert promote/demote succeeds or fails as intended | Tests check the result of `promote` but do not verify the failure mode appears in status warnings before the operation | + +--- + +### DR Failover & Failback + +| Operation | Helper | How to Assert Correctly | Gaps | +|---|---|---|---| +| Promote DR secondary | `testhelpers.PromoteDRSecondary(t, drCluster)` | `GetRepStatusUntil(ctx, t, drClient, isDR=true, mode=="primary")` | Also assert `known_secondaries == []` immediately after promotion | +| Update DR secondary primary | `testhelpers.UpdatePrimaryDRSecondary(t, cluster, token, caFile)` | `GetRepStatus(t, client, isDR=true)["mode"] == "secondary"` and `["secondary_id"] == expectedID` | Also assert old primary shows the new secondary in its `known_secondaries` | +| Demote DR primary | `client.Logical().Write("sys/replication/dr/primary/demote", ...)` | `WaitForReplicationStatus(client, isDR=true, mode=="secondary")` | No check that the demoted node's perf replication also stops | +| Full DR failback (A→B→A) | `SealCores` → `PromoteDRSecondary` → `EnsureCoresUnsealed` → `demote` → `UpdatePrimaryDRSecondary` | Final: both DR and perf `known_secondaries` match expected IDs; `mode == "primary"` on restored node | Data written during B-is-primary window is never verified to survive the failback to A | +| Write DR operation | `testhelpers.WriteDROperation(t, client, path, token)` | Assert `resp != nil`; `GetRepStatus` after to confirm state change took effect | Return value contents are not verified, only non-nil | +| Create DR op batch token | `testhelpers.CreatePathBatchToken(t, client, path)` | Assert the token is non-empty and has correct policy via a test read/write against the path | Token is used directly without verifying its capability scope | +| No replication loop after failover | `vault.TestClusterCoreGetReplFSM(activeCore).State()` | `state == StreamWALs`; subscribe `changeCh`, assert no state change for 5s | FSM loop check is not consistently applied on both the promote *and* the failback path | +| Perf secondary reconnects after DR failover | `EnablePerformanceSecondaryNoWait(t, token, newPri, sec, updatePrimary=true)` | `GetRepStatusUntil(secClient, perf, mode=="secondary" && primary_cluster_addr == newClusterAddr)` | `WaitForPerfReplicationWorking(newPrimary, perfSec)` is missing after reconnect in several tests | + +--- + +### Perf Replication Topology Changes + +| Operation | Helper | How to Assert Correctly | Gaps | +|---|---|---|---| +| Revoke perf secondary | `clusters.RevokePerfReplicationSecondary(t)` | Assert secondary no longer in primary's `known_secondaries`; secondary mode becomes `"disabled"` | No assertion that writes to the secondary after revocation return errors | +| Disable perf secondary | `clusters.DisablePerfReplicationSecondary(t)` | `WaitForReplicationStatus(secClient, DR=false, mode=="disabled")` | No assertion that previously replicated data is still locally readable after disable | +| Promote perf secondary | `client.Logical().Write("sys/replication/performance/secondary/promote", ...)` | `GetRepStatusUntil(secClient, perf, mode=="primary")` and `known_secondaries == []` | Missing: verify old primary can no longer reach this node as a secondary | +| Reconnect secondary to new primary | `EnablePerformanceSecondary(t, token, newPri, sec, updatePrimary=true, ...)` | `WaitForPerfReplicationWorking(newPri, sec)` | Use `api.RequireState(priorState)` on secondary client to guarantee reads reflect pre-reconnect writes | + +--- + +### Assert Replication Health & State + +| Operation | Helper | How to Assert Correctly | Gaps | +|---|---|---|---| +| Check replication health (health endpoint) | `testhelpers.CheckRepHealth(t, client, coreIdx, clusterName, perfMode, drMode)` | Checks `ReplicationPerformanceMode` and `ReplicationDRMode` | Does not assert `Sealed`, `Standby`, or `Initialized` — a sealed node can return stale mode strings | +| Get raw replication status | `testhelpers.GetRepStatus(t, client, isDR)` | Assert `mode`, `known_secondaries`, `secondary_id`, `primary_cluster_addr` | `last_wal`, `last_remote_wal`, `state` frequently unchecked | +| Poll replication status with predicate | `testhelpers.GetRepStatusUntil(ctx, t, client, isDR, pred)` | Context timeout surfaces last observed status | Predicates usually only check `mode`; should also validate WAL and connection fields | +| Replication status secondaries list | `compareReplicationStatusSecondaries(t, secondaries, knownSecondaries)` | Asserts `len` equal and `[0].NodeID` matches | Only checks index 0; does not assert full list or ordering | +| Replication metrics correctness | Retry loop over `testhelpers.SysMetricsReq`; check gauge values per cluster role | Roles held: gauge `== 1`; roles not held: gauge `== 0` | No assertion that metrics reset after role change (e.g. after DR promotion, old primary's `dr.primary` gauge does not drop to 0) | +| Raft voter configuration | `testhelpers.VerifyRaftVoters(t, client, expected map[nodeID]isVoter)` | Full voter map comparison via `cmp.Diff` | Should be called after every topology change but is often omitted | + +--- + +### Simulate Replication Lag / Network Conditions + +| Operation | Helper | How to Assert Correctly | Gaps | +|---|---|---|---| +| Latency injection | `latencyInjector.SetLatency(d)` on `physical.TransactionalLatencyInjector` | `getReplicationLag(client) >= injectedLatency` after injection; `< 1s` before, via `RetryUntil` | No assertion that operations still succeed under latency; no assertion on error rates | +| Block Raft FSM applies | `testhelpers.BlockRaftAppliesUntil412Seen(tc, &retryCounter)` | Assert `retryCounter > 0` after the operation; always `defer cleanup()` | No assertion on data outcome after unblocking; the write may have been lost | +| Replication canary lag | `client.Sys().ReplicationPerformanceStatusWithContext(ctx).Secondaries[0].ReplicationPrimaryCanaryAgeMillis` | Assert `< threshold` before injection; `>= injectedLatency` after, via `RetryUntil` | No assertion that canary age resets after latency is removed | +| WAL wait duration config | `testhelpers.ReplicationConfig(waitDuration)` combined with `WaitForDRWAL` | Assert secondary WAL catches up within the configured window | Missing: assert no data visible on secondary before WAL catches up | + +--- + +### Storage/Mount Operations That Test Replication Propagation + +| Operation | Helper / API | How to Assert Correctly | Gaps | +|---|---|---|---| +| Write on primary, read on secondary | `WaitForPerfReplicationWorking` / `WaitForPerfReplicationWorkingClients` | Write probe key; read on secondary and assert non-nil | Assert the *value* matches: `require.Equal(t, expected, secret.Data["bar"])`; current tests only check non-nil | +| Replication causality | `api.RecordState(&state)` on write; `api.RequireState(state)` on secondary read | Secondary read guaranteed to see the write via X-Vault-Index header enforcement | Not used consistently — most tests rely on `time.Sleep`, making them timing-sensitive | +| Mount replication | `testhelpers.IsMounted(t, secClient, ns, mount)` | Assert `true` on secondary after mount on primary | Assert the mount is *absent* before replication settles to catch false positives | +| Cross-namespace remount | `client.Sys().Remount(oldPath, newPath)` | Read data at new path on secondary; assert accessible at new path and gone from old path | Tests only assert the API call succeeds; secondary read at both old and new paths is missing | +| Manual reindex | `client.Logical().Write("sys/replication/reindex", ...)` | Wait for `"verified reindex"` log message; then re-check Merkle roots match | Fragile log-string matching; no assertion that data integrity holds post-reindex | +| Local-path data after DR failover | Mount with `Local: true` or `PassthroughWithLocalPathsFactory`, then promote DR | Assert local-path data is *absent* on DR secondary after promotion — it must never replicate | Entirely absent in current tests; local data on primary is never verified to be missing on DR secondary | + +--- + +## Cache Invalidation on Secondaries + +Understanding how cache invalidation works is essential for writing correct replication tests. + +### The Two Invalidation Paths + +#### 1. Sync Invalidation (`syncInvalidate`) — Critical Tables + +Triggered directly by the replication FSM as each WAL entry is applied. **Synchronous and blocking** — the FSM apply does not return until invalidation completes. + +| Path Key | What Gets Invalidated | +|---|---| +| `coreMountConfigPath` | Mount table (add/remove/tune mounts) | +| `coreAuthConfigPath` | Auth method table | +| `coreAuditConfigPath` | Audit device table | +| `namespaceConfigPath` | Namespace table | +| `pluginCatalogPath` | Plugin catalog | +| `coreKeyringCanaryPath` | Keyring / rekey | +| `apiLockStateFullPath` | API lock state | +| `coreReplicatedSecondaryFilteredPathsPath` | Filtered paths (perf secondaries only) | + +#### 2. Async Invalidation (`asyncInvalidateKey`) — Everything Else + +The replication FSM drops the storage path onto `invalidationCh` (buffered channel, size `AsyncInvalidationChannelSize = 256`). A background goroutine drains it: + +``` +Primary write → WAL entry → replicated to secondary FSM +→ secondary FSM applies entry to storage +→ storage path sent to invalidationCh +→ asyncInvalidateHandler goroutine picks it up +→ asyncInvalidateKey resolves path → backend via router +→ backend.InvalidateKey(ctx, backendPathSuffix) +``` + +--- + +## Cache Invalidation Test Patterns + +### Pattern 1: `require.Eventually` Poll (Most Common) + +Write on primary, poll secondary until new value appears: + +```go +// Write on primary +_, err := primaryClient.Logical().Write("sys/config/my-feature", data) +require.NoError(t, err) + +// Poll secondary until cache is invalidated and new value is visible +require.Eventually(t, func() bool { + secret, err := secondaryClient.Logical().Read("sys/config/my-feature") + if err != nil || secret == nil || secret.Data == nil { + return false + } + return secret.Data["field"] == expectedValue +}, 30*time.Second, 500*time.Millisecond, "update should replicate to secondary") +``` + +**When to use**: Default for cross-cluster (perf/DR) invalidation tests. + +### Pattern 2: `api.RecordState` + `api.RequireState` (Causal Consistency) + +Captures WAL index from primary write, enforces it on secondary read. Secondary returns HTTP 412 if WAL hasn't caught up: + +```go +var state string +primaryClient.WithResponseCallbacks(api.RecordState(&state)).Logical().Write(path, data) + +// Secondary read blocked (412) until WAL index >= state +secondaryClient.WithRequestCallbacks(api.RequireState(state)).Logical().Read(path) +``` + +To verify 412 actually fired (invalidation was async): + +```go +var retryCount atomic.Int32 +cleanup := testhelpers.BlockRaftAppliesUntil412Seen(secondary.Cores[1], &retryCount) +defer cleanup() +// ... do write + RequireState read ... +require.Greater(t, retryCount.Load(), int32(0), "secondary was behind, should have retried") +``` + +**When to use**: Within a single cluster (perf standbys); proves causal ordering. + +**Note**: Does not work cross-cluster — only within a single cluster's perf standbys. + +### Pattern 3: Force Synchronous Invalidation (Deterministic Tests) + +Set `AsyncInvalidationChannelSize = 0` to force FSM to block until invalidation completes: + +```go +oldSize := vault.AsyncInvalidationChannelSize +vault.AsyncInvalidationChannelSize = 0 +defer func() { vault.AsyncInvalidationChannelSize = oldSize }() + +// Now writes and secondary reads are synchronous — no Eventually needed +``` + +**When to use**: When deterministic tests without timing sensitivity are required. + +### Summary + +| Pattern | Mechanism | Use Case | +|---|---|---| +| `require.Eventually` poll | Retry read until value matches | Cross-cluster (perf/DR) invalidation | +| `RecordState` + `RequireState` | WAL-index via X-Vault-Index header; 412 retry | Single cluster perf standbys; causal ordering | +| `AsyncInvalidationChannelSize = 0` | Force async → sync | Deterministic tests without timing sensitivity | diff --git a/vault/development/skills/testing/SKILL.md b/vault/development/skills/testing/SKILL.md new file mode 100644 index 0000000..e81de75 --- /dev/null +++ b/vault/development/skills/testing/SKILL.md @@ -0,0 +1,218 @@ +--- +name: vault-testing +description: Write and run tests for HashiCorp Vault. Use when writing unit tests, integration tests, running specific tests, understanding build tags. Covers CE/EE testing, race detection, and common test patterns. +compatibility: Requires Go 1.22+, Docker for integration tests +--- + +# Vault Testing + +## Fast Test Execution + +Run tests directly with `go test` using proper tags and environment variables for faster execution (skips prep/generation steps): + +```bash +# Basic enterprise tests +CGO_ENABLED=0 \ +VAULT_ADDR= \ +VAULT_TOKEN= \ +VAULT_DEV_ROOT_TOKEN_ID= \ +VAULT_ACC= \ +go test -tags='enterprise,testonly' ./... -timeout=45m -parallel=20 + +# With race detector (requires CGO) +CGO_ENABLED=1 \ +VAULT_ADDR= \ +VAULT_TOKEN= \ +VAULT_DEV_ROOT_TOKEN_ID= \ +VAULT_ACC= \ +go test -tags='enterprise,testonly' -race ./... -timeout=60m -parallel=20 +``` + +## Running Specific Tests + +### Basic Patterns + +```bash +# Run all tests in package +CGO_ENABLED=0 VAULT_ADDR= VAULT_TOKEN= VAULT_DEV_ROOT_TOKEN_ID= VAULT_ACC= \ +go test -tags='enterprise,testonly' ./vault/identity -timeout=45m -parallel=20 + +# Run specific test function +CGO_ENABLED=0 VAULT_ADDR= VAULT_TOKEN= VAULT_DEV_ROOT_TOKEN_ID= VAULT_ACC= \ +go test -tags='enterprise,testonly' ./vault -run TestSpecificTest -timeout=45m -parallel=20 + +# Verbose output +CGO_ENABLED=0 VAULT_ADDR= VAULT_TOKEN= VAULT_DEV_ROOT_TOKEN_ID= VAULT_ACC= \ +go test -tags='enterprise,testonly' ./vault -run TestJWT -v -timeout=45m -parallel=20 + +# Run multiple times (stability) +for i in {1..10}; do \ + CGO_ENABLED=0 VAULT_ADDR= VAULT_TOKEN= VAULT_DEV_ROOT_TOKEN_ID= VAULT_ACC= \ + go test -tags='enterprise,testonly' ./vault -run TestJWT -timeout=45m -parallel=20; \ +done +``` + +### Test Filtering + +The `-run` flag uses **Go regex patterns**: + +```bash +# Match any test starting with TestJWT +-run TestJWT + +# Exact match only +-run TestJWT$ + +# Match tests with JWT anywhere +-run '.*JWT.*' + +# Multiple patterns (OR) +-run 'Test(JWT|OIDC)' + +# Subtests +-run TestJWT/with_role +``` + +## Test Both CE and EE + +When changing shared code, always test both editions: + +```bash +# Community Edition +CGO_ENABLED=0 go test -tags='testonly' ./vault/identity -timeout=45m -parallel=20 + +# Enterprise Edition +CGO_ENABLED=0 VAULT_ADDR= VAULT_TOKEN= VAULT_DEV_ROOT_TOKEN_ID= VAULT_ACC= \ +go test -tags='enterprise,testonly' ./vault/identity -timeout=45m -parallel=20 +``` + +## Build Tags in Code + +### File Naming + +``` +feature.go # Shared (CE + EE) +feature_oss.go # CE / Open Source only +feature_ent.go # EE only +feature_test.go # Shared tests +feature_ent_test.go # EE tests only +``` + +### Tag Syntax + +```go +//go:build enterprise +// Enterprise-only code + +//go:build !enterprise +// CE-only code + +//go:build ent +// +build ent +// Enterprise test +``` + +## Writing Tests + +### Table-Driven Pattern + +```go +func TestCreateEntity(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + input string + expectError bool + }{ + {"valid input", "test-entity", false}, + {"empty input", "", true}, + } + + for _, tt := range tests { + tt := tt // Capture range variable + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + // Test implementation + }) + } +} +``` + +### Test Cluster Setup + +```go +func TestWithCluster(t *testing.T) { + t.Parallel() + + cluster := vault.NewTestCluster(t, &vault.CoreConfig{}, + &vault.TestClusterOptions{HandlerFunc: vaulthttp.Handler}) + t.Cleanup(cluster.Cleanup) + + core := cluster.Cores[0].Core + vault.TestWaitActive(t, core) // CRITICAL: Always wait + + client := cluster.Cores[0].Client + // Test with client +} +``` + +## Integration and Acceptance Tests + +### Integration Tests + +```bash +# Basic integration tests +CGO_ENABLED=0 \ +VAULT_SKIP_LOGGING_LEASE_EXPIRATIONS=1 \ +VAULT_ADDR= \ +VAULT_TOKEN= \ +VAULT_ACC= \ +go test -tags='enterprise,testonly' ./integ -v -timeout=120m -parallel=4 + +# With race detector +CGO_ENABLED=1 \ +VAULT_SKIP_LOGGING_LEASE_EXPIRATIONS=1 \ +VAULT_ADDR= \ +VAULT_TOKEN= \ +VAULT_ACC= \ +go test -tags='enterprise,testonly' -race ./integ -v -timeout=120m -parallel=4 +``` + +### Acceptance Tests + +**WARNING**: These may incur costs or modify real resources. + +```bash +VAULT_ACC=1 \ +CGO_ENABLED=0 \ +go test -tags='enterprise,testonly' ./... -v -timeout=60m +``` + +## Common Failures & Fixes + +| Error | Cause | Fix | +|-------|-------|-----| +| `undefined: ConstantName` | Missing build tags | Use `-tags='enterprise,testonly'` | +| `nil pointer dereference` | Uninitialized map/struct | `m := make(map[string]string)` | +| `context deadline exceeded` | Cluster not ready | Add `vault.TestWaitActive(t, core)` | +| `data race detected` | Concurrent access | Add mutex, run with `-race` and `CGO_ENABLED=1` | +| `redeclared type` | Build tag issue | Check `//go:build` tags | + +## Pre-PR Checklist + +- [ ] Enterprise tests pass: `CGO_ENABLED=0 VAULT_ADDR= VAULT_TOKEN= VAULT_DEV_ROOT_TOKEN_ID= VAULT_ACC= go test -tags='enterprise,testonly' ./path -timeout=45m -parallel=20` +- [ ] CE tests pass: `CGO_ENABLED=0 go test -tags='testonly' ./path -timeout=45m -parallel=20` +- [ ] Race detector passes: `CGO_ENABLED=1 VAULT_ADDR= VAULT_TOKEN= VAULT_DEV_ROOT_TOKEN_ID= VAULT_ACC= go test -tags='enterprise,testonly' -race ./path -timeout=60m -parallel=20` +- [ ] Tests pass multiple runs +- [ ] **`make fmt` applied** (always the final step) + +## Final Step: Format Code + +**Always run `make fmt` as the last step** in any development or test authoring cycle: + +```bash +make fmt +``` + +This ensures all Go files are properly formatted before committing or submitting a PR. diff --git a/vault/understanding/skills/acronym-helper/SKILL.md b/vault/understanding/skills/acronym-helper/SKILL.md new file mode 100644 index 0000000..b9b13f0 --- /dev/null +++ b/vault/understanding/skills/acronym-helper/SKILL.md @@ -0,0 +1,46 @@ +--- +name: vault-acronym-helper +description: Provide context and definitions for HashiCorp Vault acronyms, terminology, and usage guidelines. Use when encountering unfamiliar Vault-specific acronyms (e.g., PKI, KMIP, DR, HVD, VSO), normalizing terminology in documentation, understanding and reviewing Vault code, docs/blogs/UI copy, or understanding Vault concepts like auth methods, secrets engines, or storage backends. +--- + +# Vault Acronym Helper + +Lookup and normalize HashiCorp Vault acronyms, terminology, and usage conventions. + +## Quick Lookup + +For acronym definitions and terminology guidance, consult [references/acronyms.md](references/acronyms.md). + +## Key Terminology Corrections + +Apply these corrections when writing or reviewing Vault content: + +| Deprecated Term | Correct Term | +|-----------------|--------------| +| auth backend | **auth method** | +| secrets backend | **secrets engine** | +| master key | **root key** | +| Raft (as feature) | **Integrated Storage** | +| generic secrets | **KV secrets** | + +## Capitalization Rules + +- **Lowercase in body text**: secret, token, cluster, secrets engine, auth method +- **Capitalize proper names**: Vault Agent, Shamir Seal, Integrated Storage +- **Init Caps in headings only** + +## Common Acronyms (Quick Reference) + +| Acronym | Meaning | +|---------|---------| +| DR | Disaster Recovery | +| PKI | Public Key Infrastructure | +| KMIP | Key Management Interoperability Protocol | +| HVD | HCP Vault Dedicated | +| HVE | HashiCorp Vault Enterprise | +| HVS | HCP Vault Secrets | +| VSO | Vault Secrets Operator | +| DEK | Data Encryption Key | +| KEK | Key Encryption Key | + +For complete acronym list with detailed explanations, see [references/acronyms.md](references/acronyms.md). diff --git a/vault/understanding/skills/acronym-helper/references/acronyms.md b/vault/understanding/skills/acronym-helper/references/acronyms.md new file mode 100644 index 0000000..3466bdf --- /dev/null +++ b/vault/understanding/skills/acronym-helper/references/acronyms.md @@ -0,0 +1,113 @@ +# Vault Acronyms and Terminology Reference + +## Table of Contents +- [Acronyms](#acronyms) +- [Style & Usage Rules](#style--usage-rules) +- [Vault Terms and Concepts](#vault-terms-and-concepts) + +--- + +## Acronyms + +| Acronym | Expansion | Meaning | +|---------|-----------|---------| +| **ACME** | Automatic Certificate Management Environment | Protocol for automated certificate issuance | +| **ADP** | Advanced Data Protection | Vault Enterprise module for protecting secrets in external systems | +| **BYOK** | Bring Your Own Key | Customers generate/manage keys locally | +| **CLM** | Certificate Lifecycle Management | Certificate creation, revocation, expiration | +| **DEK** | Data Encryption Key | Vault's encryption key, protected by root key | +| **DR** | Disaster Recovery | Strategies for site/datacenter failover | +| **EGP** | Endpoint Governing Policy | Sentinel policy for specific Vault path (Enterprise) | +| **EMR** | Electronic Medical Record | Digital clinical data (e.g., Epic) | +| **FDE** | Full Disk Encryption | Encrypts all data on disk | +| **FIPS** | Federal Information Processing Standard | US/Canadian crypto standards | +| **FPE** | Format Preserving Encryption | Ciphertext preserves input format | +| **FSM** | Finite State Machine | Deterministic state machine for log ordering | +| **GDPR** | General Data Protection Regulation | EU data protection regulation | +| **GRC** | Governance, Risk and Compliance | Strategy combining governance, risk, compliance | +| **HIPAA** | Healthcare Insurance Portability and Accountability Act | US healthcare data protection | +| **HVD** | HCP Vault Dedicated | HashiCorp-managed Vault Enterprise on HCP | +| **HVE** | HashiCorp Vault Enterprise | Self-managed Vault Enterprise | +| **HVS** | HCP Vault Secrets | Cloud-native secrets management service | +| **KEK** | Key Encryption Key | Key that encrypts another key | +| **KMIP** | Key Management Interoperability Protocol | OASIS protocol for key lifecycle management | +| **KMSE** | Key Management Secret Engine | Vault secrets engine for key management | +| **NIST** | National Institute of Standards and Technology | US standards body | +| **OIDC** | OpenID Connect | Identity protocol for SSO (Okta, Ping, Google) | +| **PAM** | Privileged Access Management | Managing privileged credentials | +| **PCI** | Payment Card Information | PCI DSS security standard | +| **PHI** | Protected Health Information | HIPAA-governed healthcare data | +| **PII** | Personally Identifiable Information | Data identifying a person | +| **PKI** | Public Key Infrastructure | Certificate and public-key crypto management | +| **PQC** | Post-Quantum Cryptography | Algorithms secure against quantum attacks | +| **PR** | Performance Replication / Replica | Enterprise replication mode | +| **PRNG** | PseudoRandom Number Generator | Algorithm for key/nonce generation | +| **RGP** | Role Governing Policy | Sentinel RBAC policy (Enterprise) | +| **RUM** | Resources Under Management | Measure of Vault-managed resources | +| **SDP** | Software-Defined Perimeter | Dynamic infrastructure security model | +| **SSRF** | Server-Side Request Forgery | Web vulnerability for unintended requests | +| **TDE** | Transparent Database Encryption | DB encryption at rest | +| **VCS** | Vault Cloud Secrets | Former name for HCP Vault Secrets | +| **VSI** | Vault Secure Introduction | Process of obtaining initial client token | +| **VSO** | Vault Secrets Operator | Kubernetes secrets integration | + +--- + +## Style & Usage Rules + +### Term Corrections + +| Incorrect | Correct | Notes | +|-----------|---------|-------| +| auth backend | **auth method** | Old term | +| secrets backend | **secrets engine** | Old term | +| master key | **root key** | Being renamed | +| Raft | **Integrated Storage** | Feature name is Integrated Storage | +| generic secrets | **KV secrets** | Prefer KV secrets engine | +| Secret (capitalized) | **secret** | Lowercase in body text | +| Token (capitalized) | **token** | Lowercase in body text | +| Vault Cluster | **Vault cluster** | Lowercase "cluster" in body | +| HCP Portal | **HCP portal** | Lowercase "portal" in body | +| Secrets Engine | **secrets engine** | Lowercase in body text | + +### Proper Capitalization (Always Capitalize) + +- **Vault Agent** - Proper name +- **Shamir Seal** - Named algorithm +- **Integrated Storage** - Product feature name + +### Key Clarifications + +**Unseal keys vs Shamir keys:** +- Default Shamir seal: unseal keys ARE Shamir keys +- Auto-unseal: recovery keys are Shamir keys; unseal key from provider (e.g., AWS) + +**Storage Backend vs Integrated Storage:** +- Integrated Storage is ONE of the supported storage backends +- Integrated Storage is the only internal storage option +- Don't use "Raft" as the feature name + +**GPG vs PGP:** +- GPG software → **GnuPG** +- The keys → **PGP keys** (regardless of creation tool) + +**Client token vs Vault token:** +- Both terms are interchangeable +- CLI output often shows "client token" + +--- + +## Vault Terms and Concepts + +| Term | Definition | +|------|------------| +| **backend** | Historical term. auth backend → auth method; secrets backend → secrets engine; storage backend remains | +| **cipher** | Algorithm encrypting plaintext to ciphertext (e.g., AES-256-GCM, RSA) | +| **cryptographic barrier** | Encryption layer protecting all Vault data at rest using AES-256-GCM | +| **FedRAMP** | US program for cloud security assessment/authorization | +| **quorum** | Majority in peer set: floor(n/2) + 1 (e.g., 5 nodes → quorum = 3) | +| **Shamir's** | Shamir's Secret Sharing Algorithm for manual unseal | +| **tokenization** | Replacing sensitive data with tokens; vaultless tokenization preserves format | +| **Tweak** | Non-secret value in Transformation Secrets Engine for FPE context | +| **Watchtower** | Former code name for **Boundary** | +| **Workload IdP** | Workload Identity Provider in HCP context |