Summary
There is no scale testing or documented performance characteristics for the operator. Users don't know how many realms/clients the operator can handle or when they should deploy multiple operators.
Problem
Currently unknown:
- Maximum realms per operator instance
- Maximum clients per realm
- Reconciliation latency at scale (100, 500, 1000 resources)
- Memory usage growth with resource count
- CPU usage during mass reconciliation events
Proposed Solution
1. Create Benchmark Test Suite
# tests/benchmark/test_scale.py
@pytest.mark.benchmark
class TestScaleCharacteristics:
async def test_100_realms_reconciliation_time(self):
"""Measure time to reconcile 100 realms from scratch."""
async def test_500_clients_per_realm(self):
"""Measure reconciliation with 500 clients in single realm."""
async def test_operator_memory_at_scale(self):
"""Track RSS memory with increasing resource counts."""
2. Run in CI (Optional, Long-Running)
Add workflow that runs weekly or on-demand:
- Spin up larger Kind cluster (multi-node)
- Create 100, 500, 1000 resources
- Measure and record metrics
- Fail if regression detected
3. Document Results
Add to documentation:
## Performance Characteristics
| Metric | Value | Notes |
|--------|-------|-------|
| Realms per operator | ~100 | Tested configuration |
| Clients per realm | ~500 | Beyond this, consider sharding |
| Reconciliation p99 | <5s | With rate limiting at defaults |
| Memory per 100 resources | ~50MB | Approximate growth |
Acceptance Criteria
References
- ADR-049: Horizontal scaling via multiple operator deployments
- Review finding: TODO/SENIOR_SRE_REVIEW.md
Summary
There is no scale testing or documented performance characteristics for the operator. Users don't know how many realms/clients the operator can handle or when they should deploy multiple operators.
Problem
Currently unknown:
Proposed Solution
1. Create Benchmark Test Suite
2. Run in CI (Optional, Long-Running)
Add workflow that runs weekly or on-demand:
3. Document Results
Add to documentation:
Acceptance Criteria
docs/operations/scaling.mdReferences