Skip to content

Add scale benchmark tests and document performance limits #175

Description

@vriesdemichael

Summary

There is no scale testing or documented performance characteristics for the operator. Users don't know how many realms/clients the operator can handle or when they should deploy multiple operators.

Problem

Currently unknown:

  • Maximum realms per operator instance
  • Maximum clients per realm
  • Reconciliation latency at scale (100, 500, 1000 resources)
  • Memory usage growth with resource count
  • CPU usage during mass reconciliation events

Proposed Solution

1. Create Benchmark Test Suite

# tests/benchmark/test_scale.py
@pytest.mark.benchmark
class TestScaleCharacteristics:
    async def test_100_realms_reconciliation_time(self):
        """Measure time to reconcile 100 realms from scratch."""
        
    async def test_500_clients_per_realm(self):
        """Measure reconciliation with 500 clients in single realm."""
        
    async def test_operator_memory_at_scale(self):
        """Track RSS memory with increasing resource counts."""

2. Run in CI (Optional, Long-Running)

Add workflow that runs weekly or on-demand:

  • Spin up larger Kind cluster (multi-node)
  • Create 100, 500, 1000 resources
  • Measure and record metrics
  • Fail if regression detected

3. Document Results

Add to documentation:

## Performance Characteristics

| Metric | Value | Notes |
|--------|-------|-------|
| Realms per operator | ~100 | Tested configuration |
| Clients per realm | ~500 | Beyond this, consider sharding |
| Reconciliation p99 | <5s | With rate limiting at defaults |
| Memory per 100 resources | ~50MB | Approximate growth |

Acceptance Criteria

  • Benchmark test suite created
  • Results documented in docs/operations/scaling.md
  • Guidance on when to deploy multiple operators
  • Memory and CPU recommendations based on scale

References

  • ADR-049: Horizontal scaling via multiple operator deployments
  • Review finding: TODO/SENIOR_SRE_REVIEW.md

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions