Feature: Enable scaling existing clusters by adding more worker nodes

[mrgb7]

Feature Request: Cluster Scaling - Add Worker Nodes to Existing Clusters

Overview

Enable users to dynamically scale existing clusters by adding more worker nodes after the initial cluster creation. This feature would provide flexibility to accommodate changing workload demands and resource requirements.

Use Cases

• Increased workload demands: Scale up when applications require more compute resources
• High availability: Add nodes for better fault tolerance and load distribution
• Seasonal scaling: Temporarily increase capacity during peak usage periods
• Growth accommodation: Scale infrastructure as the organization grows
• Resource optimization: Add specialized nodes for specific workloads

Proposed Features

1. Scale Command

playground scale cluster <cluster-name> --workers <count>
playground scale cluster <cluster-name> --add-workers 2
playground scale cluster <cluster-name> --set-workers 5

2. Configuration Options

• Specify number of worker nodes to add
• Set total desired worker node count
• Configure new node specifications (CPU, memory, disk)
• Choose node placement and availability zones
• Set node labels and taints for workload scheduling

3. Validation and Safety

• Validate cluster exists and is in running state
• Check resource availability before scaling
• Prevent scaling beyond infrastructure limits
• Graceful handling of scaling failures

4. Progress Monitoring

• Real-time progress updates during scaling operation
• Status reporting for node provisioning
• Integration with existing cluster status commands

Implementation Considerations

Technical Requirements

• Extend cluster configuration to support dynamic scaling
• Implement node provisioning logic for existing clusters
• Update cluster state management
• Add validation for scaling operations
• Ensure compatibility with different infrastructure providers

User Experience

• Clear progress indicators during scaling
• Informative error messages for scaling failures
• Documentation with scaling best practices
• Examples for common scaling scenarios

Acceptance Criteria

• playground scale command implemented with worker node options
• Support for both additive (--add-workers) and absolute (--set-workers) scaling
• Validation prevents invalid scaling operations
• Progress monitoring during scaling operations
• Updated documentation with scaling examples
• Integration tests for scaling functionality
• Graceful error handling and rollback capabilities

Additional Features (Future Enhancements)

• Auto-scaling based on resource utilization
• Scheduled scaling operations
• Integration with monitoring systems
• Multi-zone scaling strategies
• Cost optimization recommendations

Benefits

• Flexibility: Adapt cluster size to changing requirements
• Resource efficiency: Scale up only when needed
• High availability: Distribute workloads across more nodes
• User convenience: Simple commands for complex operations
• Cost management: Right-size clusters for current needs