First of all - really cool project! The idea of transparently scaling pods to zero with checkpoint/restore and near-instant activation is great, and it works impressively well. Thanks for building it.
Problem
Enabling zeropod on an existing workload is currently an all-or-nothing decision. To find out how a pod would behave - how often it would scale down, whether the scaledown-duration is tuned right, how often incoming traffic would trigger restores - you have to actually let zeropod checkpoint and restore it. For production or production-like workloads this makes evaluation risky: a checkpoint failure, restore latency on first connection, or an application that doesn't survive checkpointing all have real user impact.
zeropod.ctrox.dev/disable-checkpointing doesn't help here, since it still stops the process (restore just becomes a restart).
Proposal
Add a dry-run mode, e.g. via a new annotation:
zeropod.ctrox.dev/dry-run: "true"
When enabled, the shim tracks activity and runs the scale-down timer exactly as it does today, but when the timer fires it does not checkpoint or stop the container. Instead it:
- logs that the container would have been scaled down (and later, when TCP activity resumes, that it would have been restored),
- emits metrics so the behaviour can be evaluated over time, e.g. counters like
zeropod_dry_run_checkpoints_total / zeropod_dry_run_restores_total, or alternatively a dry_run label on the existing running/checkpoint metrics,
Since the eBPF activity tracking works independently of checkpointing, "would restore" events can be derived from the first tracked connection after a would-be scale-down, giving a realistic picture of the checkpoint/restore cycle the pod would go through.
Use cases
- Safely evaluating zeropod on existing production workloads before opting in.
- Tuning
scaledown-duration against real traffic patterns without any impact.
- Estimating potential resource savings (time the pod would have spent scaled down) to decide which workloads are worth migrating to the zeropod runtime class.
Alternatives considered
disable-checkpointing: still stops the workload, so it's not usable for risk-free evaluation.
- Estimating from external traffic metrics: doesn't use zeropod's actual activity tracking (eBPF TCP tracking, probe detection), so results can diverge from what zeropod would really do.
First of all - really cool project! The idea of transparently scaling pods to zero with checkpoint/restore and near-instant activation is great, and it works impressively well. Thanks for building it.
Problem
Enabling zeropod on an existing workload is currently an all-or-nothing decision. To find out how a pod would behave - how often it would scale down, whether the
scaledown-durationis tuned right, how often incoming traffic would trigger restores - you have to actually let zeropod checkpoint and restore it. For production or production-like workloads this makes evaluation risky: a checkpoint failure, restore latency on first connection, or an application that doesn't survive checkpointing all have real user impact.zeropod.ctrox.dev/disable-checkpointingdoesn't help here, since it still stops the process (restore just becomes a restart).Proposal
Add a dry-run mode, e.g. via a new annotation:
When enabled, the shim tracks activity and runs the scale-down timer exactly as it does today, but when the timer fires it does not checkpoint or stop the container. Instead it:
zeropod_dry_run_checkpoints_total/zeropod_dry_run_restores_total, or alternatively adry_runlabel on the existing running/checkpoint metrics,Since the eBPF activity tracking works independently of checkpointing, "would restore" events can be derived from the first tracked connection after a would-be scale-down, giving a realistic picture of the checkpoint/restore cycle the pod would go through.
Use cases
scaledown-durationagainst real traffic patterns without any impact.Alternatives considered
disable-checkpointing: still stops the workload, so it's not usable for risk-free evaluation.