SK-255 // update KWOK stages to include configurable delays by ogorman89 · Pull Request #255 · acrlabs/simkube

ogorman89 · 2026-06-04T23:27:30Z

Description and Rationale

we want to include reasonable container startup and image pull delays in our KWOK stages to present

How

KWOK supports configurable stages and provides detailed examples for how to properly chain them in their repo
added a new stage pod-create that includes a delay and jitter to simulate container startup time
added a delay and jitter to our existing pod-ready to simulate image pull time
currently using the defaults delays and jitter from the kwok examples but expect to fine tune these defaults

delay:
    durationMilliseconds: 1000  # indicates the default delay in milliseconds
    durationFrom:  # if provided via annotations overrides the durationMilliseconds 
      expressionFrom: .metadata.annotations["pod-create.stage.kwok.x-k8s.io/delay"]
    jitterDurationMilliseconds: 5000  # indicates the default jitter in milliseconds
    jitterDurationFrom:  # if provided via annotations overrides the jitterDurationMilliseconds 
      expressionFrom: .metadata.annotations["pod-create.stage.kwok.x-k8s.io/jitter-delay"]

JitterDurationMilliseconds is the duration plus an additional amount chosen uniformly at random from the interval between DurationMilliseconds and JitterDurationMilliseconds. ~ kwok source

Test Steps

I ran local simulations with the old stages and new stages

Old Stages

pods reach running nearly immediately

New Stages

pods now have two slight delays w/ jitter

Other Notes

I think the next step for this feature is a PR to update the annotations which override the defaults set in this PR, the annotations receptors are ready on the KWOK stages but pods will need to be assigned matching matching annotations to override the defaults
[ X ] I certify that this PR does not contain any code that has been generated with GitHub Copilot or any other AI-based code generation tool, in accordance with this project's policies.

linear-code · 2026-06-04T23:27:33Z

SK-255

codecov · 2026-06-04T23:29:39Z

Codecov Report

❌ Patch coverage is 70.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.64%. Comparing base (1beaf19) to head (649b980).

Files with missing lines	Patch %	Lines
sk-cli/src/run.rs	0.00%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #255      +/-   ##
==========================================
- Coverage   78.69%   78.64%   -0.05%     
==========================================
  Files          62       62              
  Lines        3919     3939      +20     
==========================================
+ Hits         3084     3098      +14     
- Misses        835      841       +6

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

drmorr0 · 2026-06-05T17:35:39Z

Ok I'm writing this down for my benefit, because I was getting really confused looking at the stage files.

Stages are only triggered for pods that have been scheduled to a node that KWOK manages, which is why there's no condition on the stages to check if the pod has been scheduled, because it must have been.
In the existing set of stages (i.e., before this PR) we use the existing of a pod IP to determine if/whether to apply the pod running/ready stage. The template for pod-ready.yml sets the pod IP using the builtin-to-KWOK PodWithIP template function.
In the new set of stages, we have a pod-created stage which is simulating "the pod is pulling the image but isn't running yet" and then we modify the pod-ready stage to be "the image pull is complete and we should start it up". We now use the presence of a pod IP to determine whether to apply this stage.
Once the pod transitions to the "pod created" stage (after some jitter), we set the container statuses for all of the containers to ContainerCreating (which seems correct, the ready field is false here); we also set two conditions, ContainersReady and Ready to False. (I initially read the "type" as ContainersNotReady here and was very confused because I thought it was saying that ContainersNotReady was False, e.g., containers ARE ready, but OK now I'm tracking).
Finally, we transition to the pod is running/ready for Initialized pods that are Pending, but that don't have any "container StartedAt" values.

Also, just for reference, we use another label (simkube.kwok.io/stage-complete) to indicate pods that shouldn't run forever. I regularly get confused as to why that label is there and how it gets applied, but the answer is "the mutation webhook applies it if a bunch of other conditions apply that say "this pod should terminate based on the information we have in the trace".

@ogorman89 it might (????) be nice to document this in a README in the kwok stages directory?

drmorr0 · 2026-06-05T16:57:07Z

    kind: Pod
  selector:
    matchExpressions:
-      - key: .metadata.deletionTimestamp


This is a dumb nitpick but can we not add quotes unless we need them?

(And can we also use double-quotes where needed, unless we have to use single-quotes?)

I'm adding linting rules for this!

Ok I added what we talked about to a yamllint config and put it in its own config file .config/.yamllint.yaml

N.B., we already have a .yamllint config in here: https://github.com/acrlabs/simkube/blob/main/.yamllint

However, stuffing this into .config seems nicer than having it at the top level, can you merge the existing file into the one you created?

drmorr0 · 2026-06-05T17:04:51Z

+  delay:
+    durationMilliseconds: 1000
+    durationFrom:
+      expressionFrom: .metadata.annotations["pod-create.stage.kwok.x-k8s.io/delay"]


In pod-complete.yml we pull the value from simkube.kwok.io/stage-complete-time. I don't totally remember why I used this format (if I had to hazard a guess, it's probably because I thought this annotation format was ugly; which, I still think, lol), but I think we should be consistent.

Maybe this is simkube.kwok.io/stage-create-delay and simkube.kwok.io/stage-create-delay-jitter? And similar for stage-running? Or if you have a better idea that's fine too.

We should also add these values into constants.rs so we can reference them later.

That made sense to me with one minor change: istead of calling it running I am using ready since that maps directly to the kwok stage: e.g. simkube.kwok.io/stage-ready-delay -> pod-ready kwok stage.

OK I have one dumb extra request here; in #256, for "making my life easier" reasons I changed all the simkube annotation prefixes to be simkube.io; the labels/annotations got renamed to simkube.io/kwok-stage-complete-time, etc. Can you make the corresponding change here? I can rebase on top of your change once mine is ready and that way we don't have messy merge conflicts.

drmorr0 · 2026-06-08T23:26:20Z

Kubernetes Object DAG

%%{init: {'themeVariables': {'mainBkg': '#ddd'}}}%%
graph LR

classDef default color:#000
subgraph global
  direction LR
  global/simkube[<b>Namespace</b><br>simkube]
%% DELETED OBJECTS START
%% DELETED OBJECTS END
end

subgraph sk-tracer
  direction LR
  simkube/sk-tracer-svc[<b>Service</b><br>sk-tracer-svc]
  simkube/sk-tracer-depl[<b>Deployment</b><br>sk-tracer-depl]
  simkube/sk-tracer-sa[<b>ServiceAccount</b><br>sk-tracer-sa]
  sk-tracer/sk-tracer-crb[<b>ClusterRoleBinding</b><br>sk-tracer-crb]
  simkube/sk-tracer-tracer-config[<b>ConfigMap</b><br>sk-tracer-tracer-config]
  simkube/sk-tracer-sa--->simkube/sk-tracer-depl
  sk-tracer/sk-tracer-crb--->simkube/sk-tracer-depl
  simkube/sk-tracer-tracer-config--->simkube/sk-tracer-depl
%% DELETED OBJECTS START
%% DELETED OBJECTS END
end

subgraph sk-ctrl
  direction LR
  simkube/sk-ctrl-depl[<b>Deployment</b><br>sk-ctrl-depl]
  simkube/sk-ctrl-sa[<b>ServiceAccount</b><br>sk-ctrl-sa]
  sk-ctrl/sk-ctrl-crb[<b>ClusterRoleBinding</b><br>sk-ctrl-crb]
  simkube/sk-ctrl-sa--->simkube/sk-ctrl-depl
  sk-ctrl/sk-ctrl-crb--->simkube/sk-ctrl-depl
%% DELETED OBJECTS START
%% DELETED OBJECTS END
end

global--->sk-tracer
global--->sk-ctrl

%% STYLE DEFINITIONS START
%% STYLE DEFINITIONS END

New object
Deleted object
Updated object
Updated object (causes pod recreation)

Detailed Diff

drmorr0 · 2026-06-10T05:35:44Z

    pub remote_write_endpoint: Option<String>,

+    #[arg(long, long_help = "image pull delay in milliseconds", default_value = IMAGE_PULL_DELAY)]
+    pub image_pull_delay: i32,


nit: why i32 instead of u32? I don't think these can ever be negative.

drmorr0 · 2026-06-10T05:36:29Z

 pub const DEFAULT_METRICS_SVC_ACCOUNT: &str = "prometheus-k8s";
 pub const DRIVER_ADMISSION_WEBHOOK_PORT: &str = "8888";
 pub const SK_LEASE_NAME: &str = "sk-lease";
+pub const POD_STARTUP_DELAY: &str = "0";


I don't think we should hard-code a bunch of 0 constants in here, it's fine to just write default_value = "0" above.

drmorr0 · 2026-06-10T05:41:06Z

    pub speed: Option<f64>,
    pub paused_time: Option<DateTime<Utc>>,
    pub hooks: Option<SimulationHooksConfig>,
+    pub image_pull_delay: Option<i32>,


Optional: it might make sense to pull these out into their own struct (SimulationDelayParameters or something like that)? Similar to how we have the Driver params and the metrics configs.

Yea that makes sense, I created a SimulationDelayParameters struct to house them.

drmorr0 · 2026-06-10T05:41:33Z

    )]
    pub remote_write_endpoint: Option<String>,

+    #[arg(long, long_help = "image pull delay in milliseconds", default_value = IMAGE_PULL_DELAY)]


Optional: it might make sense to put these in their own section/heading under the help output?

I put them in a section called Delays, it could use a better name.

"Lifecycle Configuration"?

drmorr0 · 2026-06-10T19:35:15Z

    pub speed: Option<f64>,
    pub paused_time: Option<DateTime<Utc>>,
    pub hooks: Option<SimulationHooksConfig>,
+    pub simulation_delay_parameters: SimulationDelayParameters,


Maybe also "lifecycle parameters" here?

I called this lifecycle_params to stay consistent, it looks like we typically use the shorter "_config" and "_params" to be concise.

ogorman89 added 3 commits June 3, 2026 16:23

kwok stages: add pod-create, update pod-ready

50d1e77

Merge remote-tracking branch 'origin/main' into ian/kwok-stages

8a7209d

update selectors

bf3ef1b

ogorman89 self-assigned this Jun 4, 2026

ogorman89 requested a review from drmorr0 June 4, 2026 23:27

drmorr0 requested changes Jun 5, 2026

View reviewed changes

set delays and jitter via skcli

b9b0618

lints

3dc48aa

ogorman89 requested a review from drmorr0 June 8, 2026 23:50

drmorr0 reviewed Jun 10, 2026

View reviewed changes

ogorman89 added 4 commits June 10, 2026 10:03

annotations simkube.kwok.io -> simkube.io, update precommit

ec483b9

updated stage labels, add SimulationDelayParameters, add help subsection

5daf7cb

lints for unquoted strings

1ef7a32

bump codecov version

be15202

drmorr0 approved these changes Jun 10, 2026

View reviewed changes

renaming: lifecycle_params

649b980

ogorman89 merged commit 0ae43f0 into main Jun 11, 2026
8 of 9 checks passed

ogorman89 deleted the ian/kwok-stages branch June 11, 2026 02:36

Uh oh!

Uh oh!

Conversation

ogorman89 commented Jun 4, 2026

Description and Rationale

How

Test Steps

Old Stages

New Stages

Other Notes

Uh oh!

linear-code Bot commented Jun 4, 2026

Uh oh!

codecov Bot commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

drmorr0 commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ogorman89 Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

drmorr0 commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Kubernetes Object DAG

Detailed Diff

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Jun 4, 2026 •

edited

Loading

drmorr0 commented Jun 5, 2026 •

edited

Loading

ogorman89 Jun 8, 2026 •

edited

Loading

drmorr0 commented Jun 8, 2026 •

edited

Loading