Feat/add CFS scheduler

[panicking]

Add CFS (Completely Fair Scheduler) to CPS VM thread execution

Motivation

The CPS VM's CpsThreadGroup.run() currently executes all runnable threads in insertion (ID) order to exhaustion in a single pass. A standing // TODO at that method acknowledges the problem: a tight Groovy loop that rarely suspends can monopolize the CPS VM thread, delaying all other pipeline branches in the same build. Later-created threads (higher IDs) always execute after earlier ones — there is no fairness guarantee.

Changes

Two commits that build on each other:

1. Performance Instrumentation (CpsFlowExecution)

Adds measurement hooks so every subsequent optimization can be benchmarked:

• 4 new TimingKind values: serializationRead, serializationWrite, compilationTotal, schedulerWait — recorded during normal pipeline execution and visible in thread dumps and support bundles.
• Per-thread execution counters on CpsThread: totalChunksRun, totalNanosExecuting, becameRunnableAt — foundation for vruntime tracking.
• Per-pass scheduler metrics on CpsThreadGroup: thread counts, runnable counts, pass duration, queue depth — logged at FINE level.

Zero behavioral change. All new fields are either transient (diagnostic) or primitive long (transparent to River serialization).

2. CFS Scheduler (CpsThreadGroup.run())

Replaces the "run all runnable threads" loop with a fairness algorithm adapted from Linux's Completely Fair Scheduler:

• Each CpsThread accumulates virtual runtime (vruntime) proportional to its wall-clock execution time divided by weight:

  vruntime += (elapsedNs * 1024) / max(1, weight)
  

• Each scheduler pass picks the thread with the lowest vruntime instead of iterating all threads in ID order.
• New threads start at min_vruntime so they get prompt attention rather than being starved.
• A dynamic quantum scales with system load:

  quantum = 50ms / max(1, activeBuilds / loadFactor)    floor = 5ms
  

The quantum is cooperative — checked after each CPS chunk completes, never during execution. Nearly all chunks suspend naturally at step boundaries (sleep, sh, semaphore) in microseconds, so the 50ms default generates zero additional overhead. The real fairness comes from vruntime ordering: CPU-heavy threads accumulate higher vruntime and naturally lose priority on subsequent passes.

Configuration

Property	Type	Default	Effect
org.jenkinsci.plugins.workflow.cps.CFS.baseQuantumMs	long (ms)	50	Maximum wall-clock time per thread per pass. Set to 0 to disable quantum enforcement while keeping CFS ordering.
org.jenkinsci.plugins.workflow.cps.CFS.loadFactor	int	10	Controls how aggressively the quantum shrinks under load. Lower = shrinks sooner.

Example JVM arguments:

-Dorg.jenkinsci.plugins.workflow.cps.CFS.baseQuantumMs=100
-Dorg.jenkinsci.plugins.workflow.cps.CFS.loadFactor=20

Files Changed

 doc/cfs-scheduler.md                               | 153 ++++++++++++
 .../jenkinsci/plugins/workflow/cps/CpsFlowExecution.java     |  26 ++-
 .../jenkinsci/plugins/workflow/cps/CpsThread.java  |  52 ++++-
 .../jenkinsci/plugins/workflow/cps/CpsThreadGroup.java       | 235 ++++++++++++++-----
 .../workflow/cps/CpsFlowExecutionTest.java         |   4 +-
 5 files changed, 412 insertions(+), 58 deletions(-)

How It Compares to Before

Aspect	Before	After
Thread selection	All runnable threads, ID order	One thread, lowest vruntime
Fairness	Later threads (higher IDs) always run last	Equal share among all threads
New threads	Starved until existing threads complete	Start at min_vruntime
Scheduler wait	Measured but not acted on	Measured and used for ordering
Configuration	None	Two system properties
Per-pass overhead	O(n) scan + execute all runnable	O(n) scan + execute one thread

Backward Compatibility

The new vruntime (long) and weight (int) fields on CpsThread are Java primitives, handled automatically by the existing River serialization. Old program.dat files deserialize with vruntime=0 and weight=0; the resume() method corrects weight=0 to the default 1024 on first use. No migration is needed.

Testing

All 79 CPS tests pass:

CpsFlowExecutionTest:        31 ✓
CpsThreadDumpTest:             4 ✓
CpsThreadTest:                 1 ✓
CpsScriptTest:                 7 ✓
SandboxContinuableTest:        1 ✓
ReplayActionTest:              9 ✓
ReplayPipelineCommandTest:     1 ✓
LoadStepTest:                  3 ✓
ParallelStepTest:             15 ✓
RestartingLoadStepTest:        7 ✓

Serialization round-trips are verified by the restart tests (RestartingLoadStepTest, ReplayActionTest).

Documentation

Complete documentation in doc/cfs-scheduler.md covering:

• vruntime tracking and scheduling weight
• Fair queue selection algorithm
• New thread initialization
• Dynamic quantum formula with scaling examples
• Configuration via system properties
• Performance characteristics
• Observability (FINE-level logging, schedulerWait timing)
• Future enhancement directions
  Deployment in our infrastructure for more testing

Submitter checklist

• Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
• Ensure that the pull request title represents the desired changelog entry
• Please describe what you did
• Link to relevant issues in GitHub or Jira
• Link to relevant pull requests, esp. upstream and downstream changes
• Ensure you have provided tests that demonstrate the feature works or the issue is fixed

[panicking]

@jglick can someone take a look? I'm testing it but I don't know is somenthing we want to explore or have

[jglick]

If you are referring to

workflow-cps-plugin/plugin/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java

Line 461 in e007b7f

// TODO: maybe instead of running all the thread, run just one thread in round robin

this dates to 95e7372 in #46 from a decade ago. I never expected this code to be touched again. Is there a concrete problem you are trying to solve here?

a tight Groovy loop that rarely suspends

suggests poorly written Pipeline script; Groovy code should be completing on the order of milliseconds.

[panicking]

If you are referring to

workflow-cps-plugin/plugin/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java

Line 461 in e007b7f

// TODO: maybe instead of running all the thread, run just one thread in round robin

this dates to 95e7372 in #46 from a decade ago. I never expected this code to be touched again. Is there a concrete problem you are trying to solve here?

a tight Groovy loop that rarely suspends

suggests poorly written Pipeline script; Groovy code should be completing on the order of milliseconds.

Yes, I have started from a TODO, so what you need are metrics that are showing some improvements at different loads. Right now, I'm using in my infrastructure but I can prepare some metrics