Skip to content

Per-deploy Rholang execution scales backwards with CPU #493

Description

@elmiringos

What did you expect to see?

Per-deploy block creation and replay times that stay flat or decrease as the container's CPU limit grows. A TransferTerm deploy is a small, single-shard; doubling CPU should at worst leave it unchanged.

Concretely: a 1-deploy block replayed on a 2-CPU node in 0.8 s should not take longer to replay on an 8-CPU node of the same hardware family.

What did you see instead?

Replay (and propose-core) time grows with CPU limit:

k8s CPU limit (cgroup quota) 1-deploy block replay 10-deploy block replay
2 CPU (minimum) 0.75–0.82 s not measured (test ramp stops earlier)
8 CPU (mid) 2.1–3.7 s (3–5× slower) 14–26 s

The downstream effect: with three validators concurrently proposing at the same height under heartbeat-off conditions, mid+ profiles can't accumulate.

FT returns zero finalizations even though the DAG is being populated (43 distinct blocks observed, 0 finalized). With a 15-minute window, the chain can converge — at BestCap = 12, BestTPS = 0.20 deploys/sec — which is still ~5× below the 1.0 deploys/sec/profile we'd extrapolate from the minimum-profile baseline.

Steps to reproduce the bug

  1. Deploy a 3-validator shard on Kubernetes with explicit cgroup CPU limits — e.g. one shard at 2 CPU/pod, another at 8 CPU/pod.
  2. Disable heartbeat on both shards (heartbeat-enabled = false in rnode.conf).
  3. From outside the validator pods, run a tight loop on each validator in parallel:
    • flood N transfer deploys via the public HTTP /api/deploy endpoint
    • call POST /api/propose on the admin port (40405)
    • wait for the validator's tip to advance via /api/blocks/1
    • repeat for ~3-15 minutes
  4. Compare per-deploy block replay time across the two shards.

What version of F1R3FLY Node are you using?

  • Source: f1r3node Rust 0.4.13

What is your environment?

  • OS / kernel: Linux (k3s host)
  • Container runtime: containerd via k3s
  • Orchestrator: k3s single-node cluster
  • CPU limits: tested across cgroup CFS quotas of 2, 4, 8, 16, 32 CPUs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions