Haven't any idea what's wrong with scx_loader

[Anifyuli]

Hi, I encounter some problem with scx_scheds setup. It sometimes happen, and sometimes not. But this issue usually appear on https://github.com/whitehara/kernel-patch-fedora than this official build. It looks confusing because it often happen on official build too. It problem like this

● scx_loader.service - DBUS on-demand loader of sched-ext schedulers
     Loaded: loaded (/usr/lib/systemd/system/scx_loader.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (running) since Sun 2026-07-05 16:44:59 WIB; 2s ago
 Invocation: d5fed147f5b14127adc3eb7388473432
   Main PID: 731742 (scx_loader)
      Tasks: 11 (limit: 9140)
     Memory: 7.3M (peak: 28.2M)
        CPU: 463ms
     CGroup: /system.slice/scx_loader.service
             └─731742 /usr/bin/scx_loader

Jul 05 16:45:00 ThinkPad-X280 scx_loader[732177]: makes BPF programs fail to load with 'func_proto incompatible with
Jul 05 16:45:00 ThinkPad-X280 scx_loader[732177]: vmlinux'. This happens when the kernel was built with pahole < 1.26.
Jul 05 16:45:00 ThinkPad-X280 scx_loader[732177]: Fix: boot a kernel whose BTF was generated with pahole >= 1.26.
Jul 05 16:45:00 ThinkPad-X280 scx_loader[732177]: Affected distros include Ubuntu 24.04 LTS. See kernel commit
Jul 05 16:45:00 ThinkPad-X280 scx_loader[732177]: 9edd04c4189e ("docs: Raise minimum pahole version to 1.26 for
Jul 05 16:45:00 ThinkPad-X280 scx_loader[732177]: KF_IMPLICIT_ARGS kfuncs").
Jul 05 16:45:00 ThinkPad-X280 scx_loader[732177]: Caused by:
Jul 05 16:45:00 ThinkPad-X280 scx_loader[732177]:     0: Failed to load BPF program
Jul 05 16:45:00 ThinkPad-X280 scx_loader[732177]:     1: Invalid argument (os error 22)
Jul 05 16:45:01 ThinkPad-X280 scx_loader[731742]: [ERROR]: Failed to start scheduler (attempt 5/5)

Any idea what's wrong with it? I report it on Whitehara repo too for tracing it. This is my config

$ cat /etc/scx_loader.toml
default_sched = "scx_bpfland"
default_mode = "Auto"

[scheds.scx_bpfland]
auto_mode = ["-f","-k","-m","auto","-P"]
gaming_mode = ["-m","performance","-f"]
lowlatency_mode = ["-m","performance", "-w"]
powersave_mode = ["-s","20000","-t","100","-I","100"]
server_mode = ["-s","2000","-S"]

[scheds.scx_tickless]
auto_mode = []
gaming_mode = []
lowlatency_mode = []
powersave_mode = []
server_mode = []

[scheds.scx_flash]
auto_mode = []
gaming_mode = []
lowlatency_mode = []
powersave_mode = []
server_mode = []

[scheds.scx_rusty]
auto_mode = []
gaming_mode = []
lowlatency_mode = []
powersave_mode = []
server_mode = []

[scheds.scx_p2dq]
auto_mode = []
gaming_mode = []
lowlatency_mode = ["-y"]
powersave_mode = []
server_mode = ["--keep-running"]

[scheds.scx_lavd]
auto_mode = []
gaming_mode = ["--performance"]
lowlatency_mode = ["--performance"]
powersave_mode = ["--powersave"]
server_mode = []

Thanks for your hard work :)

[Enekololo]

[11:06 PM]-[eneko@fedora]-[~]-
$ sudo scx_lavd
2026-07-05T21:06:18.826968Z  INFO ThreadId(01) scx_lavd: scheds/rust/scx_lavd/src/main.rs:338: Autopilot mode is enabled.
2026-07-05T21:06:18.827136Z  INFO ThreadId(01) scx_lavd: scheds/rust/scx_lavd/src/main.rs:380: Energy model won't be used for CPU preference order.
2026-07-05T21:06:18.827143Z  INFO ThreadId(01) scx_lavd: scheds/rust/scx_lavd/src/main.rs:393: Pinned task slice mode is enabled (5000 us). Pinned tasks will use per-CPU DSQs.
2026-07-05T21:06:18.827149Z  INFO ThreadId(01) scx_lavd: scheds/rust/scx_lavd/src/main.rs:1066: Opts {
    verbose: 0,
    autopilot: true,
    autopower: false,
    performance: false,
    powersave: false,
    balanced: false,
    slice_max_us: 5000,
    slice_min_us: 500,
    lat_load_target_pct: 100,
    mig_delta_pct: 0,
    lb_low_util_pct: 25,
    lb_local_dsq_util_pct: 10,
    pinned_slice_us: Some(
        5000,
    ),
    preempt_shift: 6,
    cpu_pref_order: "",
    no_use_em: true,
    no_futex_boost: false,
    no_fast_lb: false,
    no_preemption: false,
    no_wake_sync: false,
    no_slice_boost: false,
    per_cpu_dsq: false,
    enable_cpu_bw: false,
    partial: false,
    no_core_compaction: false,
    no_freq_scaling: false,
    stats: None,
    monitor: None,
    monitor_sched_samples: None,
    log_level: "info",
    version: false,
    run_id: None,
    help_stats: false,
    libbpf: LibbpfOpts {
        relaxed_maps: None,
        pin_root_path: None,
        kconfig: None,
        btf_custom_path: None,
        bpf_token_path: None,
    },
    topology: None,
}
2026-07-05T21:06:18.936054Z  INFO ThreadId(01) scx_lavd: scheds/rust/scx_lavd/src/main.rs:613: capacity bound:  1024 (12.5%)
  primary CPUs:  [0]
  overflow CPUs: [4, 1, 5, 2, 6, 3, 7]
2026-07-05T21:06:18.936074Z  INFO ThreadId(01) scx_lavd: scheds/rust/scx_lavd/src/main.rs:613: capacity bound:  8192 (100%)
  primary CPUs:  [0]
  overflow CPUs: [1, 2, 3, 4, 5, 6, 7]
2026-07-05T21:06:19.010689Z  WARN ThreadId(01) scx_utils::libbpf_logger: rust/scx_utils/src/libbpf_logger.rs:12: libbpf: extern (func ksym) 'scx_bpf_error_bstr': func_proto [431] incompatible with vmlinux [60030]

2026-07-05T21:06:19.010775Z  WARN ThreadId(01) scx_utils::libbpf_logger: rust/scx_utils/src/libbpf_logger.rs:12: libbpf: failed to load BPF skeleton 'bpf_bpf': -EINVAL

Error: the running kernel's BTF has malformed scx kfunc prototype(s): __scx_bpf_dsq_insert_vtime, __scx_bpf_select_cpu_and, scx_bpf_cpu_rq, scx_bpf_cpuperf_cap, scx_bpf_cpuperf_cur, scx_bpf_cpuperf_set, scx_bpf_dsq_insert___v2, scx_bpf_error_bstr, scx_bpf_get_idle_smtmask, scx_bpf_get_idle_smtmask_node, scx_bpf_kick_cpu, scx_bpf_pick_any_cpu, scx_bpf_pick_any_cpu_node, scx_bpf_select_cpu_dfl, scx_bpf_sub_dispatch, scx_bpf_task_cgroup.

These kfuncs are KF_IMPLICIT_ARGS but their public BTF prototype
still carries the implicit 'struct bpf_prog_aux *' argument, which
makes BPF programs fail to load with 'func_proto incompatible with
vmlinux'. This happens when the kernel was built with pahole < 1.26.

Fix: boot a kernel whose BTF was generated with pahole >= 1.26.
Affected distros include Ubuntu 24.04 LTS. See kernel commit
9edd04c4189e ("docs: Raise minimum pahole version to 1.26 for
KF_IMPLICIT_ARGS kfuncs").

Caused by:
    0: Failed to load BPF program
    1: Invalid argument (os error 22)

the same happens for me too this is a bug

[0ldMonk]

I am running into same bug. From what i have gathered, scx works fine with kernel-cachyos-lts but not with latest one.

[sirlucjan]

These kfuncs are KF_IMPLICIT_ARGS but their public BTF prototype
still carries the implicit 'struct bpf_prog_aux *' argument, which
makes BPF programs fail to load with 'func_proto incompatible with
vmlinux'. This happens when the kernel was built with pahole < 1.26.

Fix: boot a kernel whose BTF was generated with pahole >= 1.26.
Affected distros include Ubuntu 24.04 LTS. See kernel commit
9edd04c4189e ("docs: Raise minimum pahole version to 1.26 for
KF_IMPLICIT_ARGS kfuncs").

Apparently, it's the older version of pahole that's causing the problem.

[uladziislau]

Regression: kernel-cachyos 7.0.12 → 7.1.2 — broken BTF breaks all sched-ext schedulers

Confirmed on Fedora 44 (kernel-cachyos from COPR):

🟢 Works

• 7.0.12-cachyos — scx_bpfland, scx_rusty, and all 14 schedulers work fine

🔴 Fails

• 7.1.2-cachyos — no sched-ext scheduler loads. Exact error:

Error: the running kernel BTF has malformed scx kfunc prototype(s)
These kfuncs are KF_IMPLICIT_ARGS but their public BTF prototype
still carries the implicit "struct bpf_prog_aux *" argument, which
makes BPF programs fail to load with "func_proto incompatible with
vmlinux". This happens when the kernel was built with pahole < 1.26.
Fix: boot a kernel whose BTF was generated with pahole >= 1.26.

This affects all 14 installed sched-ext schedulers: bpfland, rusty, layered, lavd, cosmos, flash, rustland, beerland, p2dq, tickless, cake, chaos, pandemonium, and the loader itself.

Root cause

Kernel 7.1.2-cachyos was built with pahole < 1.26. Since kernel commit 9edd04c4189e ("docs: Raise minimum pahole version to 1.26 for KF_IMPLICIT_ARGS kfuncs"), all KF_IMPLICIT_ARGS kfuncs require pahole ≥ 1.26 — actually ≥ 1.27, because decl_tag_kfuncs support was added in pahole 1.27 (see acmel/dwarves@72e88f29c).

The kernel Makefile gates --btf_features=decl_tag_kfuncs on pahole ≥ 1.26 (scripts/Makefile.btf). Without this flag, pahole emits no DECL_TAG BTF entries for __bpf_kfunc-annotated functions → resolve_btfids skips btf2btf() → vmlinux BTF keeps the 3-param proto with implicit aux argument → BPF programs expecting 2 params fail with "func_proto incompatible".

Kernel 7.0.12-cachyos works fine — so something changed in the COPR build environment between the two releases (possibly a dwarves package downgrade or missing update).

Workaround

sudo grubby --set-default /boot/vmlinuz-7.0.12-cachyos1.fc44.x86_64
sudo reboot

Fix for maintainers

Ensure dwarves >= 1.27 (currently pahole 1.30 is available in Fedora 44 repos) is installed in the COPR build root, then rebuild kernel-cachyos-7.1.2.

References

• Kernel patch raising minimum to 1.26: 9edd04c4189e
• pahole decl_tag_kfuncs: 72e88f29c

[sirlucjan]

The reason why kernel 7.0 works but 7.1 doesn't: torvalds/linux@9edd04c4189e

[SigonLegacy]

Issue persists in the current 7.2.0 release.

sigonlegacy@Polonius:~$ uname -a
Linux Polonius 7.2.0-cachyos1.fc44.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Aug 21 08:54:10 UTC 2026 x86_64 GNU/Linux
sigonlegacy@Polonius:~$ sudo scx_cake
[sudo] password for sigonlegacy: 
13:31:45 [INFO] 🍰 scx_cake 1.2.1 x86_64-unknown-linux-gnu
13:31:45 [INFO]    cores   12 physical + 8 SMT = 20 CPUs
13:31:45 [INFO]    slice   3000µs · queues 20 per-CPU vtime + 1 global wake
13:31:45 [INFO]    kernel  queued_wakeup on · dsq_peek native
13:31:45 [INFO]    irq     interrupt sinks tracked live by handler-time share
Error: the running kernel's BTF has malformed scx kfunc prototype(s): __scx_bpf_dsq_insert_vtime, scx_bpf_cid_override, scx_bpf_cid_to_cpu, scx_bpf_cid_topo, scx_bpf_cidperf_cap, scx_bpf_cpu_to_cid, scx_bpf_cpuperf_cap, scx_bpf_cpuperf_cur, scx_bpf_cpuperf_set, scx_bpf_dispatch_nr_slots, scx_bpf_get_idle_smtmask_node, scx_bpf_kick_cid, scx_bpf_kick_cpu, scx_bpf_pick_any_cpu, scx_bpf_select_cpu_dfl, scx_bpf_sub_dispatch, scx_bpf_task_cgroup, scx_bpf_task_set_dsq_vtime, scx_bpf_task_set_slice, scx_bpf_test_and_clear_cpu_idle.

These kfuncs are KF_IMPLICIT_ARGS but their public BTF prototype
still carries the implicit 'struct bpf_prog_aux *' argument, which
makes BPF programs fail to load with 'func_proto incompatible with
vmlinux'. This happens when the kernel was built with pahole < 1.26.

Fix: boot a kernel whose BTF was generated with pahole >= 1.26.
Affected distros include Ubuntu 24.04 LTS. See kernel commit
9edd04c4189e ("docs: Raise minimum pahole version to 1.26 for
KF_IMPLICIT_ARGS kfuncs").

Caused by:
    0: Failed to load BPF program
    1: Invalid argument (os error 22)

[sirlucjan]

It’s not the kernel’s fault. You need to have pahole 1.31.

[TheHENOOB]

It’s not the kernel’s fault. You need to have pahole 1.31.

I have pahole 1.31 installed, issue persists

[MY_USER_NAME]@[MY_COMPUTER_NAME]:~$ uname -a
Linux [MY_COMPUTER_NAME] 7.2.0-cachyos1.fc44.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Aug 21 08:54:10 UTC 2026 x86_64 GNU/Linux
[MY_USER_NAME]@[MY_COMPUTER_NAME]:~$ pahole --version
v1.31
[MY_USER_NAME]@[MY_COMPUTER_NAME]:~$ systemctl status scx_loader
● scx_loader.service - DBUS on-demand loader of sched-ext schedulers
     Loaded: loaded (/usr/lib/systemd/system/scx_loader.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (running) since Mon 2026-08-24 12:47:18 -03; 2min 4s ago
 Invocation: 648ee80aef19467e8761b430acb01340
   Main PID: 846 (scx_loader)
      Tasks: 14 (limit: 8495)
     Memory: 21.4M (peak: 48.2M)
        CPU: 1.149s
     CGroup: /system.slice/scx_loader.service
             └─846 /usr/bin/scx_loader

ago 24 12:47:19 [MY_COMPUTER_NAME] scx_loader[1110]: makes BPF programs fail to load with 'func_proto incompatible with
ago 24 12:47:19 [MY_COMPUTER_NAME] scx_loader[1110]: vmlinux'. This happens when the kernel was built with pahole < 1.26.
ago 24 12:47:19 [MY_COMPUTER_NAME] scx_loader[1110]: Fix: boot a kernel whose BTF was generated with pahole >= 1.26.
ago 24 12:47:19 [MY_COMPUTER_NAME] scx_loader[1110]: Affected distros include Ubuntu 24.04 LTS. See kernel commit
ago 24 12:47:19 [MY_COMPUTER_NAME] scx_loader[1110]: 9edd04c4189e ("docs: Raise minimum pahole version to 1.26 for
ago 24 12:47:19 [MY_COMPUTER_NAME] scx_loader[1110]: KF_IMPLICIT_ARGS kfuncs").
ago 24 12:47:19 [MY_COMPUTER_NAME] scx_loader[1110]: Caused by:
ago 24 12:47:19 [MY_COMPUTER_NAME] scx_loader[1110]:     0: Failed to load BPF program
ago 24 12:47:19 [MY_COMPUTER_NAME] scx_loader[1110]:     1: Invalid argument (os error 22)
ago 24 12:47:19 [MY_COMPUTER_NAME] scx_loader[846]: [ERROR]: Failed to start scheduler (attempt 5/5)
[MY_USER_NAME]@[MY_COMPUTER_NAME]:~$

[sirlucjan]

It’s not enough just to install it. You need to compile the kernel on it.

[TheHENOOB]

It’s not enough just to install it. You need to compile the kernel on it.

If I need to recompile the whole kernel to fix this issue, wouldn't that be a problem with the kernel build provided in the copr repo?

Also can you show me a resource about how to recompile the cachyos kernel for Fedora? or a different way to fix this bug in my machine?

[sirlucjan]

As for the pahole version – https://copr.fedorainfracloud.org/coprs/bieszczaders/kernel-cachyos-addons/build/10843517/ – here is the version you’re interested in.

Steps for kernel:

1. Download git repo
2. cd to proper directory

sudo dnf install rpmdevtools koji
rpmdev-setuptree
sudo dnf builddep <specname>.spec
rpmbuild -bb --target=x86_64 <specname>.spec

I’m going by memory here; I’ve never had Fedora for more than two months.

[TheHENOOB]

Issue persists on current copr build of linux-cachyos 7.2.2

[MY_USER_NAME]@[MACHINE_NAME]:~$ uname -r
7.2.2-cachyos1.fc44.x86_64
[MY_USER_NAME]@[MACHINE_NAME]:~$ systemctl status scx_loader
● scx_loader.service - DBUS on-demand loader of sched-ext schedulers
     Loaded: loaded (/usr/lib/systemd/system/scx_loader.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (running) since Fri 2026-08-28 18:04:46 -03; 3min 1s ago
 Invocation: ed3af67ccfb24983a6157a3aaf4423a7
   Main PID: 811 (scx_loader)
      Tasks: 14 (limit: 8496)
     Memory: 21M (peak: 47.6M)
        CPU: 788ms
     CGroup: /system.slice/scx_loader.service
             └─811 /usr/bin/scx_loader

ago 28 18:04:47 [MACHINE_NAME] scx_loader[1233]: makes BPF programs fail to load with 'func_proto incompatible with
ago 28 18:04:47 [MACHINE_NAME] scx_loader[1233]: vmlinux'. This happens when the kernel was built with pahole < 1.26.
ago 28 18:04:47 [MACHINE_NAME] scx_loader[1233]: Fix: boot a kernel whose BTF was generated with pahole >= 1.26.
ago 28 18:04:47 [MACHINE_NAME] scx_loader[1233]: Affected distros include Ubuntu 24.04 LTS. See kernel commit
ago 28 18:04:47 [MACHINE_NAME] scx_loader[1233]: 9edd04c4189e ("docs: Raise minimum pahole version to 1.26 for
ago 28 18:04:47 [MACHINE_NAME] scx_loader[1233]: KF_IMPLICIT_ARGS kfuncs").
ago 28 18:04:47 [MACHINE_NAME] scx_loader[1233]: Caused by:
ago 28 18:04:47 [MACHINE_NAME] scx_loader[1233]:     0: Failed to load BPF program
ago 28 18:04:47 [MACHINE_NAME] scx_loader[1233]:     1: Invalid argument (os error 22)
ago 28 18:04:47 [MACHINE_NAME] scx_loader[811]: [ERROR]: Failed to start scheduler (attempt 5/5)
[MY_USER_NAME]@[MACHINE_NAME]:~$

[sirlucjan]

As I said above: rebuild kernel on top newer pahole is needed. COPR is using older version.

[TheHENOOB]

As I said above: rebuild kernel on top newer pahole is needed. COPR is using older version.

I know but I think I have a theory to why the provided builds are bugged.

https://packages.fedoraproject.org/pkgs/dwarves/dwarves/

The build spec (or the fedora cloud) probably is collecting the dwarves package from the fedora repo, which currently is not updated to version 1.31.

Edit: I am a little bit illiterate lol.

There's probably a way to force the usage of a copr package to use on the build instead of using the official dwarves provided binaries.

In case I have the time I'll be sending a PR.