xlurm

A minimal Linux single-host, multi-user NVIDIA (nv) / Moore Threads (mtt) / MetaX (mx) / Huawei Ascend (asc) / T-Head PPU (ppu) / Hygon HCU (hcu) accelerator scheduler. Written in Rust, it runs tasks directly through process executors without tmux. One shared scheduler allocates whole-machine devices, and each task runs with the submitting user's identity.

中文文档：README_zh.md

Install all commands with one command from the repository root:

cargo install --path . --locked

This installs xlurm, xrun, xbatch, xqueue, xcancel, and xinfo to ~/.cargo/bin (which must be in PATH). There is no need to specify each --bin or add --bins. --locked installs dependencies according to the repository lock file and may be omitted; installation uses a release build by default.

On a shared server, an administrator installs the binaries system-wide and starts the shared scheduler:

# Build all binaries and install them to /usr/local/bin using sudo
./install.sh
sudo xlurm start

# Run as a regular user, without sudo
xinfo                                    # Show devices
xrun -g 1 python train.py                 # Queue, run, stream logs, and return the task exit code
xbatch -g 2 train.sh                     # Submit a bash script in the background and return a job ID
xqueue                                   # Show queued and running tasks
xcancel 3                                # Cancel a task and its process group

install.sh requires Cargo and sudo. It builds with --release --locked, then installs all six binaries as root-owned files with mode 0755. It does not start or restart the scheduler. An already-running daemon and the workers it launches continue using the previous executable until the daemon is restarted.

Commands

Command	Purpose
xrun [OPTIONS] COMMAND [ARGS...]	Wait for a task in the foreground, forward stdout/stderr, and cancel on Ctrl-C
xbatch [OPTIONS] SCRIPT [ARGS...]	Snapshot and submit a script without waiting for resources
xqueue [JOB_ID]	Show the latest 100 active tasks across users with start, wait, and run times; with an ID, show detailed results for your task
xcancel JOB_ID	Cancel a task and its process group; equivalent to xqueue --cancel JOB_ID
xinfo	Show devices, external usage, and allocations
sudo xlurm clean	Remove all logs while the scheduler is stopped and no task is running

Resource options include -g/--gpus N (also --devices, default 1), -i/--device-ids ID[,ID...], -x/--exclude-device-ids ID[,ID...], and --device nv|mtt|mx|asc|ppu|hcu; -n/--name NAME and -t/--time-limit SECONDS are also available. Full backend names nvidia|mthreads|metax|ascend|ppu|hcu are accepted. -i requests exact device IDs, for example xrun -i 0,2 python train.py; all IDs must belong to one backend. On a single-backend machine the backend is inferred. If the IDs exist on multiple backends, use --device to disambiguate. -x excludes device IDs from automatic allocation, for example xrun -x 0,2 -g 1 python train.py; it cannot be combined with -i. -g may be combined with -i, but the count must match; when omitted, the count is inferred from the IDs. -g 0 still submits a CPU task.

xrun --device ascend -g 2 python train_npu.py
xrun --device ppu -g 1 python train.py
xrun --device metax -i 1 python train.py
xrun --device hcu -i 1 python train.py
xrun --device nvidia -g 1 -t 3600 python train.py --epochs 10
xrun -i 0,2 -g 2 python train.py
xrun --device nvidia -x 0,2 -g 1 python train.py
xrun -g 0 bash -c 'echo hello; exit 7'     # xrun also returns 7
xbatch --device ascend -g 2 train.sh --epochs 10
xbatch -g 1 --wrap 'python prepare.py && python train.py'
xqueue --all                            # Include completed tasks
xqueue 3                                # Show your task's status and exit code
xqueue 3 --log                          # Read your task's output
xqueue --all --json
xinfo --json

xqueue displays at most the latest 100 matching tasks, including with --all; all users see host-wide queue summaries. For pending jobs, its DEVICES column shows the requested type and count (for example, asc:2, auto:1, or cpu) or the requested IDs. Once a job starts, the column lists assigned devices, grouping multiple IDs from one vendor as asc:[0,1] or nv:[0,1]. Start times use UTC+8: the default list shows MM-DD HH:MM:SS, while --all shows YYYY-MM-DD HH:MM:SS. Job details retain the year. Durations use HH:MM:SS, or D-HH:MM:SS after 24 hours. A pending job's wait time and a running job's run time continue increasing. Jobs that have not started show - for the start and run times.

Options for xrun go before the command name. Starting at the command name, all later arguments are passed to the task, including options such as --help and --gpus. The -- separator is optional, so the older form xrun -g 1 -- python train.py remains supported.

Arguments are passed directly to the process; they are not assembled into a shell command. To use pipes, redirection, or variable expansion, explicitly use bash -c or xbatch --wrap. Batch scripts always run under bash and receive arguments unchanged; directives such as #SBATCH are not parsed. Tasks retain the working directory and environment from submission, so you can activate conda or a virtual environment before submitting.

xrun (or xlurm run) prints status messages to stderr with a UTC timestamp and Job ID. After waiting for more than one second it prints Waiting for resources... once; when execution starts it prints Task started. once. Tasks that start immediately also receive the start message. Task logs continue to be written normally, while execution errors such as failure to start a task are reported to stderr. After a background submission, xbatch only returns the task ID for later inspection or cancellation.

[xlurm] 2026-09-20 12:34:56 UTC Job 3: Waiting for resources...
[xlurm] 2026-09-20 12:35:10 UTC Job 3: Task started.

xrun follows foreground logs but does not provide an interactive terminal; task stdin is /dev/null, and stdout/stderr are stored together. Use python -u when Python must flush logs immediately.

Scheduler

sudo xlurm start                             # Start in the background and discover installed accelerator drivers
sudo xlurm start --backend ascend            # Manage only Ascend
sudo xlurm start --backend mtt               # Manage only Moore Threads GPUs
sudo xlurm start --backend metax             # Manage only MetaX GPUs
sudo xlurm start --backend ppu               # Manage only T-Head PPU devices
sudo xlurm start --backend hcu               # Manage only Hygon HCU devices
sudo xlurm start --backend ascend --exclude-device-ids 2,3  # Temporarily disable suspect cards
sudo xlurm start --backend none              # CPU mode; no device driver required
sudo xlurm daemon --backend auto --max-running 32  # Run in the foreground
sudo xlurm exclude 2,3                       # Disable cards while the scheduler is running
sudo xlurm include 2                         # Re-enable card 2 without restarting
sudo xlurm stop                              # Stop scheduling; already-started tasks continue
sudo xlurm restart                           # Stop and start; refuse if any task is running
sudo xlurm clean                             # Remove logs after all tasks have finished

xlurm clean is an offline administrator action. It refuses to run if the scheduler is active or if persisted state contains a RUNNING task. After both checks pass, it removes every task .log file and daemon.log while preserving queued tasks, task history, results, and other spool files. If a worker finishes while the scheduler is stopped, restart the scheduler once so it can collect the result before stopping it and running clean. New tasks and a subsequent background start create new log files.

start does not modify an already-running scheduler; stop it first to change the backend or concurrency limit. Use exclude/include to change device exclusions while it is running. A subsequent start takes over running tasks, reads results completed while it was offline, and continues queued tasks. After a machine reboot or unexpected worker disappearance, running tasks without results are marked FAILED and are not rerun automatically.

restart is equivalent to stop followed by start and accepts the same backend, concurrency, and device exclusion options as start. Unlike stop, it refuses to stop the scheduler while any task is RUNNING; queued tasks do not prevent a restart.

start, daemon, and restart accept --exclude-device-ids ID[,ID...] (repeatable) to temporarily remove suspect cards from scheduling for all users. IDs are the logical device IDs shown by xinfo; with multiple backends, the same ID is excluded in every selected backend, so use --backend to limit the scope. Unknown IDs and duplicates are rejected. Excluded cards remain visible in xinfo as disabled, with a job ID if already allocated. Both automatic allocation and exact -i requests respect this restriction; new requests exceeding the enabled pool are rejected. Previously queued tasks that need excluded cards remain pending. Running tasks adopted after stop/start continue without interruption. Exclusions are not persisted: start or restart without this option to re-enable the cards. This scheduler option is separate from the per-task xrun/xbatch -x option.

While the scheduler is running, administrators can use xlurm exclude ID[,ID...] to add cards to the exclusion set and xlurm include ID[,ID...] to remove them. Both commands take effect for subsequent allocations without restarting or interrupting running tasks; queued tasks resume when their cards are re-enabled and available. Repeating either operation is harmless. The entire ID list is validated before any change. These live changes are also temporary and are cleared when the daemon exits. Ordinary users cannot change the exclusion set; a non-root test daemon permits its own account. New clients require an updated daemon for these commands; older daemons reject the new requests without changing state.

Ascend prefers the DCMI v2 API. If the installed library does not export the dcmiv2_* symbols, xlurm uses the compatible dcmi_* API from dcmi_interface_api.h; it never falls back to npu-smi. The Ascend backend requires the library, v2 symbols, and successful initialization; failures are reported as discovery errors. Library search checks /usr/local/dcmi/libdcmi.so, /usr/local/Ascend/driver/lib64/driver/libdcmi.so, then the dynamic loader's libdcmi.so search. Set XLURM_DCMI_LIBRARY=/absolute/path/libdcmi.so in the administrator's daemon environment to use a different trusted library; an explicit path disables that search. This setting applies at daemon startup, including restart, and does not come from submitted jobs.

MetaX library search checks /opt/mxdriver/lib/libmxsml.so, /opt/maca/lib/libmxsml.so, /opt/mxn100/lib/libmxsml.so, then the dynamic loader's libmxsml.so search. Set XLURM_MXSML_LIBRARY=/absolute/path/libmxsml.so in the administrator's daemon environment to select another trusted library and disable fallback search. The library must provide the MXSML extension APIs and mxSmlGetDeviceUnavailableReason. Missing libraries or symbols and initialization/discovery errors skip MetaX in automatic mode and stop startup with --backend metax. Submitted job environments cannot change this setting.

PPU library search checks /usr/local/PPU_SDK/lib/libhgml.so, /usr/local/PPU_SDK/targets/x86_64-linux/lib/libhgml.so, then the dynamic loader's libhgml.so search. Set XLURM_HGML_LIBRARY=/absolute/path/libhgml.so in the administrator's daemon environment to select another trusted library and disable fallback search. HGML v2 initialization, discovery, and compute-process symbols are required; failures skip PPU in automatic mode and stop startup with --backend ppu. Submitted job environments cannot change this setting.

HCU library search checks /opt/hyhal/lib/librocm_smi64.so, /opt/dtk/.hyhal/rocm_smi/lib/librocm_smi64.so, then the dynamic loader's librocm_smi64.so. Administrators may override the library with XLURM_HCU_SMI_LIBRARY=/absolute/path/librocm_smi64.so and the health executable with XLURM_HCU_HEALTH_TOOL=/absolute/path/hy-smi in the daemon environment. Explicit overrides disable the respective search and apply at daemon startup, never from submitted job environments. Missing symbols or initialization failures stop HCU discovery.

For DTK installations that provide /opt/dtk/env.sh, source it in the administrator's Bash shell before starting the HCU daemon so it inherits the driver library paths. The preferred DTK hy-smi health executable may otherwise fail with librocm_smi64.so.2: cannot open shared object file, leaving devices unknown and unavailable for allocation. sudo or a service manager may discard these environment settings; initialize DTK in the shell or service that actually launches the daemon. For an existing daemon, apply the environment when restarting after running jobs finish.

# Run in the administrator/root Bash shell inside the HCU environment.
source /opt/dtk/env.sh
xlurm start --backend hcu

The default shared state directory is /var/lib/xlurm. All users automatically connect to its Unix socket without starting their own service. Set XLURM_HOME to another short absolute path owned by the administrator; all clients and the scheduler must use the same path. There is no configuration file and no network listening port.

The state directory is root-owned with mode 0755, and the socket has mode 0666 so local users can connect. jobs/ has mode 0700, while state, environment, and logs have mode 0600. Users can connect to the socket but cannot modify queue files or directly read another user's logs. All log access is authorized by the server.

$XLURM_HOME/
  xlurm.sock       Local Unix socket
  daemon.lock      Single-scheduler lock
  daemon.log       Diagnostics such as device discovery and startup failures
  state.json       Queue and history
  jobs/            Per-task description, logs, lock, and exit result

Completed tasks (successful, failed, cancelled, or timed out) are retained for 7 days from their completion time. The scheduler cleans them at startup and once per minute while running, deleting expired task logs, descriptions, results, cancellation markers, locks, and temporary files, and removing their records from state.json; job IDs are never reused. Queued and running tasks are unaffected.

After cleanup, xqueue --all no longer lists the task, and querying its ID or reading its logs returns job not found. Export logs that must be retained before cleanup. This policy only removes old tasks; it does not limit the log size of running tasks or clean the task working directory or daemon.log.

Execution and Resources

xrun / xbatch / xqueue / xcancel / xinfo
               │ Unix socket
       SO_PEERCRED authentication → single-threaded Scheduler
               │ Executor: start / poll / cancel
          ProcessExecutor
               │
       Dedicated worker → drop privileges to submitter → task process group
                    Logs + atomic result file

• The scheduler tries pending tasks requesting more devices first, keeping submission order for equal counts. It starts each task when resources are available; smaller tasks may run while a larger task waits. At most 32 tasks run at once, configurable with --max-running.
• Each device slot is allocated exclusively, and a task uses only one vendor's device pool. Automatic selection tries NVIDIA, then Ascend, Moore Threads, MetaX, PPU, and HCU. Multi-chip Ascend cards are allocated by individual compute chip.
• NVIDIA devices are discovered and monitored with nvidia-smi; CUDA_VISIBLE_DEVICES is set by GPU UUID to avoid differences between CUDA and management-tool index order.
• Ascend defaults to loading libdcmi.so once and querying logical device IDs, per-chip health, and process occupancy directly through DCMI v2, using a single-threaded scheduler. ASCEND_RT_VISIBLE_DEVICES uses logical IDs; DCMI v2 queries use these logical IDs directly, without physical card/chip mappings. The Ascend backend requires the library, v2 symbols, and successful initialization; failures are reported as discovery errors. Discovery errors exclude Ascend in automatic backend discovery and stop startup when --backend ascend is selected; monitoring failures or missing devices show unknown and pause allocation. Nonzero health shows unhealthy and excludes the chip from allocation.
• Moore Threads devices are discovered through the read-only MTML API (libmtml.so); external process use is read through MTML process-memory records. xlurm does not invoke mthreads-gmi. Jobs receive allocated UUIDs in MTHREADS_VISIBLE_DEVICES and device indices in MUSA_VISIBLE_DEVICES. The display label is mtt.
• MetaX loads libmxsml.so once and discovers device indices, names, and UUIDs through MXSML. Availability and per-device compute processes are queried through the API; mx-smi is not required. Unavailable devices show unhealthy; query errors, invalid responses, or a changed UUID show unknown and pause allocation for that device. Jobs receive allocated device indices in CUDA_VISIBLE_DEVICES and MACA_VISIBLE_DEVICES. The display label is mx.
• T-Head PPU devices are discovered by index, UUID, and name through the SDK’s libhgml.so HGML v2 API; per-device compute processes are queried directly through HGML. ppu-smi is not required. Query errors, invalid responses, or a changed UUID show unknown and pause allocation for that device. Jobs receive allocated UUIDs in CUDA_VISIBLE_DEVICES; the display label is ppu.
• Hygon HCU devices are discovered by Unique ID and monitored for process occupancy directly through the driver's librocm_smi64.so API; JSON output and PATH no longer affect these probes. Comprehensive health checks still require hy-smi --healthcheck, preferring /opt/dtk/.hyhal/bin/hy-smi over PATH. Unhealthy devices are excluded and failed probes pause allocation. ROCR_VISIBLE_DEVICES selects allocated device indices; HIP_VISIBLE_DEVICES and CUDA_VISIBLE_DEVICES select the resulting 0..N-1 view. These environment variables control runtime visibility, not access-control isolation. Driver process reporting must be verified under load before relying on it to detect external workloads; successful empty results cannot prove there are no hidden processes. The display label is hcu.
• External compute processes reported by the driver are checked every two seconds. Occupied devices are not allocated; query failures are shown as unknown and pause allocation. Driver command calls wait at most three seconds. DCMI, MXSML, HGML, and HCU SMI calls run synchronously in the scheduler and rely on the driver's return time; they have no application-level timeout.
• Client connections wake the idle scheduler immediately. Queue responses use buffered socket writes to avoid excessive system calls when job history is large; the scheduler remains single-threaded.
• Workers hold inherited file locks, allowing the scheduler to take over tasks after a restart without relying on bare PIDs to determine liveness. Exit results are written atomically.
• Cancellation first sends SIGTERM to the process group, then sends SIGKILL if it is still running after one second. When the main task process exits, the same group is cleaned up, orphan processes are reaped, and devices are released.

Ascend monitoring failures

For Ascend monitoring failures, xinfo prints the failed query and driver error below the device row; xinfo --json includes an optional monitoring_error field. This requires an updated daemon to supply the details. The daemon logs changes to the error and successful recovery, without repeating the same error on every poll. Diagnostics contain driver query information, not task commands or environments.

A healthy chip can still show unknown when process occupancy cannot be queried. For example, a process-memory query can return DCMI error -8001 when the driver lists an exited PID while surviving Python processes still hold device handles. This code alone does not establish that cause. On the affected host, an administrator can inspect remaining handles with sudo fuser -v /dev/davinci* and confirm the corresponding tasks with their owners before terminating them. A handle on /dev/davinci_manager alone does not identify which chip a process uses; the daemon itself also holds one. Do not infer availability from a missing /proc/PID entry or a failed query's zero process count. Allocation resumes only after monitoring succeeds and reports the device idle.

xlurm cleans up its task's process group on cancellation, timeout, and normal completion. Tasks started outside xlurm, and descendants that deliberately leave their process group, require separate lifecycle management; xlurm does not automatically terminate them or reset devices.

Multi-user Permissions

Operation	Regular user	Root administrator
xinfo, queue summary	Can see the whole host	Can see the whole host
Submit a task	Runs with the user's UID/GID	Runs as root
Task details, command, environment, logs	Own tasks only	All tasks
Cancel a task	Own tasks only	All tasks
Exclude/include devices	Not allowed	Allowed
Start/stop the shared scheduler	Not allowed	Allowed

Identity comes from the kernel's SO_PEERCRED; clients cannot choose task ownership through JSON or the USER environment variable. Before execution, the scheduler re-resolves the local account and supplementary groups, sets supplementary groups plus real/effective/saved GID and UID, then enters the user's working directory and executes the command. Files owned by the user are accessed with that user's permissions, and supplementary groups required for driver access are preserved.

Tasks set no_new_privs, so a task cannot rely on sudo or setuid programs for privilege escalation. Queue summaries are host-wide and contain only the task name, owner, resources, and status. Commands, scripts, and environments are never included; task details, logs, and cancellation are restricted to the owner or root. Scheduling prioritizes requested device count, with no user quotas, configurable priority classes, or billing.

This project provides multi-user identity and control permissions. GPU/NPU access is described through visible-device environment variables and is not yet enforced with cgroups or device-node restrictions. Programs that bypass the scheduler can still compete for devices. MIG, memory partitioning, and intentionally detached background services are unsupported. CPU tasks set all supported device visibility variables to -1.

Development

There are only five runtime dependencies: anyhow, clap, libc, serde, and serde_json. The project does not use an async runtime, database, HTTP, web UI, or tmux.

cargo fmt --check
cargo test --locked
cargo clippy --locked --all-targets -- -D warnings

# Integration test with two actual system accounts: requires root,
# creates no accounts, and uses no real devices
cargo build --bins --locked
sudo python3 tests/multiuser.py

End-to-end tests use simulated drivers to cover NVIDIA, Ascend, MetaX, HCU, and PPU, exclusive allocation, external usage and query failures, foreground exit codes, script snapshots, cancellation, timeouts, and scheduler crash takeover. Permission tests cover owner spoofing, cross-user querying/log access/cancellation, administrator permissions, and queue privacy. Ordinary cargo test does not require root or accelerator hardware; the DCMI, MXSML, HGML, and HCU SMI tests compile simulated shared libraries and require a C compiler (cc).

tests/multiuser.py uses the existing nobody and daemon accounts to verify real UID/GID/supplementary-group switching, output-file ownership, private queues, and unauthorized access rejection. It uses temporary directories and CPU tasks without modifying account configuration. For development, start a non-root test instance accessible only to the current user with XLURM_HOME=/tmp/my-xlurm xlurm daemon --backend none.

Interface semantics reference CUDA_VISIBLE_DEVICES, Ascend visible devices, MetaX MXSML APIs, HCU SMI APIs, PPU management library, and Linux Unix socket credentials.

This project was strongly inspired by gflow.