Robust KernelBench

This is generally a robust version inspired by KernelBench (https://github.com/ScalingIntelligence/KernelBench). This code fixes a lot of the runtime problems the original code had: (Like fixing the GPU errors that go unoticed and corrupt the final results, adding distributed compilation, and adding more detailed metrics.)

This repo is also for the paper: "FIL Hypothesis"... (Citation coming soon).

Getting started:

1. Installation

# ENV for the model (sglang)
conda create -n env_kb python=3.11
conda activate env_kb
pip3 install -r requirements_new_sglang.txt

# ENV for kernelbench (newer requirements breaks)
conda create -n env_kb2 python=3.11
conda activate env_kb2
pip3 install -r requirements.txt
# conda install cuda-nvcc_linux-64 -c conda-forge # in case you are on conda you need this. 
# Maybe needed?
# conda install cuda-toolkit

# OTHER:
# conda install -c conda-forge gxx_linux-64 gcc_linux-64 make #you don't really need this, but it should resolve the cxx warning.
# export PIP_CACHE_DIR=/workspace/cache
# export TMPDIR=/workspace/cache

Running the code:

1. Multi round end-2-end scripts (Slurm):

bash ./scripts/slurm_job_v2.sh

2. (older) End-2-end scripts: (1. Generate Kernels, 2. Compile Kernels (distributed), 3. Evaluate Kernels)

bash ./scripts/run_tsp_jobs.sh #local server (see task-spooler below)
# OR
bash ./scripts/run_slurm_jobs.sh

Sequentially running each part seperately

0. Running all in one go (newer):

CUDA_VISIBLE_DEVICES=2
jobs=(
 "1 1 kernelbench"
)
python3 robust_kernelbench/run_main.py --version "v9_4" --model Qwen3_Coder_30B_A3B_Instruct     --parent_prompt_type "kernelbench" --num_samples 1 "${jobs[@]}" > out.txt

1. Model Inference scripts (1. Generate Kernels)

#This is the command that will be run...
python3 robust_kernelbench/run_inference_test_time_scaling.py --experiment_name "exp_local_1" --num_items 3 --trial 1

2. Compilation: (2. Compile Kernels (distributed / sequential))

python3 robust_kernelbench/compile_all_sequential.py --experiment_name exp_local_1 --trial 1 --num_processes 8

python3 robust_kernelbench/compile_all.py --experiment_name exp_local_1 --trial 1 --num_processes 8

3. Evaluation: (3. Benchmark and Evaluate Kernels)

python3 robust_kernelbench/evaluate_all.py --experiment_name "exp_local_1" --trial 1

Running the Analysis:

1. Installation:

pip3 install -r requirements_analysis.txt

2. Running: See README_ANALYSIS.md

---

Task Spooler (local server SLURM alternative)

1. Installation

apt update
apt install task-spooler

2. Usage:

 tsp timeout 3600 python3 python_script.py \ 
                        --param_1 "experiment_name" 
#this will timeout in 60 minutes (3600 seconds)

tsp #this shows current available jobs
tsp -c #shows you the output of the last job

References :

• https://github.com/bstee615/shared-task-spooler
• https://github.com/justanhduc/task-spooler

---

CONDA INSTALL LINUX:

https://www.anaconda.com/docs/getting-started/miniconda/install#linux-2

# cd /workspace
mkdir miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ./miniconda3/miniconda.sh
bash ./miniconda3/miniconda.sh -b -u -p ./miniconda3

To activate it next time

source /workspace/miniconda3/etc/profile.d/conda.sh
# OR
source ~/miniconda3/etc/profile.d/conda.sh

conda activate env_robust_kernelbench 

conda activate env_kb
#it was created using `conda create -n env_robust_kernelbench python=3.11

Slurm:

1. Job

sbatch scripts/slurm_job_v2.sh

2. Interactive

# salloc --partition=interactive-gpu --gres=gpu:h200:1 --time=01:00:00 --ntasks=1
salloc --partition=interactive-gpu --gres=gpu:h200:3 --time=08:00:00 --ntasks=3
# salloc --partition=interactive-gpu --time=08:00:00 --gres=gpu:h200:1
srun --pty --overlap --jobid=100854 bash
tmux attach -t 0
tmux capture-pane -t 0:0.0 -S - && tmux save-buffer ./output.txt

3. other commands

scontrol ping
scontrol show partition interactive-gpu
scontrol show assoc user=$USER

Tmux:

https://tmux.info/docs/cheatsheet

1. Getting it running

tmux

2. Switching between windows:

CTRL+b w #switch between windows
CTRL+b c #create new window
Ctrl+b " #split vertical
Ctrl+b d #detach from tmux

LLM Serving

1. Serving LLM

python3 -m sglang.launch_server \
--model-path Qwen/Qwen3-Coder-Next \
--tensor-parallel-size 2 \
--tool-call-parser qwen3_coder \
--host 0.0.0.0 \
--port 30000

2. Testing it is running:

curl -v http://0.0.0.0:30000/health
curl -v http://0.0.0.0:30000/model_info

---

(C) Nikolai Rozanov - 2025 - Present