From e449b0efbdac4605cac64d15747ec7a0167c8946 Mon Sep 17 00:00:00 2001 From: varunr Date: Tue, 27 Jan 2026 20:23:18 -0800 Subject: [PATCH] Add FarmShare experiment documentation to README - SSH multiplexing configuration - One-time setup instructions - Code syncing workflow - Single and batch experiment execution - Job monitoring and result retrieval - Troubleshooting guide - Updated project structure with cluster/ directory Co-Authored-By: Claude (claude-opus-4-5) --- README.md | 195 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 194 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 893fb01..9f762b4 100644 --- a/README.md +++ b/README.md @@ -73,11 +73,202 @@ Configuration: cluster_spec=v100:4|p100:4|k80:4, policy=FIFO, seed=42, num_total Results: average JCT=36721.82, utilization=0.14, makespan=215530.17 ``` +## Running Experiments on FarmShare + +For large-scale experiments (e.g., replicating Gavel paper figures), use Stanford's FarmShare cluster. + +### Prerequisites + +1. **SSH access to FarmShare** - Ensure you can SSH to `rice.stanford.edu` +2. **SSH multiplexing (recommended)** - Keep a persistent connection for faster commands + +Add to `~/.ssh/config`: +``` +Host farmshare + HostName rice.stanford.edu + User + ControlMaster auto + ControlPath ~/.ssh/sockets/%r@%h-%p + ControlPersist 600 +``` + +Create the socket directory and connect: +```bash +mkdir -p ~/.ssh/sockets +ssh farmshare # Keep this terminal open +``` + +### Initial Setup (One-Time) + +```bash +# 1. Sync code to FarmShare (from local machine) +rsync -avz --exclude='.venv' --exclude='__pycache__' --exclude='results*' \ + /path/to/gavel farmshare:~/ + +# 2. SSH to FarmShare and run setup +ssh farmshare +cd ~/gavel/cluster +chmod +x setup_farmshare.sh +./setup_farmshare.sh +``` + +The setup script: +- Creates a Python virtual environment +- Installs dependencies (cvxpy, numpy, etc.) +- Generates experiment configurations +- Creates log directories + +### Syncing Code Changes + +When you modify code locally, sync to FarmShare before running experiments: + +```bash +# Sync scheduler code +rsync -avz src/scheduler/ farmshare:~/gavel/src/scheduler/ + +# Sync cluster scripts +rsync -avz cluster/ farmshare:~/gavel/cluster/ +``` + +### Running Experiments + +#### Single Experiment (Interactive) + +```bash +ssh farmshare +cd ~/gavel/cluster +source ~/.venv/bin/activate + +# Run experiment index 0 +python3 run_benchmark.py \ + --index 0 \ + --experiments-file experiments_full.json \ + --output-dir results_full \ + --scheduler-dir ~/gavel/src/scheduler +``` + +#### Batch Experiments (SLURM) + +Submit all experiments as a SLURM job array: + +```bash +ssh farmshare "cd ~/gavel/cluster && sbatch submit_full.sbatch" +``` + +The sbatch file (`cluster/submit_full.sbatch`) configures: +- `--array=0-311` - Run experiments 0 through 311 +- `--time=08:00:00` - 8-hour timeout per experiment +- `--mem=8G` - Memory allocation +- `--partition=normal` - FarmShare partition + +#### Monitoring Jobs + +```bash +# Check job status +ssh farmshare "squeue -u \$USER" + +# Count completed experiments +ssh farmshare "ls ~/gavel/cluster/results_full/*/summary.txt | wc -l" + +# View a specific job's output +ssh farmshare "cat ~/gavel/cluster/slurm_logs/full-_.out" + +# Check for errors +ssh farmshare "cat ~/gavel/cluster/slurm_logs/full-_.err" +``` + +#### Canceling Jobs + +```bash +# Cancel all your jobs +ssh farmshare "scancel -u \$USER" + +# Cancel a specific job array +ssh farmshare "scancel " +``` + +### Retrieving Results + +Sync results back to your local machine: + +```bash +# Sync summary files (excludes large simulation.log files) +./cluster/sync_results.sh results_full + +# Or manually with rsync +rsync -avz --exclude='simulation.log' \ + farmshare:~/gavel/cluster/results_full/ \ + ./cluster/results_full/ +``` + +### Managing Disk Space + +Simulation logs can be large (100MB+ each). Compress completed experiments: + +```bash +# Compress logs for completed experiments only +ssh farmshare "cd ~/gavel/cluster && ./compress_completed_logs.sh results_full" + +# Check disk usage +ssh farmshare "du -sh ~/gavel/cluster/results_full" +``` + +### Experiment Configuration Files + +| File | Description | +|------|-------------| +| `experiments_full.json` | Full paper replication (312 experiments) | +| `experiments_pilot.json` | Pilot runs for testing (subset) | +| `submit_full.sbatch` | SLURM batch script for full experiments | +| `submit_retry.sbatch` | Retry failed experiments | + +### Generating New Experiments + +```bash +cd cluster + +# Generate full experiment set (Figures 9, 10, 11) +python3 generate_full_experiments.py + +# Generate pilot experiments +python3 generate_pilot_experiments.py +``` + +### Troubleshooting + +**Job stuck in pending (PD) state:** +```bash +# Check why job is pending +ssh farmshare "squeue -u \$USER -o '%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R'" +``` + +**Stale file handle errors:** +- NFS issues on FarmShare - retry the failed experiments +- Use `submit_retry.sbatch` with failed indices + +**Solver failures (ECOS):** +- See `docs/2025-01-27-ecos-solver-failures-research.md` for analysis +- Mostly affects `finish_time_fairness` at high loads +- These experiments are near saturation anyway + +**Out of memory:** +- Increase `--mem` in the sbatch file +- Default is 8G, try 16G for complex experiments + ## Project Structure ``` . -├── docs/plans/ # Design documents +├── cluster/ # FarmShare experiment infrastructure +│ ├── run_benchmark.py # Main experiment runner +│ ├── generate_*.py # Experiment config generators +│ ├── submit_*.sbatch # SLURM batch scripts +│ ├── sync_results.sh # Result retrieval script +│ ├── experiments_*.json # Experiment configurations +│ └── results_*/ # Experiment outputs +├── docs/ +│ ├── plans/ # Design documents +│ └── *.md # Research notes and analysis ├── requirements-sim.txt # macOS-compatible dependencies ├── src/ │ └── scheduler/ @@ -100,6 +291,8 @@ Results: average JCT=36721.82, utilization=0.14, makespan=215530.17 | `policies/` | Policy implementations (what we'll extend) | | `scripts/sweeps/` | Entry points for running experiments | | `simulation_throughputs.json` | Job throughput profiles by GPU type | +| `cluster/run_benchmark.py` | FarmShare experiment runner with telemetry | +| `cluster/experiments_full.json` | Full paper replication configs (312 experiments) | ## Contributing