Add FarmShare experiment documentation to README

README.md

@@ -73,11 +73,202 @@ Configuration: cluster_spec=v100:4|p100:4|k80:4, policy=FIFO, seed=42, num_total
 Results: average JCT=36721.82, utilization=0.14, makespan=215530.17
 ```
 
+## Running Experiments on FarmShare
+
+For large-scale experiments (e.g., replicating Gavel paper figures), use Stanford's FarmShare cluster.
+
+### Prerequisites
+
+1. **SSH access to FarmShare** - Ensure you can SSH to `rice.stanford.edu`
+2. **SSH multiplexing (recommended)** - Keep a persistent connection for faster commands
+
+Add to `~/.ssh/config`:
+```
+Host farmshare
+    HostName rice.stanford.edu
+    User <your-sunetid>
+    ControlMaster auto
+    ControlPath ~/.ssh/sockets/%r@%h-%p
+    ControlPersist 600
+```
+
+Create the socket directory and connect:
+```bash
+mkdir -p ~/.ssh/sockets
+ssh farmshare  # Keep this terminal open
+```
+
+### Initial Setup (One-Time)
+
+```bash
+# 1. Sync code to FarmShare (from local machine)
+rsync -avz --exclude='.venv' --exclude='__pycache__' --exclude='results*' \
+    /path/to/gavel farmshare:~/
+
+# 2. SSH to FarmShare and run setup
+ssh farmshare
+cd ~/gavel/cluster
+chmod +x setup_farmshare.sh
+./setup_farmshare.sh
+```
+
+The setup script:
+- Creates a Python virtual environment
+- Installs dependencies (cvxpy, numpy, etc.)
+- Generates experiment configurations
+- Creates log directories
+
+### Syncing Code Changes
+
+When you modify code locally, sync to FarmShare before running experiments:
+
+```bash
+# Sync scheduler code
+rsync -avz src/scheduler/ farmshare:~/gavel/src/scheduler/
+
+# Sync cluster scripts
+rsync -avz cluster/ farmshare:~/gavel/cluster/
+```
+
+### Running Experiments
+
+#### Single Experiment (Interactive)
+
+```bash
+ssh farmshare
+cd ~/gavel/cluster
+source ~/.venv/bin/activate
+
+# Run experiment index 0
+python3 run_benchmark.py \
+    --index 0 \
+    --experiments-file experiments_full.json \
+    --output-dir results_full \
+    --scheduler-dir ~/gavel/src/scheduler
+```
+
+#### Batch Experiments (SLURM)
+
+Submit all experiments as a SLURM job array:
+
+```bash
+ssh farmshare "cd ~/gavel/cluster && sbatch submit_full.sbatch"
+```
+
+The sbatch file (`cluster/submit_full.sbatch`) configures:
+- `--array=0-311` - Run experiments 0 through 311
+- `--time=08:00:00` - 8-hour timeout per experiment
+- `--mem=8G` - Memory allocation
+- `--partition=normal` - FarmShare partition
+
+#### Monitoring Jobs
+
+```bash
+# Check job status
+ssh farmshare "squeue -u \$USER"
+
+# Count completed experiments
+ssh farmshare "ls ~/gavel/cluster/results_full/*/summary.txt | wc -l"
+
+# View a specific job's output
+ssh farmshare "cat ~/gavel/cluster/slurm_logs/full-<jobid>_<index>.out"
+
+# Check for errors
+ssh farmshare "cat ~/gavel/cluster/slurm_logs/full-<jobid>_<index>.err"
+```
+
+#### Canceling Jobs
+
+```bash
+# Cancel all your jobs
+ssh farmshare "scancel -u \$USER"
+
+# Cancel a specific job array
+ssh farmshare "scancel <jobid>"
+```
+
+### Retrieving Results
+
+Sync results back to your local machine:
+
+```bash
+# Sync summary files (excludes large simulation.log files)
+./cluster/sync_results.sh results_full
+
+# Or manually with rsync
+rsync -avz --exclude='simulation.log' \
+    farmshare:~/gavel/cluster/results_full/ \
+    ./cluster/results_full/
+```
+
+### Managing Disk Space
+
+Simulation logs can be large (100MB+ each). Compress completed experiments:
+
+```bash
+# Compress logs for completed experiments only
+ssh farmshare "cd ~/gavel/cluster && ./compress_completed_logs.sh results_full"
+
+# Check disk usage
+ssh farmshare "du -sh ~/gavel/cluster/results_full"
+```
+
+### Experiment Configuration Files
+
+| File | Description |
+|------|-------------|
+| `experiments_full.json` | Full paper replication (312 experiments) |
+| `experiments_pilot.json` | Pilot runs for testing (subset) |
+| `submit_full.sbatch` | SLURM batch script for full experiments |
+| `submit_retry.sbatch` | Retry failed experiments |
+
+### Generating New Experiments
+
+```bash
+cd cluster
+
+# Generate full experiment set (Figures 9, 10, 11)
+python3 generate_full_experiments.py
+
+# Generate pilot experiments
+python3 generate_pilot_experiments.py
+```
+
+### Troubleshooting
+
+**Job stuck in pending (PD) state:**
+```bash
+# Check why job is pending
+ssh farmshare "squeue -u \$USER -o '%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R'"
+```
+
+**Stale file handle errors:**
+- NFS issues on FarmShare - retry the failed experiments
+- Use `submit_retry.sbatch` with failed indices
+
+**Solver failures (ECOS):**
+- See `docs/2025-01-27-ecos-solver-failures-research.md` for analysis
+- Mostly affects `finish_time_fairness` at high loads
+- These experiments are near saturation anyway
+
+**Out of memory:**
+- Increase `--mem` in the sbatch file
+- Default is 8G, try 16G for complex experiments
+
 ## Project Structure
 
 ```
 .
-├── docs/plans/              # Design documents
+├── cluster/                 # FarmShare experiment infrastructure
+│   ├── run_benchmark.py     # Main experiment runner
+│   ├── generate_*.py        # Experiment config generators
+│   ├── submit_*.sbatch      # SLURM batch scripts
+│   ├── sync_results.sh      # Result retrieval script
+│   ├── experiments_*.json   # Experiment configurations
+│   └── results_*/           # Experiment outputs
+├── docs/
+│   ├── plans/               # Design documents
+│   └── *.md                 # Research notes and analysis
 ├── requirements-sim.txt     # macOS-compatible dependencies
 ├── src/
 │   └── scheduler/
@@ -100,6 +291,8 @@ Results: average JCT=36721.82, utilization=0.14, makespan=215530.17
 | `policies/` | Policy implementations (what we'll extend) |
 | `scripts/sweeps/` | Entry points for running experiments |
 | `simulation_throughputs.json` | Job throughput profiles by GPU type |
+| `cluster/run_benchmark.py` | FarmShare experiment runner with telemetry |
+| `cluster/experiments_full.json` | Full paper replication configs (312 experiments) |
 
 ## Contributing