Jiayu/gavelfgdexperiment - #10
Open
Turquoise-T wants to merge 5 commits into
Open
Conversation
Add fragmentation penalty to MaxMinFairness LP to discourage allocation to GPU types with high fragmentation. This creates a feedback loop from FGD placement to LP allocation decisions. Changes: - Add fgd_frag_penalty_weight parameter to Scheduler - Track per-GPU-type fragmentation EMA in scheduler - Add set_fragmentation_context() to MaxMinFairness policy - Modify LP objective: max(min(throughput)) - lambda * frag_penalty When fgd_frag_penalty_weight=0 (default), behavior is unchanged.
- Fix missing `from time import time` in set_queue.py (runtime NameError) - Fix trailing comma bug in throughput_estimator.py (wrong data type) - Remove 9 dead functions from utils.py and unused imports - Remove orphaned lease.py (never imported) - Remove dead Job.from_proto() method - Consolidate _solve_with_fallback into policy.py base module - Remove unused imports across fgd.py, alibaba_trace_parser.py, job_id_pair.py, run_evaluation.py, run_standalone.py - Consolidate duplicate build_workload_from_distribution and _weighted_choice into alibaba_trace_parser.py - Remove demo functions and __main__ block from fgd.py - Remove dead get_round_fragmentation() from fgd_placement.py Made-with: Cursor
Implements FGD paper's scoring improvements and adds experiment suites: ## FGD Improvements (src/fgd/fgd.py) - **Paper scoring**: Sigmoid-based scoring `int(100/(1+exp(delta)))` with integer quantization for tie-breaking (use_paper_scoring) - **60% popularity threshold**: Filter workload to top 60% most popular task types as per paper's GetTypicalPods (popularity_threshold) - **Buddy-aware tie-breaking**: Prefer allocations leaving 2^n free GPUs on nodes for better future packing (use_buddy_tiebreak) - **Per-cluster fragmentation**: Option to compute fragmentation delta across all nodes instead of single node (use_cluster_fragmentation) ## Gavel+FGD Integration (src/scheduler/) - Added parameters to scheduler.py and fgd_placement.py to expose all FGD options through the Gavel scheduler interface - Dynamic fragmentation penalty weight already integrated ## Experiments - **fgd_improved**: 80 experiments comparing strided, fgd_vanilla, fgd_paper, fgd_buddy, fgd_cluster (8 loads x 5 methods x 2 seeds) - **gavelfgd_improved**: 42 experiments comparing gavel, gavelfgd, gavelfgd_improved (7 loads x 3 configs x 2 seeds) - **gavelfgd_all_improvements**: 56 experiments comparing gavel, gavelfgd_vanilla, gavelfgd_paper, gavelfgd_full with dynamic penalty (7 loads x 4 configs x 2 seeds) ## Results - gavelfgd_full (with dynamic penalty): 4.7% JCT improvement at 310 jph - gavelfgd_paper (paper+buddy): 3.6% improvement at 310 jph - FGD placement over strided: 2-4% improvement at high loads
- Update default threshold in fgd.py filter_by_popularity_threshold() - Update comments in fgd_placement.py - Update all experiment configs to use 85% threshold - Add new fgd_improved_v2 experiment config with unified improvements (paper scoring + buddy tiebreak + dynamic penalty) - Add submit_fgd_improved_v2.sbatch for running experiments
clubzip
approved these changes
Mar 8, 2026
clubzip
left a comment
clubzip
left a comment
Collaborator
There was a problem hiding this comment.
As mentioned in the meeting, the paper uses a 95% popularity threshold. Is there a reason you used 85% instead? I think there might have been some miscommunication.
Since I don’t want to block this PR, I’ll approve for now. It would be great if you could double-check this part before merging.
GavelFGD+ combines four improvements over base GavelFGD: - Paper-accurate FGD sigmoid scoring - 95% popularity threshold (raised from 85%) - Buddy tiebreak for equal-score placement options - Placement-Opportunity-Aware (POA) LP bonus (mu=0.10) The POA bonus replaces the earlier LP fragmentation penalty. The key insight: LP penalties on allocation cannot fix placement-caused fragmentation. POA instead rewards allocation to GPU types where jobs can currently be placed (measured by fit-fraction per demand size), steering the LP toward types with open node capacity rather than away from historically fragmented ones. Results on Alibaba cluster (7 load levels, 2 seeds each): - At 85% utilization: GavelFGD+ reduces frag rate by 76.7% over GavelFGD (2.01% -> 0.47%) and JCT by 2.1% (22275s -> 21810s) - At moderate utilization (<72%): neutral (POA signal undifferentiated) - Effect is load-dependent by design: POA vanishes when fit fractions are uniformly high (low load), activates when they diverge (high load) New files: - experiments/combined/configs/phase_gavelfgd_combined.json - experiments/combined/slurm/submit_gavelfgd_combined.sbatch - experiments/combined/plot_gavelfgd_combined.py - experiments/combined/results/gavelfgd_combined/ (42 experiment results) - experiments/combined/results/figures/gavelfgd_combined_*.png (3 figures) - docs/paper-evaluation-section.md (rewritten for GavelFGD+) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Title: feat: FGD paper improvements with Gavel
integration
Description:
Summary
This PR implements FGD paper's scoring algorithm
improvements and integrates them with the Gavel
scheduler.
Key Features
(1 + exp(delta))) for quantized tie-breaking
types covering 85% of workload (corrected from
initial 60%)
leaving power-of-2 free GPUs on nodes
fragmentation penalty based on cluster
utilization
Experiment Suites
comparing strided vs fgd_vanilla vs fgd_improved
on Cluster H (5592 GPUs)
2 seeds
paper+buddy+dynamic penalty
Gavel+FGD integration
with full feature comparison
Results
improvements
tie-breaking