GSVD Merging

Standalone experiment project for running GSVD model-merging tests on top of the fusion_bench framework.

This repository is meant to hold:

• the GSVD method implementation
• the GSVD experiment configs
• the GSVD example launch scripts

The framework itself is still provided by the installed fusion_bench package.

What This Project Contains

Current supported GSVD experiment families:

• CLIP vision continual merging
  • ViT-B/16
  • ViT-B/32
  • ViT-L/14
  • TA8, TALL14, TALL20 task sets
• FLAN-T5 base continual merging
  • full fine-tuned GLUE checkpoints
  • LoRA GLUE checkpoints

Installation

From this directory:

pip install -e .

This project depends on an installed fusion_bench package for the underlying framework classes and model/task implementations.

Main Entry Point

After installation, the project exposes a gsvd_merge command. The example scripts in this repo invoke the local module directly with python -m src.scripts.cli, which is usually the most reliable option from a checkout.

Default root config:

• config/fabric_model_fusion.yaml

You can also invoke the CLI directly:

python -m src.scripts.cli

Run CLIP Tests

Example scripts:

• examples/gsvd/main_vitb16.sh
• examples/gsvd/main_vitb16_final_eval.sh
• examples/gsvd/main_vitb32.sh
• examples/gsvd/main_vitb32_final_eval.sh
• examples/gsvd/main_vitl14.sh
• examples/gsvd/main_vitl14_final_eval.sh

Run one of them from this repository root:

bash examples/gsvd/main_vitb16.sh

The default main_*.sh scripts evaluate at each merge step. Use the matching *_final_eval.sh variant to skip intermediate evaluation and only evaluate at the end.

Manual example:

python -m src.scripts.cli \
  method=gsvd \
  modelpool=CLIPVisionModelPool/clip-vit-base-patch32_TA8_model_only \
  taskpool=CLIPVisionModelTaskPool/clip-vit-classification_TA8

Run FLAN-T5 Tests

Example scripts:

• examples/gsvd/flan_t5_base.sh
• examples/gsvd/flan_t5_base_lora16.sh

Run one of them from this repository root:

bash examples/gsvd/flan_t5_base.sh

Manual examples:

Full fine-tuned checkpoints:

python -m src.scripts.cli \
  method=gsvd_general \
  modelpool=Seq2SeqLMPool/flan-t5-base_glue \
  taskpool=flan-t5_glue_text_generation

LoRA checkpoints:

python -m src.scripts.cli \
  method=gsvd_general \
  modelpool=Seq2SeqLMPool/flan-t5-base_glue_lora16 \
  taskpool=flan-t5_glue_text_generation

Ready-to-Run Root Configs

Included starter configs:

• config/_get_started/flan_t5_base_gsvd.yaml
• config/_get_started/flan_t5_base_gsvd_lora16.yaml

Example:

python -m src.scripts.cli --config-name _get_started/flan_t5_base_gsvd

Project Structure

Important directories:

• src/method/
  • GSVD implementation code
• config/method/
  • GSVD method configs
• config/modelpool/
  • local modelpool configs needed by GSVD tests
• config/taskpool/
  • local taskpool configs needed by GSVD tests
• config/model/
  • supporting model fragments for CLIP and FLAN-T5
• config/dataset/
  • supporting dataset fragments for CLIP task configs
• examples/gsvd/
  • runnable shell scripts for GSVD experiments

Important Note

This repository is not a full fork of the FusionBench framework in the same way as the standalone dop repository.

Instead, it is a GSVD-focused experiment project that still relies on the installed fusion_bench package for:

• framework programs
• modelpool/taskpool classes
• utility code
• benchmark task implementations

So the intended setup is:

1. install fusion_bench
2. install this project
3. run GSVD experiments from here