music-generation-copyright

This is a codebase for music generation copyright project

Readme

Environment

It's recommended to use conda environment for convenience.

# python 3.10 is tested though some other versions may work as well
conda create -n music-copyright python=3.10
conda activate music-copyright

It's recommended to install 11.8 for traker==0.3.1 (latest version for 2023/12/22).

# check your nvcc version
nvcc --version

If the version is not 11.8 or even nvcc is not installed in your environment. You can install it through

# refer to https://anaconda.org/nvidia/cuda-toolkit
conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit

Then install following packages in order.

TODO: Add a requirements.txt

#as of TRAK 0.3.2, pip versions 23 and above are incompatible with a dependency (fast-jl). For now, reccommended to install pip 22.3.1
pip install pip==22.3.1

# torch 2.1 is tested
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

# install TRAK library with CUDA acceleration
pip install traker[fast]==0.3.1

# Patch a PR
# https://github.com/MadryLab/trak/pull/58

# other dependencies
pip install pretty_midi
pip install scipy

Data Preparation

Currently we only use magenta(v.2.0.0) dataset for our experiment.

1. Download the maestro-v2.0.0-midi.zip from magenta(v.2.0.0)
2. Preprocess the dataset, the /path/to/processed/maestro-v2.0.0-processed will be used for future training and generation.

cd music-generation-models/MusicTransformer-Pytorch-paper-1211
python preprocess_midi.py -output_dir /path/to/processed/maestro-v2.0.0-processed /path/to/downloaded-midi/maestro-v2.0.0

Experiment

In the experiment, we have several splited process. (TODO: combine them in a unified script).

Attribution score calculation

1. Train a model with full dataset for music generation. TODO: the paramter should be cleaned.

Here is a recommended training setting. Be sure to follow it through out the whole experiment. It could lead to false result. The training could take ~40 minutes on A40.. (TODO: make it less than 10min).

cd music-generation-models/MusicTransformer-Pytorch-paper-1211
python3 train.py --no_tensorboard\
                 -input_dir /home/user/music-copyright-project/dataset/maestro-v2.0.0-processed\
                 -output_dir /home/user/music-copyright-project/checkpoint/paper-1211-reproduce/model-checkpoints/100p/${SLURM_ARRAY_TASK_ID}\
                 -epochs 40\
                 -max_sequence 256\
                 -train_sample_seed 0\
                 --rpr\
                 --fixed_crop\
                 -train_seed ${SLURM_ARRAY_TASK_ID}\
                 -train_sample_ratio 1\
                 -weight_modulus 5\
                 -batch_size 64\
                 --train_sample_shuffle

2. Independently train 50 (subject to change) models with 50% dataset for data attribution score calculation. The training of each model could take ~20 minutes on A40.. (TODO: make it less than 10min).

cd music-generation-models/MusicTransformer-Pytorch-paper-1211
python3 train.py --no_tensorboard\
                 -input_dir /home/user/music-copyright-project/dataset/maestro-v2.0.0-processed\
                 -output_dir /home/user/music-copyright-project/checkpoint/paper-1211-reproduce/model-checkpoints/100p/${SLURM_ARRAY_TASK_ID}\
                 -epochs 40\
                 -max_sequence 256\
                 -train_sample_seed ${SLURM_ARRAY_TASK_ID}\
                 --rpr\
                 --fixed_crop\
                 -train_seed 0\
                 -train_sample_ratio 0.5\
                 -weight_modulus 5\
                 -batch_size 64\
                 --train_sample_shuffle

3. Generate some music segment by the fully trained model as the test set for data attribution. The generation of all test set (178 samples) is less than 10 minutes on A40.

cd music-generation-models/MusicTransformer-Pytorch-paper-1211
python3 generate.py -midi_root /home/user/music-copyright-project/dataset/maestro-v2.0.0-processed\
                    -output_dir /home/user/music-copyright-project/checkpoint/paper-1211-reproduce/generated-music/256\
                    -model_weights /home/user/music-copyright-project/checkpoint/paper-1211-reproduce/model-checkpoints/100p/0/results/best_acc_weights.pickle\
                    --rpr\
                    -target_seq_length 512\
                    -num_prime 256\
                    -max_sequence 256

4. Calculate the attribution score through TRAK. The score calculation takes about ~8 hours on a single A40.

cd data-attribution-models/TRAK-paper-1211
python3 get_score.py -midi_root /home/user/music-copyright-project/dataset/maestro-v2.0.0-processed\
                     -trak_model_checkpoints_root /home/user/music-copyright-project/checkpoint/paper-1211-reproduce/model-checkpoints/50p\
                     -trak_model_checkpoints_number 49\
                     -train_windows_size 256\
                     --rpr\
                     -num_prime 256\
                     -trak_save_dir /home/user/music-copyright-project/checkpoint/paper-1211-reproduce/data-attribution-score/256-together\
                     --test_windows_size 511\
                     -max_sequence 256\
                     -trak_test_batchsize 2\
                     --trak_generated_music_root /home/user/music-copyright-project/checkpoint/paper-1211-reproduce/generated-music/256\
                     -generate_length 256

Evaluation

Ranking correlation

1. Independently train ~100 models with 50% dataset and their sampled data index.

# same as previous steps for 50% dataset retraining.
# Or you can just reuse the previous 50 50% dataset trained models

2. Calculate the actural (golden standard) model output function result

cd data-attribution-models/TRAK-paper-1211
python3 get_model_output.py -midi_root /home/user/music-copyright-project/dataset/maestro-v2.0.0-processed\
                            -trak_model_checkpoints_root /home/user/music-copyright-project/checkpoint/paper-1211-reproduce/model-checkpoints/50p\
                            -trak_model_checkpoints_number 49\
                            -train_windows_size 256\
                            --rpr\
                            -num_prime 256\
                            --test_windows_size 511\
                            -max_sequence 256\
                            -trak_train_batchsize 8\
                            --trak_generated_music_root /home/user/music-copyright-project/checkpoint/paper-1211-reproduce/generated-music/256\
                            -generate_length 256

3. Evaluate the spearman ranking correlation according to counterfactual prediction.

cd data-attribution-models/TRAK-paper-1211
python get_spearman_corr.py -trak_model_checkpoints_number 49 -trak_model
_checkpoints_root /home/user/music-copyright-project/checkpoint/paper-1211-reproduce/model-checkpoints/50p

Similarity correlation

1. Save the topKth largest score trainging sample and the test sample as pairs and calculate the similarities between these pairs.

cd data-attribution-models/TRAK-paper-1211
python calculate_similarity.py --trak_generated_music_root /home/user/music-copyright-project/checkpoint/paper-1211-reproduce/generated-music/256 --test_windows_size 511 -output_dir /home/user/music-copyright-project/checkpoint/scratch/256-gen -midi_root /home/user/music-copyright-project/dataset/maestro-v2.0.0-processed -train_windows_size 256

Experiment for Adversarial attack

Duplication

Duplicatation means that if we duplicate one of the sample in training set. Would the score (remarkably, the score should be the sum of all duplicate samples) be increased?

In this experiment, the only thing you need to change is the way you train the models (both for music generation(full dataset) and attribution score calculation(subset)).

Currently, there is a sampler be passed to the train loader looks like this.

sampler = EPianoDatasetSampler(train_dataset, seed=args.train_sample_seed,
                               ratio=args.train_sample_ratio, saving_root=args.output_dir,
                               shuffle=args.train_sample_shuffle,
                               duplicate=[0, 100])  # duplicate index 0 in 5000 training samples 100 times

Then you can follow the previous settings to (following #attribution-score-calculation)

1. Train a 100% (5100 samples) model for music generation
2. Train 10 50% (2550 samples) model for attribution score
3. Generate 178 music segment by test prompt using 100% (5100 samples) model. (for this experiment, genearte 1 token should be enough.)
4. Calculate the score

Copy-and-Paste

Copy and paste means that if we just copy some events (like a sub-segment) from high influential training samples to some randomly chosed data sample, could we make the score of that data sample higher?

In this experiment, the only thing you need to change is the way you train the models (both for music generation(full dataset) and attribution score calculation(subset)).

# change index 0's first 128 events to be the same as index 4663's first 128 events
modification_map=[
    {"target_index": SANITY_CHECK_LENGTH_256_TOTAL_5000[0],
     "source_index": SANITY_CHECK_LENGTH_256_TOTAL_5000[4663],
     "target_range": [0, 128],
     "source_range": [0, 128],
     "if_label": False}
]
train_dataset, _, _ = create_epiano_datasets(args.input_dir,
                                             args.max_sequence,
                                             full_version=args.fixed_crop,
                                             modification_map=modification_map)

Then you can follow the previous settings to (following #attribution-score-calculation)

1. Train a 100% (5000 samples) model for music generation
2. Train 10 50% (2500 samples) model for attribution score
3. Generate 178 music segment by test prompt using 100% (5000 samples) model. (for this experiment, genearte 1 token should be enough.)
4. Calculate the score

Gradient based

TODO

Version

• pytorch 2.1
• CUDA 11.8
• traker 0.3.1 (with my patch MadryLab/trak#58)