How to replace the new dataset

[a897456]

Hi @v-iashin
If I want to retrain the model with a new dataset, such as LJSpeech, which .py file should I start with?

[v-iashin]

Hi, thank you for your question.

Assuming you are only considering training the first stage (autoencoder), here are a few hints:

1. Create a dataset module similar to VGGSound in specvqgan/data/vggsound.py.
2. Adapt the config configs/vggsound_codebook.yaml for your dataset.
3. Then follow the instructions Training a spectrogram codebook

[a897456]

Hi @v-iashin
Thank you for your reply.
I obtained the mel_spectrum from the LJSpeech dataset through tacotron2 and generated 13100 .npy files, but the spec_len of the mel_spectrum in these files was disorganized, and the .npy file whose spec_len was larger than 848 (the number required in your code) was only 1/3, about 470 files.
What should I do to keep going? Please give me some hints. Thank you.

[a897456]

E:\ProgramData\anaconda3\envs\py39\python.exe C:\Users\User1\Downloads\SpecVQGAN-main\train.py python train.py --base configs/LJSpeech_codebook.yaml -t True --gpus 0,
2024-03-06T20-39-39_LJSpeech_codebook
Global seed set to 23
Running on GPUs 0,
loaded pretrained LPAPS loss from specvqgan/modules/autoencoder/lpaps\vggishish16.pt
VQLPAPSWithDiscriminator running with hinge loss.
E:\ProgramData\anaconda3\envs\py39\lib\site-packages\pytorch_lightning\utilities\distributed.py:68: UserWarning: ModelCheckpoint(save_last=True, monitor=None) is a redundant configuration. You can save the last checkpoint with ModelCheckpoint(save_top_k=None, monitor=None).
warnings.warn(*args, **kwargs)
We will not save audio for conditioning and conditioning_rec
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Global seed set to 23
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/1
[W C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\distributed\c10d\socket.cpp:558] [c10d] The client socket has failed to connect to [localhost.sangfor.com.cn]:64851 (system error: 10049 - �����������У�������ĵ�ַ��Ч��).
[W C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\distributed\c10d\socket.cpp:558] [c10d] The client socket has failed to connect to [localhost.sangfor.com.cn]:64851 (system error: 10049 - �����������У�������ĵ�ַ��Ч��).
accumulate_grad_batches = 1
Setting learning rate to 1.35e-05 = 1 (accumulate_grad_batches) * 1 (num_gpus) * 3 (batchsize) * 4.50e-06 (base_lr)
Project config
model:
base_learning_rate: 4.5e-06
target: specvqgan.models.vqgan.VQModel
params:
embed_dim: 256
n_embed: 1024
ddconfig:
double_z: false
z_channels: 256
resolution: 848
in_channels: 1
out_ch: 1
ch: 128
ch_mult:
- 1
- 1
- 2
- 2
- 4
num_res_blocks: 2
attn_resolutions:
- 53
dropout: 0.0
lossconfig:
target: specvqgan.modules.losses.vqperceptual.VQLPAPSWithDiscriminator
params:
disc_conditional: false
disc_in_channels: 1
disc_start: 30001
disc_weight: 0.8
codebook_weight: 1.0
min_adapt_weight: 1.0
max_adapt_weight: 1.0
perceptual_weight: 1.0
data:
target: train.SpectrogramDataModuleFromConfig
params:
batch_size: 3
num_workers: 8
spec_dir_path: ./data/LJSpeech
sample_rate: 22050
mel_num: 80
spec_len: 860
spec_crop_len: 848
random_crop: false
train:
target: specvqgan.data.LJSpeech_data.LJSpeechSpecsTrain
params:
specs_dataset_cfg: null
validation:
target: specvqgan.data.LJSpeech_data.LJSpeechSpecsValidation
params:
specs_dataset_cfg: null
test:
target: specvqgan.data.LJSpeech_data.LJSpeechSpecsTest
params:
specs_dataset_cfg: null
python: null
train:
py: null

Lightning config
callbacks:
image_logger:
target: train.ImageLogger
params:
for_specs: true
vocoder_cfg:
target: train.VocoderMelGan
params:
ckpt_vocoder: ./vocoder/logs/vggsound/
trainer:
sync_batchnorm: true
distributed_backend: ddp
gpus: 0,

E:\ProgramData\anaconda3\envs\py39\lib\site-packages\omegaconf\basecontainer.py:225: UserWarning: cfg.pretty() is deprecated and will be removed in a future version.
Use OmegaConf.to_yaml(cfg)

warnings.warn(

| Name | Type | Params

0 | encoder | Encoder | 29.3 M
1 | decoder | Decoder | 42.4 M
2 | loss | VQLPAPSWithDiscriminator | 17.5 M
3 | quantize | VectorQuantizer | 262 K
4 | quant_conv | Conv2d | 65.8 K
5 | post_quant_conv | Conv2d | 65.8 K

74.9 M Trainable params
14.7 M Non-trainable params
89.6 M Total params
358.463 Total estimated model params size (MB)
Epoch 0: 0%| | 0/151 [00:00<?, ?it/s] E:\ProgramData\anaconda3\envs\py39\lib\site-packages\pytorch_lightning\utilities\distributed.py:68: RuntimeWarning: You are using LearningRateMonitor callback with models that have no learning rate schedulers. Please see documentation for configure_optimizers method.
warnings.warn(*args, **kwargs)

Hi @v-iashin
Did I succeed?

[a897456]

Hi @v-iashin
I am so sorry, I'm a novice and I may ask a lot of silly questions, please forgive me. thank you.
Can you tell me what do these numbers mean?

[v-iashin]

these are the number of times each codebook code was used during previous epoch, e.g. code No.4 was used 48 times. You want these counts to be as uniform as possible (less zeros).

[a897456]

Hi @v-iashin
Thank you for your reply

1. validating shows 0%, whether this is because I did not add a validation set?
2. Now epoch=12, there seem to be more and more zeros. What can I do to reduce them?
3. What is the number of the loss and the epoch in your pre-trained model?
   

[a897456]

Hi @v-iashin Thank you for your reply. I obtained the mel_spectrum from the LJSpeech dataset through tacotron2 and generated 13100 .npy files, but the spec_len of the mel_spectrum in these files was disorganized, and the .npy file whose spec_len was larger than 848 (the number required in your code) was only 1/3, about 470 files. What should I do to keep going? Please give me some hints. Thank you.

Hi @v-iashin
Can you give me some guidance on this problem?