How do you actually make new songs with this?

[JELSTUDIO]

Normally you'd enter some lyrics and have the AI generate a song from that.

I can get the demos to output songs (The inference runs fine and finishes without errors and I get an output-song)

But how does one create a new song with new lyrics?

There are no instructions detailing this (At least I can't seem to find any)

If I create a json-file with the suggested structure:

[
{"start": 2.2, "end": 2.5, "One": "First word of lyrics"},
{"start": 2.5, "end": 3.7, "word": "Second word of lyrics"},
{"start": 4.5, "end": 6.7, "three": "Third word of lyrics"}
]

I get this error:

Traceback (most recent call last):
File "D:\AI\jamify\src\jam\dataset.py", line 160, in process_sample_safely
return self.process_sample(sample)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\AI\jamify\src\jam\dataset.py", line 232, in process_sample
word_text = word["word"]
~~~~^^^^^^^^
KeyError: 'word'

Process 0: Skipped batch 0 - failed to process

We are (Hopefully) not supposed to sit and edit the 3 sections "char", "word", and "segment" manually?? (The json demo-file is over 7 thousand lines, which would be impossible to do manually in any sensible way)

I'm sure I'm misunderstanding something about how this jamify works.

Does anybody know?

Thank you.

[zhouzhao01]

I use ASR tools (like Whisper) to transcribe an existing music piece and obtain word-level timestamps. Then I use an LLM to clean and modify the captions, producing the lyric file the model needs.

[JELSTUDIO]

I use ASR tools (like Whisper) to transcribe an existing music piece and obtain word-level timestamps. Then I use an LLM to clean and modify the captions, producing the lyric file the model needs.

Thank you. I was hoping for a way to create completely new songs though :) (Songs that haven't been recorded and therefore doesn't haven any timings yet)