I have tried your llama example and the output is random and took 770 second to finish:
commmand :
python src/run_generation.py --model_type llama --model_name_or_path meta-llama/Llama-2-7b-chat-hf --prefix "<s>[INST] <<SYS>>\n You are a helpful assistant. Answer with detailed responses according to the entire instruction or question. \n<</SYS>>\n\n Summarize the following book: " --prompt example_inputs/harry_potter_full.txt --suffix " [/INST]" --test_unlimiformer --fp16 --layer_begin 16 --index_devices 0 --datastore_device 0 --length 10000
output :
||| This chapter is from "Harry Potter and the Philosopher's Stone" by J.K. Rowling. Harry Potter, a young wizard, has just saved the
world from the Dark Lord Voldemort and is returning to Hogwarts School for the
summer break. He, Ron, and Hermione, are leaving, and they've just caught,
There are no obstacles here.
"I don't know
"wanted to,
"They
"seem
"mind
"hard
"ever
"making
"them
"still
"not
"Just
"how
"hard
"know
"like
"fright
"through
"the
"consider
"him
"Other
"it's
"for
"out
"made
"dead
"good
"can't
"t
"Harry
"add
"Harry
"make
"dead
"extra
"Harry
"bra
"They
"There
"ded
"Harry
"he
"There
"wiz
"Thank
Harry
"Before
Harry."
"You
they
" "lived
"you.
"I
" onlived.
"No one
"I
"distantly
"Harry?
"which
"
"Yes,
"was
"I suppose
"No,
"I
"... Harry
"No...
"Before
"lived
"you?
"No,"
"Harry
"you.
"what
"lived."
"I reckoned
"no
"No,"Harry."
"lived
"Dumbled
"you
"...I reckoned me,"said
"No," said
"Harry."
"Harry."
"Dumbled
"No," said
I have tried your llama example and the output is random and took 770 second to finish:
commmand :
output :
||| This chapter is from "Harry Potter and the Philosopher's Stone" by J.K. Rowling. Harry Potter, a young wizard, has just saved the
world from the Dark Lord Voldemort and is returning to Hogwarts School for the
summer break. He, Ron, and Hermione, are leaving, and they've just caught,
There are no obstacles here.
"I don't know
"wanted to,
"They
"seem
"mind
"hard
"ever
"making
"them
"still
"not
"Just
"how
"hard
"know
"like
"fright
"through
"the
"consider
"him
"Other
"it's
"for
"out
"made
"dead
"good
"can't
"t
"Harry
"add
"Harry
"make
"dead
"extra
"Harry
"bra
"They
"There
"ded
"Harry
"he
"There
"wiz
"Thank
Harry
"Before
Harry."
"You
they
" "lived
"you.
"I
" onlived.
"No one
"I
"distantly
"Harry?
"which
"
"Yes,
"was
"I suppose
"No,
"I
"... Harry
"No...
"Before
"lived
"you?
"No,"
"Harry
"you.
"what
"lived."
"I reckoned
"no
"No,"Harry."
"lived
"Dumbled
"you
"...I reckoned me,"said
"No," said
"Harry."
"Harry."
"Dumbled
"No," said