CUDA_VISIBLE_DEVICES=0 python run.py --model_name "deepseek-ai/deepseek-coder-1.3b-base" --dataset_name "tianyang/repobench_python_v1.1" --start_date "2023-12-01" --end_date "2023-12-31" --language "python" --max_token_nums 15800 --levels "2k" "4k" "8k" "12k" "16k" --temperature 0.2 --top_p 0.95 --max_new_tokens 128 --batch_size 1
/network/scratch/n/nizar.islah/.conda/envs/lngctx/lib/python3.10/site-packages/transformers/utils/hub.py:106: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
warnings.warn(
(…)-00000-of-00002-baebda7f3a6e980a.parquet: 100%|█| 32.1M/32.1M [00:00<00:00, 39.4MB/s]
(…)-00001-of-00002-5780ed62c5162a3e.parquet: 100%|███| 132M/132M [00:02<00:00, 47.1MB/s]
(…)-00000-of-00001-523c739d72005569.parquet: 100%|███| 153M/153M [00:02<00:00, 55.4MB/s]
(…)-00000-of-00001-9721c68f6224a587.parquet: 100%|███| 156M/156M [00:04<00:00, 37.5MB/s]
Generating cross_file_first split: 100%|███| 8033/8033 [00:01<00:00, 5671.98 examples/s]
Generating cross_file_random split: 100%|██| 7618/7618 [00:01<00:00, 6352.26 examples/s]
Generating in_file split: 100%|████████████| 7910/7910 [00:01<00:00, 6462.95 examples/s]
Filter: 100%|█████████████████████████████| 2528/2528 [00:00<00:00, 14161.03 examples/s]
Filter: 100%|█████████████████████████████| 2237/2237 [00:00<00:00, 13471.42 examples/s]
Filter: 100%|█████████████████████████████| 2373/2373 [00:00<00:00, 13274.12 examples/s]
tokenizer_config.json: 100%|███████████████████████████| 793/793 [00:00<00:00, 11.4MB/s]
tokenizer.json: 100%|██████████████████████████████| 1.37M/1.37M [00:00<00:00, 23.3MB/s]
special_tokens_map.json: 100%|█████████████████████████| 482/482 [00:00<00:00, 8.39MB/s]
config.json: 100%|█████████████████████████████████████| 631/631 [00:00<00:00, 9.49MB/s]
pytorch_model.bin: 100%|████████████████████████████| 2.69G/2.69G [00:23<00:00, 114MB/s]
generation_config.json: 100%|██████████████████████████| 119/119 [00:00<00:00, 1.74MB/s]
model.safetensors: 100%|████████████████████████████| 2.69G/2.69G [00:24<00:00, 112MB/s]
Generating cross_file_first: 2%|▍ | 8/377 [00:20<15:26, 2.51s/it]
Generating cross_file_first: 69%|███████████
Generating cross_file_first: 69%|███████████
Generating cross_file_first: 71%|███████████
Generating cross_file_first: 71%|███████████
Generating cross_file_first: 72%|███████████
Generating cross_file_first: 75%|███████████
Generating cross_file_first: 75%|███████████
Generating cross_file_first: 100%|████████████████████| 377/377 [14:12<00:00, 2.26s/it]
Generating cross_file_random: 100%|██████████Generating cross_file_random: 100%|███████████████████| 313/313 [11:47<00:00, 2.26s/it]
Generating in_file: 0%| Generating in_file: 100%|█████████████████████████████| 307/307 [11:20<00:00, 2.22s/it]
ERROR: Could not consume arg: 8k
Usage: run.py --model_name deepseek-ai/deepseek-coder-1.3b-base --dataset_name tianyang/repobench_python_v1.1 --start_date 2023-12-01 --end_date 2023-12-31 --language python --max_token_nums 15800 --levels 2k 4k 8k 12k 16k --temperature 0.2 --top_p 0.95 --max_new_tokens
For detailed information on this command, run:
run.py --model_name deepseek-ai/deepseek-coder-1.3b-base --dataset_name tianyang/repobench_python_v1.1 --start_date 2023-12-01 --end_date 2023-12-31 --language python --max_token_nums 15800 --levels 2k 4k 8k 12k 16k --temperature 0.2 --top_p 0.95 --max_new_tokens --help
hi I ran the README example code and ran into this. any idea what could be causing this error and how to fix it? Thanks
hi I ran the README example code and ran into this. any idea what could be causing this error and how to fix it? Thanks