I started a 3-nodes dynamo serve and I get this error, not everytime, but most of the times:
$ genai-perf profile -m deepseek-ai/DeepSeek-R1-Distill-Llama-8B --url dynamo_host:8000 --endpoint-type chat --synthetic-input-tokens-mean 128 --synthetic-input-tokens-stddev 0 --output-tokens-mean 20 --output-tokens-stddev 0 --streaming --request-count 10 --warmup-request-count 2
[2025-07-29 11:38:23] DEBUG Inferred tokenizer from model name: deepseek-ai/DeepSeek-R1-Distill-Llama-8B config_tokenizer.py:74
[2025-07-29 11:38:23] INFO Profiling these models: deepseek-ai/DeepSeek-R1-Distill-Llama-8B create_config.py:57
[2025-07-29 11:38:23] INFO Model name 'deepseek-ai/DeepSeek-R1-Distill-Llama-8B' cannot be used to create artifact directory. Instead, perf_analyzer_config.py:154
'deepseek-ai_DeepSeek-R1-Distill-Llama-8B' will be used.
[2025-07-29 11:38:28] INFO Running Perf Analyzer : 'perf_analyzer -m deepseek-ai/DeepSeek-R1-Distill-Llama-8B --async --stability-percentage 999 common.py:184
--warmup-request-count 2 --request-count 10 -i http -u dynamo_host:8000 --service-kind openai --endpoint
v1/chat/completions --input-data artifacts/deepseek-ai_DeepSeek-R1-Distill-Llama-8B-openai-chat-concurrency1/inputs.json
--profile-export-file artifacts/deepseek-ai_DeepSeek-R1-Distill-Llama-8B-openai-chat-concurrency1/profile_export.json'
[2025-07-29 11:38:31] INFO Loading response data from profile_data_parser.py:66
'artifacts/deepseek-ai_DeepSeek-R1-Distill-Llama-8B-openai-chat-concurrency1/profile_export.json'
[2025-07-29 11:38:31] INFO Parsing total 10 requests. llm_profile_data_parser.py:123
Progress: 0%| | 0/10 [00:00<?, ?requests/s][2025-07-29 11:38:31] ERROR
Failed to parse JSON string: utils.py:103
'{"id":"chatcmpl-b6ee7ccf-9a85-410f-8433-99d475ffc777","choices":[{"index":0,"delta":{"content":"ym","function_call":null,"tool_
calls":null,"role":"assistant","refusal":null},"finish_reason":null,"logprobs":null}],"created":1753778310,"model":"deepseek-ai/
DeepSeek-R1-Distill-Llama-8B","service_tier":null,"system_fingerprint":null,"object":"chat.completion.chunk","usage":{"prompt_to
kens":133,"completion_tokens"'
Progress: 80%|███████████████████████████████████████████████████████████████████████████████████████████████▏ | 8/10 [00:00<00:00, 560.73requests/s]
Traceback (most recent call last):
File "/opt/dynamo/venv/lib/python3.12/site-packages/genai_perf/main.py", line 52, in main
run()
File "/opt/dynamo/venv/lib/python3.12/site-packages/genai_perf/main.py", line 45, in run
args.func(config, extra_args)
File "/opt/dynamo/venv/lib/python3.12/site-packages/genai_perf/subcommand/profile.py", line 78, in profile_handler
data_parser = calculate_metrics(config, perf_analyzer_config, tokenizer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/dynamo/venv/lib/python3.12/site-packages/genai_perf/subcommand/common.py", line 87, in calculate_metrics
return LLMProfileDataParser(
^^^^^^^^^^^^^^^^^^^^^
File "/opt/dynamo/venv/lib/python3.12/site-packages/genai_perf/profile_data_parser/llm_profile_data_parser.py", line 90, in __init__
super().__init__(filename, goodput_constraints)
File "/opt/dynamo/venv/lib/python3.12/site-packages/genai_perf/profile_data_parser/profile_data_parser.py", line 69, in __init__
self._parse_profile_data(data)
File "/opt/dynamo/venv/lib/python3.12/site-packages/genai_perf/profile_data_parser/llm_profile_data_parser.py", line 100, in _parse_profile_data
llm_metrics = self._parse_requests(requests)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/dynamo/venv/lib/python3.12/site-packages/genai_perf/profile_data_parser/llm_profile_data_parser.py", line 130, in _parse_requests
self._preprocess_response(res_timestamps, res_outputs)
File "/opt/dynamo/venv/lib/python3.12/site-packages/genai_perf/profile_data_parser/llm_profile_data_parser.py", line 328, in _preprocess_response
elif self._is_empty_response(responses[0]):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/dynamo/venv/lib/python3.12/site-packages/genai_perf/profile_data_parser/llm_profile_data_parser.py", line 515, in _is_empty_response
text = self._extract_text_output(response)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/dynamo/venv/lib/python3.12/site-packages/genai_perf/profile_data_parser/llm_profile_data_parser.py", line 437, in _extract_text_output
return extract_method(response)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/dynamo/venv/lib/python3.12/site-packages/genai_perf/profile_data_parser/llm_profile_data_parser.py", line 466, in _extract_openai_chat_text_output
data = load_json_str(response)
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/dynamo/venv/lib/python3.12/site-packages/genai_perf/utils.py", line 100, in load_json_str
return func(json.loads(json_str))
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
^^^^^^^^^^^^^^^^^^^^^^
json.decoder.JSONDecodeError: Expecting ':' delimiter: line 1 column 413 (char 412)
[2025-07-29 11:38:31] ERROR Expecting ':' delimiter: line 1 column 413 (char 412)
I started a 3-nodes dynamo serve and I get this error, not everytime, but most of the times:
I've removed the JSON trim in the error so that it displays the whole response
It looks like an unfinished response is processed
However, with normal curl requests, the dynamo instance works very well