Critical(need help): benchmark fails with aiperf0.4 version but aiperf0.3 succeeds

[Maze999]

with 0.4.0:

it starts diplaing the UI metrics and then in 30 sec to a min it fails

Full log(from another run): i had to kill the run from above snap and hence complete log is not avaialble but issue seems consistent

2026-01-16 10:49:41 - SyntheticDatasetComposer - INFO - Using default sampling strategy for synthetic dataset: shuffle
2026-01-16 10:49:41 - dataset_manager_ce93faa4 - INFO - Generating inputs.json file at /workdir/AIE_1.10.0/llama-3.1-70b-instruct/nim_1.10/new-comp2/aiperf/aiperf-artifact/ISL200_OSL200/CON512/inputs.json
2026-01-16 10:49:41 - dataset_manager_ce93faa4 - INFO - inputs.json file generated in 0.01 seconds
2026-01-16 10:49:41 - dataset_manager_ce93faa4 - INFO - Dataset configured in 0.33 seconds
2026-01-16 10:49:48 - InferenceResultParser_c0198a97 - INFO - Initialized tokenizers: {'meta/llama-3.1-70b-instruct': {'class': 'PreTrainedTokenizerFast', 'name_or_path': 'meta-llama/Llama-3.1-70B-Instruct'}} in 9.62 seconds
2026-01-16 10:49:48 - system_controller - INFO - All services configured in 9.62 seconds
2026-01-16 10:49:48 - system_controller - INFO - AIPerf System is CONFIGURED
2026-01-16 10:49:48 - timing_manager_7b415879 - INFO - Credit issuing strategy for Request_Rate started
2026-01-16 10:50:48 - system_controller - ERROR - Error running Hook(func=<bound method SystemController._start_services of <SystemController system_controller (state=starting)>>, params=None) hook for SystemController: Failed to perform operation 'Start Profiling'
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/aiperf/common/mixins/hooks_mixin.py", line 186, in run_hooks
await hook(**kwargs)
File "/usr/local/lib/python3.12/dist-packages/aiperf/common/hooks.py", line 115, in call
await self.func(**kwargs)
File "/usr/local/lib/python3.12/dist-packages/aiperf/controller/system_controller.py", line 212, in _start_services
await self._start_profiling_all_services()
File "/usr/local/lib/python3.12/dist-packages/aiperf/controller/system_controller.py", line 244, in _start_profiling_all_services
self._parse_responses_for_errors(responses, "Start Profiling")
File "/usr/local/lib/python3.12/dist-packages/aiperf/controller/system_controller.py", line 267, in _parse_responses_for_errors
raise LifecycleOperationError(
aiperf.common.exceptions.LifecycleOperationError: Failed to perform operation 'Start Profiling'
2026-01-16 10:50:48 - system_controller - ERROR - Failed for SystemController (id=system_controller): SystemController._start_services: Failed to perform operation 'Start Profiling'
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-16 10:50:48 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')

Execution paramters:

NIM: nvcr.io/nim/meta/llama-3.1-70b-instruct-pb25h1:1.10(2 replicas)

Execution paramters:

./aiperf-bench.sh --url http://x.x.x.x --model meta/llama-3.1-70b-instruct --tokenizer meta-llama/Llama-3.1-70B-Instruct --concurrency-values 1,2,4,8,16,32,64,128,256,512 --use-cases Search,Summarization,Translation --benchmark-duration 900 --benchmark-grace-period 0 --profile-export-file llama-31-70b-fp8-tp2-pp1-latency-2nim-aiperfv04-run4

Everything being as is it succeeds after i remove 0.4 and install 0.3 version :

[Maze999]

i looked into the logs w.r.t 0.3 version and even though it also has error entries run completes sucessfully
2026-01-20 01:58:57 - aiperf.common.utils - ERROR - Failed to parse JSON string: '{"id":"chatcmpl-39fc0933-bedf-4b5c-9675-fbcef38c5fd6","object":"chat.completion.chunk","created":1768874298,"model":"meta/llama-3.1-70b-instruct","choices":[{"index":0,"delta":{"content":"' - JSONDecodeError('unexpected end of data: line 1 column 189 (char 188)')
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/aiperf/common/utils.py", line 94, in load_json_str
return func(orjson.loads(json_str))
^^^^^^^^^^^^^^^^^^^^^^
orjson.JSONDecodeError: unexpected end of data: line 1 column 189 (char 188)
2026-01-20 01:58:57 - aiperf.common.utils - ERROR - Failed to parse JSON string: '{"id":"chatcmpl-39fc0933-bedf-4b5c-9675-fbcef38c5fd6","object":"chat.completion.chunk","created":1768874298,"model":"meta/llama-3.1-70b-instruct","choices":[{"index":0,"delta":{"content":"' - JSONDecodeError('unexpected end of data: line 1 column 189 (char 188)')
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/aiperf/common/utils.py", line 94, in load_json_str
return func(orjson.loads(json_str))
^^^^^^^^^^^^^^^^^^^^^^
orjson.JSONDecodeError: unexpected end of data: line 1 column 189 (char 188)
2026-01-20 02:11:30 - RequestRateStrategy - INFO - Total timeout (900.0s + 0.0s grace) elapsed for profiling phase
2026-01-20 02:11:30 - RequestRateStrategy - NOTICE - Phase force-completed due to grace period timeout: type=CreditPhase.PROFILING total_expected_requests=None expected_duration_sec=900.0 start_ns=1768874190150527464 sent_end_ns=1768875090246703454 end_ns=1768875090248498463 sent=5972 completed=5717 requests_sent=5717
2026-01-20 02:11:30 - records_manager_42abc09a - INFO - Sent 5,972 conversations. Waiting for all to complete...
2026-01-20 02:11:30 - records_manager_42abc09a - NOTICE - All requests have completed, please wait for the results to be processed (currently 5,716 of 5,717 records processed)...
2026-01-20 02:11:30 - records_manager_42abc09a - INFO - Processed 5716 valid requests and 0 errors (5716 total).
2026-01-20 02:11:30 - records_manager_42abc09a - INFO - Processing records results...
2026-01-20 02:11:30 - system_controller - INFO - Received credits complete from timing_manager_6d02ee50
2026-01-20 02:11:30 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-20 02:11:30 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-20 02:11:30 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-20 02:11:30 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-20 02:11:30 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-20 02:11:30 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-20 02:11:30 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-20 02:11:30 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-20 02:11:30 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-20 02:11:30 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-20 02:11:30 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-20 02:11:30 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-20 02:11:30 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-20 02:11:30 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-20 02:11:30 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-20 02:11:30 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-20 02:11:30 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-20 02:11:30 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-20 02:11:30 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-20 02:11:30 - AioHttpClient - ERROR - Error in aiohttp request: RuntimeError('Connection closed.')
2026-01-20 02:11:31 - ExporterManager - INFO - Exporting all records
2026-01-20 02:11:31 - ExporterManager - INFO - Exporting console data

[ganeshku1]

@Maze999 Sorry you’re running into this issue. While we investigate on our side, could you please share the complete reproduction steps for the behavior you’re seeing? That will help us narrow this down more quickly.

[ajcasagrande]

@Maze999 it is possible this is due to the known issue from 0.4.0 where if aiperf is unable to probe your endpoint at /metrics for prometheus metrics, it will cause this timeout error.

For now, try running with --no-server-metrics to disable the server metrics service, or use --server-metrics url1/metrics ur2/metrics , etc. to specify the url to prometheus endpoints to scrape.

See: https://github.com/ai-dynamo/aiperf/blob/main/docs/server_metrics/server-metrics.md

[Maze999]

@ajcasagrande
Thank you for the quick turn around , it is working as expected after using --no-server-metrics