aiperf plot arg "--no-dashboard" is not working #604
Description
As written on title --no-dashboard not working
Log of aiperf plot --no-dashboard with verbose
aiperf plot --no-dashboard --verbose
08:05:30.981908 INFO Plot logging initialized with level: DEBUG logging.py:107
08:05:30.984854 INFO Log file: artifacts/plots/aiperf_plot.log logging.py:108
Using config: /root/.aiperf/plot_config.yaml
08:05:31.038981 INFO Server metrics downsampling: 5.0s windows, mean aggregation plot_controller.py:77
08:05:31.040139 INFO Found 1 unique run directories mode_detector.py:138
08:05:31.040745 INFO Detected SINGLE_RUN mode: 1 run directory found mode_detector.py:75
08:05:31.041333 INFO Generating single-run plots (1 run) plot_controller.py:143
08:05:31.041894 INFO Loading run from: artifacts/nvidia_llama-3.1-nemotron-nano-8b-v1-openai-chat-concurrency300 data_loader.py:240
08:05:31.090068 INFO Loaded 300 records from artifacts/nvidia_llama-3.1-nemotron-nano-8b-v1-openai-chat-concurrency300/profile_export.jsonl data_loader.py:678
08:05:31.091037 INFO Loaded aggregated data from artifacts/nvidia_llama-3.1-nemotron-nano-8b-v1-openai-chat-concurrency300/profile_export_aiperf.json data_loader.py:770
08:05:31.091743 INFO Extracted telemetry data with 0 endpoints data_loader.py:432
08:05:31.092300 DEBUG No GPUs found in telemetry data data_loader.py:487
08:05:31.092803 DEBUG Skipping derived GPU metrics: telemetry data not available or no GPUs found data_loader.py:507
08:05:31.094382 INFO Loaded 24 server metrics from artifacts/nvidia_llama-3.1-nemotron-nano-8b-v1-openai-chat-concurrency300/server_metrics_export.json (0 data_loader.py:972
timeslice data points)
08:05:31.095245 INFO Loaded server metrics from JSON (Parquet not available) data_loader.py:332
08:05:31.095874 INFO Found 21 available metrics: ['benchmark_duration', 'input_sequence_length', 'inter_chunk_latency', 'inter_token_latency', data_loader.py:561
'max_response_timestamp', 'min_request_timestamp', 'output_sequence_length', 'output_token_count', 'output_token_throughput',
'output_token_throughput_per_user', 'prefill_throughput_per_user', 'request_count', 'request_latency', 'request_throughput',
'time_to_first_output_token', 'time_to_first_token', 'time_to_second_token', 'total_isl', 'total_osl', 'total_output_tokens',
'total_token_throughput']
08:05:31.096857 INFO Classification disabled for plot 'ttft_over_time': using groups=['run_name'] config.py:452
08:05:31.097500 INFO Classification disabled for plot 'ttft_timeline': using groups=['run_name'] config.py:452
08:05:31.098091 INFO Classification disabled for plot 'timeslices_ttft': using groups=['run_name'] config.py:452
08:05:31.098675 INFO Classification disabled for plot 'timeslices_itl': using groups=['run_name'] config.py:452
08:05:31.099277 INFO Classification disabled for plot 'gpu_utilization_and_throughput_over_time': using groups=['run_name'] config.py:452
08:05:31.197667 DEBUG Successfully registered PlotlyJSONEncoder. kaleido.py:36
08:05:31.201349 DEBUG Using selector: EpollSelector selector_events.py:64
08:05:31.202306 DEBUG Timeout: 90 kaleido.py:159
08:05:31.202851 DEBUG Attempting to open new browser. browser_async.py:100
08:05:31.204384 INFO Chromium init'ed with kwargs {} chromium.py:182
08:05:31.205068 DEBUG Looking for a chrome browser. chromium.py:81
08:05:31.205574 DEBUG Looking for browser, skipping local? False _which.py:64
08:05:31.207252 DEBUG Looking for at local chrome download path: /usr/local/lib/python3.12/dist-packages/choreographer/cli/browser_exe/chrome-linux64/chrome _which.py:76
08:05:31.207861 DEBUG Local chrome not found at path: /usr/local/lib/python3.12/dist-packages/choreographer/cli/browser_exe/chrome-linux64/chrome. _which.py:81
08:05:31.208704 DEBUG Looking for a chromium browser. chromium.py:81
08:05:31.209206 DEBUG Looking for browser, skipping local? False _which.py:64
08:05:31.209736 DEBUG Looking for at local chrome download path: /usr/local/lib/python3.12/dist-packages/choreographer/cli/browser_exe/chrome-linux64/chrome _which.py:76
08:05:31.210268 DEBUG Local chrome not found at path: /usr/local/lib/python3.12/dist-packages/choreographer/cli/browser_exe/chrome-linux64/chrome. _which.py:81
08:05:31.210916 DEBUG Looking for a edge browser. chromium.py:81
08:05:31.211388 DEBUG Looking for browser, skipping local? False _which.py:64
08:05:31.211935 DEBUG Looking for at local chrome download path: /usr/local/lib/python3.12/dist-packages/choreographer/cli/browser_exe/chrome-linux64/chrome _which.py:76
08:05:31.212456 DEBUG Local chrome not found at path: /usr/local/lib/python3.12/dist-packages/choreographer/cli/browser_exe/chrome-linux64/chrome. _which.py:81
08:05:31.213120 DEBUG Looking for a brave browser. chromium.py:81
08:05:31.213617 DEBUG Looking for browser, skipping local? False _which.py:64
08:05:31.214134 DEBUG Looking for at local chrome download path: /usr/local/lib/python3.12/dist-packages/choreographer/cli/browser_exe/chrome-linux64/chrome _which.py:76
08:05:31.214682 DEBUG Local chrome not found at path: /usr/local/lib/python3.12/dist-packages/choreographer/cli/browser_exe/chrome-linux64/chrome. _which.py:81
08:05:31.215310 DEBUG Looking for a vivaldi browser. chromium.py:81
08:05:31.215809 DEBUG Looking for browser, skipping local? False _which.py:64
08:05:31.216323 DEBUG Looking for at local chrome download path: /usr/local/lib/python3.12/dist-packages/choreographer/cli/browser_exe/chrome-linux64/chrome _which.py:76
08:05:31.216876 DEBUG Local chrome not found at path: /usr/local/lib/python3.12/dist-packages/choreographer/cli/browser_exe/chrome-linux64/chrome. _which.py:81
08:05:31.217691 ERROR Failed to generate ttft_over_time: single_run.py:71
Kaleido requires Google Chrome to be installed.
Either download and install Chrome yourself following Google's instructions for your operating system,
or install it from your terminal by running:
$ plotly_get_chrome
08:05:31.362599 DEBUG Using selector: EpollSelector selector_events.py:64
08:05:31.363603 DEBUG Timeout: 90 kaleido.py:159
08:05:31.364177 DEBUG Attempting to open new browser. browser_async.py:100
08:05:31.365876 INFO Chromium init'ed with kwargs {} chromium.py:182
08:05:31.366472 DEBUG Looking for a chrome browser. chromium.py:81
08:05:31.366967 DEBUG Looking for browser, skipping local? False _which.py:64
08:05:31.367545 DEBUG Looking for at local chrome download path: /usr/local/lib/python3.12/dist-packages/choreographer/cli/browser_exe/chrome-linux64/chrome _which.py:76
08:05:31.368089 DEBUG Local chrome not found at path: /usr/local/lib/python3.12/dist-packages/choreographer/cli/browser_exe/chrome-linux64/chrome. _which.py:81
08:05:31.368895 DEBUG Looking for a chromium browser. chromium.py:81
08:05:31.369374 DEBUG Looking for browser, skipping local? False _which.py:64
08:05:31.369880 DEBUG Looking for at local chrome download path: /usr/local/lib/python3.12/dist-packages/choreographer/cli/browser_exe/chrome-linux64/chrome _which.py:76
08:05:31.370435 DEBUG Local chrome not found at path: /usr/local/lib/python3.12/dist-packages/choreographer/cli/browser_exe/chrome-linux64/chrome. _which.py:81
08:05:31.371066 DEBUG Looking for a edge browser. chromium.py:81
08:05:31.371582 DEBUG Looking for browser, skipping local? False _which.py:64
08:05:31.372187 DEBUG Looking for at local chrome download path: /usr/local/lib/python3.12/dist-packages/choreographer/cli/browser_exe/chrome-linux64/chrome _which.py:76
08:05:31.372734 DEBUG Local chrome not found at path: /usr/local/lib/python3.12/dist-packages/choreographer/cli/browser_exe/chrome-linux64/chrome. _which.py:81
08:05:31.373419 DEBUG Looking for a brave browser. chromium.py:81
08:05:31.373922 DEBUG Looking for browser, skipping local? False _which.py:64
08:05:31.374454 DEBUG Looking for at local chrome download path: /usr/local/lib/python3.12/dist-packages/choreographer/cli/browser_exe/chrome-linux64/chrome _which.py:76
08:05:31.374980 DEBUG Local chrome not found at path: /usr/local/lib/python3.12/dist-packages/choreographer/cli/browser_exe/chrome-linux64/chrome. _which.py:81
08:05:31.375640 DEBUG Looking for a vivaldi browser. chromium.py:81
08:05:31.376114 DEBUG Looking for browser, skipping local? False _which.py:64
08:05:31.376661 DEBUG Looking for at local chrome download path: /usr/local/lib/python3.12/dist-packages/choreographer/cli/browser_exe/chrome-linux64/chrome _which.py:76
08:05:31.377185 DEBUG Local chrome not found at path: /usr/local/lib/python3.12/dist-packages/choreographer/cli/browser_exe/chrome-linux64/chrome. _which.py:81
08:05:31.378113 ERROR Failed to generate ttft_timeline: single_run.py:71
Kaleido requires Google Chrome to be installed.
Either download and install Chrome yourself following Google's instructions for your operating system,
or install it from your terminal by running:
$ plotly_get_chrome
08:05:31.379260 DEBUG Skipping timeslices_ttft - required data not available single_run.py:60
08:05:31.379829 DEBUG Skipping timeslices_itl - required data not available single_run.py:60
08:05:31.380376 DEBUG Skipping gpu_utilization_and_throughput_over_time - required data not available single_run.py:60
08:05:31.381254 DEBUG Summary saved to artifacts/plots/summary.txt base.py:60
Saved 0 plots to: artifacts/plots
Logs: artifacts/plots/aiperf_plot.log