Skip to content

Auto GPU detection + Updated run args generation for 5.0 #433

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 22 commits into
base: dev
Choose a base branch
from
Draft

Auto GPU detection + Updated run args generation for 5.0 #433

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
4baae86
Update batch size to state
anandhu-eng May 19, 2025
7cfca34
batch size - priority to state dict
anandhu-eng May 19, 2025
06d81d6
[Automated Commit] Format Codebase [skip ci]
github-actions[bot] May 19, 2025
61e8eaa
update run command arguments for v5.0
anandhu-eng May 20, 2025
24ff3a0
[Automated Commit] Format Codebase [skip ci]
github-actions[bot] May 20, 2025
618996a
extract nvidia gpu name automatically
anandhu-eng May 20, 2025
537e647
[Automated Commit] Format Codebase [skip ci]
github-actions[bot] May 20, 2025
8b2ccfc
Update customize.py
anandhu-eng May 20, 2025
796c363
update batch size extraction logic
anandhu-eng May 20, 2025
cdca719
[Automated Commit] Format Codebase [skip ci]
github-actions[bot] May 20, 2025
90ec0df
update packages
anandhu-eng May 20, 2025
c279990
add version for nvidia harness
anandhu-eng May 20, 2025
f707c9f
Update meta.yaml
anandhu-eng May 20, 2025
07046e7
updated package version
anandhu-eng May 20, 2025
79de08b
Merge branch 'dev' into dev
arjunsuresh May 22, 2025
ec27be0
modify batch size format based on inference version
anandhu-eng May 24, 2025
572d538
revert gpu and dla info extraction from state
anandhu-eng May 25, 2025
7605329
[Automated Commit] Format Codebase [skip ci]
github-actions[bot] May 25, 2025
f41b152
Update meta.yaml
anandhu-eng May 25, 2025
d500b5f
Merge branch 'dev' into dev
arjunsuresh May 26, 2025
0b0c1fd
Merge branch 'dev' into dev
arjunsuresh May 28, 2025
15ddcd6
Merge branch 'dev' into dev
arjunsuresh Jun 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions script/app-mlperf-inference-nvidia/customize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ def preprocess(i):
if os_info['platform'] == 'windows':
return {'return': 1, 'error': 'Windows is not supported in this script yet'}
env = i['env']
state = i['state']

if is_true(env.get('MLC_RUN_STATE_DOCKER', '')):
return {'return': 0}
Expand Down Expand Up @@ -518,12 +519,22 @@ def preprocess(i):
if dla_inference_streams:
run_config += f" --dla_inference_streams={dla_inference_streams}"

gpu_batch_size = env.get('MLC_MLPERF_NVIDIA_HARNESS_GPU_BATCH_SIZE')
gpu_batch_size = state.get('batch_size', env.get(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Guard this for v5.0?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed in commit 796c363.

For 5.0, the correct format is --dla/gpu_batch_size=model:batch_size. Since users could provide batch size manually, I have placed MLC_MLPERF_NVIDIA_HARNESS_GPU_BATCH_SIZE as alternative source to state.

'MLC_MLPERF_NVIDIA_HARNESS_GPU_BATCH_SIZE'))
if gpu_batch_size:
if env.get('MLC_MLPERF_INFERENCE_VERSION', '') == "5.0":
gpu_batch_size = ",".join(
f"{key}:{value}" for key,
value in gpu_batch_size.items())
run_config += f" --gpu_batch_size={gpu_batch_size}"

dla_batch_size = env.get('MLC_MLPERF_NVIDIA_HARNESS_DLA_BATCH_SIZE')
dla_batch_size = state.get('dla_batch_size', env.get(
'MLC_MLPERF_NVIDIA_HARNESS_DLA_BATCH_SIZE'))
if dla_batch_size:
if env.get('MLC_MLPERF_INFERENCE_VERSION', '') == "5.0":
dla_batch_size = ",".join(
f"{key}:{value}" for key,
value in dla_batch_size.items())
run_config += f" --dla_batch_size={dla_batch_size}"

input_format = env.get('MLC_MLPERF_NVIDIA_HARNESS_INPUT_FORMAT')
Expand Down
71 changes: 71 additions & 0 deletions script/app-mlperf-inference-nvidia/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1699,13 +1699,23 @@ variations:
env:
MLC_MLPERF_NVIDIA_HARNESS_USE_GRAPHS: 'True'
MLC_MLPERF_LOADGEN_OFFLINE_TARGET_QPS: 0.6

l4,sdxl,offline,run_harness,batch_size.1:
state:
batch_size:
sdxl: 1

l4,sdxl,offline,run_harness,num-gpu.8:
default_variations:
batch-size: batch_size.1
env:
MLC_MLPERF_NVIDIA_HARNESS_USE_GRAPHS: 'True'
MLC_MLPERF_LOADGEN_OFFLINE_TARGET_QPS: 4.8

l4,sdxl,offline,run_harness,num-gpu.8,batch_size.1:
state:
batch_size:
sdxl: 1

l4,sdxl,server,run_harness,num-gpu.1:
default_variations:
Expand All @@ -1714,6 +1724,11 @@ variations:
MLC_MLPERF_NVIDIA_HARNESS_USE_GRAPHS: 'True'
MLC_MLPERF_LOADGEN_SERVER_TARGET_QPS: 0.55
MLC_MLPERF_NVIDIA_HARNESS_SDXL_SERVER_BATCHER_TIME_LIMIT: 0

l4,sdxl,server,run_harness,num-gpu.1,batch_size.1:
state:
batch_size:
sdxl: 1

l4,sdxl,server,run_harness,num-gpu.8:
default_variations:
Expand All @@ -1723,6 +1738,11 @@ variations:
MLC_MLPERF_LOADGEN_SERVER_TARGET_QPS: 5.05
MLC_MLPERF_NVIDIA_HARNESS_SDXL_SERVER_BATCHER_TIME_LIMIT: 0

l4,sdxl,server,run_harness,num-gpu.8,batch_size.1:
state:
batch_size:
sdxl: 1

l4,resnet50:
default_env:
MLC_MLPERF_LOADGEN_OFFLINE_TARGET_QPS: 10500
Expand All @@ -1738,6 +1758,11 @@ variations:
MLC_MLPERF_NVIDIA_HARNESS_GPU_INFERENCE_STREAMS: "1"
MLC_MLPERF_NVIDIA_HARNESS_USE_GRAPHS: 'True'

l4,resnet50,offline,run_harness,batch_size.32:
state:
batch_size:
resnet50: 32

l4,resnet50,server,run_harness:
default_variations:
batch-size: batch_size.16
Expand All @@ -1749,10 +1774,20 @@ variations:
MLC_MLPERF_NVIDIA_HARNESS_DEQUE_TIMEOUT_USEC: 2000
MLC_MLPERF_NVIDIA_HARNESS_USE_CUDA_THREAD_PER_DEVICE: 'True'

l4,resnet50,server,run_harness,batch_size.16:
state:
batch_size:
resnet50: 16

l4,retinanet,offline,run_harness:
default_variations:
batch-size: batch_size.2

l4,retinanet,offline,run_harness,batch_size.2:
state:
batch_size:
retinanet: 2

l4,retinanet,server,run_harness:
default_variations:
batch-size: batch_size.2
Expand All @@ -1763,10 +1798,20 @@ variations:
MLC_MLPERF_NVIDIA_HARNESS_DEQUE_TIMEOUT_USEC: 30000
MLC_MLPERF_NVIDIA_HARNESS_WORKSPACE_SIZE: 20000000000

l4,retinanet,server,run_harness,batch_size.2:
state:
batch_size:
retinanet: 2

l4,bert_,offline,run_harness:
default_variations:
batch-size: batch_size.16

l4,bert_,offline,run_harness,batch_size.16:
state:
batch_size:
bert: 16

l4,bert_,server,run_harness:
default_variations:
batch-size: batch_size.16
Expand All @@ -1776,14 +1821,29 @@ variations:
MLC_MLPERF_NVIDIA_HARNESS_SOFT_DROP: "1.0"
MLC_MLPERF_NVIDIA_HARNESS_USE_SMALL_TILE_GEMM_PLUGIN: "True"

l4,bert_,server,run_harness,batch_size.16:
state:
batch_size:
bert: 16

l4,3d-unet_,offline,run_harness:
default_variations:
batch-size: batch_size.1

l4,3d-unet_,offline,run_harness,batch_size.1:
state:
batch_size:
3d-unet: 1

l4,rnnt,offline,run_harness:
default_variations:
batch-size: batch_size.512

l4,rnnt,offline,run_harness,batch_size.512:
state:
batch_size:
rnnt: 512

l4,rnnt,server,run_harness:
default_variations:
batch-size: batch_size.512
Expand All @@ -1792,9 +1852,20 @@ variations:
MLC_MLPERF_NVIDIA_HARNESS_AUDIO_BUFFER_NUM_LINES: "1024"
MLC_MLPERF_NVIDIA_HARNESS_NUM_WARMUPS: "1024"

l4,rnnt,server,run_harness,batch_size.512:
state:
batch_size:
rnnt: 512

l4,dlrm_,offline,run_harness:
default_variations:
batch-size: batch_size.1400

l4,dlrm_,offline,run_harness,batch_size.1400:
state:
batch_size:
dlrm: 1400

t4:
group: gpu-name
env:
Expand Down
9 changes: 9 additions & 0 deletions script/app-mlperf-inference/customize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,16 @@ def preprocess(i):
env = i['env']
state = i['state']

logger = i['automation'].logger

if env.get('MLC_MLPERF_IMPLEMENTATION', '') == 'nvidia':
if "nvidia" in env.get('MLC_CUDA_DEVICE_PROP_GPU_NAME', '').lower() and env.get(
'MLC_NVIDIA_GPU_NAME', '') == '':
# extract the Nvidia GPU model name automatically
env['MLC_NVIDIA_GPU_NAME'] = env['MLC_CUDA_DEVICE_PROP_GPU_NAME'].lower(
).split()[-1].strip()
logger.info(
f"Extracted Nvidia GPU name: {env['MLC_NVIDIA_GPU_NAME']}")
if env.get('MLC_NVIDIA_GPU_NAME', '') in [
"rtx_4090", "a100", "t4", "l4", "orin", "custom"]:
env['MLC_NVIDIA_HARNESS_GPU_VARIATION'] = "_" + \
Expand Down
Loading