Skip to content

Commit cb128fb

Browse files
arjunsuresharjunsuresh
authored
Merge branch 'dev' into fix/gptj-add-tp-pp-size-variations
2 parents 01c6cba + b616fe2 commit cb128fb

File tree

36 files changed

+1048
-72
lines changed

36 files changed

+1048
-72
lines changed

‎.github/workflows/run_tests_on_modified_meta_with_secrets.yml‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ name: Test script on modified meta (with secret)
22

33
on:
44
pull_request_target:
5-
branches: [ "main", "dev" ]
5+
branches: [ "main" ]
66
paths:
77
- 'script/**meta.yaml'
88

@@ -16,7 +16,7 @@ jobs:
1616
- name: 'Checkout'
1717
uses: actions/checkout@v4
1818
with:
19-
fetch-depth: 2
19+
fetch-depth: 0
2020

2121
- name: Setup Python
2222
uses: actions/setup-python@v2
@@ -27,12 +27,15 @@ jobs:
2727
run: |
2828
pip install pyyaml
2929
30+
- name: Fetch PR head
31+
run: |
32+
git fetch origin pull/${{ github.event.pull_request.number }}/head:pr-head
33+
3034
- name: Get changed files
3135
id: modified-files
3236
run: |
33-
git remote add upstream ${{ github.event.pull_request.base.repo.clone_url }}
34-
git fetch upstream
35-
changed_files=$(git diff upstream/${{ github.event.pull_request.base.ref }} --name-only | paste -sd, -)
37+
git fetch origin ${{ github.event.pull_request.base.ref }}
38+
changed_files=$(git diff origin/${{ github.event.pull_request.base.ref }}...pr-head --name-only | paste -sd, -)
3639
echo "$changed_files" | python3 .github/scripts/list_modified_scripts_with_secrets.py
3740
3841
process_modified_files:

‎.github/workflows/test-mlc-based-submission-generation.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ jobs:
1414
fail-fast: false
1515
matrix:
1616
os: [ubuntu-latest, windows-latest, macos-latest]
17-
python-version: [ "3.12" ]
17+
python-version: [ "3.13" ]
1818
division: ["closed", "open", "closed-open"]
1919
category: ["datacenter", "edge"]
2020
case: ["closed", "closed-no-compliance", "closed-power", "closed-failed-power-logs", "case-1", "case-2", "case-3", "case-5", "case-6", "case-7", "case-8"]

‎.github/workflows/test-mlperf-inference-abtf-poc.yml‎

Lines changed: 2 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -16,25 +16,17 @@ jobs:
1616
fail-fast: false
1717
matrix:
1818
os: [ubuntu-24.04, ubuntu-latest, macos-latest, macos-15, windows-latest]
19-
python-version: [ "3.8", "3.12" ]
19+
python-version: [ "3.9", "3.14" ]
2020
backend: [ "pytorch" ]
2121
implementation: [ "python" ]
2222
docker: [ "", " --docker --docker_mlc_repo=${{ github.event.pull_request.head.repo.html_url }} --docker_mlc_repo_branch=${{ github.event.pull_request.head.ref }} --docker_dt" ]
2323
extra-args: [ "--adr.compiler.tags=gcc", "--env.MLC_MLPERF_LOADGEN_BUILD_FROM_SRC=off" ]
2424
exclude:
25-
- os: ubuntu-latest
26-
python-version: "3.8"
27-
- os: windows-latest
28-
python-version: "3.8"
2925
- os: windows-latest
3026
extra-args: "--adr.compiler.tags=gcc"
3127
- os: windows-latest
3228
docker: " --docker --docker_mlc_repo=${{ github.event.pull_request.head.repo.html_url }} --docker_mlc_repo_branch=${{ github.event.pull_request.head.ref }} --docker_dt"
3329
# windows docker image is not supported in MLC yet
34-
- os: macos-latest
35-
python-version: "3.8"
36-
- os: macos-15
37-
python-version: "3.8"
3830
- os: macos-latest
3931
docker: " --docker --docker_mlc_repo=${{ github.event.pull_request.head.repo.html_url }} --docker_mlc_repo_branch=${{ github.event.pull_request.head.ref }} --docker_dt"
4032
- os: macos-15
@@ -114,4 +106,4 @@ jobs:
114106
115107
- name: Test MLPerf Inference ABTF POC using ${{ matrix.backend }} on ${{ matrix.os }}
116108
run: |
117-
mlcr run-abtf,inference,_poc-demo --test_query_count=2 --adr.cocoeval.version_max=1.5.7 --adr.cocoeval.version_max_usable=1.5.7 --quiet ${{ matrix.extra-args }} ${{ matrix.docker }} -v
109+
mlcr run-abtf,inference,_poc-demo --test_query_count=2 --quiet ${{ matrix.extra-args }} ${{ matrix.docker }} -v

‎.github/workflows/test-mlperf-inference-resnet50.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ jobs:
1717
fail-fast: false
1818
matrix:
1919
os: [ubuntu-latest, windows-latest, macos-latest]
20-
python-version: [ "3.12" ]
20+
python-version: [ "3.13" ]
2121
backend: [ "onnxruntime", "tf" ]
2222
implementation: [ "python", "cpp" ]
2323
exclude:

‎.github/workflows/test-mlperf-inference-retinanet.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ jobs:
1616
fail-fast: false
1717
matrix:
1818
os: [ubuntu-latest, windows-latest, macos-latest]
19-
python-version: [ "3.12" ]
19+
python-version: [ "3.13" ]
2020
backend: [ "onnxruntime", "pytorch" ]
2121
implementation: [ "python", "cpp" ]
2222
compiler-string: [ "", "--adr.compiler.tags=aocc --env.MLC_AOCC_ACCEPT_EULA=yes" ]

‎.github/workflows/test-mlperf-inference-rgat.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ jobs:
1717
fail-fast: false
1818
matrix:
1919
os: [ubuntu-latest]
20-
python-version: [ "3.12" ]
20+
python-version: [ "3.14" ]
2121
backend: [ "pytorch" ]
2222
implementation: [ "python" ]
2323

‎automation/script/module.py‎

Lines changed: 48 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -3621,13 +3621,17 @@ def _run_deps(self, deps, clean_env_keys_deps, env, state, const, const_state, a
36213621
if d.get(key):
36223622
d[key] = {}
36233623

3624-
# print(f"ii = {ii}, d = {d}")
36253624
utils.merge_dicts(
36263625
{'dict1': ii, 'dict2': d, 'append_lists': True, 'append_unique': True})
36273626

36283627
r = self.action_object.access(ii)
36293628
if r['return'] > 0:
3630-
return r
3629+
if is_true(d.get('continue_on_error')):
3630+
# Warning printed by mlcflow
3631+
# logger.warning(f"Dependency with tags: {d['tags']} failed. Ignoring the failure as 'continue_on_error' is set for the dependency call")
3632+
pass
3633+
else:
3634+
return r
36313635

36323636
run_state['version_info'] = run_state_copy.get(
36333637
'version_info')
@@ -5466,53 +5470,55 @@ def prepare_and_run_script_with_postprocessing(i, postprocess="postprocess"):
54665470

54675471
rc = os.system(cmd)
54685472

5469-
if rc > 0 and not i.get('ignore_script_error', False):
5470-
# Check if print files when error
5471-
print_files = meta.get('print_files_if_script_error', [])
5472-
if len(print_files) > 0:
5473-
for pr in print_files:
5474-
if os.path.isfile(pr):
5475-
r = utils.load_txt(file_name=pr)
5476-
if r['return'] == 0:
5477-
logger.info(
5478-
"========================================================")
5479-
logger.info("Print file {}:".format(pr))
5480-
logger.info("")
5481-
logger.info(r['string'])
5482-
logger.info("")
5483-
5484-
# Check where to report errors and failures
5485-
repo_to_report = run_state.get(
5486-
'script_entry_repo_to_report_errors', '')
5487-
5488-
if repo_to_report == '':
5489-
script_repo_alias = run_state.get('script_repo_alias', '')
5490-
script_repo_git = run_state.get('script_repo_git', False)
5491-
5492-
if script_repo_git and script_repo_alias != '':
5493-
repo_to_report = 'https://github.com/' + \
5494-
script_repo_alias.replace('@', '/') + '/issues'
5495-
5496-
if repo_to_report == '':
5497-
repo_to_report = 'https://github.com/mlcommons/mlperf-automations/issues'
5498-
5499-
note = '''
5473+
if rc > 0:
5474+
if not is_true(i.get('ignore_script_error', False)):
5475+
# Check if print files when error
5476+
print_files = meta.get('print_files_if_script_error', [])
5477+
if len(print_files) > 0:
5478+
for pr in print_files:
5479+
if os.path.isfile(pr):
5480+
r = utils.load_txt(file_name=pr)
5481+
if r['return'] == 0:
5482+
logger.info(
5483+
"========================================================")
5484+
logger.info("Print file {}:".format(pr))
5485+
logger.info("")
5486+
logger.info(r['string'])
5487+
logger.info("")
5488+
5489+
# Check where to report errors and failures
5490+
repo_to_report = run_state.get(
5491+
'script_entry_repo_to_report_errors', '')
5492+
5493+
if repo_to_report == '':
5494+
script_repo_alias = run_state.get('script_repo_alias', '')
5495+
script_repo_git = run_state.get('script_repo_git', False)
5496+
5497+
if script_repo_git and script_repo_alias != '':
5498+
repo_to_report = 'https://github.com/' + \
5499+
script_repo_alias.replace('@', '/') + '/issues'
5500+
5501+
if repo_to_report == '':
5502+
repo_to_report = 'https://github.com/mlcommons/mlperf-automations/issues'
5503+
5504+
note = '''
55005505
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
55015506
Please file an issue at {} along with the full MLC command being run and the relevant
55025507
or full console log.
55035508
'''.format(repo_to_report)
55045509

5505-
rr = {
5506-
'return': 2,
5507-
'error': 'MLC script failed (name = {}, return code = {})\n\n{}'.format(
5508-
meta['alias'],
5509-
rc,
5510-
note)}
5510+
rr = {
5511+
'return': 2,
5512+
'error': f"""Native run script failed inside MLC script (name = {meta['alias']}, return code = {rc})\n\n{note}"""
5513+
}
55115514

5512-
if repro_prefix != '':
5513-
dump_repro(repro_prefix, rr, run_state)
5515+
if repro_prefix != '':
5516+
dump_repro(repro_prefix, rr, run_state)
55145517

5515-
return rr
5518+
return rr
5519+
else:
5520+
logger.warn(
5521+
f"""Native run script failed inside MLC script (name = {meta['alias']}, return code = {rc}. Ignoring as ignore_script_error is set.)\n""")
55165522

55175523
# Load updated state if exists
55185524
if tmp_file_run_state != '' and os.path.isfile(tmp_file_run_state):

‎script/app-mlperf-automotive-mlcommons-python/README.md‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# README for app-mlperf-automotive-mlcommons-python
2+
This README is automatically generated. Add custom content in [info.md](info.md). Please follow the [script execution document](https://docs.mlcommons.org/mlcflow/targets/script/execution-flow/) to understand more about the MLC script execution.
3+
4+
`mlcflow` stores all local data under `$HOME/MLC` by default. So, if there is space constraint on the home directory and you have more space on say `/mnt/$USER`, you can do
5+
```
6+
mkdir /mnt/$USER/MLC
7+
ln -s /mnt/$USER/MLC $HOME/MLC
8+
```
9+
You can also use the `ENV` variable `MLC_REPOS` to control this location but this will need a set after every system reboot.
10+
11+
## Setup
12+
13+
If you are not on a Python development environment please refer to the [official docs](https://docs.mlcommons.org/mlcflow/install/) for the installation.
14+
15+
```bash
16+
python3 -m venv mlcflow
17+
. mlcflow/bin/activate
18+
pip install mlcflow
19+
```
20+
21+
- Using a virtual environment is recommended (per `pip` best practices), but you may skip it or use `--break-system-packages` if needed.
22+
23+
### Pull mlperf-automations
24+
25+
Once `mlcflow` is installed:
26+
27+
```bash
28+
mlc pull repo mlcommons@mlperf-automations --pat=<Your Private Access Token>
29+
```
30+
- `--pat` or `--ssh` is only needed if the repo is PRIVATE
31+
- If `--pat` is avoided, you'll be asked to enter the password where you can enter your Private Access Token
32+
- `--ssh` option can be used instead of `--pat=<>` option if you prefer to use SSH for accessing the github repository.
33+
## Run Commands
34+
35+
```bash
36+
mlcr automotive,mlcommons,reference,run-mlperf-inference,object-detection,abtf-model,demo
37+
```
38+
39+
### Script Inputs
40+
41+
| Name | Description | Choices | Default |
42+
|------|-------------|---------|------|
43+
| `--device` | | | `` |
44+
| `--count` | | | `` |
45+
| `--docker` | | | `` |
46+
| `--hw_name` | | | `` |
47+
| `--imagenet_path` | | | `` |
48+
| `--max_batchsize` | | | `` |
49+
| `--mode` | | | `accuracy` |
50+
| `--num_threads` | | | `` |
51+
| `--threads` | Alias for num_threads | | `` |
52+
| `--dataset` | | | `` |
53+
| `--model` | | | `` |
54+
| `--output_dir` | | | `` |
55+
| `--power` | | | `` |
56+
| `--power_server` | | | `` |
57+
| `--ntp_server` | | | `` |
58+
| `--max_amps` | | | `` |
59+
| `--max_volts` | | | `` |
60+
| `--regenerate_files` | | | `` |
61+
| `--rerun` | | | `` |
62+
| `--scenario` | | | `Offline` |
63+
| `--test_query_count` | | | `10` |
64+
| `--clean` | | | `` |
65+
| `--dataset_args` | | | `` |
66+
| `--target_qps` | | | `` |
67+
| `--target_latency` | | | `` |
68+
| `--offline_target_qps` | | | `` |
69+
| `--server_target_qps` | | | `` |
70+
| `--constantstream_target_qps` | | | `` |
71+
| `--singlestream_target_latency` | | | `` |
72+
| `--multistream_target_latency` | | | `` |
73+
| `--output` | | | `` |
74+
### Generic Script Inputs
75+
76+
| Name | Description | Choices | Default |
77+
|------|-------------|---------|------|
78+
| `--input` | Input to the script passed using the env key `MLC_INPUT` | | `` |
79+
| `--output` | Output from the script passed using the env key `MLC_OUTPUT` | | `` |
80+
| `--outdirname` | The directory to store the script output | | `cache directory ($HOME/MLC/repos/local/cache/<>) if the script is cacheable or else the current directory` |
81+
| `--outbasename` | The output file/folder name | | `` |
82+
| `--name` | | | `` |
83+
| `--extra_cache_tags` | Extra cache tags to be added to the cached entry when the script results are saved | | `` |
84+
| `--skip_compile` | Skip compilation | | `False` |
85+
| `--skip_run` | Skip run | | `False` |
86+
| `--accept_license` | Accept the required license requirement to run the script | | `False` |
87+
| `--skip_system_deps` | Skip installing any system dependencies | | `False` |
88+
| `--git_ssh` | Use SSH for git repos | | `False` |
89+
| `--gh_token` | Github Token | | `` |
90+
| `--hf_token` | Huggingface Token | | `` |
91+
| `--verify_ssl` | Verify SSL | | `False` |
92+
## Variations
93+
94+
### Batch-size
95+
96+
- `batch_size.#` _(# can be substituted dynamically)_
97+
98+
### Device
99+
100+
- `cpu` (default)
101+
- `cuda`
102+
103+
### Framework
104+
105+
- `onnxruntime`
106+
- `pytorch` (default)
107+
108+
### Implementation
109+
110+
- `python` (default)
111+
112+
### Loadgen-scenario
113+
114+
- `constantstream`
115+
- `singlestream`
116+
117+
### Models
118+
119+
- `abtf-demo-model`
120+
- `abtf-poc-model` (default)
121+
- `bevformer`
122+
- `deeplabv3plus`
123+
- `ssd`
124+
125+
### Ungrouped
126+
127+
- `multistream`
128+
- `mvp_demo`
129+
- `offline`
130+
- `server`

‎script/app-mlperf-automotive-mlcommons-python/meta.yaml‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -414,8 +414,6 @@ variations:
414414
- tags: get,generic-python-lib,_pycocotools
415415
- tags: get,generic-python-lib,_package.torchmetrics
416416
- tags: get,generic-python-lib,_package.faster-coco-eval
417-
version_max: "1.5.7"
418-
version_max_usable: "1.5.7"
419417
names:
420418
- cocoeval
421419
- tags: get,dataset,raw,mlcommons-cognata
@@ -441,8 +439,6 @@ variations:
441439
- tags: get,generic-python-lib,_pycocotools
442440
- tags: get,generic-python-lib,_package.torchmetrics
443441
- tags: get,generic-python-lib,_package.faster-coco-eval
444-
version_max: "1.5.7"
445-
version_max_usable: "1.5.7"
446442
names:
447443
- cocoeval
448444
- tags: get,dataset,raw,mlcommons-cognata,_abtf-poc

‎script/get-aocc/customize.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,7 @@ def preprocess(i):
7373
def detect_version(i):
7474
logger = i['automation'].logger
7575

76-
r = i['automation'].parse_version({'match_text': r'CLANG:\s(?:AOCC_)?([\d.]+(?:-[\w#]+)*(?:-Build#\d+)?|Unknown-Revision)(?=[ )])',
76+
r = i['automation'].parse_version({'match_text': r'CLANG:\s(?:AOCC_)?([\d.]+(?:pre)?(?:-[\w#]+)*(?:-Build#\d+)?|Unknown-Revision)(?=[ )])',
7777
'group_number': 1,
7878
'env_key': 'MLC_AOCC_VERSION',
7979
'which_env': i['env']})

0 commit comments

Comments
 (0)