Skip to content

Commit c4270cb

Browse files
arjunsureshadminactions-user
authored
Changes for SCC24 (#18)
* Added doc files * Added doc files * Added publish file * Added mkdocs.yml file * Support generic result repositories * Fixes for SCC24 results * Fixes for SCC24 results * Added submission checker gh action * Use v4.1 version for scc24 * Results on system test * Restrict CLA check to MLC * Fix submission checker version * Added results updater GH action * Updated results summary * Updated results summary * Update update-results.yml * Updated results summary * Added results updater GH action * Updated results summary * Added results updater GH action * Updated results summary * Use incremental dbversion * Use incremental dbversion * Updated results summary * Delete summary_results.json * Updated results summary * Update update-results.yml * Updated results summary * Update publish.yml * Updated results summary * Update update-results.yml * Updated results summary * Results from self hosted Github actions - NVIDIARTX4090 * Updated results summary * Results from self hosted Github actions - NVIDIARTX4090 * Updated results summary * Results from self hosted Github actions - NVIDIARTX4090 * Updated results summary --------- Co-authored-by: admin <[email protected]> Co-authored-by: GitHub Action <[email protected]>
1 parent 6678c20 commit c4270cb

File tree

105 files changed

+5751
-2
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

105 files changed

+5751
-2
lines changed

‎.github/workflows/cla.yml

+1
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ on:
88

99
jobs:
1010
cla-check:
11+
if: github.repository_owner == 'mlcommons'
1112
runs-on: ubuntu-latest
1213
steps:
1314
- name: "MLCommons CLA bot check"

‎.github/workflows/publish.yml

+31
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
name: Publish visualization site
2+
3+
4+
on:
5+
release:
6+
types: [published]
7+
push:
8+
branches:
9+
- mlperf-inference-results-scc24
10+
- docs
11+
12+
jobs:
13+
14+
publish:
15+
name: Publish the site
16+
runs-on: ubuntu-latest
17+
18+
steps:
19+
- name: Checkout repository normally
20+
uses: actions/checkout@v3
21+
22+
- name: Set up Python
23+
uses: actions/setup-python@v4
24+
with:
25+
python-version: "3.11"
26+
27+
- name: Init docs site
28+
run: bash run.sh
29+
30+
- name: Run Mkdocs deploy
31+
run: mkdocs gh-deploy --force

‎.github/workflows/run-mlperf-inference-submission-checker.yml

+31
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
2+
3+
name: MLPerf inference submission checker
4+
5+
on:
6+
pull_request:
7+
branches: [ "main", "mlperf-inference-results-scc24" ]
8+
9+
jobs:
10+
build:
11+
12+
runs-on: ubuntu-latest
13+
env:
14+
CM_INDEX: "on"
15+
strategy:
16+
fail-fast: false
17+
matrix:
18+
python-version: [ "3.10" ]
19+
20+
steps:
21+
- uses: actions/checkout@v3
22+
- name: Set up Python ${{ matrix.python-version }}
23+
uses: actions/setup-python@v3
24+
with:
25+
python-version: ${{ matrix.python-version }}
26+
- name: Install dependencies
27+
run: |
28+
python3 -m pip install cm4mlops
29+
- name: Run MLPerf Inference Submission Checker
30+
run: |
31+
cm run script --tags=run,mlperf,inference,submission,checker,_short-run --adr.submission-checker-src.tags=_repo.https://github.com/gateoverflow/inference --src_version=v4.1 --quiet --extra_args=" --skip-extra-files-in-root-check" --submission_dir=./

‎.github/workflows/update-results.yml

+51
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
2+
3+
name: MLPerf inference results updater
4+
5+
6+
on:
7+
push:
8+
branches: [ "main", "mlperf-inference-results-scc24" ]
9+
10+
jobs:
11+
build:
12+
13+
runs-on: ubuntu-latest
14+
env:
15+
CM_INDEX: "on"
16+
strategy:
17+
fail-fast: false
18+
matrix:
19+
python-version: [ "3.10" ]
20+
21+
steps:
22+
- uses: actions/checkout@v3
23+
- name: Set up Python ${{ matrix.python-version }}
24+
uses: actions/setup-python@v3
25+
with:
26+
python-version: ${{ matrix.python-version }}
27+
28+
- name: Install dependencies
29+
run: |
30+
python3 -m pip install cm4mlops
31+
32+
- name: Run MLPerf Inference Submission Checker and generate results summary
33+
run: |
34+
cm run script --tags=run,mlperf,inference,submission,checker,_short-run --src_version=v4.1 --adr.submission-checker-src.tags=_repo.https://github.com/gateoverflow/inference,_branch.improve_result_generation --quiet --extra_args=" --skip-extra-files-in-root-check" --submission_dir=./ > >(tee -a out.txt) 2> >(tee -a checker_log.txt >&2)
35+
cmr "convert from-csv to-md" --csv_file=summary.csv --md_file=README.md
36+
37+
USER="GitHub Action"
38+
[email protected]
39+
40+
git config --global user.name "$USER"
41+
git config --global user.email "$EMAIL"
42+
git remote set-url origin https://x-access-token:${{ secrets.GITHUB_TOKEN_TOKEN }}@github.com/${{ github.repository }}
43+
git add summary*
44+
echo -e 'Please download [summary.xlsx](summary.xlsx) to view the most recent results. \n ```' > temp
45+
tail -n 16 checker_log.txt >> temp
46+
echo -e '\n```\n' >> temp
47+
cat temp | cat - README.md > temp1
48+
head -n 100 temp1 > README.md
49+
git add README.md
50+
51+
git diff-index --quiet HEAD || (git commit -am "Updated results summary" && git push origin)

‎README.md

+29
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
Please download [summary.xlsx](summary.xlsx) to view the most recent results.
2+
```
3+
[2024-10-01 12:37:20,009 submission_checker1.py:2936 INFO] Results=6, NoResults=0, Power Results=0
4+
[2024-10-01 12:37:20,009 submission_checker1.py:2943 INFO] ---
5+
[2024-10-01 12:37:20,009 submission_checker1.py:2944 INFO] Closed Results=0, Closed Power Results=0
6+
7+
[2024-10-01 12:37:20,009 submission_checker1.py:2949 INFO] Open Results=6, Open Power Results=0
8+
9+
[2024-10-01 12:37:20,009 submission_checker1.py:2954 INFO] Network Results=0, Network Power Results=0
10+
11+
[2024-10-01 12:37:20,009 submission_checker1.py:2959 INFO] ---
12+
[2024-10-01 12:37:20,009 submission_checker1.py:2961 INFO] Systems=5, Power Systems=0
13+
[2024-10-01 12:37:20,009 submission_checker1.py:2962 INFO] Closed Systems=0, Closed Power Systems=0
14+
[2024-10-01 12:37:20,009 submission_checker1.py:2967 INFO] Open Systems=5, Open Power Systems=0
15+
[2024-10-01 12:37:20,009 submission_checker1.py:2972 INFO] Network Systems=0, Network Power Systems=0
16+
[2024-10-01 12:37:20,009 submission_checker1.py:2977 INFO] ---
17+
[2024-10-01 12:37:20,009 submission_checker1.py:2982 INFO] SUMMARY: submission looks OK
18+
INFO:root: ! call "postprocess" from /home/runner/CM/repos/mlcommons@cm4mlops/script/run-mlperf-inference-submission-checker/customize.py
19+
20+
```
21+
22+
| | Organization | Availability | Division | SystemType | SystemName | Platform | Model | MlperfModel | Scenario | Result | Accuracy | number_of_nodes | host_processor_model_name | host_processors_per_node | host_processor_core_count | accelerator_model_name | accelerators_per_node | Location | framework | operating_system | notes | compliance | errors | version | inferred | has_power | Units | weight_data_types |
23+
|---:|:---------------|:---------------|:-----------|:-------------|:-------------|:-------------------------------------------------------|:--------------------|:--------------------|:-----------|----------:|:--------------------------------------------------------------|------------------:|:----------------------------|---------------------------:|----------------------------:|:-------------------------|------------------------:|:----------------------------------------------------------------------------------------------------------|:---------------|:------------------------------------------------|:----------------------------------|-------------:|---------:|:----------|-----------:|:------------|:----------|:--------------------|
24+
| 0 | MLCommons | available | open | datacenter | 48ed6105bd85 | 48ed6105bd85-nvidia-gpu-TensorRT-scc24-main | stable-diffusion-xl | stable-diffusion-xl | Offline | 1.13292 | CLIP_SCORE: 15.586050063371658 FID_SCORE: 236.8087101317688 | 1 | Intel(R) Xeon(R) w7-2495X | 1 | 24 | NVIDIA GeForce RTX 4090 | 1 | open/MLCommons/results/48ed6105bd85-nvidia-gpu-TensorRT-scc24-main/stable-diffusion-xl/offline | TensorRT | Ubuntu 20.04 (linux-6.2.0-39-generic-glibc2.31) | Automated by MLCommons CM v2.3.6. | 1 | 0 | v4.1 | 0 | False | Samples/s | int8 |
25+
| 1 | MLCommons | available | open | datacenter | 48ed6105bd85 | 48ed6105bd85-nvidia-gpu-TensorRT-scc24-base | stable-diffusion-xl | stable-diffusion-xl | Offline | 1.13598 | CLIP_SCORE: 15.586050063371658 FID_SCORE: 236.8087101317688 | 1 | Intel(R) Xeon(R) w7-2495X | 1 | 24 | NVIDIA GeForce RTX 4090 | 1 | open/MLCommons/results/48ed6105bd85-nvidia-gpu-TensorRT-scc24-base/stable-diffusion-xl/offline | TensorRT | Ubuntu 20.04 (linux-6.2.0-39-generic-glibc2.31) | Automated by MLCommons CM v2.3.6. | 1 | 0 | v4.1 | 0 | False | Samples/s | int8 |
26+
| 2 | MLCommons | available | open | edge | gh_action | gh_action-reference-gpu-pytorch_v2.4.1-default_config | gptj-99 | gptj-99 | Offline | 52.9478 | nan | 1 | Intel(R) Xeon(R) w7-2495X | 1 | 24 | NVIDIA GeForce RTX 4090 | 1 | open/MLCommons/results/gh_action-reference-gpu-pytorch_v2.4.1-default_config/gptj-99/offline | pytorch v2.4.1 | Ubuntu 22.04 (linux-6.2.0-39-generic-glibc2.35) | Automated by MLCommons CM v2.3.4. | 1 | 0 | v4.1 | 0 | False | Tokens/s | fp32 |
27+
| 3 | MLCommons | available | open | edge | gh_action | gh_action-reference-gpu-pytorch_v2.4.1-default_config | stable-diffusion-xl | stable-diffusion-xl | Offline | 0.345721 | CLIP_SCORE: 15.18544016778469 FID_SCORE: 235.69504308101006 | 1 | Intel(R) Xeon(R) w7-2495X | 1 | 24 | NVIDIA GeForce RTX 4090 | 1 | open/MLCommons/results/gh_action-reference-gpu-pytorch_v2.4.1-default_config/stable-diffusion-xl/offline | pytorch v2.4.1 | Ubuntu 22.04 (linux-6.2.0-39-generic-glibc2.35) | Automated by MLCommons CM v2.3.4. | 1 | 0 | v4.1 | 0 | False | Samples/s | fp32 |
28+
| 4 | MLCommons | available | open | datacenter | 48ed6105bd85 | 48ed6105bd85-reference-gpu-pytorch_v2.1.0a0-scc24-base | stable-diffusion-xl | stable-diffusion-xl | Offline | 0.373636 | CLIP_SCORE: 15.236237794160843 FID_SCORE: 238.78369342212613 | 1 | Intel(R) Xeon(R) w7-2495X | 1 | 24 | NVIDIA GeForce RTX 4090 | 1 | open/MLCommons/results/48ed6105bd85-reference-gpu-pytorch_v2.1.0a0-scc24-base/stable-diffusion-xl/offline | TensorRT | Ubuntu 20.04 (linux-6.2.0-39-generic-glibc2.31) | Automated by MLCommons CM v2.3.6. | 1 | 0 | v4.1 | 0 | False | Samples/s | fp32 |
29+
| 5 | MLCommons | available | open | datacenter | f9ac88850adc | f9ac88850adc-reference-gpu-pytorch_v2.4.1-scc24-base | stable-diffusion-xl | stable-diffusion-xl | Offline | 0.376944 | CLIP_SCORE: 15.18544016778469 FID_SCORE: 235.69504308101006 | 1 | Intel(R) Xeon(R) w7-2495X | 1 | 24 | NVIDIA GeForce RTX 4090 | 1 | open/MLCommons/results/f9ac88850adc-reference-gpu-pytorch_v2.4.1-scc24-base/stable-diffusion-xl/offline | pytorch v2.4.1 | Ubuntu 22.04 (linux-6.2.0-39-generic-glibc2.35) | Automated by MLCommons CM v2.3.9. | 1 | 0 | v4.1 | 0 | False | Samples/s | fp32 |

‎dbversion

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
100

‎docinit.sh

+43
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
#!/bin/bash
2+
3+
if [ ! -e docs ]; then
4+
git clone https://github.com/GATEOverflow/inference_results_visualization_template.git docs
5+
test $? -eq 0 || exit $?
6+
fi
7+
8+
python3 -m pip install -r docs/requirements.txt
9+
10+
if [ ! -e overrides ]; then
11+
cp -r docs/overrides overrides
12+
test $? -eq 0 || exit $?
13+
fi
14+
15+
repo_owner=${INFERENCE_RESULTS_REPO_OWNER:-mlcommons}
16+
repo_branch=${INFERENCE_RESULTS_REPO_BRANCH:-main}
17+
repo_name=${INFERENCE_RESULTS_REPO_NAME:-inference_results_${INFERENCE_RESULTS_VERSION}}
18+
ver_num=$(cat dbversion)
19+
let ver_num++
20+
echo "ver_num=$ver_num" > dbversion
21+
if [ ! -e docs/javascripts/config.js ]; then
22+
if [ -n "${INFERENCE_RESULTS_VERSION}" ]; then
23+
echo "const results_version=\"${INFERENCE_RESULTS_VERSION}\";" > docs/javascripts/config.js;
24+
echo "var repo_owner=\"${repo_owner}\";" >> docs/javascripts/config.js;
25+
echo "var repo_branch=\"${repo_branch}\";" >> docs/javascripts/config.js;
26+
echo "var repo_name=\"${repo_name}\";" >> docs/javascripts/config.js;
27+
#ver_num=`echo ${INFERENCE_RESULTS_VERSION} | tr -cd '0-9'`
28+
echo "const dbVersion =\"${ver_num}\";" >> docs/javascripts/config.js;
29+
else
30+
echo "Please export INFERENCE_RESULTS_VERSION=v4.1 or the corresponding version";
31+
exit 1
32+
fi
33+
fi
34+
35+
if [ ! -e docs/thirdparty/tablesorter ]; then
36+
cd docs/thirdparty && git clone https://github.com/Mottie/tablesorter.git && cd -
37+
test $? -eq 0 || exit $?
38+
fi
39+
40+
python3 process.py
41+
test $? -eq 0 || exit $?
42+
python3 process_results_table.py
43+
test $? -eq 0 || exit $?

‎mkdocs.yml

+43
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
site_name: MLPerf Inference Results Comparison
2+
repo_url: https://github.com/mlcommons/inference_results_v4.0
3+
theme:
4+
name: material
5+
logo: img/logo_v2.svg
6+
favicon: img/logo_v2.svg
7+
palette:
8+
primary: deep purple
9+
accent: yellow
10+
custom_dir: overrides
11+
features:
12+
- content.tabs.link
13+
- content.code.copy
14+
- navigation.expand
15+
- navigation.sections
16+
- navigation.indexes
17+
- navigation.instant
18+
- navigation.tabs
19+
- navigation.tabs.sticky
20+
- navigation.top
21+
- toc.follow
22+
nav:
23+
- Results: index.md
24+
- Compare: compare/index.md
25+
- Top Results: top_results/index.md
26+
27+
markdown_extensions:
28+
- pymdownx.tasklist:
29+
custom_checkbox: true
30+
- pymdownx.details
31+
- admonition
32+
- attr_list
33+
- def_list
34+
- footnotes
35+
- pymdownx.superfences:
36+
custom_fences:
37+
- name: mermaid
38+
class: mermaid
39+
format: !!python/name:pymdownx.superfences.fence_code_format
40+
- pymdownx.tabbed:
41+
alternate_style: true
42+
plugins:
43+
- macros

‎open/MLCommons/code/gptj-99/README.md

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
TBD

‎open/MLCommons/measurements/48ed6105bd85-reference-gpu-pytorch_v2.1.0a0-scc24-base/README.md

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
| Model | Scenario | Accuracy | Throughput | Latency (in ms) |
2+
|---------------------|------------|-----------------------|--------------|-------------------|
3+
| stable-diffusion-xl | offline | (15.23624, 238.78369) | 0.374 | - |

‎open/MLCommons/measurements/48ed6105bd85-reference-gpu-pytorch_v2.1.0a0-scc24-base/stable-diffusion-xl/offline/48ed6105bd85-reference-gpu-pytorch_v2.1.0a0-scc24-base.json

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{
2+
"starting_weights_filename": "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0",
3+
"retraining": "no",
4+
"input_data_types": "fp32",
5+
"weight_data_types": "fp32",
6+
"weight_transformations": "no"
7+
}

‎open/MLCommons/measurements/48ed6105bd85-reference-gpu-pytorch_v2.1.0a0-scc24-base/stable-diffusion-xl/offline/README.md

+57
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
This experiment is generated using the [MLCommons Collective Mind automation framework (CM)](https://github.com/mlcommons/cm4mlops).
2+
3+
*Check [CM MLPerf docs](https://docs.mlcommons.org/inference) for more details.*
4+
5+
## Host platform
6+
7+
* OS version: Linux-6.2.0-39-generic-x86_64-with-glibc2.29
8+
* CPU version: x86_64
9+
* Python version: 3.8.10 (default, Sep 11 2024, 16:02:53)
10+
[GCC 9.4.0]
11+
* MLCommons CM version: 2.3.6
12+
13+
## CM Run Command
14+
15+
See [CM installation guide](https://docs.mlcommons.org/inference/install/).
16+
17+
```bash
18+
pip install -U cmind
19+
20+
cm rm cache -f
21+
22+
cm pull repo mlcommons@cm4mlops --checkout=735b581a345d5a04f8cd7195f6fbff3a37c538d8
23+
24+
cm run script \
25+
--tags=run-mlperf,inference,_r4.1-dev,_short,_scc24-base \
26+
--model=sdxl \
27+
--implementation=reference \
28+
--framework=pytorch \
29+
--category=datacenter \
30+
--scenario=Offline \
31+
--execution_mode=test \
32+
--device=cuda \
33+
--quiet \
34+
--precision=float16
35+
```
36+
*Note that if you want to use the [latest automation recipes](https://docs.mlcommons.org/inference) for MLPerf (CM scripts),
37+
you should simply reload mlcommons@cm4mlops without checkout and clean CM cache as follows:*
38+
39+
```bash
40+
cm rm repo mlcommons@cm4mlops
41+
cm pull repo mlcommons@cm4mlops
42+
cm rm cache -f
43+
44+
```
45+
46+
## Results
47+
48+
Platform: 48ed6105bd85-reference-gpu-pytorch_v2.1.0a0-scc24-base
49+
50+
Model Precision: fp32
51+
52+
### Accuracy Results
53+
`CLIP_SCORE`: `15.23624`, Required accuracy for closed division `>= 31.68632` and `<= 31.81332`
54+
`FID_SCORE`: `238.78369`, Required accuracy for closed division `>= 23.01086` and `<= 23.95008`
55+
56+
### Performance Results
57+
`Samples per second`: `0.373636`

‎open/MLCommons/measurements/48ed6105bd85-reference-gpu-pytorch_v2.1.0a0-scc24-base/stable-diffusion-xl/offline/accuracy_console.out

Whitespace-only changes.

0 commit comments

Comments
 (0)