HLM Correlation and Accuracy

[oluwatimilehin]

Hi team, thank you for making this tool open-source. I have a few questions about the HLM in Polaris:

• Why does it report the geometric mean of the ratio of the HLM to the silicon target score by default instead of the HLM to the actual silicon score? It seems like the latter would be a better measure of its accuracy, but I suspect I am missing something.

• Why does the correlation tool report a geometric mean rather than a mean absolute percentage error? The mean absolute percentage error for the number of cycles predicted seems more common for assessing simulation performance, so I'm curious about the choice here.

Thanks!

[oluwatimilehin]

I made some changes to the code and came up with these numbers after running:

python tools/run_ttsi_corr.py \
    --tag 03nov25 \
    --workloads-config config/ttsi_correlation_workloads.yaml \
    --arch-config config/tt_wh.yaml \
    --output-dir __CORRELATION_OUTPUT

Workload	Batch	Precision	Si-Score	HLM-Score	Ratio HLM/Si	HLM Cycles	Si Cycles	Abs % Error
BERT-Large	8	bf8	270.0	476.06	1.76	16,804,745	29,629,630	43.28%
ResNet-50	16	bf8	4700.0	6121.27	1.30	2,613,837	3,404,256	23.22%
Sentence-Bert	8	bf8	433.0	1251.53	2.89	6,392,177	18,475,751	65.40%
UNet (VGG19)	1	bf8	198.0	286.62	1.45	3,488,974	5,050,506	30.92%
ViT-base	8	bf8	1370.0	969.40	0.71	8,252,494	5,839,417	41.32%
YOLOv8s	1	bf8	215.0	679.48	3.16	1,471,722	4,651,163	68.36%

It looks like Polaris has a mean absolute error of ~45% compared to running on an actual Wormhole n150 device for these workloads. Do these numbers look right?

If you're curious, you can find the changes I made here: https://github.com/oluwatimilehin/polaris/pull/1/files