Skip to content

Investigate accuracy issues in working models #739

Open
9 of 30 issues completed
Open
Bug
9 of 30 issues completed
@ayerofieiev-tt

Description

@ayerofieiev-tt

Among the 66 models running end-to-end in TT-NN, the following models exhibit accuracy deviations:

Models with Accuracy Issues:

  1. Bloom: 43.3
  2. albert/albert-base-v2: 69.04
  3. albert/albert-large-v2: 23.63
  4. albert/albert-xlarge-v2: 53.42
  5. mobilenetv1_100.ra4_e3600_r224_in1k: -6.46
  6. regnet_x_16gf: 1.01
  7. regnet_x_32gf: 3.94

Models with Accuracy Marked as N/A:

  1. Unet-brain: N/A
  2. ssd300_vgg16: N/A

Action Required:

  1. Investigate and diagnose the causes of the accuracy deviations.
  2. If the issue is in TT-NN operation, determine what operation on what input causes an issue. Fire a ticket to tt-metal, link here
  3. Validate expected accuracy baselines for each model.
  4. Assess Unet-brain to determine if accuracy reporting is feasible.

Sub-issues

Metadata

Metadata

Assignees

Labels

Type

Projects

Status

In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions