HPU support

[IlyasMoutawwakil]

What does this PR do?

This PR introduces upstream support for HPU torch device/backend:

• HPU is the device name for Intel Gaudi Accelerators, a very powerful, energy efficient ASIC for AI workloads.
• Gaudi1 is available on AWS since 2021, Gaudi2/Gaudi3 on Intel Dev Cloud and soon on IBM Cloud.
• The documentation of the torch device is available here.

This is part of three PRs:

• Safetensors support hpu:0 safetensors#578
• Accelerate HPU support #3378 (this one)
• Transformers HPU support transformers#36424

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[muellerzr]

Thanks a bunch! From the accelerate side this looks fine; do you want to do big model inference while you're at it? (if not all good).

I assume accelerate test etc went well? 🤗

[IlyasMoutawwakil]

@muellerzr I forgot to mark it as a draft 😅.
I'm still debugging some issues, both from accelerate and optimum-habana side, should be ready by next week.

[SunMarc]

Nice thanks for the PR ! +1 for big model inference also for a folow-up PR for example

[muellerzr]

No worries @IlyasMoutawwakil ! Just let us know when you're all set to go 🫡

[IlyasMoutawwakil]

All tests that don't require fp16/fp8 are passing on gaudi1.
I'm slowly reenabling some tests that are now passing on gaudi2 with Synapse 1.20+Pytorch 2.6, like pippy.
I added an explanation of every test skipped on HPU, there are mainly three reasons for skipping:

• unsupported hpu device indexing (hpu:1).
• unsupported empty_cache() op.
• missing bnb support.

one last test that fails with no explanation is test_multi_device_merge_fsdp_weights, for now all I'm seeing is # Synapse detected a device critical error that requires a restart, I can investigate it later outside of the scope of this PR.

[IlyasMoutawwakil]

will be removed and only schedule will stay before merge.

[IlyasMoutawwakil]

it seems that is_fp8_available() only checks for libraries availability and not for device capability.
check_fp8_capability is used for that but that's confusing no ? because is_fp16/bf16 do check for device capability of using those dtypes.

[IlyasMoutawwakil]

I didn't change the behavior of is_fp8_available() to not break bc but it would make sense to have a single source of truth.

[muellerzr]

I can see the logic, I'll change it in a follow up and can see why that'd be confusing