[Feature Request] Add full static-shape Telum EP integration for ONNX Runtime on s390x

[k8ika0s]

Describe the feature request

Summary

Add full static-shape Telum EP integration for ONNX Runtime on s390x (z16+), including robust capability gating, explicit CPU fallback behavior, and reproducible offload/performance validation.

Problem

Current Telum EP support is functional but not yet complete across real transformer-like graphs and validation workflows.
Key gaps are:

• partial NNPA function availability across hardware generations
• clear fallback observability when nodes are not offloaded
• repeatable proof (offload + acceleration) suitable for upstream review without native Telum CI hardware

Proposed Feature

1. Keep static-shape-first scope for Telum EP.
2. Use runtime NNPA capability query as source of truth for kernel registration/partitioning.
3. Default to CPU fallback (non-strict), with strict mode available for fail-fast.
4. Add reproducible benchmark and validation workflow showing:
   • node placement on Telum
   • CPU vs Telum performance deltas on deterministic static models

Success Criteria

• Telum-assigned nodes are clearly reported.
• Unsupported NNPA paths fall back predictably (or fail in strict mode).
• Reproducible benchmark evidence demonstrates acceleration on Telum-capable hosts.
• Non-Telum builds remain unaffected.

Non-Goals

• Dynamic-shape full offload in this phase.
• Forcing new external CI infrastructure in upstream repo.

Request For Maintainer Feedback

• Acceptable CI/testing contract when Telum hardware is unavailable in upstream CI.

  Because upstream CI has no Telum hardware and does not allow third-party runners, we should test Telum capability logic through a stubbed capability interface. The EP keeps the real zDNN-backed capability path in production, but tests can inject deterministic “snapshot” capability profiles (NNPA present/absent, function bits, max-dim limits) to validate GetCapability decisions, partitioning behavior, and CPU fallback paths. This gives reliable regression coverage in standard CI without requiring real Telum execution, while preserving true hardware validation on dedicated s390x/Telum hosts. Runtime behavior on hardware remains unchanged by default.

• Preferred PR slicing for review.

  Suggested:
  1: runtime/provider behavior
  2: kernel/coverage expansion
  3: performance/tooling
  4: Docs

Initial testing results:

Telum_EP_Perf_Validation.md

Note: I see another change pending around s390x. I am not involved with that work, but I would be happy to review and collaborate on any merge conflicts.

Describe scenario use case

Use Case Scenario

A production inference service runs transformer-style models on IBM z16/z17 systems.
Operators need:

• deterministic static-shape inference,
• maximal Telum offload where available,
• safe CPU fallback where unavailable,
• visibility into fallback decisions,
• measurable latency/throughput improvements vs CPU-only execution.

[k8ika0s]

Opened Telum EP stack PRs:\n\n- PR1 (upstream, ready for review): https://github.com/microsoft/onnxruntime/pull/27337\n- PR2 draft (stacked in fork): https://github.com/k8ika0s/onnxruntime/pull/1\n- PR3 draft (stacked in fork): https://github.com/k8ika0s/onnxruntime/pull/2\n- PR4 draft (stacked in fork): https://github.com/k8ika0s/onnxruntime/pull/3\n\nPR1 contains full stack context and notes that PR2-PR4 will be retargeted/upstreamed incrementally after PR1 review direction.

[yuslepukhin]

These are great contributions. We are migrating to a plug-in EP APIs where EPs are no longer part of the ONNXRuntime source code base. There is a number of PRs that expose a stable API for the EP to implement that have been checked in.
All of our major EPs are migrating. I will check if we have a good public document for that.

[k8ika0s]

Awesome! Seems like a great way to do it. I'll review any docs and look for PRs and realign around the new plug-in architecutre. Any additional info is appreciated.

[k8ika0s]

Thanks for the guidance on moving to the plug-in EP APIs.

After reviewing the current plugin EP docs and recent merged PRs, we’re pivoting Telum upstreaming from in-tree EP registration to the plugin EP integration path.

This means the current in-tree stack (PR #27337 and the fork draft stack) is now superseded for final upstreaming, and we’ll submit a new plugin-based stack in smaller reviewable slices.

Planned replacement stack:

1. Telum backend extraction + plugin-ready interface layer
2. Telum plugin EP skeleton (factory/device/capability/compile path)
3. Core op port (MatMul/Gemm/elementwise/activations/Softmax/LayerNorm)
4. Plugin-EP-aligned validation/perf/docs

Validation approach:

• CI-safe coverage with stubbed backend + plugin integration tests
• s390x Telum hardware runs for functional/perf evidence
• Explicit CPU fallback for unsupported/non-static-shape paths

If there is a preferred landing pattern (in-tree plugin sample first vs external plugin repo flow), we’ll follow that.

[yuslepukhin]

https://onnxruntime.ai/docs/execution-providers/plugin-ep-libraries/
Documentation PR in progress: #27340

https://github.com/microsoft/onnxruntime/blob/main/include/onnxruntime/core/session/onnxruntime_ep_c_api.h
https://github.com/microsoft/onnxruntime/blob/main/include/onnxruntime/core/session/onnxruntime_ep_device_ep_metadata_keys.h

https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/core/session/plugin_ep

https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/test/autoep/library/example_plugin_ep/example_plugin_ep.cc

With the new EP plug-in approach, you are not required to host the source code alongside with ORT. This allows you a considerable freedom as to hosting it and running CIs.

[k8ika0s]

Update: we closed the previous in-tree/draft PR stack to do a clean plugin-EP refactor pass before re-opening upstream PRs.

Current refactor progress (local/fork branch):

• Telum plugin scaffold + registration/inference tests are in place.
• Internal scaffold symbol cleanup completed (Example* -> Telum*).
• Added explicit partial-support/fallback assignment coverage for Telum plugin path.
  Once the next slices are complete and validated, we will open a fresh, plugin-aligned PR stack.