[bug] scalar metrics comparison not working due to KFP SDK v1 / Kubeflow v2 incompatibility

[ziv-maxu]

Environment

• KFP version:
  backend:v2.3.0
  frontend:v2.15.0
• KFP SDK version:
  kfp sdk: 1.8.22

Steps to reproduce

1.Create a pipeline component that outputs scalar metrics following the [v1 SDK output viewer] documentation(https://www.kubeflow.org/docs/components/pipelines/legacy-v1/sdk/output-viewer/#v1-sdk-writing-out-metadata-for-the-output-viewers), writing mlpipeline-metrics artifact in the v1 format:

2.Run the pipeline successfully — scalar metrics are recorded as expected.
3.Navigate to "Compare Runs" in the Kubeflow Pipelines UI and select multiple runs to compare.
4.Open the Metrics tab to compare scalar metrics across runs.
5.Observe in the browser developer console that the UI makes a request to /api/metrics (or similar endpoint), which returns 405 Method Not Allowed.

Expected result

The Compare Runs Metrics tab should display and compare scalar metrics (e.g., accuracy, loss) across selected runs, as it did in KFP v1.

Materials and reference

Labels

---

Impacted by this bug? Give it a 👍.

[yashasvi290906]

Hi @ziv-maxu ,

I’d like to work on this issue. From the description, it seems related to how the backend handles the /api/metrics endpoint for v1-format metrics.

I’ll start by checking the endpoint implementation and method handling to identify why it returns 405.

Please let me know if this aligns with the expected direction. Thanks!

[ziv-maxu]

Hi @yashasvi290906 , thanks for picking this up!

Yes, your direction is correct. Here's some additional context that might help:

The core issue is that KFP v2 backend removed (or never implemented) the /api/metrics endpoint that the v1 SDK relies on for scalar metrics comparison. When using KFP SDK v1 (kfp==1.8.22) with a KFP v2 backend (v2.3.0), the pipeline runs successfully and metrics are recorded, but the "Compare Runs" UI fails because it calls an endpoint that returns 405 Method Not Allowed.

Looking forward to your findings!

[yashasvi290906]

Thanks for the clarification, @ziv-maxu !

I investigated the backend and found that:

• metrics are already stored in the run_metrics table
• parsing and conversion to v1 format already exist (api_converter.go)
• but there is no /api/metrics endpoint implemented

I’m planning to add a compatibility endpoint /api/metrics that:

• fetches metrics from existing run storage
• uses the existing conversion logic to format them
• returns a response compatible with the v1 UI

Please let me know if this aligns with your expectations before I proceed.

[ziv-maxu]

Hi @yashasvi290906 , great findings!
That aligns perfectly with my expectations.
Since the run_metrics table and api_converter.go conversion logic already exist, adding a compatibility /api/metrics endpoint sounds like the right approach.
Please go ahead, thanks in advance.

[yashasvi290906]

Hi @ziv-maxu , I traced the current flow and found that Compare Runs uses metrics already attached to runs, rather than a separate /api/metrics endpoint.

The likely issue is in the metrics ingestion/parsing path for v1-style mlpipeline-metrics artifacts. In particular, the backend was normalizing camelCase numberValue to snake_case number_value using fragile string replacement.

I prepared a fix that makes this normalization more robust via structured JSON parsing so v1 SDK metrics are consistently ingested and surfaced in run.metrics.

Here is the PR : #13205

Please review it and let me know if any changes are needed.
Thank you!