RuntimeStatsMetric Reporter for table scans in Iceberg Connector #24904

j-sund · 2025-04-10T20:28:05Z

Description

Adds a RuntimeStatsMetricsReporter class that exposes iceberg metrics.

Motivation and Context

Exposing these metrics can improve query observability, which thus makes future improvements to the Iceberg integration a little easier. This also enables better performance monitoring.

Impact

Allows for better visibility into the Iceberg integration.

Test Plan

Ensure the metric is visible and being populated. Basic tests are included in IcebergDistributedSmokeTestBase.java class

Contributor checklist

Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
Documented new properties (with its default value), SQL syntax, functions, or other functionality.
If release notes are required, they follow the release notes guidelines.
[] Adequate tests were added if applicable.
CI passed.

Release Notes

== NO RELEASE NOTE ==

linux-foundation-easycla · 2025-04-10T20:28:09Z

The committers listed above are authorized under a signed CLA.

✅ login: j-sund / name: Joshua Sundararaman (e2b5904, 2bfd588, bdce0b2, 1920330, 59ca7d7, 96a3229, 25f23fb, c0cfde0, 7b2f748, 4d29467, a586c2a, d601da6, eb56227, 51f3946, db9c0ac, a05f395, 17ce34e, 8b19e78, 2cf2a5f, a6f40d7, c21c4b8)

ZacBlanco · 2025-04-14T19:19:01Z

@j-sund Can you sign the CLA using the link in the comment above? You can just sign as a personal contributor.

Also, before your next push try running ./mvnw validate -pl presto-iceberg to ensure your code passes all the checkstyle and license rules. I see the maven checks / maven-checks CI action is failing with the following:

Warning:  Missing header in: /home/runner/work/presto/presto/presto-iceberg/src/main/java/com/facebook/presto/iceberg/RuntimeStatsMetricsReporter.java

ZacBlanco

Just minor things. This looks great!

presto-iceberg/src/main/java/com/facebook/presto/iceberg/RuntimeStatsMetricsReporter.java

presto-iceberg/src/test/java/com/facebook/presto/iceberg/IcebergDistributedSmokeTestBase.java

presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergSplitManager.java

presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergUtil.java

… for remaining metrics

hantangwangd

This looks like a great feature! Some nits, and a minor problem in the test code.

hantangwangd · 2025-05-17T17:48:11Z

presto-iceberg/src/main/java/com/facebook/presto/iceberg/RuntimeStatsMetricsReporter.java

+    /**
+     * Helper method to construct the full metric name for a table scan.
+     *
+     * @param tableName  the name of the table
+     * @param metricName the name of the metric
+     * @return the composed metric name in the format: table.scan.metric
+     */
+    private String tableScanString(final String tableName, final String metricName)
+    {
+        return tableName + ".scan." + metricName;
+    }


nit: move this private method to the bottom, and it can be declared static.

hantangwangd · 2025-05-17T17:52:29Z

presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergSplitManager.java

@@ -96,16 +97,19 @@ public ConnectorSplitSource getSplits(
            return new ChangelogSplitSource(session, typeManager, icebergTable, scan);
        }
        else if (table.getIcebergTableName().getTableType() == EQUALITY_DELETES) {
+            RuntimeStats runtimeStats = session.getRuntimeStats();


Couldn't comment above, so commenting here. Should we also register metrics reporter for CHANGELOG table scan?

hantangwangd · 2025-05-17T17:55:19Z

presto-iceberg/src/main/java/com/facebook/presto/iceberg/TableStatisticsMaker.java

@@ -168,14 +168,15 @@ public static TableStatistics getTableStatistics(
            Table icebergTable,
            List<IcebergColumnHandle> columns)
    {
-        return new TableStatisticsMaker(icebergTable, session, typeManager).makeTableStatistics(statisticsFileCache, tableHandle, currentPredicate, constraint, columns);
+        return new TableStatisticsMaker(icebergTable, session, typeManager).makeTableStatistics(statisticsFileCache, tableHandle, currentPredicate, constraint, columns, session);


nit: newly added parameter session is useless.

hantangwangd · 2025-05-17T17:56:00Z

presto-iceberg/src/main/java/com/facebook/presto/iceberg/TableStatisticsMaker.java

    }

    private TableStatistics makeTableStatistics(StatisticsFileCache statisticsFileCache,
            IcebergTableHandle tableHandle,
            Optional<TupleDomain<IcebergColumnHandle>> currentPredicate,
            Constraint constraint,
-            List<IcebergColumnHandle> selectedColumns)
+            List<IcebergColumnHandle> selectedColumns,
+            ConnectorSession session)


nit: newly added parameter session is useless.

hantangwangd · 2025-05-17T18:07:46Z

presto-iceberg/src/test/java/com/facebook/presto/iceberg/IcebergDistributedSmokeTestBase.java

+
+        assertTrue(runtimestats
+                .getMetrics()
+                .get("iceberg.tpch.orders.scan.totalPlanningDuration")


We shouldn't hardcode iceberg.tpch.orders as the table name here and below, as it may very across different catalog test subclasses. This seems to be the core reason of failures in TestIcebergSmokeHive and TestIcebergSmokeRestNestedNameSpace.

hantangwangd · 2025-05-22T04:04:04Z

I have submit a PR #25140 to make sure that the table name loaded from all catalogs have consistent formatting <catalog>.<schema>.<table>. So I think after that change being merged, we can use the following code to build table name uniformly:

        String catalog = getSession().getCatalog().get();
        String schema = getSession().getSchema().get();
        String tableName = catalog + "." + schema + ".orders";

j-sund marked this pull request as ready for review April 12, 2025 03:16

j-sund requested review from hantangwangd, ZacBlanco and a team as code owners April 12, 2025 03:16

j-sund requested a review from presto-oss April 12, 2025 03:16

j-sund force-pushed the runtimeStatsReporter branch from cfc3890 to 67e5b9b Compare April 23, 2025 16:42

ZacBlanco requested changes Apr 30, 2025

View reviewed changes

j-sund force-pushed the runtimeStatsReporter branch 2 times, most recently from 72ef7e5 to 6124180 Compare May 5, 2025 21:42

ZacBlanco requested changes May 5, 2025

View reviewed changes

presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergUtil.java Outdated Show resolved Hide resolved

j-sund force-pushed the runtimeStatsReporter branch from 6124180 to 5b96c26 Compare May 15, 2025 19:49

j-sund added 17 commits May 16, 2025 13:49

Initial template for RuntimeStatsMetricsReporter

25f23fb

More method stubs for MetricsReporter

e2b5904

Added totalDuration metric tracker

4d29467

Added resultDataFiles and resultDeleteFiles Metric and added template…

96a3229

… for remaining metrics

Added 4 more metrics, totalManifests related

2bfd588

Filled out rest of metric templates

7b2f748

Added guards for possible null values

db9c0ac

First 3 Metrics for MetricsReporter

1920330

Added query creation

bdce0b2

Deleted Testing File, didn't work, and should be placed elsewhere

a6f40d7

Added Metrics Reporter to Iceberg Connector

59ca7d7

Added a tableString builder method and fixed naming conventions

2cf2a5f

Added testing for RuntimestatsMetricReporter

a586c2a

Finished basic test for RuntimeStatsMetricReporter

eb56227

Adjusted RuntimeStatsMetricsReporter to follow checkstyle practice

c0cfde0

Fixed checkstyle for runtimestatsmetricreporter

51f3946

Removed comments and added better assert checks

a05f395

j-sund added 3 commits May 16, 2025 13:49

Added RuntimeStatsMetricReporter to all newScans in IcebergUtil

c21c4b8

Fixed test case and passed runtimeStats besides session

17ce34e

Attempt test case fix and added reporter to TableStatisticsMaker

8b19e78

j-sund force-pushed the runtimeStatsReporter branch from 5b96c26 to 8b19e78 Compare May 16, 2025 18:50

Added reporter to PartitionTable newScan

d601da6

hantangwangd requested changes May 17, 2025

View reviewed changes

hantangwangd mentioned this pull request May 22, 2025

[Iceberg]Prefix table (loaded from Hive Catalog) with catalog name #25140

Open

6 tasks

RuntimeStatsMetric Reporter for table scans in Iceberg Connector #24904

Are you sure you want to change the base?

RuntimeStatsMetric Reporter for table scans in Iceberg Connector #24904

Uh oh!

Conversation

j-sund commented Apr 10, 2025

Description

Motivation and Context

Impact

Test Plan

Contributor checklist

Release Notes

Uh oh!

linux-foundation-easycla bot commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ZacBlanco commented Apr 14, 2025

Uh oh!

ZacBlanco left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hantangwangd left a comment

Choose a reason for hiding this comment

Uh oh!

hantangwangd May 17, 2025

Choose a reason for hiding this comment

Uh oh!

hantangwangd May 17, 2025

Choose a reason for hiding this comment

Uh oh!

hantangwangd May 17, 2025

Choose a reason for hiding this comment

Uh oh!

hantangwangd May 17, 2025

Choose a reason for hiding this comment

Uh oh!

hantangwangd May 17, 2025

Choose a reason for hiding this comment

Uh oh!

hantangwangd commented May 22, 2025

Uh oh!

Uh oh!

linux-foundation-easycla bot commented Apr 10, 2025 •

edited

Loading