-
Notifications
You must be signed in to change notification settings - Fork 5.5k
RuntimeStatsMetric Reporter for table scans in Iceberg Connector #24904
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
j-sund
wants to merge
22
commits into
prestodb:master
Choose a base branch
from
j-sund:runtimeStatsReporter
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+286
−30
Open
Changes from all commits
Commits
Show all changes
22 commits
Select commit
Hold shift + click to select a range
3bd3c8e
Initial template for RuntimeStatsMetricsReporter
j-sund afe94e5
More method stubs for MetricsReporter
j-sund f46a0e5
Added totalDuration metric tracker
j-sund e01409f
Added resultDataFiles and resultDeleteFiles Metric and added template…
j-sund d3b78a6
Added 4 more metrics, totalManifests related
j-sund e416ff6
Filled out rest of metric templates
j-sund 0831294
Added guards for possible null values
j-sund e5c733f
First 3 Metrics for MetricsReporter
j-sund b66ff2a
Added query creation
j-sund 27fc795
Deleted Testing File, didn't work, and should be placed elsewhere
j-sund 0a4b2b8
Added Metrics Reporter to Iceberg Connector
j-sund 1625a0f
Added a tableString builder method and fixed naming conventions
j-sund 2254777
Added testing for RuntimestatsMetricReporter
j-sund d944e2f
Finished basic test for RuntimeStatsMetricReporter
j-sund 0bea095
Adjusted RuntimeStatsMetricsReporter to follow checkstyle practice
j-sund 7bcdc29
Fixed checkstyle for runtimestatsmetricreporter
j-sund 6bc1e84
Removed comments and added better assert checks
j-sund 84c3cf8
Added RuntimeStatsMetricReporter to all newScans in IcebergUtil
j-sund f4eecbd
Fixed test case and passed runtimeStats besides session
j-sund 9c23516
Attempt test case fix and added reporter to TableStatisticsMaker
j-sund 9ef6b65
Added reporter to PartitionTable newScan
j-sund 8c38914
addressed review feedback
j-sund File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
183 changes: 183 additions & 0 deletions
183
presto-iceberg/src/main/java/com/facebook/presto/iceberg/RuntimeStatsMetricsReporter.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,183 @@ | ||
| /* | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
| package com.facebook.presto.iceberg; | ||
|
|
||
| import com.facebook.presto.common.RuntimeStats; | ||
| import com.facebook.presto.common.RuntimeUnit; | ||
| import org.apache.iceberg.metrics.MetricsReport; | ||
| import org.apache.iceberg.metrics.MetricsReporter; | ||
| import org.apache.iceberg.metrics.ScanReport; | ||
|
|
||
| /** | ||
| * A MetricsReporter implementation for reporting | ||
| * Iceberg scan metrics to Presto's RuntimeStats. | ||
| */ | ||
|
|
||
| public final class RuntimeStatsMetricsReporter | ||
| implements MetricsReporter | ||
| { | ||
| /** | ||
| * RuntimeStats variable used for storing scan metrics from Iceberg reports. | ||
| */ | ||
| private final RuntimeStats runtimeStats; | ||
|
|
||
| /** | ||
| * Constructs a RuntimeStatsMetricsReporter. | ||
| * | ||
| * @param runtimeStat the RuntimeStats instance to report metrics to | ||
| */ | ||
| public RuntimeStatsMetricsReporter(final RuntimeStats runtimeStat) | ||
| { | ||
| this.runtimeStats = runtimeStat; | ||
| } | ||
|
|
||
| @Override | ||
| public void report(final MetricsReport report) | ||
| { | ||
| if (!(report instanceof ScanReport)) { | ||
| return; | ||
| } | ||
|
|
||
| ScanReport scanReport = (ScanReport) report; | ||
| String tableName = scanReport.tableName(); | ||
|
|
||
| if (scanReport.scanMetrics().totalPlanningDuration() != null) { | ||
| runtimeStats.addMetricValue( | ||
| tableScanString(tableName, "totalPlanningDuration"), | ||
| RuntimeUnit.NANO, | ||
| scanReport.scanMetrics().totalPlanningDuration() | ||
| .totalDuration().toNanos()); | ||
| } | ||
|
|
||
| if (scanReport.scanMetrics().resultDataFiles() != null) { | ||
| runtimeStats.addMetricValue( | ||
| tableScanString(tableName, "resultDataFiles"), | ||
| RuntimeUnit.NONE, | ||
| scanReport.scanMetrics().resultDataFiles().value()); | ||
| } | ||
|
|
||
| if (scanReport.scanMetrics().resultDeleteFiles() != null) { | ||
| runtimeStats.addMetricValue( | ||
| tableScanString(tableName, "resultDeleteFiles"), | ||
| RuntimeUnit.NONE, | ||
| scanReport.scanMetrics().resultDeleteFiles().value()); | ||
| } | ||
|
|
||
| if (scanReport.scanMetrics().totalDataManifests() != null) { | ||
| runtimeStats.addMetricValue( | ||
| tableScanString(tableName, "totalDataManifests"), | ||
| RuntimeUnit.NONE, | ||
| scanReport.scanMetrics().totalDataManifests().value()); | ||
| } | ||
|
|
||
| if (scanReport.scanMetrics().totalDeleteManifests() != null) { | ||
| runtimeStats.addMetricValue( | ||
| tableScanString(tableName, "totalDeleteManifests"), | ||
| RuntimeUnit.NONE, | ||
| scanReport.scanMetrics().totalDeleteManifests().value()); | ||
| } | ||
|
|
||
| if (scanReport.scanMetrics().scannedDataManifests() != null) { | ||
| runtimeStats.addMetricValue( | ||
| tableScanString(tableName, "scannedDataManifests"), | ||
| RuntimeUnit.NONE, | ||
| scanReport.scanMetrics().scannedDataManifests().value()); | ||
| } | ||
|
|
||
| if (scanReport.scanMetrics().skippedDataManifests() != null) { | ||
| runtimeStats.addMetricValue( | ||
| tableScanString(tableName, "skippedDataManifests"), | ||
| RuntimeUnit.NONE, | ||
| scanReport.scanMetrics().skippedDataManifests().value()); | ||
| } | ||
|
|
||
| if (scanReport.scanMetrics().totalFileSizeInBytes() != null) { | ||
| runtimeStats.addMetricValue( | ||
| tableScanString(tableName, "totalFileSizeInBytes"), | ||
| RuntimeUnit.BYTE, | ||
| scanReport.scanMetrics().totalFileSizeInBytes() | ||
| .value()); | ||
| } | ||
|
|
||
| if (scanReport.scanMetrics().totalDeleteFileSizeInBytes() != null) { | ||
| runtimeStats.addMetricValue( | ||
| tableScanString(tableName, "totalDeleteFileSizeInBytes"), | ||
| RuntimeUnit.BYTE, | ||
| scanReport.scanMetrics().totalDeleteFileSizeInBytes() | ||
| .value()); | ||
| } | ||
|
|
||
| if (scanReport.scanMetrics().skippedDataFiles() != null) { | ||
| runtimeStats.addMetricValue( | ||
| tableScanString(tableName, "skippedDataFiles"), | ||
| RuntimeUnit.NONE, | ||
| scanReport.scanMetrics().skippedDataFiles() | ||
| .value()); | ||
| } | ||
|
|
||
| if (scanReport.scanMetrics().skippedDeleteFiles() != null) { | ||
| runtimeStats.addMetricValue( | ||
| tableScanString(tableName, "skippedDeleteFiles"), | ||
| RuntimeUnit.NONE, | ||
| scanReport.scanMetrics().skippedDeleteFiles().value()); | ||
| } | ||
|
|
||
| if (scanReport.scanMetrics().scannedDeleteManifests() != null) { | ||
| runtimeStats.addMetricValue( | ||
| tableScanString(tableName, "scannedDeleteManifests"), | ||
| RuntimeUnit.NONE, | ||
| scanReport.scanMetrics().scannedDeleteManifests().value()); | ||
| } | ||
|
|
||
| if (scanReport.scanMetrics().skippedDeleteManifests() != null) { | ||
| runtimeStats.addMetricValue( | ||
| tableScanString(tableName, "skippedDeleteManifests"), | ||
| RuntimeUnit.NONE, | ||
| scanReport.scanMetrics().skippedDeleteManifests().value()); | ||
| } | ||
|
|
||
| if (scanReport.scanMetrics().indexedDeleteFiles() != null) { | ||
| runtimeStats.addMetricValue( | ||
| tableScanString(tableName, "indexedDeleteFiles"), | ||
| RuntimeUnit.NONE, | ||
| scanReport.scanMetrics().indexedDeleteFiles().value()); | ||
| } | ||
|
|
||
| if (scanReport.scanMetrics().equalityDeleteFiles() != null) { | ||
| runtimeStats.addMetricValue( | ||
| tableScanString(tableName, "equalityDeleteFiles"), | ||
| RuntimeUnit.NONE, | ||
| scanReport.scanMetrics().equalityDeleteFiles().value()); | ||
| } | ||
|
|
||
| if (scanReport.scanMetrics().positionalDeleteFiles() != null) { | ||
| runtimeStats.addMetricValue( | ||
| tableScanString(tableName, "positionalDeleteFiles"), | ||
| RuntimeUnit.NONE, | ||
| scanReport.scanMetrics().positionalDeleteFiles().value()); | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * Helper method to construct the full metric name for a table scan. | ||
| * | ||
| * @param tableName the name of the table | ||
| * @param metricName the name of the metric | ||
| * @return the composed metric name in the format: table.scan.metric | ||
| */ | ||
| private static String tableScanString(final String tableName, final String metricName) | ||
| { | ||
| return tableName + ".scan." + metricName; | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Couldn't comment above, so commenting here. Should we also register metrics reporter for CHANGELOG table scan?