Skip to content

Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria #8137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bhattmanish98 wants to merge 7 commits into
base: trunk
Choose a base branch
from
+2,609 −669
Open

Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria #8137

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
4acc11d
Metrics Changes: Ways to emit aggregated metrics + reduce number of m…
bhattmanish98 Oct 1, 2025
18d2be9
Merge branch 'trunk' of https://github.com/ABFSDriver/AbfsHadoop into…
bhattmanish98 Dec 16, 2025
abf1f1e
Removed unwanted changes
bhattmanish98 Dec 16, 2025
9111f5e
Fixed Checkstyle
bhattmanish98 Dec 16, 2025
1af17c1
Fixed checkstyle
bhattmanish98 Dec 17, 2025
e6a6087
Fixed checkstyle + rename few variables
bhattmanish98 Dec 17, 2025
ba2961c
Checkstyle fix
bhattmanish98 Dec 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 57 additions & 9 deletions hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -324,10 +324,6 @@ public class AbfsConfiguration{
DefaultValue = DEFAULT_METRIC_ANALYSIS_TIMEOUT_MS)
private int metricAnalysisTimeout;

@StringConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_METRIC_URI,
DefaultValue = EMPTY_STRING)
private String metricUri;

@StringConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_METRIC_ACCOUNT_NAME,
DefaultValue = EMPTY_STRING)
private String metricAccount;
Expand All @@ -336,6 +332,34 @@ public class AbfsConfiguration{
DefaultValue = EMPTY_STRING)
private String metricAccountKey;

@BooleanConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_METRICS_COLLECTION_ENABLED,
DefaultValue = DEFAULT_METRICS_COLLECTION_ENABLED)
private boolean metricsCollectionEnabled;

@BooleanConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_SHOULD_EMIT_METRICS_ON_IDLE_TIME,
DefaultValue = DEFAULT_SHOULD_EMIT_METRICS_ON_IDLE_TIME)
private boolean shouldEmitMetricsOnIdleTime;

@LongConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_METRICS_EMIT_THRESHOLD,
DefaultValue = DEFAULT_METRICS_EMIT_THRESHOLD)
private long metricsEmitThreshold;

@LongConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_METRICS_EMIT_THRESHOLD_INTERVAL_SECS,
DefaultValue = DEFAULT_METRICS_EMIT_THRESHOLD_INTERVAL_SECS)
private long metricsEmitThresholdIntervalInSecs;

@LongConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_METRICS_EMIT_INTERVAL_MINS,
DefaultValue = DEFAULT_METRICS_EMIT_INTERVAL_MINS)
private long metricsEmitIntervalInMins;

@IntegerConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_MAX_METRICS_CALLS_PER_SECOND,
DefaultValue = DEFAULT_MAX_METRICS_CALLS_PER_SECOND)
private int maxMetricsCallsPerSecond;

@BooleanConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_BACKOFF_RETRY_METRICS_ENABLED,
DefaultValue = DEFAULT_BACKOFF_RETRY_METRICS_ENABLED)
private boolean backoffRetryMetricsEnabled;

@IntegerConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_ACCOUNT_OPERATION_IDLE_TIMEOUT,
DefaultValue = DEFAULT_ACCOUNT_OPERATION_IDLE_TIMEOUT_MS)
private int accountOperationIdleTimeout;
Expand Down Expand Up @@ -1290,10 +1314,6 @@ public int getMetricAnalysisTimeout() {
return this.metricAnalysisTimeout;
}

public String getMetricUri() {
return metricUri;
}

public String getMetricAccount() {
return metricAccount;
}
Expand All @@ -1302,6 +1322,34 @@ public String getMetricAccountKey() {
return metricAccountKey;
}

public boolean isMetricsCollectionEnabled() {
return metricsCollectionEnabled;
}

public boolean shouldEmitMetricsOnIdleTime() {
return shouldEmitMetricsOnIdleTime;
}

public long getMetricsEmitThreshold() {
return metricsEmitThreshold;
}

public long getMetricsEmitIntervalInMins() {
return metricsEmitIntervalInMins;
}

public long getMetricsEmitThresholdIntervalInSecs() {
return metricsEmitThresholdIntervalInSecs;
}

public int getMaxMetricsCallsPerSecond() {
return maxMetricsCallsPerSecond;
}

public boolean isBackoffRetryMetricsEnabled() {
return backoffRetryMetricsEnabled;
}

public int getAccountOperationIdleTimeout() {
return accountOperationIdleTimeout;
}
Expand Down Expand Up @@ -1390,7 +1438,7 @@ public TracingHeaderFormat getTracingHeaderFormat() {
}

public MetricFormat getMetricFormat() {
return getEnum(FS_AZURE_METRIC_FORMAT, MetricFormat.EMPTY);
return getEnum(FS_AZURE_METRIC_FORMAT, MetricFormat.INTERNAL_METRIC_FORMAT);
}

public AuthType getAuthType(String accountName) {
Expand Down
48 changes: 33 additions & 15 deletions hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsCountersImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -201,20 +201,39 @@ public void initializeWriteResourceUtilizationMetrics() {


@Override
public void initializeMetrics(MetricFormat metricFormat) {
public void initializeMetrics(final MetricFormat metricFormat,
final AbfsConfiguration abfsConfiguration) {
switch (metricFormat) {
case INTERNAL_BACKOFF_METRIC_FORMAT:
abfsBackoffMetrics = new AbfsBackoffMetrics();
break;
case INTERNAL_FOOTER_METRIC_FORMAT:
abfsReadFooterMetrics = new AbfsReadFooterMetrics();
break;
case INTERNAL_METRIC_FORMAT:
abfsBackoffMetrics = new AbfsBackoffMetrics();
abfsReadFooterMetrics = new AbfsReadFooterMetrics();
break;
default:
break;
case INTERNAL_BACKOFF_METRIC_FORMAT:
abfsBackoffMetrics = new AbfsBackoffMetrics(
abfsConfiguration.isBackoffRetryMetricsEnabled());
break;
case INTERNAL_FOOTER_METRIC_FORMAT:
initializeReadFooterMetrics();
Copy link
Contributor

@anmolanmol1234 anmolanmol1234 Dec 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

break missing here

case INTERNAL_METRIC_FORMAT:
abfsBackoffMetrics = new AbfsBackoffMetrics(
abfsConfiguration.isBackoffRetryMetricsEnabled());
initializeReadFooterMetrics();
break;
default:
break;
}
}

/**
* Initialize the read footer metrics.
* In case the metrics are already initialized,
* create a new instance with the existing map.
*/
private void initializeReadFooterMetrics() {
if (abfsReadFooterMetrics == null) {
abfsReadFooterMetrics = new AbfsReadFooterMetrics();
} else {
//In case metrics is emitted based on total count, there could be a chance
// that file type for which we have calculated the type will be lost.
// To avoid that, creating a new instance with existing map.
abfsReadFooterMetrics = new AbfsReadFooterMetrics(
abfsReadFooterMetrics.getFileTypeMetricsMap());
}
}

Expand Down Expand Up @@ -375,8 +394,7 @@ public DurationTracker trackDuration(String key) {
public String toString() {
String metric = "";
Copy link
Contributor

@anmolanmol1234 anmolanmol1234 Dec 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use empty string constant

if (abfsBackoffMetrics != null) {
long totalNoRequests = getAbfsBackoffMetrics().getMetricValue(TOTAL_NUMBER_OF_REQUESTS);
if (totalNoRequests > 0) {
if (getAbfsBackoffMetrics().getMetricValue(TOTAL_NUMBER_OF_REQUESTS) > 0) {
metric += "#BO:" + getAbfsBackoffMetrics().toString();
}
}
Expand Down
13 changes: 1 addition & 12 deletions ...p-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,7 @@ public void initialize(URI uri, Configuration configuration)
.withBlockFactory(blockFactory)
.withBlockOutputActiveBlocks(blockOutputActiveBlocks)
.withBackReference(new BackReference(this))
.withFileSystemId(this.fileSystemId)
.build();

this.abfsStore = new AzureBlobFileSystemStore(systemStoreBuilder);
Expand Down Expand Up @@ -828,18 +829,6 @@ public synchronized void close() throws IOException {
if (isClosed()) {
return;
}
if (getAbfsStore().getClient().isMetricCollectionEnabled()) {
TracingContext tracingMetricContext = new TracingContext(
clientCorrelationId,
fileSystemId, FSOperationType.GET_ATTR, true,
tracingHeaderFormat,
listener, abfsCounters.toString());
try {
getAbfsClient().getMetricCall(tracingMetricContext);
} catch (IOException e) {
throw new IOException(e);
}
}
// does all the delete-on-exit calls, and may be slow.
super.close();
LOG.debug("AzureBlobFileSystem.close");
Expand Down
12 changes: 9 additions & 3 deletions ...ls/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -258,7 +258,7 @@ public AzureBlobFileSystemStore(
boolean useHttps = (usingOauth || abfsConfiguration.isHttpsAlwaysUsed()) ? true : abfsStoreBuilder.isSecureScheme;
this.abfsPerfTracker = new AbfsPerfTracker(fileSystemName, accountName, this.abfsConfiguration);
this.abfsCounters = abfsStoreBuilder.abfsCounters;
initializeClient(uri, fileSystemName, accountName, useHttps);
initializeClient(uri, fileSystemName, accountName, useHttps, abfsStoreBuilder.fileSystemId);
final Class<? extends IdentityTransformerInterface> identityTransformerClass =
abfsStoreBuilder.configuration.getClass(FS_AZURE_IDENTITY_TRANSFORM_CLASS, IdentityTransformer.class,
IdentityTransformerInterface.class);
Expand Down Expand Up @@ -1717,7 +1717,7 @@ public boolean isInfiniteLeaseKey(String key) {
* @throws IOException
*/
private void initializeClient(URI uri, String fileSystemName,
String accountName, boolean isSecure)
String accountName, boolean isSecure, String fileSystemId)
throws IOException {
if (this.getClient() != null) {
return;
Expand Down Expand Up @@ -1795,7 +1795,7 @@ private void initializeClient(URI uri, String fileSystemName,
this.clientHandler = new AbfsClientHandler(baseUrl, creds,
abfsConfiguration,
tokenProvider, sasTokenProvider, encryptionContextProvider,
populateAbfsClientContext());
populateAbfsClientContext(), fileSystemId);

this.setClient(getClientHandler().getClient());
LOG.trace("AbfsClient init complete");
Expand Down Expand Up @@ -1966,6 +1966,7 @@ public static final class AzureBlobFileSystemStoreBuilder {
private DataBlocks.BlockFactory blockFactory;
private int blockOutputActiveBlocks;
private BackReference fsBackRef;
private String fileSystemId;

public AzureBlobFileSystemStoreBuilder withUri(URI value) {
this.uri = value;
Expand Down Expand Up @@ -2007,6 +2008,11 @@ public AzureBlobFileSystemStoreBuilder withBackReference(
return this;
}

public AzureBlobFileSystemStoreBuilder withFileSystemId(String fileSystemId) {
this.fileSystemId = fileSystemId;
return this;
}

public AzureBlobFileSystemStoreBuilder build() {
return this;
}
Expand Down
10 changes: 8 additions & 2 deletions ...hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,14 @@ public final class ConfigurationKeys {
public static final String FS_AZURE_ACCOUNT_KEY_PROPERTY_NAME = "fs.azure.account.key";
public static final String FS_AZURE_METRIC_ACCOUNT_NAME = "fs.azure.metric.account.name";
public static final String FS_AZURE_METRIC_ACCOUNT_KEY = "fs.azure.metric.account.key";
public static final String FS_AZURE_METRIC_URI = "fs.azure.metric.uri";
public static final String FS_AZURE_METRIC_FORMAT = "fs.azure.metric.format";
Copy link
Contributor

@anujmodi2021 anujmodi2021 Dec 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add jvadoc for all these with {@value} tag.

Copy link
Contributor

@anujmodi2021 anujmodi2021 Dec 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep all the metric related configs name consistent with same prefix.
fs.azure.metrics....

Copy link
Contributor Author

@bhattmanish98 bhattmanish98 Dec 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

metric account name, key and format have "metric", other configurations have "metrics". I have kept it intentionally. Do you want to change this?

Copy link
Contributor

@anujmodi2021 anujmodi2021 Dec 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I think we should have common prefix for all the metric related configs

public static final String FS_AZURE_METRICS_COLLECTION_ENABLED = "fs.azure.metrics.collection.enabled";
public static final String FS_AZURE_SHOULD_EMIT_METRICS_ON_IDLE_TIME = "fs.azure.should.emit.metrics.on.idle.time";
public static final String FS_AZURE_METRICS_EMIT_THRESHOLD = "fs.azure.metrics.emit.threshold";
public static final String FS_AZURE_METRICS_EMIT_THRESHOLD_INTERVAL_SECS = "fs.azure.metrics.emit.threshold.interval.secs";
public static final String FS_AZURE_METRICS_EMIT_INTERVAL_MINS = "fs.azure.metrics.emit.interval.mins";
public static final String FS_AZURE_MAX_METRICS_CALLS_PER_SECOND = "fs.azure.max.metrics.calls.per.second";
public static final String FS_AZURE_BACKOFF_RETRY_METRICS_ENABLED = "fs.azure.backoff.retry.metrics.enabled";

public static final String FS_AZURE_ACCOUNT_KEY_PROPERTY_NAME_REGX = "fs\\.azure\\.account\\.key\\.(.*)";
public static final String FS_AZURE_SECURE_MODE = "fs.azure.secure.mode";
Expand Down Expand Up @@ -231,7 +238,6 @@ public final class ConfigurationKeys {
* character constraints are not satisfied. **/
public static final String FS_AZURE_CLIENT_CORRELATIONID = "fs.azure.client.correlationid";
public static final String FS_AZURE_TRACINGHEADER_FORMAT = "fs.azure.tracingheader.format";
public static final String FS_AZURE_METRIC_FORMAT = "fs.azure.metric.format";
public static final String FS_AZURE_CLUSTER_NAME = "fs.azure.cluster.name";
public static final String FS_AZURE_CLUSTER_TYPE = "fs.azure.cluster.type";
public static final String FS_AZURE_SSL_CHANNEL_MODE_KEY = "fs.azure.ssl.channel.mode";
Expand Down
7 changes: 7 additions & 0 deletions ...azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,13 @@ public final class FileSystemConfigurations {
public static final boolean DEFAULT_ENABLE_AUTOTHROTTLING = false;
public static final int DEFAULT_METRIC_IDLE_TIMEOUT_MS = 60_000;
public static final int DEFAULT_METRIC_ANALYSIS_TIMEOUT_MS = 60_000;
public static final boolean DEFAULT_METRICS_COLLECTION_ENABLED = true;
Copy link
Contributor

@anujmodi2021 anujmodi2021 Dec 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to keep it enabled by default for all Cxs?

Copy link
Contributor Author

@bhattmanish98 bhattmanish98 Dec 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this feature will be enabled by default.

public static final boolean DEFAULT_SHOULD_EMIT_METRICS_ON_IDLE_TIME = false;
public static final long DEFAULT_METRICS_EMIT_THRESHOLD = 100_000L;
public static final long DEFAULT_METRICS_EMIT_THRESHOLD_INTERVAL_SECS = 60;
Copy link
Contributor

@anujmodi2021 anujmodi2021 Dec 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Genrally names here also contains FS. Format was exact config variable name prefixed with DEFAULT_

I know this isn't followed everywhere but let's try to resolve this here.

Copy link
Contributor Author

@bhattmanish98 bhattmanish98 Dec 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, earlier as well we keep it DEFAULT_. We can discuss it offline as it will make the variable name bigger.

public static final long DEFAULT_METRICS_EMIT_INTERVAL_MINS = 60;
public static final int DEFAULT_MAX_METRICS_CALLS_PER_SECOND = 3;
public static final boolean DEFAULT_BACKOFF_RETRY_METRICS_ENABLED = false;
public static final boolean DEFAULT_FS_AZURE_ACCOUNT_LEVEL_THROTTLING_ENABLED = true;
public static final int DEFAULT_ACCOUNT_OPERATION_IDLE_TIMEOUT_MS = 60_000;
public static final int DEFAULT_ANALYSIS_PERIOD_MS = 10_000;
Expand Down
Loading