Cloud Monitoring POC for hadoop operations metrics

[dheerajsngh]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request establishes a foundational framework for enhanced observability by integrating Google Cloud Monitoring for Hadoop operations. It leverages OpenCensus to capture and push critical GCS operation duration metrics, enabling users to monitor connector performance more effectively. The newly introduced configurable export interval provides flexibility in how frequently these metrics are reported, allowing for tailored monitoring strategies.

Highlights

• Cloud Monitoring Integration: Introduced initial support for exporting Hadoop operation metrics to Google Cloud Monitoring (Stackdriver) using OpenCensus.
• Configurable Metrics Export Interval: Added a new configuration property fs.gs.metrics.export.interval.ms to allow users to specify how frequently metrics are exported, defaulting to 60 seconds.
• GCS Operation Duration Tracking: Implemented tracking and export of GCS operation durations, tagged by operation type (e.g., 'open', 'read'), to provide detailed performance insights.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• gcs/src/main/java/com/google/cloud/hadoop/fs/gcs/GhfsGlobalStorageStatistics.java
  • Imported necessary OpenCensus and CloudMonitoringMetricsRecorder classes.
  • Added logic to record GCS operation durations to OpenCensus if Cloud Monitoring is enabled, including dynamic tag creation for operation types.
• gcs/src/main/java/com/google/cloud/hadoop/fs/gcs/GoogleHadoopFileSystemConfiguration.java
  • Defined a new Hadoop configuration property GCS_METRICS_EXPORT_INTERVAL_MS with a default value of 60000 milliseconds.
  • Updated the getGcsOptionsBuilder method to set the metrics export interval from the new configuration property.
• gcsio/src/main/java/com/google/cloud/hadoop/gcsio/CloudMonitoringMetricsRecorder.java
  • Changed the class visibility from package-private to public.
  • Defined TYPE as a TagKey and GCS_DURATION as a MeasureLong for OpenCensus metrics.
  • Updated the default EXPORT_INTERVAL from 5000ms to 60000ms.
  • Modified the create and setupCloudMonitoringExporter methods to accept and utilize a configurable exportIntervalMs.
  • Added a isCloudMonitoringEnabled static method to check if the recorder is initialized.
  • Registered a new OpenCensus view for spark/driver/connector/gcs/duration to capture GCS operation latencies.
• gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorageImpl.java
  • Modified the instantiation of CloudMonitoringMetricsRecorder to pass the metricsExportIntervalMs from GoogleCloudStorageOptions.
• gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorageOptions.java
  • Added a new abstract method getMetricsExportIntervalMs to retrieve the metrics export interval.
  • Updated the default builder to set metricsExportIntervalMs to 60000L.
  • Added an abstract setMetricsExportIntervalMs method to the Builder class.
• pom.xml
  • Added a new dependency for google-http-client-jackson2 version 1.42.3.

Activity

• No specific activity (comments, reviews, or progress updates) has been recorded for this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[gemini-code-assist]

Code Review

This pull request introduces a proof-of-concept for exporting Hadoop operation metrics to Google Cloud Monitoring using OpenCensus. The changes involve adding configuration for enabling metrics and setting the export interval, creating a CloudMonitoringMetricsRecorder to handle metric publishing, and instrumenting GhfsGlobalStorageStatistics to record operation durations.

My review focuses on improving the usability and cleanliness of the new metrics implementation. I've identified that the metric name is too specific to Spark and should be made more generic for broader Hadoop ecosystem use. I've also pointed out some dead code that can be removed. Overall, this is a great step towards better observability.

[gemini-code-assist]

The metric name spark/driver/connector/gcs/duration is specific to Spark. Since this is a general Hadoop connector that can be used with other tools like MapReduce and Hive, a more generic metric name would be more appropriate and less confusing. Consider changing it to something like gcs/connector/duration.

Suggested change

	public static final MeasureLong GCS_DURATION =
	MeasureLong.create("spark/driver/connector/gcs/duration", "Duration of GCS operations", "ms");
	public static final MeasureLong GCS_DURATION =
	MeasureLong.create("gcs/connector/duration", "Duration of GCS operations", "ms");

[gemini-code-assist]

To maintain consistency with the suggested change for the GCS_DURATION measure, this view name should also be updated to use a more generic name, for example gcs/connector/duration.

Suggested change

	Name.create("spark/driver/connector/gcs/duration"),
	Name.create("gcs/connector/duration"),

[gemini-code-assist]

The EXPORT_INTERVAL constant is no longer used since the export interval is now configurable and passed in as a parameter. This constant should be removed to eliminate dead code.