Skip to content

[b456377401] Cache Spark YARN applications #1029

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dominik-burdzy merged 1 commit into from
Jan 5, 2026
Merged

[b456377401] Cache Spark YARN applications #1029

dominik-burdzy merged 1 commit into from
Jan 5, 2026

Conversation

@dominik-burdzy
Copy link
Collaborator

@dominik-burdzy dominik-burdzy commented Nov 19, 2025 •
edited
Loading

  1. Collected Spark YARN applications need to be cached as they will be needed in the upcoming task connecting to Spark Server History and fetching their source and used Spark version from event logs.
  2. Make ClouderaAPIHostsTask as the only source of truth for caching hosts as this is more reliable than CMF endpoint. It simplifies the logic and make it less error-prone.
  3. Move common methods used in multiple tests into utils class.

@dominik-burdzy dominik-burdzy marked this pull request as draft November 19, 2025 13:02
@dominik-burdzy dominik-burdzy force-pushed the feature/b456377401/cache-spark-yarn-apps branch from 21441db to a572910 Compare November 20, 2025 10:22
@github-actions
Copy link

github-actions bot commented Nov 20, 2025 •
edited
Loading

Code Coverage Report

Overall Project 62.23% 🍏
Files changed 100% 🍏

File Coverage
YarnApplicationType.java 100% 🍏
ClouderaAPIHostsTask.java 100% 🍏
ClouderaCMFHostsTask.java 100% 🍏
ClouderaYarnApplicationTypeTask.java 95.44% 🍏
ClouderaManagerHandle.java 90.7% 🍏

@dominik-burdzy dominik-burdzy marked this pull request as ready for review November 20, 2025 10:33
@dominik-burdzy dominik-burdzy force-pushed the feature/b456377401/cache-spark-yarn-apps branch 3 times, most recently from 9384236 to c611ec9 Compare January 5, 2026 09:43
@dominik-burdzy
[b456377401] Cache Spark YARN applications
6792120 
1. Collected Spark YARN applications need to be cached as they will be needed in the upcoming task connecting to Spark Server History and fetching their source and used Spark version from event logs.
2. Make ClouderaAPIHostsTask as the only source of truth for caching hosts as this is more reliable than CMF endpoint. It simplifies the logic and make it less error-prone.
3. Move common methods used in multiple tests into utils class.
@dominik-burdzy dominik-burdzy force-pushed the feature/b456377401/cache-spark-yarn-apps branch from c611ec9 to 6792120 Compare January 5, 2026 09:48
@dominik-burdzy dominik-burdzy merged commit 2dee75a into main Jan 5, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vladislav-sidorovich vladislav-sidorovich vladislav-sidorovich approved these changes

@zaldis zaldis Awaiting requested review from zaldis

@kajgol kajgol Awaiting requested review from kajgol

@shevek-google shevek-google Awaiting requested review from shevek-google shevek-google is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dominik-burdzy @vladislav-sidorovich