-
Notifications
You must be signed in to change notification settings - Fork 4.9k
[Improvement-17670][Worker-monitoring] Add disk usage monitoring for data.basedir.path directory #17677
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: dev
Are you sure you want to change the base?
Conversation
…theses to make the operator precedence explicit
…asedir.path directory (apache#17670)
…asedir.path directory (apache#17670)Test
…asedir.path directory (apache#17670)format
…asedir.path directory (apache#17670)format
…asedir.path directory (apache#17670)Add data basedir disk usage threshold but set to 1.0 (100%) to effectively disable the check
.../java/org/apache/dolphinscheduler/server/master/config/MasterServerLoadProtectionConfig.java
Fixed
Show fixed
Hide fixed
.../java/org/apache/dolphinscheduler/server/master/config/MasterServerLoadProtectionConfig.java
Fixed
Show fixed
Hide fixed
…asedir.path directory (apache#17670) test docker-compose.yaml
…asedir.path directory (apache#17670) Fix field shadowing and improve code clarity
.../java/org/apache/dolphinscheduler/server/master/config/MasterServerLoadProtectionConfig.java
Fixed
Show fixed
Hide fixed
|
…asedir.path directory (apache#17670) fix :
…asedir.path directory (apache#17670) fix :MasterServerLoadProtectionConfig
…asedir.path directory (apache#17670) move disk monitoring to workers only - Remove master disk checks, keep only for workers - Clean up related configs and constructors - Fix config reference in WorkerServerLoadProtection
|
Hi @SbloodyS @ruanwenjun @EricGao888 |
The newest build error (No plugin found for prefix 'sonar') is from a missing SonarQube plugin in CI—unrelated to my PR changes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adding max-data-basedir-disk-usage-percentage-thresholds will conflict with the current max-disk-usage-percentage-thresholds, which will make it more difficult for users to understand.
I think we should configure multiple directories in the following two ways
1.
max-disk-usage-percentage-thresholds:
/data1: 0.8
/data2: 0.9
max-disk-usage-percentage-thresholds:
path: /data1,/data2
percentage: 0.9
This needs to be discussed. cc @ruanwenjun @zhongjiajie @Gallardot


Purpose of the pull request
close #17670
Brief change log
Feature Enhancement
Added disk usage monitoring for the data.basedir.path directory
Added dataBasedirPathDiskUsagePercentage field in Worker heartbeat data
Added display of dataBasedir disk usage on the frontend monitoring page
Added internationalization support (Chinese and English)
Implemented load protection based on disk usage of the data.basedir.path directory
Added maxDataBasedirDiskUsagePercentageThresholds configuration item in BaseServerLoadProtectionConfig
Implemented disk usage check logic for the dataBasedir path in BaseServerLoadProtection
Added max-data-basedir-disk-usage-percentage-thresholds configuration option in Worker config files
Configuration Updates
Kubernetes Deployment Configuration
Added description of the environment variable WORKER_SERVER_LOAD_PROTECTION_MAX_DATA_BASEDIR_DISK_USAGE_PERCENTAGE_THRESHOLDS in README.md
Added corresponding configuration items in values.yaml
Docker Deployment Configuration
Added WORKER_SERVER_LOAD_PROTECTION_MAX_DATA_BASEDIR_DISK_USAGE_PERCENTAGE_THRESHOLDS configuration in all test docker-compose.yaml files
UI Improvements
Adjusted layout of the Worker monitoring page
Added data directory disk usage metric; increased number of icons per row from 4 to 5, ensuring all monitoring metrics are displayed on the same line
These changes enhance DolphinScheduler's disk monitoring capabilities by providing fine-grained monitoring and overload protection for the data.basedir.path directory, helping prevent service issues caused by insufficient disk space.
Verify this pull request
This pull request is already covered by existing tests, such as WorkerServerLoadProtectionTest.
Pull Request Notice
Pull Request Notice
If your pull request contains incompatible change, you should also add it to
docs/docs/en/guide/upgrade/incompatible.md