fix for issue #4638 -- create in-alarm variable to prevent excessive …

[sTarunRaaj]

…elasticsearch health alerts

Fixes

Fixes #4638

Description

This pull request introduces cooldown logic for the Elasticsearch cluster healthcheck DAG to prevent redundant Slack alerts. Specifically, it tracks the timestamp of the last alert using an Airflow Variable. If a new DAG run detects the same issue within the past 2 hours, it suppresses the alert. Once the cluster recovers (status returns to green), the timestamp is cleared to allow future alerts.

Testing Instructions

1. Trigger the DAG with a failing healthcheck (e.g., simulate status=red).
2. Confirm an alert is sent and that an Airflow Variable (es_last_alert_time_<env>) is created.
3. Trigger the DAG again within 6 hours — confirm that no alert is sent.
4. Trigger with a healthy status — confirm the variable is deleted.
5. Trigger again with another failure after the variable was cleared — confirm a new alert is sent.

Checklist

• My pull request has a descriptive title.
• My pull request targets the main branch.
• My commit messages follow best practices.
• My code follows the repository’s code style.
• I added or updated tests for the changes I made.
• I ran the DAG locally and verified the cooldown behavior.