Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[10-10EZ/10-10CG] Update monitor documentation #104688

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
+4 −1
Open

[10-10EZ/10-10CG] Update monitor documentation #104688

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8bc0631
update 1010ez sidekiq amount
coope93 Mar 7, 2025
d684538
add context for 1010cg monitor
coope93 Mar 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions ...0CG Form/Endpoint Monitoring & Zero Silent Failures/10-10CG Monitor Playbook.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,9 @@ _last updated: 01-07-2025_
## Slack & Email Alert: "Triggered: 1010CG submission job has failed ten retries"

- At least one application submission has been retried 10 times, and has failed. The submission will retry 6 more times before failing completely
- A statsD metric should have been incremented for `vets_api.statsd.api_form1010cg_async_failed_ten_retries` with a tag containing the `claim_id`. That `claim_id` can be used to find the `claim` record in the database.
- If the submission ultimately fails, we would expect the "Triggered: 1010CG submission job has failed with no retries left" monitor to trigger within 2 days of this monitor being triggered.
- To validate a submission has succeeded after this monitor is triggered, the `claim_id` can be used to check if there is a `claim` record in the database. If it is no longer present the submission has since succeeded, and if it has yet to succeed the record will still exist.

### Required access

Expand Down
2 changes: 1 addition & 1 deletion ...0EZ Form/Endpoint Monitoring & Zero Silent Failures/10-10EZ Monitor Playbook.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ _last updated: 01-06-2025_

## Slack & Email Alert: "Triggered: 1010EZ submission job has failed with no retries left"

- At least one application submission has been retried 14 times, has failed and will not be retried again.
- At least one application submission has been retried 16 times, has failed and will not be retried again.

### Required access

Expand Down