workqueue: add dead-letter support

[tcnghia]

No description provided.

[Copilot]

Pull Request Overview

This PR introduces dead-letter support to the workqueue system by adding new functionality to permanently remove tasks after a maximum number of retry attempts. Key changes include:

• Adding new methods (Fail and GetAttempts) to the OwnedInProgressKey interface and their implementations in both in-memory and GCS-backed queues.
• Updating dispatcher logic to optionally use Fail when a task’s retry count exceeds a configured maxRetry value.
• Enhancing test coverage and configuration (Terraform variables and dispatcher configuration) to support maximum retry functionality.

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pkg/workqueue/interface.go	Updated the interface to include Fail and GetAttempts methods.
pkg/workqueue/inmem/inmem_test.go	Added conformance testing for the maximum retry behavior.
pkg/workqueue/inmem/inmem.go	Provided in-memory implementations for the new methods.
pkg/workqueue/gcs/gcs_test.go	Added tests to validate dead-letter behavior using maxRetry.
pkg/workqueue/gcs/gcs.go	Implemented dead-letter support with metadata updates and metrics in the GCS-based queue.
pkg/workqueue/dispatcher/handler.go	Extended the Handler function to accept and pass through a maxRetry value.
pkg/workqueue/dispatcher/dispatcher.go	Updated dispatcher functions to handle maxRetry logic and correctly switch to Fail when needed.
pkg/workqueue/conformance/conformance.go	Added a new test scenario to validate maximum retry behavior.
modules/workqueue/variables.tf	Added a new Terraform variable to configure the maximum retry threshold.
modules/workqueue/dispatcher.tf	Injected the maxRetry configuration into the dispatcher module.
modules/workqueue/cmd/dispatcher/main.go	Updated the main dispatcher command to pass the maxRetry value to the dispatcher.