feat(csharp): Add retry-after behavior for 503 responses in Spark ADBC driver

[jadewang-db]

Description

This PR implements retry-after behavior for the Spark ADBC driver when receiving 503 responses with Retry-After headers. This is particularly useful for Databricks clusters that may return 503 responses when a cluster is starting up or experiencing temporary unavailability.

Changes

• Added new configuration parameters:
  • adbc.spark.temporarily_unavailable_retry (default: 1 - enabled)
  • adbc.spark.temporarily_unavailable_retry_timeout (default: 900 seconds)
• Created a RetryHttpHandler class that wraps the existing HttpClientHandler to handle 503 responses
• Modified SparkHttpConnection to use the new retry handler
• Added comprehensive unit tests for the retry behavior

Implementation Details

When a 503 response with a Retry-After header is received:

1. The handler will wait for the number of seconds specified in the header
2. It will then retry the request
3. If another 503 response is received, it will continue retrying
4. If the total retry time exceeds the configured timeout, it will fail with an appropriate error message

Testing

Added unit tests to verify:

• Retry behavior for 503 responses with Retry-After headers
• Timeout behavior when retry time exceeds the configured limit
• Handling of invalid or missing Retry-After headers
• Disabling retry behavior via configuration
• Parameter validationv

[CurtHagenlocher]

Thanks for the change! I've left feedback, some more optional than others.

[davidhcoe]

Can we hold off submitting this until a decision is made on #2672?

[CurtHagenlocher]

Thanks for the change! Please make sure the linter doesn't complain about whitespace issues and that the retry flag is properly handled.

[CurtHagenlocher]

I assume you'd intended to parse or otherwise analyze the string configuration value? Right now, tempUnavailableRetryValue is never set to anything other than its default value of true .

[CurtHagenlocher]

Thanks for mollifying the linter! This is still an open item.