Summary
When a connection uses regions = ["*"] and one enabled (opted-in) region's endpoint is unreachable at the network level, every multi-region query fails — slowly. There is no config mechanism that can exclude the broken region without abandoning the wildcard, and no way to make transport-level failures non-fatal.
Observed in production with me-south-1 (Bahrain): the region is opted-in at the org level but runs no workloads, and TCP connections to ec2.me-south-1.amazonaws.com time out from the querying environment. Every query against a multi-region table then fails after several minutes:
aws_vpc_security_group.listVpcSecurityGroups: api_error="operation error EC2: DescribeSecurityGroups,
exceeded maximum number of attempts, 11, https response error StatusCode: 0, RequestID: ,
request send failed, Post \"https://ec2.me-south-1.amazonaws.com/\": dial tcp 99.82.136.87:443: i/o timeout"
...
streamRows execution has failed: calling queryCache.AbortSet (aws: exceeded allowed timeout)
Mechanics
regions = ["*"] is expanded in listQueryRegionsForConnection (aws/multi_region.go) to all regions enabled for the account — including opted-in regions that are unreachable from where Steampipe runs. Patterns support * and ? only; there is no negation or exclusion.- The client uses the AWS SDK standard retryer (
getClientWithMaxRetries, aws/service.go) with the SDK's default HTTP client timeouts. request send failed / dial tcp ... i/o timeout is classified as a retryable connection error, so the SDK retries the full max_error_retry_attempts budget. Each attempt waits up to the default 30s dial connect timeout (DefaultDialConnectTimeout in aws-sdk-go-v2), plus exponential backoff between attempts — an unreachable region burns multiple minutes before returning anything.
- By then the query's context deadline has usually expired, so the whole scan aborts with
exceeded allowed timeout. One dead region fails all regions.
Why no existing config option solves this
regions: to exclude one region you must enumerate every wanted region explicitly and keep the list current as AWS launches/opts-in regions — the wildcard's main value is lost.ignore_error_codes: matches smithy.APIError codes only (shouldIgnoreErrors, aws/errors.go). A dial timeout never reaches the API layer — StatusCode 0, no error code — so it can never match.ignore_error_messages: does regex-match the full error string, so it can swallow the failure — but only after the retry budget is exhausted, so the multi-minute stall remains, and broad patterns (i/o timeout) silently hide genuine failures in healthy regions too. It is also not exposed in all managed environments (e.g. Turbot Pipes connection config), while regions, max_error_retry_attempts, min_error_retry_delay, and ignore_error_codes are.max_error_retry_attempts: lowering it makes the query fail faster, not succeed.
Proposals
- Region exclusion in config. Support negation patterns in
regions (e.g. regions = ["*", "!me-south-1"]) or a new exclude_regions argument, applied after wildcard expansion in listQueryRegionsForConnection. Small, backwards-compatible, and gives wildcard users a surgical opt-out.
- Fail fast on transport-level errors. An endpoint that cannot be dialed is not throttling; retrying it
max_error_retry_attempts times with 30s connect timeouts is wasted wall-clock. Classify request send failed connection errors as non-retryable (or cap them at 1–2 attempts) so an unreachable region surfaces its error in seconds instead of minutes.
- (Optional, opt-in) per-region error isolation. A flag that makes a region whose List call fails with a transport error contribute zero rows plus a warning, instead of failing the entire multi-region scan. Trade-off: silent data gaps — hence opt-in, not default.
Proposal 1 alone resolves the operational problem; proposal 2 improves failure behavior for everyone regardless.
Environment
- steampipe-plugin-aws (current), multi-region connection with
regions = ["*"]
- Region opted-in but unused; endpoint resolves to a Global Accelerator address (
99.82.x.x) that is unreachable from the querying network. Why the endpoint is unreachable is environment-specific and out of scope here — the plugin behavior (slow total failure, no exclusion mechanism) is the issue.
Summary
When a connection uses
regions = ["*"]and one enabled (opted-in) region's endpoint is unreachable at the network level, every multi-region query fails — slowly. There is no config mechanism that can exclude the broken region without abandoning the wildcard, and no way to make transport-level failures non-fatal.Observed in production with
me-south-1(Bahrain): the region is opted-in at the org level but runs no workloads, and TCP connections toec2.me-south-1.amazonaws.comtime out from the querying environment. Every query against a multi-region table then fails after several minutes:Mechanics
regions = ["*"]is expanded inlistQueryRegionsForConnection(aws/multi_region.go) to all regions enabled for the account — including opted-in regions that are unreachable from where Steampipe runs. Patterns support*and?only; there is no negation or exclusion.getClientWithMaxRetries,aws/service.go) with the SDK's default HTTP client timeouts.request send failed/dial tcp ... i/o timeoutis classified as a retryable connection error, so the SDK retries the fullmax_error_retry_attemptsbudget. Each attempt waits up to the default 30s dial connect timeout (DefaultDialConnectTimeoutin aws-sdk-go-v2), plus exponential backoff between attempts — an unreachable region burns multiple minutes before returning anything.exceeded allowed timeout. One dead region fails all regions.Why no existing config option solves this
regions: to exclude one region you must enumerate every wanted region explicitly and keep the list current as AWS launches/opts-in regions — the wildcard's main value is lost.ignore_error_codes: matchessmithy.APIErrorcodes only (shouldIgnoreErrors,aws/errors.go). A dial timeout never reaches the API layer — StatusCode 0, no error code — so it can never match.ignore_error_messages: does regex-match the full error string, so it can swallow the failure — but only after the retry budget is exhausted, so the multi-minute stall remains, and broad patterns (i/o timeout) silently hide genuine failures in healthy regions too. It is also not exposed in all managed environments (e.g. Turbot Pipes connection config), whileregions,max_error_retry_attempts,min_error_retry_delay, andignore_error_codesare.max_error_retry_attempts: lowering it makes the query fail faster, not succeed.Proposals
regions(e.g.regions = ["*", "!me-south-1"]) or a newexclude_regionsargument, applied after wildcard expansion inlistQueryRegionsForConnection. Small, backwards-compatible, and gives wildcard users a surgical opt-out.max_error_retry_attemptstimes with 30s connect timeouts is wasted wall-clock. Classifyrequest send failedconnection errors as non-retryable (or cap them at 1–2 attempts) so an unreachable region surfaces its error in seconds instead of minutes.Proposal 1 alone resolves the operational problem; proposal 2 improves failure behavior for everyone regardless.
Environment
regions = ["*"]99.82.x.x) that is unreachable from the querying network. Why the endpoint is unreachable is environment-specific and out of scope here — the plugin behavior (slow total failure, no exclusion mechanism) is the issue.