Skip to content

s3: commands hang on unhealthy S3 endpoint #490

Description

@nirs

Problem

ramenctl is a tool to debug unhealthy systems. When the system is not
healthy, ramenctl should complete and report the issues - not hang
forever and produce no report when killed.

Currently, S3 operations (HeadBucket, ListObjectsV2, GetObject) have no
timeout. When the S3 endpoint is slow or unreachable, ramenctl hangs
indefinitely. Since it never completes, no YAML or HTML report is
generated. The user (or AI agent) gets no useful output at all.

With timeouts, ramenctl will fail the S3 validation but continue to
produce a complete report showing which S3 profiles failed and why.

This is especially important when running ramenctl via an AI agent (e.g.
bob). The agent's harness enforces its own command timeout (typically
120-180 seconds). When ramenctl hangs, the harness kills it with no
report. The agent retries, and the command hangs and is killed again.
The agent never gets any results.

Proposed solution

Add a configurable timeout for S3 operations. Each S3 request gets its
own timeout context as a child of the command context, so command
cancellation (Ctrl+C) still works.

Alternatives considered

  • HTTP client timeout: simpler but does not use the command context. The
    S3 code already uses context for command cancellation. Using context for
    timeouts means a future global command timeout will automatically apply
    to S3 operations as well.

User experience

When S3 is unhealthy, ramenctl will complete and create a report instead
of hanging. This also works better with AI agents, avoiding termination
by the harness in most cases.

Configuration

Add s3.timeout to the config file (seconds, default 30):

s3:
  timeout: 30

The validate and gather commands read it from the config. The test commands
use the default since the test config (ramen/e2e) has no S3 concept.

Zero and negative values are rejected during config validation.

Documentation

  • Add "Configuring S3 options" section to docs/init.md.
  • Add s3.timeout to the sample config with a comment explaining the unit.
  • Add s3.timeout to the example common configuration.

Security and privacy

No security or privacy impact. The timeout only affects connection timing.

Backward compatibility

Fully backward compatible. The default 30 second timeout is applied when
the config option is not set. Existing configs without the s3 section
work unchanged.

Performance

Improves reliability by preventing indefinite hangs. No negative
performance impact.

Testing

  • Config validation test for negative timeout.
  • Config equality test for S3 timeout.
  • Golden HTML/YAML testdata updated with the new config field.

Implementation plan

  1. Refactor s3 API to use an Options struct (like the gathering package),
    no behavior change.
  2. Add Timeout field to Options with a default, hardcoded inside s3.
  3. Make the timeout configurable via config, backends, docs.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions