Backend hardcodes V1 ListObjects: cannot start against V2-only stores

[doanbutar]

Component: tempodb/backend/s3
Affects: v3.0.0 through current main (line refs below are main)

What happened
Tempo refuses to start against an S3-compatible object store that
implements ListObjectsV2 but not the legacy V1 ListObjects
(log line wrapped for width):

level=error msg="error running Tempo"
err="failed to init module services: error initialising module:
store: failed to create store: unexpected error from
ListObjects on <bucket>: Operation ListObjects is not supported"

The store answers V1 with 400 UnsupportedOperation and V2 with 200.
Captured on the wire against the same bucket, same credentials, minutes
apart (response bodies are single-line; wrapped here for width):

GET /<bucket>?delimiter=%2F&max-keys=1&prefix= -> 400
<Error><Code>UnsupportedOperation</Code>
<Message>Operation ListObjects is not supported</Message>
<Resource>/<bucket></Resource>
<RequestId>016a4825-8236-4c0e-bf60-93a979868fff</RequestId></Error>

GET /<bucket>?delimiter=%2F&list-type=2&max-keys=1&prefix= -> 200
<ListBucketResult><IsTruncated>true</IsTruncated>
<Name><bucket></Name><Prefix/><Delimiter>/</Delimiter>
<MaxKeys>1</MaxKeys>
<CommonPrefixes><Prefix>fake/</Prefix></CommonPrefixes>...

The store is an Apple-internal S3-compatible service, not publicly
reachable, so I can't hand you a repro endpoint. Any store or mock that
returns non-2xx for a list request lacking list-type=2 reproduces it.

Where

The backend is inconsistent about which list API it uses. Two sites are
hardcoded V1:

• internalNew startup "try listing objects" check
  https://github.com/grafana/tempo/blob/main/tempodb/backend/s3/s3.go#L149
• List()
  https://github.com/grafana/tempo/blob/main/tempodb/backend/s3/s3.go#L378

Two already use V2:

• ListBlocks()
  https://github.com/grafana/tempo/blob/main/tempodb/backend/s3/s3.go#L439
• Find()
  https://github.com/grafana/tempo/blob/main/tempodb/backend/s3/s3.go#L522

// s3.go:147-153
// try listing objects
if confirm {
_, err = core.ListObjects(cfg.Bucket, cfg.Prefix, "", "/", 1)
if err != nil {
return nil, fmt.Errorf(
"unexpected error from ListObjects on %s: %w", cfg.Bucket, err)
}
}

minio.Core.ListObjects is the V1 marker-based API; Core.ListObjectsV2
is the separate V2 call. There is no config option to influence this —
the S3 Config struct has no list-version field on main.

Because the failing call sits in the confirm path of internalNew, it
fails at store initialisation, before any block is written or read.
There is no degraded mode and no way to skip it. List() would then
fail identically for tenant enumeration in the blocklist poller even if
startup were bypassed.

Relationship to #7517

PR #7517 (feat: add list_objects_version option to S3 backend, open)
addresses the mirror-image problem — stores whose V2 pagination is
broken (Ceph RGW pre-Nautilus, Dell ECS) — by adding an opt-in to V1,
default V2. It does not help here, and merging it as-is leaves this gap:

• its internalNew hunk only validates the new config value; the V1
  call at L149 is untouched
• its new version-aware listObjects helper is wired into ListBlocks
  and Find only; List() keeps calling V1 directly

So after #7517 there would be a list_objects_version setting that
still cannot make Tempo start against a V2-only store. The two issues
are complementary; I'd suggest fixing them together.

Suggested fix, in preference order

1. Don't gate startup on a list call. The confirm check needs
   "can I reach this bucket", not "can I enumerate it". HeadBucket
   expresses that with no list-API dependency and no s3:ListBucket
   permission requirement. Alternatively, log a warning instead of
   failing: a store that rejects the probe may still serve every call
   Tempo actually needs, which is the case here.
2. Route every list through one version-aware helper. Extend
   #7517's listObjects to cover List() and the startup check, so
   list_objects_version governs the whole backend rather than two of
   four call sites.
3. Minimal unblock: switch L149 and L378 to ListObjectsV2. AWS has
   recommended V2 for new development since 2016, and V1 stays reachable
   for the Ceph/ECS cases via #7517's flag — a cleaner split than
   today's hardcoded mix.

Fix 1 alone would have unblocked us, since only the startup probe is hit
in a single-binary deployment until the poller runs.

Impact
Tempo is unusable with V2-only S3 implementations, and the failure mode
is opaque: unexpected error from ListObjects reads like a permissions
or connectivity problem, so the actual cause — a deliberately
unimplemented legacy API — takes a packet capture to find. Meanwhile the
store satisfies every other requirement: HEAD, ranged GET, PUT,
DELETE, multipart upload, and ListObjectsV2, all verified against it.