Kopia backup repository ignores BSL s3ForcePathStyle config - virtual-hosted-style S3 addressing not supported

[mpryc]

What steps did you take and what happened:

When a BackupStorageLocation is configured with s3ForcePathStyle: "false" and a custom s3Url endpoint, the kopia backup repository still uses path-style S3 addressing. The s3ForcePathStyle BSL config value is accepted by Velero but is never propagated to kopia's S3 storage backend - it is silently ignored.

1. Configure a BSL with a custom S3-compatible endpoint that requires virtual-hosted-style addressing:

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: example
spec:
  provider: aws
  objectStorage:
    bucket: my-bucket
    prefix: velero
  config:
    region: us-east-1
    s3Url: https://s3.example.com
    s3ForcePathStyle: "false"

2. Create a backup that triggers kopia backup repository initialization (e.g., a backup with PVC volumes using the kopia uploader).

3. Observe the BackupRepository object shows a path-style resticIdentifier and kopia uses path-style URLs to connect:

spec:
  resticIdentifier: s3:https://s3.example.com/my-bucket/velero/restic/my-namespace
  repositoryType: kopia
status:
  message: >-
    error to connect to backup repo: error to connect to storage:
    error retrieving storage config from bucket "my-bucket":
    Get "https://s3.example.com/my-bucket/velero/kopia/my-namespace/.storageconfig":
    dial tcp: lookup s3.example.com on 10.x.x.x:53: no such host
  phase: NotReady

The URL https://s3.example.com/my-bucket/... is path-style. With s3ForcePathStyle: "false", kopia should have used virtual-hosted-style: https://my-bucket.s3.example.com/....

Root cause analysis — three gaps in the code path:

Gap 1: No StoreOption constant defined

pkg/repository/udmrepo/repo_options.go defines S3-related store option constants (lines 40-46) but there is no StoreOptionS3ForcePathStyle:

StoreOptionS3KeyID            = "accessKeyID"
StoreOptionS3Provider         = "providerName"
StoreOptionS3SecretKey        = "secretAccessKey"
StoreOptionS3Token            = "sessionToken"
StoreOptionS3Endpoint         = "endpoint"
StoreOptionS3DisableTLS       = "doNotUseTLS"
StoreOptionS3DisableTLSVerify = "skipTLSVerify"
// No StoreOptionS3ForcePathStyle constant exists

Gap 2: getStorageVariables() never reads s3ForcePathStyle from BSL config

pkg/repository/provider/unified_repo.go, function getStorageVariables() (lines 544-644) maps BSL config entries to internal store options for the AWS backend. It maps s3Url, insecureSkipTLSVerify, region, etc., but never reads config["s3ForcePathStyle"]:

// Lines 597-599 — maps some config entries but not s3ForcePathStyle
result[udmrepo.StoreOptionS3Endpoint] = strings.Trim(s3URL, "/")
result[udmrepo.StoreOptionS3DisableTLSVerify] = config["insecureSkipTLSVerify"]
result[udmrepo.StoreOptionS3DisableTLS] = strconv.FormatBool(disableTLS)
// config["s3ForcePathStyle"] is never read

Gap 3: Kopia S3 backend Setup() never sets path-style option

pkg/repository/udmrepo/kopialib/backend/s3.go, function Setup() (lines 35-57) populates s3.Options fields but never sets a path-style/virtual-hosted-style field:

func (c *S3Backend) Setup(ctx context.Context, flags map[string]string, logger logrus.FieldLogger) error {
    c.options.BucketName, err = mustHaveString(udmrepo.StoreOptionOssBucket, flags)
    c.options.AccessKeyID = optionalHaveString(udmrepo.StoreOptionS3KeyID, flags)
    c.options.Endpoint = optionalHaveString(udmrepo.StoreOptionS3Endpoint, flags)
    // ... other options ...
    // Never sets any path-style / virtual-hosted-style option
}

Note: The Velero AWS plugin correctly handles s3ForcePathStyle for BSL validation and object store operations (backup metadata, etc.). Only the kopia backup repository code path is affected, creating inconsistent behavior where BSL operations succeed but backup repository operations fail.

What did you expect to happen:

When s3ForcePathStyle is set to "false" in the BSL config, kopia should use virtual-hosted-style S3 addressing (https://<bucket>.<endpoint>/<key>) when connecting to the backup repository, consistent with how the Velero AWS plugin handles this config for object store operations.

The following information will help us better understand what's going on:

This is a code-level bug — the plumbing to pass s3ForcePathStyle from BSL config to kopia's S3 backend was never implemented. No specific debug bundle is needed; the issue is reproducible by inspecting the source code and confirmed by the BackupRepository status error shown above.

Anything else you would like to add:

Upstream kopia dependency

Kopia itself currently does not expose BucketLookupDNS from the underlying minio-go SDK to its s3.Options struct. The minio-go SDK hardcodes path-style addressing for any S3 provider it doesn't explicitly recognize (AWS, GCP, Aliyun). This is tracked upstream:

• kopia/kopia#3902 — Need support for virtual-hosted-style buckets
• kopia/kopia#3734 — Related S3 addressing issue
• kopia/kopia#4462 — PR adding --virtual-hosted-style-requests parameter (currently OPEN)

Proposed Velero fix (once kopia exposes the option)

1. pkg/repository/udmrepo/repo_options.go — Add new constant:

   StoreOptionS3ForcePathStyle = "s3ForcePathStyle"

2. pkg/repository/provider/unified_repo.go in getStorageVariables() — Read from BSL config and pass through (after line 599):

   result[udmrepo.StoreOptionS3ForcePathStyle] = config["s3ForcePathStyle"]

3. pkg/repository/udmrepo/kopialib/backend/s3.go in Setup() — Read the flag and set the kopia s3.Options path-style field accordingly.

Broader impact

• Any S3-compatible provider that requires virtual-hosted-style addressing cannot be used as a kopia backup repository target.
• AWS itself is deprecating path-style URLs for general purpose buckets, making this a forward-looking issue for all providers.
• The same issue affects restic, tracked separately in velero#5102.

Related issues

• velero#5102 — Restic daemonset cannot support virtual-hosted-style (same root cause, restic side)
• kopia/kopia#3902 — Kopia needs virtual-hosted-style bucket support
• kopia/kopia#4462 — Kopia PR to add the option (OPEN)
• kopia/kopia#3734 — Related kopia S3 addressing issue

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[Lyndon-Li]

It is a known issue that Kopia doesn't support virtual-hosted-style, see #8739. Not sure when the related Kopia PRs could be merged, but once it is merged, we can go ahead to do the corresponding changes in Velero Unified Repo.

[shubham-pampattiwar]

This is a known gap. The fix requires kopia to expose a BucketLookup option on its s3.Options struct, which it currently doesn't. There's an open upstream PR for this: kopia/kopia#4462 (kopia/kopia#4462), but it's been waiting on maintainer review since April 2025.

I have commented on the upstream PR to try to move it along. Once that lands and we pick it up, the Velero-side changes are straightforward the gaps you identified are the exact plumbing we'll need to add.

If the upstream PR remains stalled, we can evaluate alternative approaches. Related: #8739 (closed), velero#5102 (#5102 side).