Skip to content

[Feature Request]: Access GCS bucket with requester pays #30747

Open
@LeoCBS

Description

@LeoCBS

What would you like to happen?

Hello, is there a way to pass the requester pay project using textio.Read("gs://my-bucket")?

Apache beam is returning this error when i try accessing a bucket with this property active:

4: SDF.SplitAndSizeRestrictions[textio.readFn] UID:4 Out:[2] 3: PCollection[ne5_splitnsized] Out:[2] 2: DataSink[S[ee5_splitnsize_i0@localhost:43511]] Coder:W;cwv_ne5_splitnsized<KV;ce5kv_ele_rest_size_lp<KV;ce5kv_ele_rest_lp<LP;c3_lp<R[fileio.Readab leFile]>,KV;c7<offsetrange.Restriction[offsetrange.Restriction;c4];c5,bool;c6>>,double;ce5size>>!GWC caused by: DoFn[UID:10, PID:e3, Name: [github.com/apache/beam/sdks/v2/go/pkg/beam/io/fileio.matchFn](http://github.com/apache/beam/sdks/v2/go/pkg/beam/io/fileio.matchFn)] failed: googleapi: Error 400: Bucket is a requester pays bucket but no user project provided., required source=/go/pkg/mod/[github.com/apache/beam/sdks/v2@v2](http://github.com/apache/beam/sdks/v2@v2). 54.0/go/pkg/beam/core/runtime/harness/harness.go:492 time=2024-03-25T11:58:07.084Z worker.ID=job-001[go-job-1-1711367886775655910]_go worker.endpoint =localhost:43511

Thanks

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions