feat(gcp_cloud_storage sink): support Parquet batch encoding

[dshmatov]

Summary

Adds Apache Parquet columnar batch encoding to the gcp_cloud_storage sink, matching the capability the aws_s3 sink already has. Enable it with batch_encoding.codec = "parquet".

When batch_encoding is set, events are encoded together as a batch into a single Parquet file per batch (instead of the standard per-event, framing-based encoding). Parquet handles its own compression internally, so:

• the top-level compression setting is bypassed (a warning is logged if it was set),
• the object Content-Type defaults to application/vnd.apache.parquet,
• the filename extension defaults to parquet.

This reuses the shared codec/batch-encoder infrastructure introduced for aws_s3 (the BatchEncoder / BatchSerializerConfig / EncoderKind machinery in lib/codecs and src/sinks/util). The change is purely the sink-side wiring plus a GcsBatchEncoding config enum — no new encoding logic.

Vector configuration

sources:
  demo:
    type: demo_logs
    format: json

sinks:
  gcs_out:
    type: gcp_cloud_storage
    inputs: [demo]
    bucket: my-bucket
    encoding:
      codec: text
    batch_encoding:
      codec: parquet
      schema_mode: auto_infer
      compression:
        algorithm: snappy

Schema-file mode is also supported:

    batch_encoding:
      codec: parquet
      schema_mode: strict
      schema_file: /etc/vector/schema.parquet
      compression:
        algorithm: zstd
        level: 10

How did you test this PR?

• cargo test --features "sinks-gcp,codecs-parquet" --lib gcp::cloud_storage — 9 new Parquet tests + the 15 existing GCS sink tests pass.
  • Config-level: TOML/YAML shape, schema_mode parsing/defaults, content-type auto-detection + user override, .parquet extension default + override, top-level-compression bypass, rejection of non-parquet codecs.
  • End-to-end encoding: parquet_encodes_valid_file runs a real batch through the sink's request builder and asserts the output is a valid Parquet file (PAR1 magic bytes, correct row count, inferred message/host columns) using the parquet reader.
• Verified the crate compiles with codecs-parquet both enabled and disabled (the feature-gated paths).
• make check-clippy, make fmt, and make check-generated-docs (regenerated gcp_cloud_storage.cue) pass.

Note: this repo currently has no GCS-cloud-storage integration-test harness (no fake-gcs-server service under scripts/integration/), so Parquet output is validated at the unit level through the real encode path rather than against a live backend. Happy to add an integration harness if maintainers would prefer one.

Change Type

• Bug fix
• New feature
• Dependencies
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

• Builds on the shared Parquet codec infrastructure from fix(releasing): enable codecs-parquet in all release feature sets #25321.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[dshmatov]

I have read the CLA Document and I hereby sign the CLA

[drichards-87]

Left a couple of suggestions from Docs and approved the PR.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2a4a585b76

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[akukhar]

Hi, I wanted to follow up on this PR. It would help fix the limitation I'm facing, and I'd really like to use it upstream.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c642a5160c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Watch schema_file for batch encoding reloads

When batch_encoding is configured with a Parquet schema_file, the serializer reads that file once during sink construction, but GcsSinkConfig still relies on the default empty files_to_watch() implementation. In deployments using config watching, edits to the schema file therefore won't trigger a sink rebuild, so GCS keeps writing with the old schema until Vector is restarted; return the configured Parquet schema path from files_to_watch() when present.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Default the unused event encoding in batch mode

For Parquet configs that omit the legacy encoding block, this new batch mode still fails deserialization because the existing flattened encoding field remains required even though the request builder ignores its serializer and framing whenever batch_encoding is set. Users have to add a dummy encoding.codec just to start the sink, so either default or bypass the normal encoding config for batch mode, or explicitly validate/document that placeholder requirement.

Useful? React with 👍 / 👎.