Skip to content

Commit ac18b48

Browse files
LeoCBSLeoCBS
committed
Merge branch 'apache:master' into master
2 parents 28f5727 + 8c7ed02 commit ac18b48

File tree

373 files changed

+7461
-4718
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

373 files changed

+7461
-4718
lines changed

‎.asf.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@ github:
5151

5252
protected_branches:
5353
master: {}
54+
release-2.65.0-postrelease: {}
5455
release-2.65: {}
5556
release-2.64.0-postrelease: {}
5657
release-2.64: {}

‎.github/gh-actions-self-hosted-runners/arc/images/Dockerfile

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -25,10 +25,10 @@ RUN docker buildx install && docker buildx version
2525

2626
USER root
2727
#Install Node
28-
RUN curl -OL https://nodejs.org/dist/v18.16.0/node-v18.16.0-linux-x64.tar.xz && \
29-
tar -C /usr/local -xf node-v18.16.0-linux-x64.tar.xz && \
30-
rm node-v18.16.0-linux-x64.tar.xz && \
31-
mv /usr/local/node-v18.16.0-linux-x64 /usr/local/node
28+
RUN curl -OL https://nodejs.org/dist/v22.14.0/node-v22.14.0-linux-x64.tar.xz && \
29+
tar -C /usr/local -xf node-v22.14.0-linux-x64.tar.xz && \
30+
rm node-v22.14.0-linux-x64.tar.xz && \
31+
mv /usr/local/node-v22.14.0-linux-x64 /usr/local/node
3232
ENV PATH="${PATH}:/usr/local/node/bin"
3333
#Install Go
3434
ARG go_version=1.24.0

‎.github/trigger_files/beam_PostCommit_Java_Nexmark_Dataflow.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
{
2-
"modification": 1
2+
"modification": 2
33
}

‎.github/trigger_files/beam_PostCommit_Python_Xlang_IO_Direct.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
{
22
"comment": "Modify this file in a trivial way to cause this test suite to run",
3-
"modification": 3
3+
"modification": 4
44
}

‎.github/workflows/beam_Inference_Python_Benchmarks_Dataflow.yml

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,9 +84,33 @@ jobs:
8484
${{ github.workspace }}/.github/workflows/load-tests-pipeline-options/beam_Inference_Python_Benchmarks_Dataflow_Pytorch_Language_Modeling_Bert_Base_Uncased.txt
8585
${{ github.workspace }}/.github/workflows/load-tests-pipeline-options/beam_Inference_Python_Benchmarks_Dataflow_Pytorch_Language_Modeling_Bert_Large_Uncased.txt
8686
${{ github.workspace }}/.github/workflows/load-tests-pipeline-options/beam_Inference_Python_Benchmarks_Dataflow_Pytorch_Imagenet_Classification_Resnet_152_Tesla_T4_GPU.txt
87+
${{ github.workspace }}/.github/workflows/load-tests-pipeline-options/beam_Inference_Python_Benchmarks_Dataflow_Pytorch_Sentiment_Streaming_DistilBert_Base_Uncased.txt
88+
${{ github.workspace }}/.github/workflows/load-tests-pipeline-options/beam_Inference_Python_Benchmarks_Dataflow_Pytorch_Sentiment_Batch_DistilBert_Base_Uncased.txt
8789
# The env variables are created and populated in the test-arguments-action as "<github.job>_test_arguments_<argument_file_paths_index>"
8890
- name: get current time
8991
run: echo "NOW_UTC=$(date '+%m%d%H%M%S' --utc)" >> $GITHUB_ENV
92+
- name: run Pytorch Sentiment Streaming using Hugging Face distilbert-base-uncased model
93+
uses: ./.github/actions/gradle-command-self-hosted-action
94+
timeout-minutes: 180
95+
with:
96+
gradle-command: :sdks:python:apache_beam:testing:load_tests:run
97+
arguments: |
98+
-PloadTest.mainClass=apache_beam.testing.benchmarks.inference.pytorch_sentiment_benchmarks \
99+
-Prunner=DataflowRunner \
100+
-PpythonVersion=3.10 \
101+
-PloadTest.requirementsTxtFile=apache_beam/ml/inference/torch_tests_requirements.txt \
102+
'-PloadTest.args=${{ env.beam_Inference_Python_Benchmarks_Dataflow_test_arguments_6 }} --mode=streaming --job_name=benchmark-tests-pytorch-sentiment-distilbert-base-uncased-streaming-${{env.NOW_UTC}} --output_table=apache-beam-testing.beam_run_inference.result_sentiment_distilbert_base_uncased' \
103+
- name: run Pytorch Sentiment Batch using Hugging Face distilbert-base-uncased model
104+
uses: ./.github/actions/gradle-command-self-hosted-action
105+
timeout-minutes: 180
106+
with:
107+
gradle-command: :sdks:python:apache_beam:testing:load_tests:run
108+
arguments: |
109+
-PloadTest.mainClass=apache_beam.testing.benchmarks.inference.pytorch_sentiment_benchmarks \
110+
-Prunner=DataflowRunner \
111+
-PpythonVersion=3.10 \
112+
-PloadTest.requirementsTxtFile=apache_beam/ml/inference/torch_tests_requirements.txt \
113+
'-PloadTest.args=${{ env.beam_Inference_Python_Benchmarks_Dataflow_test_arguments_7 }} --mode=batch --job_name=benchmark-tests-pytorch-sentiment-distilbert-base-uncased-batch-${{env.NOW_UTC}} --output_table=apache-beam-testing.beam_run_inference.result_sentiment_distilbert_base_uncased' \
90114
- name: run Pytorch Vision Classification with Resnet 101
91115
uses: ./.github/actions/gradle-command-self-hosted-action
92116
timeout-minutes: 180

‎.github/workflows/beam_PostCommit_Java_Examples_Dataflow_V2.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ jobs:
8686
- name: run PostCommit Java Distroless Examples Dataflow V2 script
8787
uses: ./.github/actions/gradle-command-self-hosted-action
8888
with:
89-
gradle-command: :runners:google-cloud-dataflow-java:examplesJavaDistrolessRunnerV2PreCommit
89+
gradle-command: :runners:google-cloud-dataflow-java:examplesJavaRunnerV2IntegrationTestDistroless
9090
max-workers: 12
9191
- name: Archive JUnit Test Results
9292
uses: actions/upload-artifact@v4

‎.github/workflows/beam_PreCommit_Java_Spark3_Versions.yml

Lines changed: 0 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -87,19 +87,12 @@ jobs:
8787
github_job: ${{ matrix.job_name }} (${{ matrix.job_phrase }})
8888
- name: Setup environment
8989
uses: ./.github/actions/setup-environment-action
90-
with:
91-
java-version: |
92-
8
93-
11
94-
# TODO(https://github.com/apache/beam/issues/32207) Run test with Java11
9590
- name: run sparkVersionsTest script
9691
uses: ./.github/actions/gradle-command-self-hosted-action
9792
with:
9893
gradle-command: :runners:spark:3:sparkVersionsTest
9994
arguments: |
10095
-PdisableSpotlessCheck=true \
101-
-PtestJavaVersion=8 \
102-
-Pjava8Home=$JAVA_HOME_8_X64 \
10396
- name: Archive JUnit Test Results
10497
uses: actions/upload-artifact@v4
10598
if: ${{ !success() }}

‎.github/workflows/load-tests-pipeline-options/beam_Inference_Python_Benchmarks_Dataflow_Pytorch_Sentiment_Batch_DistilBert_Base_Uncased.txt

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
--region=us-central1
18+
--machine_type=n1-standard-2
19+
--num_workers=20
20+
--max_num_workers=250
21+
--disk_size_gb=50
22+
--autoscaling_algorithm=THROUGHPUT_BASED
23+
--staging_location=gs://temp-storage-for-perf-tests/loadtests
24+
--temp_location=gs://temp-storage-for-perf-tests/loadtests
25+
--requirements_file=apache_beam/ml/inference/torch_tests_requirements.txt
26+
--publish_to_big_query=true
27+
--metrics_dataset=beam_run_inference
28+
--metrics_table=torch_sentiment_distilbert_base_uncased_batch
29+
--input_options={}
30+
--influx_measurement=torch_sentiment_distilbert_base_uncased
31+
--device=CPU
32+
--input_file=gs://apache-beam-ml/testing/inputs/sentences_50k.txt
33+
--runner=DataflowRunner
34+
--model_path=distilbert-base-uncased-finetuned-sst-2-english
35+
--model_state_dict_path=gs://apache-beam-ml/models/huggingface.sentiment.distilbert-base-uncased.pth

‎.github/workflows/load-tests-pipeline-options/beam_Inference_Python_Benchmarks_Dataflow_Pytorch_Sentiment_Streaming_DistilBert_Base_Uncased.txt

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
--region=us-central1
18+
--machine_type=n1-standard-2
19+
--num_workers=20
20+
--max_num_workers=250
21+
--disk_size_gb=50
22+
--autoscaling_algorithm=THROUGHPUT_BASED
23+
--staging_location=gs://temp-storage-for-perf-tests/loadtests
24+
--temp_location=gs://temp-storage-for-perf-tests/loadtests
25+
--requirements_file=apache_beam/ml/inference/torch_tests_requirements.txt
26+
--publish_to_big_query=true
27+
--metrics_dataset=beam_run_inference
28+
--metrics_table=torch_sentiment_distilbert_base_uncased
29+
--input_options={}
30+
--influx_measurement=torch_sentiment_distilbert_base_uncased
31+
--device=CPU
32+
--input_file=gs://apache-beam-ml/testing/inputs/sentences_50k.txt
33+
--runner=DataflowRunner
34+
--model_path=distilbert-base-uncased-finetuned-sst-2-english
35+
--model_state_dict_path=gs://apache-beam-ml/models/huggingface.sentiment.distilbert-base-uncased.pth

‎.github/workflows/republish_released_docker_containers.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ on:
3232
- cron: "0 6 * * 1"
3333
env:
3434
docker_registry: gcr.io
35-
release: "${{ github.event.inputs.RELEASE || '2.64.0' }}"
35+
release: "${{ github.event.inputs.RELEASE || '2.65.0' }}"
3636
rc: "${{ github.event.inputs.RC || '2' }}"
3737

3838
jobs:

‎.pre-commit-config.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,10 @@
1111
# limitations under the License.
1212

1313
repos:
14-
- repo: https://github.com/pre-commit/mirrors-yapf
14+
- repo: https://github.com/google/yapf
1515
# this rev is a release tag in the repo above and corresponds with a yapf
1616
# version. make sure this matches the version of yapf in tox.ini.
17-
rev: v0.29.0
17+
rev: v0.43.0
1818
hooks:
1919
- id: yapf
2020
files: ^sdks/python/apache_beam/

‎.test-infra/tools/refresh_looker_metrics.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,8 @@
4040
("78", ["243", "244", "245", "246", "247"]), # PyTorch Resnet 101
4141
("79", ["248", "249", "250", "251", "252"]), # PyTorch Resnet 152
4242
("80", ["253", "254", "255", "256", "257"]), # PyTorch Resnet 152 Tesla T4
43+
("82", ["263", "264", "265", "266", "267"]), # PyTorch Sentiment Streaming DistilBERT base uncased
44+
("85", ["268", "269", "270", "271", "272"]), # PyTorch Sentiment Batch DistilBERT base uncased
4345
]
4446

4547

‎CHANGES.md

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,11 +76,16 @@
7676
## New Features / Improvements
7777
* Adding Google Storage Requests Pays feature (Golang)([#30747](https://github.com/apache/beam/issues/30747)).
7878
* X feature added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).
79+
* [Python] Prism runner now auto-enabled for some Python pipelines using the direct runner ([#34921](https://github.com/apache/beam/pull/34921)).
7980
* [YAML] WriteToTFRecord and ReadFromTFRecord Beam YAML support
81+
* Python: Added JupyterLab 4.x extension compatibility for enhanced notebook integration ([#34495](https://github.com/apache/beam/pull/34495)).
8082

8183
## Breaking Changes
8284

8385
* X behavior was changed ([#X](https://github.com/apache/beam/issues/X)).
86+
* Yapf version upgraded to 0.43.0 for formatting (Python) ([#34801](https://github.com/apache/beam/pull/34801/)).
87+
* Python: Added JupyterLab 4.x extension compatibility for enhanced notebook integration ([#34495](https://github.com/apache/beam/pull/34495)).
88+
* Python: Argument abbreviation is no longer enabled within Beam. If you previously abbreviated arguments (e.g. `--r` for `--runner`), you will now need to specify the whole argument ([#34934](https://github.com/apache/beam/pull/34934)).
8489

8590
## Deprecations
8691

@@ -98,7 +103,7 @@
98103
[comment]: # ( When updating known issues after release, make sure also update website blog in website/www/site/content/blog.)
99104
* ([#X](https://github.com/apache/beam/issues/X)).
100105

101-
# [2.65.0] - Unreleased
106+
# [2.65.0] - 2025-05-12
102107

103108
## I/Os
104109

@@ -111,6 +116,10 @@
111116

112117
## Breaking Changes
113118

119+
* [Python] Cloudpickle is set as the default `pickle_library`, where previously
120+
dill was the default in [#34695](https://github.com/apache/beam/pull/34695).
121+
For known issues, reporting new issues, and understanding cloudpickle
122+
behavior refer to [#34903](https://github.com/apache/beam/issues/34903).
114123
* [Python] Reshuffle now preserves PaneInfo, where previously PaneInfo was lost
115124
after reshuffle. To opt out of this change, set the
116125
update_compatibility_version to a previous Beam version e.g. "2.64.0".
@@ -246,6 +255,7 @@ N/A
246255
## Known Issues
247256

248257
* (Java) Current version of protobuf has a [bug](https://github.com/protocolbuffers/protobuf/issues/20599) leading to incompatibilities with clients using older versions of Protobuf ([example issue](https://github.com/GoogleCloudPlatform/DataflowTemplates/issues/2191)). This issue has been seen in SpannerIO in particular. Tracked in [#34452](https://github.com/GoogleCloudPlatform/DataflowTemplates/issues/34452).
258+
* (Java) Coders that extend CustomCoder (excluding SchemaCoder and RowCoder) are update incompatible with previous versions of the same transform (Java) on some runners. This includes PubSubIO's read ([#34933](https://github.com/apache/beam/issues/34933)).
249259

250260
# [2.62.0] - 2025-01-21
251261

‎build.gradle.kts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -571,6 +571,7 @@ tasks.register("communityMetricsProber") {
571571
tasks.register("javaExamplesDataflowPrecommit") {
572572
dependsOn(":runners:google-cloud-dataflow-java:examples:preCommit")
573573
dependsOn(":runners:google-cloud-dataflow-java:examples-streaming:preCommit")
574+
dependsOn(":runners:google-cloud-dataflow-java:examplesJavaDistrolessRunnerV2PreCommit")
574575
dependsOn(":runners:google-cloud-dataflow-java:examplesJavaRunnerV2PreCommit")
575576
}
576577

0 commit comments

Comments
 (0)