Skip to content

Upgrade to Beam 2.72.0 #3557

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Abacn merged 23 commits into from
Apr 4, 2026
Merged

Upgrade to Beam 2.72.0 #3557

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
5c2e494
Update pom and requirements for 2.72.0 RC2
Amar3tto Mar 16, 2026
d59634f
Update google-ads version
Amar3tto Mar 17, 2026
09712eb
Update python requirements for 2.72.0 rc2
Amar3tto Mar 16, 2026
462f2a0
Use v23 google ads
Amar3tto Mar 17, 2026
d4d839e
Fix google-ads-to-googlecloud
Amar3tto Mar 18, 2026
fc3a09b
Fix assert
Amar3tto Mar 18, 2026
934854c
Update dataflow api version
Amar3tto Mar 19, 2026
8257035
Fix start_date_time
Amar3tto Mar 19, 2026
5378fef
Update spanner version
Amar3tto Mar 19, 2026
66ac0c5
Update spanner version
Amar3tto Mar 19, 2026
5f6a158
Remove expected exception text
Amar3tto Mar 19, 2026
4fcce6d
Fix expected exceptions
Amar3tto Mar 19, 2026
619bb0b
Use RC5
Amar3tto Mar 25, 2026
7044889
Merge 619bb0bf83ea79db9b01ff04d94170650bce9ac1 into 52f460a772452fc5f…
Amar3tto Mar 25, 2026
ce9cadc
Pin clickhouse test container to 1.20.3
Abacn Mar 25, 2026
bafe6eb
Unfork neo4j resource manager
Abacn Mar 25, 2026
b17c98e
Fork ClickHouseResourceManager
Abacn Mar 25, 2026
5956109
add driver dependency
Abacn Mar 26, 2026
d1fcfa3
fix clickhosue test containers credentials
BentsiLeviav Mar 26, 2026
fb2b4cb
Merge remote-tracking branch 'upstream/main' into pr-3505
Abacn Mar 26, 2026
aed4c43
Merge branch 'main' into pr-3505
Abacn Apr 3, 2026
e44cea2
Use Beam 2.72.0
Abacn Apr 3, 2026
d5df998
Remove intex-url
Abacn Apr 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion it/google-cloud-platform/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-spanner</artifactId>
<version>6.104.0</version>
<version>6.112.0</version>
</dependency>
<dependency>
<groupId>com.google.cloud</groupId>
Expand Down
2 changes: 1 addition & 1 deletion it/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
<awaitility.version>4.3.0</awaitility.version>
<commons-math3.version>3.6.1</commons-math3.version>
<parquet-avro.version>1.15.2</parquet-avro.version>
<dataflow-api.version>v1b3-rev20250519-2.0.0</dataflow-api.version>
<dataflow-api.version>v1b3-rev20260118-2.0.0</dataflow-api.version>
</properties>

<dependencies>
Expand Down
4 changes: 2 additions & 2 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,8 @@
<jacoco.version>0.8.13</jacoco.version>

<!-- Beam and linked versions -->
<beam.version>2.71.0</beam.version>
<beam-python.version>2.71.0</beam-python.version>
<beam.version>2.72.0</beam.version>
<beam-python.version>2.72.0</beam-python.version>
<beam-maven-repo></beam-maven-repo>

<!-- Common dependency versions -->
Expand Down
2 changes: 1 addition & 1 deletion python/default_base_bqmonitor_requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
apache-beam[gcp]==2.71.0
apache-beam[gcp]
google-cloud-bigquery-storage
setuptools
2 changes: 1 addition & 1 deletion python/default_base_python_requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
apache-beam[gcp]==2.71.0
apache-beam[gcp]
setuptools
2 changes: 1 addition & 1 deletion python/default_base_yaml_requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
apache-beam[dataframe,gcp,test,yaml]==2.71.0
apache-beam[dataframe,gcp,test,yaml]
setuptools
2 changes: 1 addition & 1 deletion python/src/main/python/bigquery-anomaly-detection/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,6 @@ version = "0.1.0"
description = "BigQuery anomaly monitoring pipeline (Dataflow Flex Template)"
requires-python = ">=3.11"
dependencies = [
"apache-beam[gcp]==2.71.0",
"apache-beam[gcp]==2.72.0",
"google-cloud-bigquery-storage",
]
1,577 changes: 843 additions & 734 deletions python/src/main/python/bigquery-anomaly-detection/requirements_all.txt
View file
Open in desktop

Large diffs are not rendered by default.

1,321 changes: 633 additions & 688 deletions python/src/main/python/job-builder-server/requirements.txt
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion python/src/main/python/streaming-llm/base_requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
apache-beam[gcp]==2.71.0
apache-beam[gcp]

@damccorm damccorm Sep 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that this doesn't work well. When there are several requirements, pip doesn't always resolve to the latest beam. For example,

DataflowTemplates/python/src/main/python/streaming-llm/requirements.txt

Line 176 in e31b279

apache-beam[gcp]==2.74.0 \
is on an old beam. This doesn't a matter much for this example template, but would be easy to miss in other instances.

I think we should add back version numbers here, proposing this in #4203

@Abacn Abacn Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This happens when your __build__ dir isn't fresh. Adding version is indeed safer, though need to change it every time on a version bump or rc validation

@damccorm damccorm Sep 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it is just that. We're getting this even when getting PRs generated by GitHub actions and I reproed on a fresh GitHub codespace - neither of these have preexisting __build__ directories

@damccorm damccorm Sep 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also note that it only failed to update streaming-llm in those passes, not the other generated requirements. This suggests that this can be caused by including certain dependencies (for now these are ml deps, but I don't think there's any guarantee that the same thing couldn't happen for other dependencies)

@damccorm damccorm Sep 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My guess is that pip resolves some of the other dependencies first, then picks a beam version which satisfies that; if the dependency (or one of its dependencies) is outside Beam's constraints, then we'll see this issue

@Abacn Abacn Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah it's possible. It's noticed some Beam's optional dependency started to depend on Beam: apache/beam#39884

torch
transformers
torchvision
Expand Down
4,565 changes: 2,546 additions & 2,019 deletions python/src/main/python/streaming-llm/requirements.txt
View file
Open in desktop

Large diffs are not rendered by default.

1,141 changes: 625 additions & 516 deletions python/src/main/python/word-count-python/requirements.txt
View file
Open in desktop

Large diffs are not rendered by default.

1,321 changes: 633 additions & 688 deletions python/src/main/python/yaml-template/requirements.txt
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion v1/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-spanner</artifactId>
<version>6.104.0</version>
<version>6.112.0</version>
</dependency>
<dependency>
<groupId>org.threeten</groupId>
Expand Down
2 changes: 1 addition & 1 deletion v2/google-ads-to-googlecloud/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
<artifactId>google-ads-to-googlecloud</artifactId>

<properties>
<google-ads.version>41.0.0</google-ads.version>
<google-ads.version>42.1.0</google-ads.version>
</properties>

<dependencies>
Expand Down
2 changes: 1 addition & 1 deletion ...googlecloud/src/main/java/com/google/cloud/teleport/v2/templates/GoogleAdsToBigQuery.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
*/
package com.google.cloud.teleport.v2.templates;

import com.google.ads.googleads.v19.services.GoogleAdsRow;
import com.google.ads.googleads.v23.services.GoogleAdsRow;
import com.google.cloud.teleport.metadata.Template;
import com.google.cloud.teleport.metadata.TemplateCategory;
import com.google.cloud.teleport.metadata.TemplateParameter;
Expand Down
2 changes: 1 addition & 1 deletion .../src/main/java/com/google/cloud/teleport/v2/transforms/GoogleAdsRowToReportRowJsonFn.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
package com.google.cloud.teleport.v2.transforms;

import com.google.ads.googleads.lib.utils.FieldMasks;
import com.google.ads.googleads.v19.services.GoogleAdsRow;
import com.google.ads.googleads.v23.services.GoogleAdsRow;
import com.google.cloud.teleport.v2.utils.GoogleAdsUtils;
import com.google.common.io.BaseEncoding;
import com.google.gson.Gson;
Expand Down
2 changes: 1 addition & 1 deletion ...oud/src/main/java/com/google/cloud/teleport/v2/utils/GoogleAdsRateLimitPolicyFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
*/
package com.google.cloud.teleport.v2.utils;

import com.google.ads.googleads.v19.errors.GoogleAdsError;
import com.google.ads.googleads.v23.errors.GoogleAdsError;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.util.concurrent.RateLimiter;
import com.google.protobuf.Message;
Expand Down
2 changes: 1 addition & 1 deletion ...e-ads-to-googlecloud/src/main/java/com/google/cloud/teleport/v2/utils/GoogleAdsUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
*/
package com.google.cloud.teleport.v2.utils;

import com.google.ads.googleads.v19.services.GoogleAdsRow;
import com.google.ads.googleads.v23.services.GoogleAdsRow;
import com.google.api.services.bigquery.model.TableSchema;
import com.google.common.base.Functions;
import com.google.common.base.Splitter;
Expand Down
12 changes: 6 additions & 6 deletions .../test/java/com/google/cloud/teleport/v2/transforms/GoogleAdsRowToReportRowJsonFnTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@
*/
package com.google.cloud.teleport.v2.transforms;

import com.google.ads.googleads.v19.enums.CampaignStatusEnum.CampaignStatus;
import com.google.ads.googleads.v19.resources.Campaign;
import com.google.ads.googleads.v19.services.GoogleAdsRow;
import com.google.ads.googleads.v23.enums.CampaignStatusEnum.CampaignStatus;
import com.google.ads.googleads.v23.resources.Campaign;
import com.google.ads.googleads.v23.services.GoogleAdsRow;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
Expand All @@ -35,14 +35,14 @@ public class GoogleAdsRowToReportRowJsonFnTest {
@Test
public void testRowToJsonWithValidInput() {
String query =
"SELECT campaign.id, campaign.name, campaign.start_date, campaign.status FROM campaign";
"SELECT campaign.id, campaign.name, campaign.start_date_time, campaign.status FROM campaign";
GoogleAdsRow row =
GoogleAdsRow.newBuilder()
.setCampaign(
Campaign.newBuilder()
.setId(1234567890L)
.setName("foo")
.setStartDate("1970-01-01")
.setStartDateTime("1970-01-01 12:00:00")
.setStatus(CampaignStatus.ENABLED))
.build();

Expand All @@ -52,7 +52,7 @@ public void testRowToJsonWithValidInput() {
.apply(ParDo.of(new GoogleAdsRowToReportRowJsonFn(query)));

String expected =
"{\"campaign_id\":1234567890,\"campaign_name\":\"foo\",\"campaign_start_date\":\"1970-01-01\",\"campaign_status\":\"ENABLED\"}";
"{\"campaign_id\":1234567890,\"campaign_name\":\"foo\",\"campaign_start_date_time\":\"1970-01-01 12:00:00\",\"campaign_status\":\"ENABLED\"}";
PAssert.that(actual).containsInAnyOrder(expected);
pipeline.run();
}
Expand Down
Loading
Loading