Skip to content

docs: Document schema preservation behavior on SaveMode.Overwrite (#1456) #1457

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yalimu-g merged 4 commits into from
Jan 23, 2026
Merged

docs: Document schema preservation behavior on SaveMode.Overwrite (#1456) #1457

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
56a5879
Updated the readme for better understanding of overwrite behaviour
sahilkumarsingh Jan 10, 2026
b1f8f84
Update README-template.md
sahilkumarsingh Jan 10, 2026
a6a8f69
added more info on schema changes
sahilkumarsingh Jan 23, 2026
2c2c581
fix: Remove trailing whitespace from README-template.md
sahilkumarsingh Jan 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions README-template.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -373,6 +373,31 @@ df.writeStream \

**Important:** The connector does not configure the GCS connector, in order to avoid conflict with another GCS connector, if exists. In order to use the write capabilities of the connector, please configure the GCS connector on your cluster as explained [here](https://github.com/GoogleCloudPlatform/bigdata-interop/tree/master/gcs).

#### Schema Behavior on Overwrite

When using `SaveMode.Overwrite` (`.mode("overwrite")`), the connector **preserves the existing table's schema**.
The data is truncated, but column types, descriptions, and policy tags are retained.

```
df.write \
.format("bigquery") \
.mode("overwrite") \
.option("temporaryGcsBucket","some-bucket") \
.save("dataset.table")
```

**Important:** If your DataFrame has a different schema than the existing table (e.g., changing a column from
`INTEGER` to `DOUBLE`), the write will fail with a type mismatch error. To change the schema, either:
- Drop the table before overwriting
- Use BigQuery DDL to alter the table schema first

Copy link
Contributor

@yalimu-g yalimu-g Jan 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For some of the schema difference the following options can work with overwrite:
Programmatic Relaxation: Set .option("allowFieldRelaxation", "true") for nullability changes and .option("allowFieldAddition", "true") for new columns.

Copy link
Contributor Author

@sahilkumarsingh sahilkumarsingh Jan 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we can add this detail. Shall I add it?

Copy link
Contributor

@yalimu-g yalimu-g Jan 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes please

Copy link
Contributor Author

@sahilkumarsingh sahilkumarsingh Jan 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, please check.

For some of the schema difference, the following options can work with overwrite:
Programmatic Relaxation: Set `.option("allowFieldRelaxation", "true")` for nullability changes and `.option("allowFieldAddition", "true")` for new columns.

This behavior was introduced between version 0.22.0 and 0.41.0 to prevent accidental schema drift.

**Note:** This behavior applies to both the `indirect` (default) and `direct` write methods.

### Running SQL on BigQuery

The connector supports Spark's [SparkSession#executeCommand](https://archive.apache.org/dist/spark/docs/3.0.0/api/java/org/apache/spark/sql/SparkSession.html#executeCommand-java.lang.String-java.lang.String-scala.collection.immutable.Map-)
Expand Down
Loading