Skip to content

Remove data type support value error in copy from gcs function #1555

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sharinetmc merged 6 commits into from
Aug 15, 2025
Merged

Remove data type support value error in copy from gcs function #1555

Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
5f49726
remove error
sharinetmc Aug 14, 2025
b563fd7
remove param
sharinetmc Aug 14, 2025
a6fa8cd
add override_data_type_check
sharinetmc Aug 14, 2025
b3193be
ruff
sharinetmc Aug 14, 2025
b708281
add accepted data types param
sharinetmc Aug 15, 2025
7ca1e5b
remove mutable defualt
sharinetmc Aug 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions parsons/google/google_bigquery.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -456,7 +456,9 @@ def copy_from_gcs(
Other arguments to pass to the underlying load_table_from_uri
call on the BigQuery client.
"""
self._validate_copy_inputs(if_exists=if_exists, data_type=data_type)
self._validate_copy_inputs(
if_exists=if_exists, data_type=data_type, override_data_type_check=True
)

job_config = self._process_job_config(
job_config=job_config,
Expand Down Expand Up @@ -1559,13 +1561,15 @@ def _fetch_query_results(self, cursor) -> Table:
ptable = petl.frompickle(temp_filename)
return Table(ptable)

def _validate_copy_inputs(self, if_exists: str, data_type: str):
def _validate_copy_inputs(
self, if_exists: str, data_type: str, override_data_type_check: bool = False
):
if if_exists not in ["fail", "truncate", "append", "drop"]:
raise ValueError(
f"Unexpected value for if_exists: {if_exists}, must be one of "
'"append", "drop", "truncate", or "fail"'
)
if data_type not in ["csv", "json"]:
if data_type not in ["csv", "json"] and not override_data_type_check:
raise ValueError(f"Only supports csv or json files [data_type = {data_type}]")

def _load_table_from_uri(
Expand Down
Loading