Bugfix: s3_file_to_parquet
function doesn't override content encoding if provided with the override kwarg
#505
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This change no longer raises ContentTypeValidationError from
s3_file_to_parquet
if appropriately overriddenRationale
-Although the "Allow overriding content encoding" bugfix
was merged into DeltaCAT and released in 1.1.30, the overriding does not properly happen during the compaction session which results in in a raised ContentTypeValidationError
when the underlying file is gzip encoded - even if the
override_content_encoding_for_parquet
is provided.Changes
s3_file_to_parquet
from no coverage including the default caseImpact
Testing
Regression Risk
override_content_encoding_for_parquet
kwarg is providedChecklist
Unit tests covering the changes have been added
E2E testing has been performed
Additional Notes
Any additional information or context relevant to this PR.