Fix(athena): Case sensitivity in CTAS property names #4171
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
erindru
commented
Sep 27, 2024
| @@ -132,6 +132,7 @@ class Generator(Trino.Generator): | |||
| TRANSFORMS = { | |||
| **Trino.Generator.TRANSFORMS, | |||
| exp.FileFormatProperty: lambda self, e: f"format={self.sql(e, 'this')}", | |||
| exp.PartitionedByProperty: lambda self, e: f"partitioned_by={self.sql(e, 'this')}", | |||
There was a problem hiding this comment.
Note that the actual reason for the uppercasing is because the Properties.NAME_TO_PROPERTY map hardcodes everything to uppercase.
Since we already had a transform to lowercase exp.FileFormatProperty, I added one for exp.PartitionedByProperty since most of the properties in Properties.NAME_TO_PROPERTY arent relevant for Athena
VaggelisD
approved these changes
Sep 27, 2024
| @@ -62,8 +62,12 @@ def test_ddl(self): | |||
|
|
|||
| # CTAS goes to the Trino engine, where the table properties cant be encased in single quotes like they can for Hive | |||
| # ref: https://docs.aws.amazon.com/athena/latest/ug/create-table-as.html#ctas-table-properties | |||
| # They're also case sentitive and need to be lowercase, otherwise you get eg "Table properties [FORMAT] are not supported." | |||
There was a problem hiding this comment.
Small typo on "sensitive".
A personal nit, feel free to ignore: I'd consider merging this comment with the first one, I think this would read a bit better e.g.
CTAS goes to the Trino engine, where the table properties:
- Must not be single quoted (Hive case)
- Must be generated in lower case as their resolution is case-sensitive
ref: ....
georgesittas
approved these changes
Sep 27, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It turns out Athena table properties are case sensitive on the Trino engine and need to be lowercase.
Fails with
Table properties [PARTITIONED_BY] are not supported.Succeeds