Destination Snowflake: break down ensureSchemaMatches #69117
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. Helpful Resources
PR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
|
2 tasks
|
|
Note Detected that there are differences in the Gradle dependencies. |
edgao
force-pushed
the
edgao/schema_evolution_test/cdk
branch
from
745f4ca to
8e02044
Compare
edgao
force-pushed
the
edgao/schema_evolution_test/snowflake
branch
from
22fd34e to
28cf33f
Compare
edgao
force-pushed
the
edgao/schema_evolution_test/cdk
branch
from
e8a9d6a to
8f8d2b0
Compare
edgao
force-pushed
the
edgao/schema_evolution_test/snowflake
branch
from
28cf33f to
2969c0d
Compare
jdpgrailsdev
reviewed
Nov 4, 2025
| @@ -1,3 +1,3 @@ | |||
| testExecutionConcurrency=-1 | |||
| cdkVersion=0.1.62 | |||
| cdkVersion=local | |||
There was a problem hiding this comment.
Nit: I assume you don't want to commit this change
jdpgrailsdev
reviewed
Nov 4, 2025
| ) | ||
| } | ||
|
|
||
| override fun diff(actualSchemaInfo: Unit, expectedSchemaInfo: Unit) { |
There was a problem hiding this comment.
Why are the parameters marked as Unit? Is this to get around some generic issue with Kotlin and what we have type-wise in Snowflake?
There was a problem hiding this comment.
sort of. it's more of a hack to support certain destinations that need to pass around more info than just column+type - #69090 (comment)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
branched from #69090
How
There are a few major hacks here:
getColumnsFromStreamactually parses out the "NOT NULL", becauseSnowflakeColumnUtils.columnsAndTypesreturns strings likeNUMBER(38,0) NOT NULL. Probably would be nicer to have columnsAndTypes return the new ColumnType struct directly, instead of destination-snowflake's bespokeColumnAndType- but that would be a lot of code churn right nowColumnDefinition, b/c it wasn't actually used anywhere. But now ColumnDefinition is identical to ColumnAndType (which, again, we probably should delete entirely)...SqlGenerator.alterTablemanually filters out nullable -> non-nullable changes, b/c destination-snowflake actually tries to respect field nullability, but this alteration feels likely to cause problems. I could be convinced to just do the blind alter though.TODO unit tests are probably completely broken
Can this PR be safely reverted and rolled back?