Skip to content

Validate strings are under 16mb instead of 128mb #67608

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Validate strings are under 16mb instead of 128mb #67608

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a8fc41f
16mb instead of 128
tryangul Oct 10, 2025
97e6fc8
Update PR link.
tryangul Oct 10, 2025
ec1f341
Imporve unit tests. Properly test nulled out long strings.
tryangul Oct 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/destination-snowflake/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ data:
connectorSubtype: database
connectorType: destination
definitionId: 424892c4-daac-4491-b35d-c6688ba547ba
dockerImageTag: 4.0.8
dockerImageTag: 4.0.9
dockerRepository: airbyte/destination-snowflake
documentationUrl: https://docs.airbyte.com/integrations/destinations/snowflake
githubIssueLabel: destination-snowflake
Expand Down
4 changes: 2 additions & 2 deletions ...in/io/airbyte/integrations/destination/snowflake/write/transform/SnowflakeValueCoercer.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,9 @@ internal val FLOAT_MIN = BigDecimal("-9007199254740991")
internal val FLOAT_RANGE = FLOAT_MIN..FLOAT_MAX

// https://docs.snowflake.com/en/sql-reference/data-types-text#varchar
internal const val VARCHAR_AND_VARIANT_LIMIT_BYTES = 134217728 // 128 MB
internal const val VARCHAR_AND_VARIANT_LIMIT_BYTES = 16777216 // 16 MB
// UTF-8 has max 4 bytes per char, so anything under this length is safe
internal const val MAX_UTF_8_STRING_LENGTH_UNDER_LIMIT = 33554432 // (134217728 / 4)
internal const val MAX_UTF_8_STRING_LENGTH_UNDER_LIMIT = 4194304 // (16777216 / 4)

fun isValid(value: AirbyteValue): Boolean {
return when (value) {
Expand Down
32 changes: 26 additions & 6 deletions ...o/airbyte/integrations/destination/snowflake/write/transform/SnowflakeValueCoercerTest.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -531,7 +531,7 @@ internal class SnowflakeValueCoercerTest {
}

@Test
fun testStringJustUnderSizeLimit() {
fun testStringAtExactSizeLimit() {
val largeString = StringValue("a".repeat(VARCHAR_AND_VARIANT_LIMIT_BYTES))
val airbyteValue =
EnrichedAirbyteValue(
Expand All @@ -547,8 +547,8 @@ internal class SnowflakeValueCoercerTest {
}

@Test
fun testStringAtExactSizeLimit() {
// Test string at exactly the 33554432 character limit
fun testStringOverSizeLimit() {
// Test string at exactly the 16777216-character limit
val exactLimitString = StringValue("a".repeat(VARCHAR_AND_VARIANT_LIMIT_BYTES + 1))
val airbyteValue =
EnrichedAirbyteValue(
Expand All @@ -561,7 +561,7 @@ internal class SnowflakeValueCoercerTest {

// This should still be valid as each 'a' is 1 byte
val result = coercer.validate(airbyteValue)
assertEquals(airbyteValue, result)
assertEquals(NullValue, result.abValue)
}

@Test
Expand Down Expand Up @@ -700,8 +700,8 @@ internal class SnowflakeValueCoercerTest {
}

@Test
fun testStringWithMultiByteCharactersNearLimit() {
// Test string with multi-byte UTF-8 characters
fun testStringWithMultiByteCharactersAtLimit() {
// Test string with multibyte UTF-8 characters
// Each emoji is 4 bytes, so we need fewer characters to hit the limit
val multiByteCount = MAX_UTF_8_STRING_LENGTH_UNDER_LIMIT
val emojiString = StringValue("🎉".repeat(multiByteCount))
Expand All @@ -718,4 +718,24 @@ internal class SnowflakeValueCoercerTest {
val result = coercer.validate(airbyteValue)
assertEquals(airbyteValue, result)
}

@Test
fun testStringWithMultiByteCharactersOverLimit() {
// Test string with multibyte UTF-8 characters
// Each emoji is 4 bytes, so we need fewer characters to hit the limit
val multiByteCount = MAX_UTF_8_STRING_LENGTH_UNDER_LIMIT
val emojiString = StringValue("🎉".repeat(multiByteCount) + 'a')

val airbyteValue =
EnrichedAirbyteValue(
abValue = emojiString,
type = StringType,
name = "emoji_string",
changes = mutableListOf(),
airbyteMetaField = null,
)

val result = coercer.validate(airbyteValue)
assertEquals(NullValue, result.abValue)
}
}
1 change: 1 addition & 0 deletions docs/integrations/destinations/snowflake.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -251,6 +251,7 @@ desired namespace.

| Version | Date | Pull Request | Subject |
|:----------------|:-----------|:-----------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 4.0.9 | 2025-10-10 | [67608](https://github.com/airbytehq/airbyte/pull/67608) | Validate strings against correct length. |
| 4.0.8 | 2025-10-09 | [67599](https://github.com/airbytehq/airbyte/pull/67599) | Improve handling of heavily interleaved streams. |
| 4.0.7 | 2025-10-09 | [67590](https://github.com/airbytehq/airbyte/pull/67590) | Use GZIP compression level 5 to improve performance. No longer explicitly `CREATE FILE FORMAT`. |
| 4.0.6 | 2025-10-09 | [67582](https://github.com/airbytehq/airbyte/pull/67582) | Schema evolution supports dropping columns with lowercase characters in their name. |
Expand Down
Loading