-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Fix int overflow in Arrow text body buffer computation #9807
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| -- This file and its contents are licensed under the Timescale License. | ||
| -- Please see the included NOTICE for copyright information and | ||
| -- LICENSE-TIMESCALE for a copy of the license. | ||
| -- Tests for body buffer overflow in columnar_result_set_row with | ||
| -- DT_ArrowText type. When vectorized text functions produce large | ||
| -- results, the body buffer growth computation could overflow int32, | ||
| -- and the total buffer size can exceed MaxAllocSize. | ||
| set max_parallel_workers_per_gather to 0; | ||
| create table t_textoverflow (time timestamptz, a text); | ||
| select table_name from create_hypertable('t_textoverflow', 'time', | ||
| chunk_time_interval => interval '1 year'); | ||
| table_name | ||
| ---------------- | ||
| t_textoverflow | ||
|
|
||
| insert into t_textoverflow | ||
| select '2020-01-01'::timestamptz + i * interval '1 hour', | ||
| repeat('x', 100000) | ||
| from generate_series(1, 10) i; | ||
| alter table t_textoverflow set ( | ||
| timescaledb.compress, | ||
| timescaledb.compress_orderby = 'time' | ||
| ); | ||
| select count(compress_chunk(c)) from show_chunks('t_textoverflow') c; | ||
| count | ||
| ------- | ||
| 1 | ||
|
|
||
| set timescaledb.debug_require_vector_agg to 'require'; | ||
| select sum(length(a || a)) from t_textoverflow; | ||
| sum | ||
| --------- | ||
| 2000000 | ||
|
|
||
| reset timescaledb.debug_require_vector_agg; | ||
| drop table t_textoverflow cascade; | ||
| -- When the actual data doesn't fit into MaxAllocSize, produce an error message | ||
| -- instead of erroring out in palloc. | ||
| create table t_text_overflow (time timestamptz not null, text_col text not null); | ||
| select table_name from create_hypertable('t_text_overflow', 'time', | ||
| chunk_time_interval => interval '1 year'); | ||
| table_name | ||
| ----------------- | ||
| t_text_overflow | ||
|
|
||
| insert into t_text_overflow | ||
| select '2020-01-01'::timestamptz + i * interval '1 hour', 'a' | ||
| from generate_series(1, 1000) i; | ||
| alter table t_text_overflow set ( | ||
| timescaledb.compress, | ||
| timescaledb.compress_orderby = 'time' | ||
| ); | ||
| select count(compress_chunk(c)) from show_chunks('t_text_overflow') c; | ||
| count | ||
| ------- | ||
| 1 | ||
|
|
||
| set timescaledb.debug_require_vector_agg to 'require'; | ||
| select count(text_col || repeat('x', 2000000)) from t_text_overflow; | ||
| ERROR: vectorized text result exceeds 1073741823 bytes |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| -- This file and its contents are licensed under the Timescale License. | ||
| -- Please see the included NOTICE for copyright information and | ||
| -- LICENSE-TIMESCALE for a copy of the license. | ||
|
|
||
| -- Tests for body buffer overflow in columnar_result_set_row with | ||
| -- DT_ArrowText type. When vectorized text functions produce large | ||
| -- results, the body buffer growth computation could overflow int32, | ||
| -- and the total buffer size can exceed MaxAllocSize. | ||
|
|
||
| set max_parallel_workers_per_gather to 0; | ||
|
|
||
| create table t_textoverflow (time timestamptz, a text); | ||
|
|
||
| select table_name from create_hypertable('t_textoverflow', 'time', | ||
| chunk_time_interval => interval '1 year'); | ||
|
|
||
| insert into t_textoverflow | ||
| select '2020-01-01'::timestamptz + i * interval '1 hour', | ||
| repeat('x', 100000) | ||
| from generate_series(1, 10) i; | ||
|
|
||
| alter table t_textoverflow set ( | ||
| timescaledb.compress, | ||
| timescaledb.compress_orderby = 'time' | ||
| ); | ||
| select count(compress_chunk(c)) from show_chunks('t_textoverflow') c; | ||
|
|
||
| set timescaledb.debug_require_vector_agg to 'require'; | ||
| select sum(length(a || a)) from t_textoverflow; | ||
| reset timescaledb.debug_require_vector_agg; | ||
|
|
||
| drop table t_textoverflow cascade; | ||
|
|
||
| -- When the actual data doesn't fit into MaxAllocSize, produce an error message | ||
| -- instead of erroring out in palloc. | ||
| create table t_text_overflow (time timestamptz not null, text_col text not null); | ||
|
|
||
| select table_name from create_hypertable('t_text_overflow', 'time', | ||
| chunk_time_interval => interval '1 year'); | ||
|
|
||
| insert into t_text_overflow | ||
| select '2020-01-01'::timestamptz + i * interval '1 hour', 'a' | ||
| from generate_series(1, 1000) i; | ||
|
|
||
| alter table t_text_overflow set ( | ||
| timescaledb.compress, | ||
| timescaledb.compress_orderby = 'time' | ||
| ); | ||
| select count(compress_chunk(c)) from show_chunks('t_text_overflow') c; | ||
|
|
||
| set timescaledb.debug_require_vector_agg to 'require'; | ||
| select count(text_col || repeat('x', 2000000)) from t_text_overflow; | ||
| reset timescaledb.debug_require_vector_agg; | ||
|
|
||
| drop table t_text_overflow cascade; | ||
|
|
||
| reset max_parallel_workers_per_gather; | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.