-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Add hypertable on/off oracle for LLM fuzzer #10295
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 9 commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
ea0124a
Add hypertable on/off oracle for LLM fuzzer
akuzm 6bac177
oops
akuzm ba4c180
look here
akuzm 08dea22
centralize options?
akuzm 453e580
remove probe error output
akuzm 2e6fb26
now look in a different place
akuzm be6c602
try another place
akuzm 558ec23
try another place
akuzm aab7530
partition chunk
akuzm 7ff4039
Merge remote-tracking branch 'origin/main' into aku/hyper-oracle
akuzm 3b1aa34
cleanup
akuzm File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
143 changes: 143 additions & 0 deletions
143
.github/workflows/llm-fuzzer/oracle/hypertable/verify.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,143 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| # This oracle runs the reproducer script with the psql variable :hyper set | ||
| # to false and true, and compares the output. The script passes the variable | ||
| # into CREATE TABLE ... WITH (tsdb.hypertable = :hyper, ...), so the same | ||
| # statements run against a plain Postgres table and a TimescaleDB | ||
| # hypertable. The admissible bug must lead to a change in script output | ||
| # between the two. | ||
| # | ||
| # An admissible repro script: | ||
| # | ||
| # Must be runnable on same database multiple times in sequence. | ||
| # | ||
| # Must run exactly the same statements no matter if it runs against the | ||
| # plain table or the hypertable. | ||
| # | ||
| # | ||
| # Must not use the psql meta-commands. | ||
| # | ||
| # Must not require superuser privileges. | ||
| # | ||
| # The output of an admissible repro script: | ||
| # | ||
| # Must be sufficiently ordered to prevent false positives (i.e. ORDER BY, no | ||
| # ties). | ||
| # | ||
| # Must not depend on floating point precision or numeric stability. | ||
| # | ||
| # Must be independent from arbitrary environmental influence like the OID values | ||
| # or chunk identifiers. | ||
| # | ||
| # Must not change when the script runs on the same database multiple times in | ||
| # sequence. | ||
| set -eu | ||
|
|
||
| psql <<<'alter database :"DBNAME" set client_min_messages to error' | ||
|
|
||
| if ! psql -v hyper=false -f "$1" > result_plain.txt | ||
| then | ||
| echo "Repro errors out, not admissible" | ||
| exit 0 | ||
| fi | ||
|
|
||
| if psql -v hyper=maybe -f "$1" &> result_probe.txt | ||
| then | ||
| echo "Repro does not use :hyper for tsdb.hypertable, not admissible" | ||
| exit 0 | ||
| fi | ||
|
|
||
| if ! psql -v hyper=off -f "$1" > result_plain_synonym.txt | ||
| then | ||
| echo "Repro errors out, not admissible" | ||
| exit 0 | ||
| fi | ||
|
|
||
|
|
||
| if ! psql -v hyper=true -f "$1" > result_hyper.txt | ||
| then | ||
| echo "Repro errors out, not admissible" | ||
| exit 0 | ||
| fi | ||
|
|
||
| if ! psql -v hyper=true -c "set enable_seqscan to off;" -f "$1" > result_hyper_noseq.txt | ||
| then | ||
| echo "Repro errors out, not admissible" | ||
| exit 0 | ||
| fi | ||
|
|
||
| if ! psql -v hyper=true -c "set enable_indexscan to off;" -f "$1" > result_hyper_noindex.txt | ||
| then | ||
| echo "Repro errors out, not admissible" | ||
| exit 0 | ||
| fi | ||
|
|
||
| if ! psql -v hyper=true -c "set enable_hashagg to off;" -f "$1" > result_hyper_nohashagg.txt | ||
| then | ||
| echo "Repro errors out, not admissible" | ||
| exit 0 | ||
| fi | ||
|
|
||
| if ! psql -v hyper=true -c " | ||
| set max_parallel_workers_per_gather = 8; | ||
| set parallel_setup_cost = 0; | ||
| set parallel_tuple_cost = 0; | ||
| set min_parallel_table_scan_size = 0; | ||
| set min_parallel_index_scan_size = 0; | ||
| " -f "$1" > result_hyper_para.txt | ||
| then | ||
| echo "Repro errors out, not admissible" | ||
| exit 0 | ||
| fi | ||
|
|
||
| if ! psql -v hyper=true -c "set work_mem = '4GB'" -f "$1" > result_hyper_mem.txt | ||
| then | ||
| echo "Repro errors out, not admissible" | ||
| exit 0 | ||
| fi | ||
|
|
||
| if ! psql -v hyper=true -c " | ||
| set enable_hashagg to off; | ||
| set enable_sort to off; | ||
| set timescaledb.enable_chunkwise_aggregation to off; | ||
| " -f "$1" > result_hyper_rowsort.txt 2> result_hyper_rowsort.err | ||
| then | ||
| echo "Repro errors out, not admissible" | ||
| exit 0 | ||
| fi | ||
|
|
||
| if ! diff -u result_plain.txt result_plain_synonym.txt \ | ||
| || ! diff -u result_hyper.txt result_hyper_noseq.txt \ | ||
| || ! diff -u result_hyper.txt result_hyper_noindex.txt \ | ||
| || ! diff -u result_hyper.txt result_hyper_nohashagg.txt \ | ||
| || ! diff -u result_hyper.txt result_hyper_para.txt \ | ||
| || ! diff -u result_hyper.txt result_hyper_mem.txt \ | ||
| || ! diff -u result_hyper.txt result_hyper_rowsort.txt | ||
| then | ||
| echo "Repro gives different results between runs, not admissible" | ||
| exit 0 | ||
| fi | ||
|
|
||
| echo | ||
| echo '```diff' | ||
| echo | ||
|
|
||
| if diff -u result_plain.txt result_hyper.txt | ||
| then | ||
| result=0 | ||
| else | ||
| result=$? | ||
| fi | ||
|
|
||
| echo | ||
| echo '```' | ||
| echo | ||
|
|
||
| if [ "${result}" -eq 0 ] | ||
| then | ||
| echo "Same result with hypertable false/true, error not reproduced" | ||
| exit 0 | ||
| fi | ||
|
|
||
| echo "Reproduced" | ||
| exit 1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is this? 🤨
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Testing the fuzzer :) I'm marking locations with known problems and checking what it finds.