feat: sql jobs outputting to s3 + streaming for high-number of rows #5704

diegoimbert · 2025-05-05T17:43:02Z

Add an annotation to stream the results of an SQL query to S3:

-- s3 prefix=bigqueries storage=secondary1 format=csv
SELECT * FROM mytable;

All arguments are optional
Supported formats: csv, json and parquet.

I did not use the annotations macro from @pyranota because it only works as a flag and does not parse arguments (maybe a future improvement to make?)

Important

Adds functionality to stream SQL query results to S3 with support for multiple formats and integrates this feature across various SQL executors.

Behavior:
- Adds support for streaming SQL query results to S3 with optional parameters for prefix, storage, and format in windmill-parser-sql/src/lib.rs.
- Supports csv, json, and parquet formats.
- Integrates with S3 for data storage using s3_helpers.rs.
SQL Execution:
- Updates bigquery_executor.rs, mssql_executor.rs, mysql_executor.rs, pg_executor.rs, and snowflake_executor.rs to handle S3 streaming.
- Parses S3 mode annotations and streams results to S3 if specified.
Dependencies:
- Adds dependencies on datafusion, tokio-stream, and tokio-util in Cargo.toml files.
Misc:
- Introduces S3ModeWorkerData struct in common.rs for managing S3 upload configurations.
- Implements upload_s3_file method in worker.rs for handling S3 uploads.

^{This description was created by}^{for cde77fe. You can customize this summary. It will automatically update as commits are pushed.}

cloudflare-workers-and-pages · 2025-05-05T17:46:40Z

Deploying windmill with Cloudflare Pages

Latest commit:	`58d5bd8`
Status:	✅ Deploy successful!
Preview URL:	https://6329338a.windmill.pages.dev
Branch Preview URL:	https://di-stream-sql-to-s3.windmill.pages.dev

View logs

ellipsis-dev

Caution

Changes requested ❌

Reviewed everything up to 3a58cb8 in 3 minutes and 11 seconds. Click for details.

Reviewed 1278 lines of code in 12 files
Skipped 0 files when reviewing.
Skipped posting 16 draft comments. View those below.
Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. backend/windmill-worker/src/worker.rs:768

Draft comment:
The run_worker loop is very long and has deeply nested and complex logic. Consider refactoring into smaller helper functions to improve readability and maintainability.
Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.

2. backend/windmill-worker/src/worker.rs:669

Draft comment:
Avoid hardcoding system paths such as '/proc/sys/vm/drop_caches'. Prefer making such paths configurable or using a dedicated abstraction so that the code works under varied environments and proper privileges can be managed.
Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.

3. backend/windmill-worker/src/worker.rs:1010

Draft comment:
There are several unwraps (or expect calls) in metrics registration and similar parts. In production code these can lead to panics when the expected environment assumptions (e.g. metric registration) are not met. Consider using proper error propagation instead.
Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.

4. backend/windmill-worker/src/worker.rs:2920

Draft comment:
The language‐specific job handling in handle_code_execution_job is very long. Consider moving each language branch into separate modules or helper functions to reduce the size of the match block and improve modularity.
Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.

5. backend/windmill-worker/src/worker.rs:1270

Draft comment:
The handling of same_worker jobs via try_recv and subsequent branching is quite intricate. Abstracting this logic into a dedicated helper function would make it easier to follow and test.
Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.

6. backend/windmill-worker/src/mssql_executor.rs:480

Draft comment:
When using S3 mode (branch starting at line 204), ensure that errors from the async stream are properly handled and that any temporary resources (like files) are cleaned up even on error.
Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.

7. backend/windmill-worker/src/pg_executor.rs:137

Draft comment:
When extracting column order from query results (lines 135–144), use explicit error handling instead of unwrap_or_default to catch cases of missing schema information.
Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.

8. backend/windmill-worker/src/worker.rs:80

Draft comment:
The run_worker function is very long and handles many responsibilities (job pulling, metrics, logging, scheduling, etc.). Consider refactoring it into smaller, modular components to improve readability and maintainability.
Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.

9. backend/windmill-worker/src/mysql_executor.rs:340

Draft comment:
The conversion functions (e.g. string_date_to_mysql_date) use unwrap_or_default() which may mask parsing errors. Consider returning a proper error when an input date string is invalid instead of defaulting silently.
Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.

10. backend/windmill-worker/src/worker.rs:387

Draft comment:
In AuthedClient::get_resource_value and similar methods, the response header values are constructed using unwrap_or defaults. Consider more robust error handling to avoid silently passing invalid header or token values.
Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.

11. backend/windmill-common/src/s3_helpers.rs:497

Draft comment:
Typo: In the comment above the RecordBatchWriterEnum (line 497), consider changing 'Originally used a Arc<Mutex<dyn RecordBatchWriter + Send>>' to 'Originally used an Arc<Mutex<dyn RecordBatchWriter + Send>>' for proper grammar.
Reason this comment was not posted:
Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 0% vs. threshold = 50% While the grammar correction is technically correct, comments about pure grammar/spelling in comments are generally not important enough to warrant a PR comment. The meaning is perfectly clear either way. This falls under the "Do NOT make comments that are obvious or unimportant" rule. The grammar error could theoretically make the code less professional or harder to maintain, and fixing it is very low effort. The impact on maintainability is negligible - developers will understand the meaning perfectly well either way. We should focus PR comments on actual code issues. Delete this comment as it's too minor and doesn't affect code functionality or maintainability in any meaningful way.

12. backend/windmill-worker/Cargo.toml:37

Draft comment:
Typo in comment on line 37: Consider replacing 'audit-worth actions' with 'audit-worthy actions' for clarity.
Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.

13. backend/windmill-worker/src/bigquery_executor.rs:43

Draft comment:
The struct name 'BigQueryResponseJobReference' uses a different capitalization (with an uppercase 'Q') compared to the other related structs (e.g. 'BigqueryResponse', 'BigqueryResponseRow', etc.). For consistency and clarity, consider aligning the capitalization (either use 'Bigquery' or 'BigQuery' consistently across the file).
Reason this comment was not posted:
Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 20% vs. threshold = 50% While this is technically correct about the inconsistency, it's a very minor code style issue that doesn't affect functionality. The comment is about a newly added struct in the diff, so it is about changes. The inconsistency isn't resolved yet. However, this kind of minor style issue may not be worth a PR comment - it's not clearly wrong enough to require a change. The inconsistency could cause confusion for future developers trying to understand the code structure. However, both "BigQuery" and "Bigquery" are commonly used variations of the product name. While consistency is good, this is an extremely minor issue that doesn't impact functionality or significantly hurt readability. The effort of changing it may not be worth the benefit. Delete this comment as it's too minor of a style issue to warrant a PR comment. The inconsistency, while real, doesn't significantly impact code quality or maintainability.

14. backend/windmill-worker/src/pg_executor.rs:90

Draft comment:
Typo: The error message 'Missing otzyp for pg arg' seems to be a misspelling. Consider correcting it to 'Missing otyp for pg arg' or a more accurate message.
Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.

15. backend/windmill-worker/src/pg_executor.rs:158

Draft comment:
Typo: In the error message comparing query size, the code uses 'MAX_RESULT_SIZE & 4' which likely is a typo. It should be 'MAX_RESULT_SIZE * 4' to match the condition check.
Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.

16. backend/windmill-worker/src/worker.rs:683

Draft comment:
Typographical error: The word 'spurrious' should be corrected to 'spurious' in the error message within the drop_cache function.
Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.

Workflow ID: wflow_frBtlmPAupSm1nFJ

^{You can customize}^{by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}

ellipsis-dev · 2025-05-09T16:51:52Z

backend/windmill-worker/src/snowflake_executor.rs

+                // let rows_stream =
+                //     rows_stream.map(|r| serde_json::value::to_value(&r?).map_err(to_anyhow));
+                // let stream = convert_json_line_stream(rows_stream.boxed(), s3.format).await?;
+                // TODO fix this


There is commented-out S3 streaming code with a TODO (lines 247–251). Please either complete the implementation or remove the commented code to avoid confusion.

Suggested change

// TODO fix this

ellipsis-dev · 2025-05-09T16:51:52Z

backend/windmill-worker/src/mysql_executor.rs

+            };
+
+            let stream = convert_json_line_stream(rows_stream.boxed(), s3.format).await?;
+            s3.upload(stream.boxed()).await?;


In the S3 branch (starting around line 77), consider adding more detailed logging and stronger error messages when executing query streaming, so that S3 upload failures can be diagnosed more easily.

i yield error in stream and everything is awaited and checked with ? operator

ellipsis-dev · 2025-05-09T16:51:52Z

backend/parsers/windmill-parser-sql/src/lib.rs

    static ref RE_NONEMPTY_SQL_BLOCK: Regex = Regex::new(r#"(?m)^\s*[^\s](?:[^-]|$)"#).unwrap();

    static ref RE_DB: Regex = Regex::new(r#"(?m)^-- database (\S+) *(?:\r|\n|$)"#).unwrap();
+    static ref RE_S3_MODE: Regex = Regex::new(r#"(?m)^-- s3( (.+))?*(?:\r|\n|$)"#).unwrap();


The regex for RE_S3_MODE is defined as:

Regex::new(r"(?m)^-- s3( (.+))?*(?:\r|\n|$)")

The * after the optional group appears redundant and may be a typographical error. Consider revising this pattern (e.g., possibly using (?m)^-- s3( (.+))?(?:\r|\n|$) instead) to ensure it behaves as expected.

Suggested change

static ref RE_S3_MODE: Regex = Regex::new(r#"(?m)^-- s3( (.+))?*(?:\r|\n|$)"#).unwrap();

static ref RE_S3_MODE: Regex = Regex::new(r#"(?m)^-- s3( (.+))?(?:\r|\n|$)"#).unwrap();

ellipsis-dev

Important

Looks good to me! 👍

Reviewed 8b87ae4 in 1 minute and 45 seconds. Click for details.

Reviewed 66 lines of code in 1 files
Skipped 0 files when reviewing.
Skipped posting 8 draft comments. View those below.
Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. backend/windmill-worker/src/snowflake_executor.rs:202

Draft comment:
The cloning of account_identifier and token for use with reqwest’s stream is acceptable, though be mindful of the overhead. The inline comment explains the necessity; consider whether documenting the potential performance implications for very high-throughput streaming might help future maintainers.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

2. backend/windmill-worker/src/snowflake_executor.rs:244

Draft comment:
The map_ok closure now uses a move lambda to capture its environment. Make sure that all captured variables (e.g. response.resultSetMetaData.rowType) remain valid throughout the stream’s lifetime.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

3. backend/windmill-worker/src/snowflake_executor.rs:254

Draft comment:
The previously commented-out code for S3 uploading has now been uncommented and integrated. Verify that convert_json_line_stream handles all expected JSON formatting edge cases and that error propagation from s3.upload is properly handled.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

4. backend/windmill-worker/src/snowflake_executor.rs:12

Draft comment:
Good to see the new import for the S3 stream conversion helper. Ensure that its behavior is well-tested and aligns with other s3_helpers utilities.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

5. backend/windmill-worker/src/snowflake_executor.rs:199

Draft comment:
Cloning the account_identifier and token to satisfy the 'static lifetime needed by reqwest::Body::wrap_stream is a workable workaround. In the future, consider using Arc to avoid multiple clones if these strings become large.
Reason this comment was not posted:
Confidence changes required: 33% <= threshold 50% None

6. backend/windmill-worker/src/snowflake_executor.rs:217

Draft comment:
The updated URL builder now uses the cloned_account_identifier and cloned_token consistently for partition requests. This correctly addresses the lifetime issue for async streaming.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

7. backend/windmill-worker/src/snowflake_executor.rs:233

Draft comment:
Adding the 'move' keyword to the map_ok closure ensures that the captured environment (including response fields) is moved into the closure. Verify that this does not hold on to heavier state than needed.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

8. backend/windmill-worker/src/snowflake_executor.rs:252

Draft comment:
The previously commented-out S3 upload block is now active. This code maps each row to a JSON value, converts the stream using convert_json_line_stream, and uploads the stream to S3. Ensure that any errors during conversion or upload are logged and handled appropriately.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

Workflow ID: wflow_RblBPwpK3tEDcw5o

^{You can customize}^{by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}

ellipsis-dev

Important

Looks good to me! 👍

Reviewed 83906ea in 1 minute and 40 seconds. Click for details.

Reviewed 142 lines of code in 1 files
Skipped 0 files when reviewing.
Skipped posting 10 draft comments. View those below.
Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. backend/windmill-common/src/s3_helpers.rs:1

Draft comment:
Imports have been re-ordered and duplicated under #[cfg(feature = "parquet")]. Consider consolidating these to avoid redundant declarations.
Reason this comment was not posted:
Confidence changes required: 30% <= threshold 50% None

2. backend/windmill-common/src/s3_helpers.rs:24

Draft comment:
The new ChannelWriter implementation using blocking_send is acceptable but be cautious when integrating synchronous blocking operations in async contexts.
Reason this comment was not posted:
Confidence changes required: 30% <= threshold 50% None

3. backend/windmill-common/src/s3_helpers.rs:562

Draft comment:
The two versions of convert_json_line_stream (#[cfg(not(feature = "parquet"))] and #[cfg(feature = "parquet")]) share the same name. Ensure usage is clear and documented to prevent confusion.
Reason this comment was not posted:
Confidence changes required: 40% <= threshold 50% None

4. backend/windmill-common/src/s3_helpers.rs:537

Draft comment:
Removal of the duplicate ChannelWriter definition is good; please ensure no lingering unused code paths remain.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

5. backend/windmill-common/src/s3_helpers.rs:1

Draft comment:
Good cleanup: the imports for 'to_anyhow' and 'rd_string' are now conditionally included under the 'parquet' feature. This ensures they’re only imported when needed.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

6. backend/windmill-common/src/s3_helpers.rs:19

Draft comment:
Combining 'Arc' and 'Mutex' into a single import is clearer and follows best practices.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

7. backend/windmill-common/src/s3_helpers.rs:508

Draft comment:
Defining RecordBatchWriterEnum under #[cfg(feature = "parquet")] limits its scope appropriately to the parquet feature.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

8. backend/windmill-common/src/s3_helpers.rs:534

Draft comment:
The new ChannelWriter struct and its Write implementation are well organized. Note that using blocking_send is acceptable in a blocking context—but be mindful if the channel saturates.
Reason this comment was not posted:
Confidence changes required: 33% <= threshold 50% None

9. backend/windmill-common/src/s3_helpers.rs:560

Draft comment:
Adding a #[cfg(not(feature = "parquet"))] version of convert_json_line_stream that yields an error clearly informs callers when the parquet feature isn’t enabled. Ensure that any caller gracefully handles this error stream.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

10. backend/windmill-common/src/s3_helpers.rs:638

Draft comment:
Removal of the duplicate ChannelWriter definition at the end reduces maintenance overhead and avoids potential inconsistencies.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

Workflow ID: wflow_F2OvSs6bJNqVPovm

^{You can customize}^{by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}

graphite-app · 2025-05-12T08:45:32Z

backend/windmill-worker/src/mysql_executor.rs

+                    Ok(mut result) => {
+                        while let Some(row) = result.next().await? {
+                            yield Ok(convert_row_to_value(row));
+                        }


The error handling in this stream needs improvement. The current implementation uses result.next().await? which will return early from the async block if an error occurs, but won't properly propagate the error through the stream.

Consider restructuring to handle errors explicitly:

while let Some(row_result) = result.next().await { match row_result { Ok(row) => yield Ok(convert_row_to_value(row)), Err(e) => yield Err(anyhow!("Error fetching row: {:?}", e)) } }

This ensures errors are properly propagated through the stream rather than terminating it prematurely.

Suggested change

Ok(mut result) => {

while let Some(row) = result.next().await? {

yield Ok(convert_row_to_value(row));

}

Ok(mut result) => {

while let Some(row_result) = result.next().await {

match row_result {

Ok(row) => yield Ok(convert_row_to_value(row)),

Err(e) => yield Err(anyhow!("Error fetching row: {:?}", e))

}

}

Spotted by Diamond

Is this helpful? React 👍 or 👎 to let us know.

ellipsis-dev

Important

Looks good to me! 👍

Reviewed 05283a8 in 1 minute and 39 seconds. Click for details.

Reviewed 26 lines of code in 2 files
Skipped 0 files when reviewing.
Skipped posting 5 draft comments. View those below.
Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. backend/windmill-worker/src/bigquery_executor.rs:4

Draft comment:
Removed TryFutureExt from the futures import. Verify that no methods from TryFutureExt are used elsewhere to avoid breaking changes.
Reason this comment was not posted:
Comment did not seem useful. Confidence is useful = 0% <= threshold 50% The comment is asking the PR author to verify that no methods from TryFutureExt are used elsewhere, which is against the rules as it asks for confirmation and verification. It does not provide a specific suggestion or point out a specific issue.

2. backend/windmill-worker/src/snowflake_executor.rs:5

Draft comment:
Removed TryFutureExt from the futures import. Ensure that removal doesn't affect any needed extensions; the use of TryStreamExt remains for stream processing.
Reason this comment was not posted:
Comment did not seem useful. Confidence is useful = 30% <= threshold 50% The comment is asking the author to ensure that the removal of TryFutureExt doesn't affect any needed extensions. This is a request for confirmation, which violates the rule against asking the PR author to confirm their intention or ensure behavior. However, it does provide a specific context about the removal of TryFutureExt and the continued use of TryStreamExt, which could be useful. Despite this, the primary focus is on ensuring the removal doesn't affect anything, which is not allowed.

3. backend/windmill-worker/src/bigquery_executor.rs:3

Draft comment:
Removed the unused import TryFutureExt; this cleans up dependency clutter. Please double-check that no methods from this trait are needed elsewhere.
Reason this comment was not posted:
Comment did not seem useful. Confidence is useful = 0% <= threshold 50% The comment asks the PR author to double-check that no methods from the removed import are needed elsewhere. This violates the rule against asking the author to double-check things. The comment does not provide a specific suggestion or point out a specific issue, so it should be removed.

4. backend/windmill-worker/src/snowflake_executor.rs:4

Draft comment:
Removed the unused import TryFutureExt while keeping TryStreamExt, which helps with stream error handling. This appears to be a tidy cleanup—ensure there is no dependency on TryFutureExt methods.
Reason this comment was not posted:
Comment looked like it was already resolved.

5. backend/windmill-worker/src/bigquery_executor.rs:32

Draft comment:
There is an inconsistency in the casing used in the names of the BigQuery-related structs. For example, the file defines BigqueryResponse (with a lowercase 'q') but then defines BigQueryResponseJobReference (with an uppercase 'Q'). For clarity and consistency, consider standardizing the spelling (either using 'Bigquery' or 'BigQuery') across the file.
Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.

Workflow ID: wflow_r90pF8ihZ07hKHfN

^{You can customize}^{by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}

ellipsis-dev

Important

Looks good to me! 👍

Reviewed cde77fe in 1 minute and 17 seconds. Click for details.

Reviewed 39 lines of code in 1 files
Skipped 0 files when reviewing.
Skipped posting 3 draft comments. View those below.
Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. backend/windmill-worker/src/mysql_executor.rs:80

Draft comment:
Returning immediately after yielding an error is clear; ensure downstream consumers handle early termination properly.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

2. backend/windmill-worker/src/mysql_executor.rs:88

Draft comment:
The loop over rows with match is explicit but consider if a while let pattern could simplify error handling.
Reason this comment was not posted:
Confidence changes required: 33% <= threshold 50% None

3. backend/windmill-worker/src/mysql_executor.rs:80

Draft comment:
Holding the connection lock for the entire streaming loop may lead to contention; verify that this is acceptable under load.
Reason this comment was not posted:
Confidence changes required: 33% <= threshold 50% None

Workflow ID: wflow_hwgPcuxsXOMUB4QX

^{You can customize}^{by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}

diegoimbert added 2 commits May 5, 2025 14:55

stream to s3 boilerplate

ea8e976

S3 works with new syntax

f29110e

diegoimbert force-pushed the di/stream-sql-to-s3 branch from 3d509a1 to f29110e Compare May 5, 2025 17:46

diegoimbert added 5 commits May 5, 2025 19:56

snowflake s3 streaming support

beba862

postgres s3 support

f7e36bf

fix postgres stream format

b6e9269

mysql s3 streaming

ed0a7a3

mssql s3 streaming

62df7af

diegoimbert force-pushed the di/stream-sql-to-s3 branch from 095178e to 62df7af Compare May 6, 2025 12:49

diegoimbert added 19 commits May 6, 2025 17:40

new s3 mode syntax

92c8925

optional folder param

6ebaf1c

rename folder to prefix

794de6b

json_stream_arr_values

b86d51b

cargo toml rollback

8990be9

convert_ndjson with datafusion

d1d4fc0

format conversion kinda works

b8c1243

Fixed not finishing the datafusion writer

dc8e041

support for pg and mssql

1d29f54

fix file ext

6b4e8bc

bigquery conversion and works with s3 streaming

8bdf4dd

fix s3 flag parser

a576f55

snowflake s3 streaming support

7609263

factor out duplicate code

2895221

remove anyhow

755b423

Err case for parse s3 mode

7440173

Send error to mpsc

967b839

bigquery s3 streaming fix for huge queries

1b261ce

remove extra stuff

3a58cb8

diegoimbert force-pushed the di/stream-sql-to-s3 branch from 84ecc8d to 3a58cb8 Compare May 9, 2025 16:46

diegoimbert marked this pull request as ready for review May 9, 2025 16:48

diegoimbert requested review from alpetric, hugocasa and rubenfiszel as code owners May 9, 2025 16:48

ellipsis-dev bot reviewed May 9, 2025

View reviewed changes

snowflake s3 streaming support

8b87ae4

diegoimbert force-pushed the di/stream-sql-to-s3 branch from 3bd2d74 to 8b87ae4 Compare May 9, 2025 17:05

ellipsis-dev bot reviewed May 9, 2025

View reviewed changes

diegoimbert added 2 commits May 9, 2025 19:09

small regex mistake

d582557

cfg(not(feature = "parquet"))

83906ea

ellipsis-dev bot reviewed May 9, 2025

View reviewed changes

diegoimbert and others added 2 commits May 9, 2025 19:30

Merge branch 'main' into di/stream-sql-to-s3

d834ca5

Merge branch 'main' into di/stream-sql-to-s3

af7468d

diegoimbert force-pushed the di/stream-sql-to-s3 branch from f8d6a82 to af7468d Compare May 12, 2025 08:43

graphite-app bot reviewed May 12, 2025

View reviewed changes

fix CI (unused import)

05283a8

ellipsis-dev bot reviewed May 12, 2025

View reviewed changes

diegoimbert and others added 2 commits May 12, 2025 16:33

error handling fix (graphite)

cde77fe

Merge branch 'main' into di/stream-sql-to-s3

bda882a

ellipsis-dev bot reviewed May 12, 2025

View reviewed changes

Merge branch 'main' into di/stream-sql-to-s3

58d5bd8

rubenfiszel changed the title ~~di/stream sql to s3~~ feat: sql jobs outputting to s3 + streaming for high-number of rows May 13, 2025

rubenfiszel merged commit c7886ea into main May 13, 2025
7 checks passed

rubenfiszel deleted the di/stream-sql-to-s3 branch May 13, 2025 08:19

github-actions bot locked and limited conversation to collaborators May 13, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: sql jobs outputting to s3 + streaming for high-number of rows #5704

feat: sql jobs outputting to s3 + streaming for high-number of rows #5704

Uh oh!

diegoimbert commented May 5, 2025 •

edited by ellipsis-dev bot

Loading

Uh oh!

cloudflare-workers-and-pages bot commented May 5, 2025 •

edited

Loading

Uh oh!

ellipsis-dev bot left a comment

Uh oh!

ellipsis-dev bot May 9, 2025

Uh oh!

ellipsis-dev bot May 9, 2025

Uh oh!

diegoimbert May 9, 2025

Uh oh!

ellipsis-dev bot May 9, 2025

Uh oh!

ellipsis-dev bot left a comment

Uh oh!

ellipsis-dev bot left a comment

Uh oh!

graphite-app bot May 12, 2025

Uh oh!

ellipsis-dev bot left a comment

Uh oh!

ellipsis-dev bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	static ref RE_S3_MODE: Regex = Regex::new(r#"(?m)^-- s3( (.+))?*(?:\r\|\n\|$)"#).unwrap();
	static ref RE_S3_MODE: Regex = Regex::new(r#"(?m)^-- s3( (.+))?(?:\r\|\n\|$)"#).unwrap();

feat: sql jobs outputting to s3 + streaming for high-number of rows #5704

feat: sql jobs outputting to s3 + streaming for high-number of rows #5704

Uh oh!

Conversation

diegoimbert commented May 5, 2025 • edited by ellipsis-dev bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloudflare-workers-and-pages bot commented May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying windmill with Cloudflare Pages

Uh oh!

ellipsis-dev bot left a comment

Choose a reason for hiding this comment

Uh oh!

ellipsis-dev bot May 9, 2025

Choose a reason for hiding this comment

Uh oh!

ellipsis-dev bot May 9, 2025

Choose a reason for hiding this comment

Uh oh!

diegoimbert May 9, 2025

Choose a reason for hiding this comment

Uh oh!

ellipsis-dev bot May 9, 2025

Choose a reason for hiding this comment

Uh oh!

ellipsis-dev bot left a comment

Choose a reason for hiding this comment

Uh oh!

ellipsis-dev bot left a comment

Choose a reason for hiding this comment

Uh oh!

graphite-app bot May 12, 2025

Choose a reason for hiding this comment

Uh oh!

ellipsis-dev bot left a comment

Choose a reason for hiding this comment

Uh oh!

ellipsis-dev bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

diegoimbert commented May 5, 2025 •

edited by ellipsis-dev bot

Loading

cloudflare-workers-and-pages bot commented May 5, 2025 •

edited

Loading