You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/core_concepts/11_persistent_storage/large_data_files.mdx
+22-1Lines changed: 22 additions & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -31,6 +31,27 @@ Windmill S3 bucket browser will not work for buckets containing more than 20 fil
31
31
ETLs can be easily implemented in Windmill using its integration with Polars and DuckDB for facilitate working with tabular data. In this case, you don't need to manually interact with the S3 bucket, Polars/DuckDB does it natively and in a efficient way. Reading and Writing datasets to S3 can be done seamlessly.
-- Write the result of a query to a different parquet file on S3
284
+
COPY (
285
+
SELECTCOUNT(*) FROM read_parquet($file1)
286
+
) TO 's3:///demo/output.pq' (FORMAT 'parquet');
287
+
288
+
```
289
+
269
290
</TabItem>
270
291
</Tabs>
271
292
@@ -283,7 +304,16 @@ With S3 as the external store, a transformation script in a flow will typically
283
304
2. Running some computation on the data.
284
305
3. Storing the result back to S3 for the next scripts to be run.
285
306
286
-
Windmill SDKs now expose helpers to simplify code and help you connect Polars or DuckDB to the Windmill workspace S3 bucket. In your usual IDE, you would need to write for _each script_:
307
+
When running a DuckDB script, Windmill automatically handles connection to your workspace storage :
308
+
309
+
```sql
310
+
-- This queries the windmill api under the hood to figure out the
If you want to use a scripting language, Windmill SDKs now expose helpers to simplify code and help you connect Polars or DuckDB to the Windmill workspace S3 bucket. In your usual IDE, you would need to write for _each script_:
0 commit comments