fix: Implement file writing directly for VortexSink instead of DataFusion demuxer (#23)

* Disable child metric registration under parent to prevent OOM (#10)

* Use tokio::sync::oneshot to prevent FuturesUnordered reentrant drop crash (#9)

* Don't return an error when we have an unsupported node, bubble up "TRUE" as in keep for that node, up to any `and` or `or` node and handle empty IN list. (#8)

* feat: Support retrieving WriterStrategyBuilder from VortexSession (#6)

* Fix session get-or-default (vortex-data#5662)

The comments described this get-or-default, but instead it was a panic

---------

Signed-off-by: Nicholas Gates <nick@nickgates.com>

* feat: Support retrieving writer strategy builder from vortex session

---------

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-authored-by: Nicholas Gates <gatesn@users.noreply.github.com>

* fix: Handle UncompressedSizeInBytes statistic correctly (#3)

* fix: Add log dependency to vortex-io

* fix: Add Debug impl for WriteStrategyBuilder and log dep for vortex-io

* fix: Update persistent module to use simplified expression handling from PR #8

* revert: Restore spiceai-51 versions of persistent module (incompatible with DF51)

* fix: Restore format.rs to spiceai-51 version

* Add Case-When Expression and tests (#12)

* fix: Ensure CastExpr/CastColumnExpr/ScalarFunctionExpr check children in can_be_pushed_down

The can_be_pushed_down function was returning true for CastExpr and CastColumnExpr
without checking if their child expressions are convertible. This caused runtime
errors when the child contained expressions like CaseExpr that convert() cannot
handle.

Also fixed ScalarFunctionExpr to recursively check its arguments.

Fixes spiceai/spiceai#9037

* Add Case-When Expression and tests

* Implement execute()

* Add additional tests and fix type issue

* Fix toml lint

* feat(case_when): implement lazy evaluation to avoid side effects in unevaluated branches

This implements proper lazy evaluation in CaseWhen expression to ensure
that THEN/ELSE branches are only evaluated for rows where they apply.
This is critical for correctness when expressions have side effects like
divide-by-zero panics.

The implementation:
1. Evaluates conditions in order, tracking which rows have been matched
2. For each condition, computes an effective mask (condition AND NOT matched)
3. Uses filter() to create a scoped array with only matching rows
4. Evaluates THEN expression only on the filtered scope
5. Uses scatter_with_mask() to expand results back to original positions
6. Short-circuits when all rows are matched or all conditions fail

This fixes TPC-DS Q73 which has a pattern like:
  CASE WHEN hd_vehicle_count > 0
       THEN hd_dep_count/hd_vehicle_count
       ELSE NULL END

Previously, the division would be evaluated for all rows including those
where hd_vehicle_count=0, causing a divide-by-zero panic. Now the division
is only evaluated for rows where the condition is true.

Added test: test_evaluate_divide_by_zero_protected_by_case_when

* Formatting

* feat(datafusion): Export DefaultExpressionConvertor (#19)

* feat: Add option for writing to target vortex file size

* fix: Actually respect target file size

* fix: Set default file size to 16mb

* fix: Update use of unwrap in tests

* test: Update tests

* fix: Always write with custom sink

* fix: Stream directly into target file

* fix: Improve safety arounded ended writer streams with remaining source

* feat: add target file size configuration for Vortex file output

- Introduced `target_file_size_mb` option to `VortexFormat` for controlling the size of output files in megabytes.
- Updated `VortexSink` to handle writing files based on the specified target size, bypassing DataFusion's default row count-based splitting.
- Implemented logic to split output files when the buffered data exceeds the target file size during the write process.
- Added tests to verify the new target file size configuration.

* refactor: clean up case expression handling and improve file size configuration logic

---------

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-authored-by: Sergei Grebnov <sergei.grebnov@gmail.com>
Co-authored-by: Luke Kim <80174+lukekim@users.noreply.github.com>
Co-authored-by: Nicholas Gates <gatesn@users.noreply.github.com>

Author: 4 people