Properly handle decoding decimals from SQLite using the decimal extension #504
+487
−38
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
peasee
approved these changes
Nov 28, 2025
ewgenius
reviewed
Nov 28, 2025
ewgenius
approved these changes
Nov 28, 2025
phillipleblanc
added a commit
that referenced
this pull request
Dec 18, 2025
* Revert "Properly handle decoding decimals from SQLite using the decimal extension (#504)" This reverts commit df8eeb1. * Remove decimal test - its invalid * Revert "Add roundtrip test and fix Decimal support" This reverts commit b9e3c14. * Use spiceai fork of sea-query * fix(sqlite): store Date32/Date64 as ISO-8601 strings in insert_batch_prepared The insert_batch_prepared function was storing Date32/Date64 values as Unix timestamps (e.g., 751248000), while the non-batch insert_batch function (via InsertBuilder) correctly stores them as ISO-8601 date strings (e.g., '1993-10-22'). When reading data back, the SQLite table provider expects date columns to contain ISO-8601 formatted strings for proper parsing/casting back to Arrow Date32 types. The Unix timestamp values failed to parse, resulting in NULL/empty date values. This change updates insert_batch_prepared to convert Date32/Date64 values to ISO-8601 strings using time::OffsetDateTime, matching the behavior of InsertBuilder::construct_insert_stmt. * Remove unneeded test
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously we were decoding Decimals from SQLite as UTF8 - this was correct because in the
spiceaiproject we were injecting a schema cast step before the datafusion execution kicked in. However, that should logically live in this project so that we can validate the datafusion logic based on the decimal types directly.