Skip to content

fix: Correct DynamoDB default time_format to include milliseconds#1530

Closed
claudespice wants to merge 25 commits into
trunkfrom
fix/dynamodb-time-format-default
Closed

fix: Correct DynamoDB default time_format to include milliseconds#1530
claudespice wants to merge 25 commits into
trunkfrom
fix/dynamodb-time-format-default

Conversation

@claudespice
Copy link
Copy Markdown
Collaborator

@claudespice claudespice commented Apr 19, 2026

Summary

  • The DynamoDB connector docs showed the default time_format as 2006-01-02T15:04:05Z07:00 (without milliseconds)
  • The code defines the actual default as 2006-01-02T15:04:05.000Z07:00 (with .000 for millisecond precision)
  • This distinction matters because Go format strings are exact patterns — the .000 suffix is required to parse timestamps that include millisecond precision

Changes

  • Updated the default format pattern from 2006-01-02T15:04:05Z07:00 to 2006-01-02T15:04:05.000Z07:00 in the time format reference table
  • Fixed across vNext and versioned docs (1.9.x, 1.10.x, 1.11.x) — the DynamoDB docs don't exist in versions before 1.9.x

Reference

Verified against spiceai/spiceai at trunk — crates/runtime/src/dataconnector/dynamodb.rs (line 80: const DEFAULT_TIME_FORMAT: &str = "2006-01-02T15:04:05.000Z07:00")

Also verified at tag v1.9.2 — same default.

Claude added 4 commits April 18, 2026 03:11
fix: Add nanosecond timestamp range limitation to MSSQL and Oracle ve…
db83140 
…rsioned docs

Commit d5341c4 documented the Arrow nanosecond timestamp range
limitation (~1677–2262) for MSSQL and Oracle connectors, but only
in website/docs/ (vNext). The limitation existed in all prior versions
too — in versions 1.5.x–1.11.x, out-of-range timestamps silently
returned 1970-01-01 UTC instead of erroring.

Propagate the limitation note to all 7 versioned doc directories
(14 files total) with version-appropriate wording.
fix: Correct PostgreSQL accelerator connection pool parameter name an…
4ead185 
…d defaults

The docs listed pg_connection_pool_min_idle (default 1) and connection_pool_size
(default 5), but the code uses pg_connection_pool_min (default 5) and
connection_pool_size (default 10).
fix: Correct IMAP ssl_mode default from tls to auto
577c5dd 
The docs stated the default ssl_mode is tls, but the code has always
used auto as the default since at least v1.5.0.
fix: Correct DynamoDB default time_format to include milliseconds
fb7b1cb 
The docs showed the default time format as 2006-01-02T15:04:05Z07:00
but the code uses 2006-01-02T15:04:05.000Z07:00 (with .000 milliseconds).
This matters because Go format strings are exact patterns — without .000,
timestamps with millisecond precision would not parse correctly.
@claudespice claudespice added the bug Something isn't working label Apr 19, 2026
@lukekim lukekim self-assigned this Apr 21, 2026
@lukekim
Copy link
Copy Markdown
Contributor

lukekim commented Apr 21, 2026

@copilot resolve the merge conflicts in this pull request

Claude and others added 20 commits April 21, 2026 18:35
@lukekim
fix: Use lowercase security_protocol values in Kafka and Debezium docs
99740c9 
The code defines kafka_security_protocol with lowercase defaults and
options (sasl_ssl, plaintext, ssl, sasl_plaintext) in both the
ParameterSpec and runtime fallback, but the documentation showed
uppercase (SASL_SSL, PLAINTEXT, etc.).

Updated across all versioned docs (1.5.x through 1.11.x) and vNext.
Verified the code has used lowercase since at least v1.6.0.

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
@lukekim
fix: Add missing DynamoDB parameters to versioned docs
a1f1dd1 
Follow-up to #1507 which added scan_interval, ready_lag, endpoint_url,
and lag_exceeds_shard_retention_behavior to the vNext docs but missed
the versioned docs.

- version-1.11.x: all 4 parameters (verified against v1.11.5 code)
- version-1.10.x: 3 parameters — scan_interval, ready_lag, endpoint_url
  (lag_exceeds_shard_retention_behavior does not exist in v1.10.4)

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
@lukekim
docs: add production deployment guides for RC and GA components (#1524)
85601a8 
Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
@lukekim
fix: Correct "Dashbord" typo in Datadog monitoring docs
98bed1a 
The Datadog import-dashboard step read "Dashbord is now configured..."
across the unversioned docs and all 7 versioned copies. Addresses one of
the items called out in #506.

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
@lukekim
fix: Remove nonexistent duckdb_connection_string and fix param name
159864a 
- Remove `duckdb_connection_string` which doesn't exist as a parameter
- Fix `open` → `duckdb_open` (component param, auto-prefixed)
- Note in-memory fallback behavior when param is omitted

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
@lukekim
fix: Remove nonexistent dremio_token parameter from deployment guide
54f8b2d 
The connector only supports username/password authentication via
dremio_username and dremio_password. Token/PAT authentication is not
implemented in the code — the only Credentials variant used is
UsernamePassword.

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
@lukekim
fix: Correct GraphQL deployment guide parameter names
f4523ed 
- Remove nonexistent `graphql_endpoint` (endpoint comes from `from:` path)
- Remove nonexistent `graphql_pagination_parameters` (pagination is automatic)
- Fix `graphql_json_pointer` → `json_pointer` (runtime param, no prefix)
- Add missing `graphql_auth_token` parameter
- Fix `graphql_auth_header` description to match actual behavior

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
@claudespice
fix: Add missing connector prefix to azure_storage_client_secret in a…
8ac9c5a 
…uth notes (#1536)

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
@claudespice
fix: Correct Postgres connector deployment guide defaults and params (#…
0ed660f 
…1535)

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
@claudespice
fix: Correct multiple inaccuracies in MySQL deployment guide (#1534)
da0a4e3 
Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
@claudespice @lukekim
fix: Document PostgreSQL replication parameters for WAL streaming (#1531
a8a0223 
)

* fix: Add nanosecond timestamp range limitation to MSSQL and Oracle versioned docs

Commit d5341c4 documented the Arrow nanosecond timestamp range
limitation (~1677–2262) for MSSQL and Oracle connectors, but only
in website/docs/ (vNext). The limitation existed in all prior versions
too — in versions 1.5.x–1.11.x, out-of-range timestamps silently
returned 1970-01-01 UTC instead of erroring.

Propagate the limitation note to all 7 versioned doc directories
(14 files total) with version-appropriate wording.

* fix: Correct PostgreSQL accelerator connection pool parameter name and defaults

The docs listed pg_connection_pool_min_idle (default 1) and connection_pool_size
(default 5), but the code uses pg_connection_pool_min (default 5) and
connection_pool_size (default 10).

* fix: Correct IMAP ssl_mode default from tls to auto

The docs stated the default ssl_mode is tls, but the code has always
used auto as the default since at least v1.5.0.

* fix: Correct DynamoDB default time_format to include milliseconds

The docs showed the default time format as 2006-01-02T15:04:05Z07:00
but the code uses 2006-01-02T15:04:05.000Z07:00 (with .000 milliseconds).
This matters because Go format strings are exact patterns — without .000,
timestamps with millisecond precision would not parse correctly.

* fix: Document PostgreSQL replication parameters for WAL streaming

The PostgreSQL connector added logical replication (WAL streaming) support
in PR #10364 with 5 new parameters (pg_replication_slot, pg_publication,
pg_replication_initial_snapshot, pg_replication_temporary_slot,
pg_replication_status_interval) that were not yet documented.

---------

Co-authored-by: Claude <claude@Claudes-Mini.localdomain>
Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
@lukekim
faq: add incremental ingestion Q and native PostgreSQL replication CD…
59d80fa 
…C option (#1541)

* faq: add incremental ingestion Q and native PostgreSQL replication CDC option

- Add new FAQ entry pointing to data-refresh docs for incremental ingestion
- Extend CDC FAQ entry with native PostgreSQL logical replication (recommended for PG)
- Add new Features > CDC > PostgreSQL Logical Replication page documenting
  wal_level=logical + pgoutput direct replication (no Debezium/Kafka)
- Update CDC overview to list native PG replication alongside Debezium

* faq, cdc: mention direct Kafka and DynamoDB Streams ingestion

* docs: fix broken links by using .md extensions for cross-page refs

* data-refresh: add combined incremental ingestion example (overlap, retention, soft deletes)
Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
@phillipleblanc
docs(otel): document runtime.telemetry.metric_prefix, properties, and…
6986a0b 
… DataDog OTLP setup (#1542)

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
@lukekim
fix: Note scylladb_ssl parameter is not yet implemented
338331f 
The scylladb_ssl parameter is defined in the ParameterSpec but
never read or applied in the session builder code. The parameter
is accepted without error but has no effect on the connection.

Added a note to the parameter description in both vNext and
1.11.x docs.

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
@lukekim
fix: Correct pg_sslmode default from verify-full to prefer
edb1f90 
The documentation incorrectly states that the default value of
pg_sslmode is verify-full. The actual default is prefer, which
matches standard libpq/tokio-postgres behavior — when no sslmode
is specified, the client tries TLS but falls back to plaintext.

For the replication (CDC) path, SslMode::from_str_or_default()
explicitly defaults to Prefer (postgres_replication/config.rs).
For the read path, no sslmode default is set in the ParameterSpec,
so the underlying tokio-postgres library applies its own default
of prefer.

Fixed across all versioned docs (1.5.x–1.11.x) and vNext.

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
@lukekim
Document multi-vector embeddings and late-interaction search
166b8b6 
Add a new Multi-Vector Search feature page covering column-of-vectors
embeddings (List<Utf8> source columns), aggregation strategies
(max/mean/sum), max_elements_per_row caps, and ColBERT-style
late-interaction multi-query vector_search.

Cross-link from the search index, vector search, and embeddings
component pages, and document the previously-missing aggregation and
max_elements_per_row fields in the datasets reference.

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
@lukekim
Replace KaTeX expression with plain text (no math plugin configured)
c6bbd20 
Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
@wyattwenzel
Release Post for v.2.0.0-rc.3 (#1546)
ff77178 
Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
@claudespice
fix: Correct IMAP ssl_mode default from tls to auto (#1529)
d172a4a 
Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
@claudespice @lukekim
fix: Correct PostgreSQL accelerator connection pool parameter name an…
384c4b8 
…d defaults (#1528)

* fix: Add nanosecond timestamp range limitation to MSSQL and Oracle versioned docs

Commit d5341c4 documented the Arrow nanosecond timestamp range
limitation (~1677–2262) for MSSQL and Oracle connectors, but only
in website/docs/ (vNext). The limitation existed in all prior versions
too — in versions 1.5.x–1.11.x, out-of-range timestamps silently
returned 1970-01-01 UTC instead of erroring.

Propagate the limitation note to all 7 versioned doc directories
(14 files total) with version-appropriate wording.

* fix: Correct PostgreSQL accelerator connection pool parameter name and defaults

The docs listed pg_connection_pool_min_idle (default 1) and connection_pool_size
(default 5), but the code uses pg_connection_pool_min (default 5) and
connection_pool_size (default 10).

---------

Co-authored-by: Claude <claude@Claudes-Mini.localdomain>
Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
@lukekim
fix: Restore MSSQL and Oracle nanosecond timestamp notes from trunk
2fe576e 
Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
Copy link
Copy Markdown

Copilot AI commented Apr 21, 2026

@copilot resolve the merge conflicts in this pull request

Resolved. The branch was rebased and now contains only the DynamoDB time_format fix (4 files) relative to trunk — the PostgreSQL, IMAP, and MSSQL/Oracle changes that were bundled in the original commits had already been merged into trunk separately. Commit 2fe576e cleaned up the stale conflicts.

Copilot AI requested a review from lukekim April 21, 2026 18:40
@lukekim lukekim closed this Apr 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lukekim lukekim Awaiting requested review from lukekim

Assignees

@lukekim lukekim

Labels

area/docs bug Something isn't working release

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@claudespice @lukekim @phillipleblanc @wyattwenzel