Skip to content

Conversation

@georgesittas
Copy link
Contributor

Addresses the issue discussed in this Slack thread: https://tobiko-data.slack.com/archives/C044BRE5W4S/p1749739413928599.

The _dt date macro variables always inject a time zone in the resulting datetime object, even if it's UTC. This is problematic in certain cases, e.g., when these values are compared to a zone-unaware timestamps. For instance, Redshift isn't clever enough when comparing zone-aware and unaware timestamps, leading to full scans instead of pruning.

The only alternative that exists today to work around this issue is to have string versions of these date values and cast them, which isn't a great experience. This PR covers this feature gap.

@georgesittas georgesittas requested a review from a team June 27, 2025 12:29
Copy link
Collaborator

@erindru erindru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might want to document this on the Redshift engine adapter page for people to consider if they're having trouble with _dt

@georgesittas georgesittas force-pushed the jo/add_dtntz_macro_variable branch from 354b6b7 to c086ce1 Compare July 1, 2025 08:03
@georgesittas georgesittas force-pushed the jo/add_dtntz_macro_variable branch from c086ce1 to 7333fe4 Compare July 1, 2025 08:04
@georgesittas georgesittas enabled auto-merge (squash) July 1, 2025 08:14
@georgesittas georgesittas merged commit 0060cab into main Jul 1, 2025
26 of 27 checks passed
@georgesittas georgesittas deleted the jo/add_dtntz_macro_variable branch July 1, 2025 08:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants