Skip to content

solana utils: refactor token accounts pipeline to account for historical states #8042

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 18 commits into
base: main
Choose a base branch
from
Draft

solana utils: refactor token accounts pipeline to account for historical states #8042

wants to merge 18 commits into from

Conversation

jeff-dude
Copy link
Member

towards CUR2-144

@github-actions github-actions bot added WIP work in progress dbt: solana covers the Solana dbt subproject labels Apr 18, 2025
jeff-dude added 17 commits April 21, 2025 10:49
@jeff-dude
Merge branch 'main' into token-accounts-full-refresh
da58475
@jeff-dude
update column names to match existing table
4855eb8
@jeff-dude
update sol transfers macro with new join
3e42db6
@jeff-dude
force incremental on transfers
2128b23
@jeff-dude
update all other transfers macros with new join
9ee7fd1
@jeff-dude
filter to incremental only
e3826d7
@jeff-dude
add tx_index to all transfers
a1eedf2
@jeff-dude
update raydium joins to token accounts
0de8487
@jeff-dude
force raydium as incremental
4e6fd6e
@jeff-dude
fix token accounts final union
e0094d3
@jeff-dude
Merge branch 'main' into token-accounts-full-refresh
a9f04f6
@jeff-dude
use address_prefix as partition universally
b555dbb
@jeff-dude
restructure token accounts pipeline to maintain compatability
c548bc5
@jeff-dude
update downstream ref naming for state history joins
caf9b7e
@jeff-dude
remove joins in raydium, values already in transfers
d25e2d7
@jeff-dude
Merge branch 'main' into token-accounts-full-refresh
fc7ac28
@jeff-dude
try without partition on raw
871479c
@jeff-dude
Copy link
Member Author

latest findings on testing run times:

  • using address_prefix on raw table(s) killed write performance on historical, CI timed out at 90m on 12-node cluster
    • compare this to when partitioning on block_date or no partition at all running in ~5-10 minutes
  • most recent run was with no partition on raw, then state history ran in normal timing (compared to block_date partition), but it looks like transfers kicking off is killing performance. we didn't see a single transfers model complete in the 90m window. this also includes a forced incremental run, meaning it's only the last one day of transfers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
dbt: solana covers the Solana dbt subproject WIP work in progress
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@jeff-dude