Skip to content

solana utils: refactor token accounts pipeline to account for historical states #8042

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 89 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
7206497
initial commit
jeff-dude Apr 18, 2025
da58475
Merge branch 'main' into token-accounts-full-refresh
jeff-dude Apr 21, 2025
4855eb8
update column names to match existing table
jeff-dude Apr 21, 2025
3e42db6
update sol transfers macro with new join
jeff-dude Apr 21, 2025
2128b23
force incremental on transfers
jeff-dude Apr 21, 2025
9ee7fd1
update all other transfers macros with new join
jeff-dude Apr 21, 2025
e3826d7
filter to incremental only
jeff-dude Apr 21, 2025
a1eedf2
add tx_index to all transfers
jeff-dude Apr 22, 2025
0de8487
update raydium joins to token accounts
jeff-dude Apr 22, 2025
4e6fd6e
force raydium as incremental
jeff-dude Apr 22, 2025
e0094d3
fix token accounts final union
jeff-dude Apr 22, 2025
a9f04f6
Merge branch 'main' into token-accounts-full-refresh
jeff-dude Apr 22, 2025
b555dbb
use address_prefix as partition universally
jeff-dude Apr 22, 2025
c548bc5
restructure token accounts pipeline to maintain compatability
jeff-dude Apr 22, 2025
caf9b7e
update downstream ref naming for state history joins
jeff-dude Apr 22, 2025
d25e2d7
remove joins in raydium, values already in transfers
jeff-dude Apr 22, 2025
fc7ac28
Merge branch 'main' into token-accounts-full-refresh
jeff-dude Apr 22, 2025
871479c
try without partition on raw
jeff-dude Apr 22, 2025
56af2f2
Merge branch 'main' into token-accounts-full-refresh
jeff-dude Apr 23, 2025
f1e68ca
update transfers macros to use decimals from fungible rather than prices
jeff-dude Apr 23, 2025
d60d8f2
remove all account type references and joins
jeff-dude Apr 23, 2025
bbd4e58
exclude spl tokens to see 2022 run fast
jeff-dude Apr 23, 2025
d2e5d52
update raydium v5 token balance owner comment
jeff-dude Apr 23, 2025
6f65730
exclude large chunk of transfers models
jeff-dude Apr 23, 2025
7e0d937
fix bug in sol transfers
jeff-dude Apr 23, 2025
ca19175
fix column name in sol transfers
jeff-dude Apr 23, 2025
d3f6f07
enable final views for transfers
jeff-dude Apr 23, 2025
9768e13
temp: remove backfills in transfers views
jeff-dude Apr 23, 2025
c21124f
update column name for token account owner in transfers for raydium
jeff-dude Apr 23, 2025
ba71b96
add tx_index to transfers
jeff-dude Apr 23, 2025
04b275e
enable spl token accounts
jeff-dude Apr 23, 2025
2092edb
test address prefix partition on raw
jeff-dude Apr 24, 2025
7586ad8
test raw partition on block date
jeff-dude Apr 24, 2025
0f8d8de
Merge branch 'main' into token-accounts-full-refresh
jeff-dude Apr 24, 2025
0c29076
finalize block_date partition on raw
jeff-dude Apr 24, 2025
3cc41f7
finalize columns in state history models
jeff-dude Apr 24, 2025
239860f
finalize columns downstream for token accounts
jeff-dude Apr 24, 2025
e299205
update schema file
jeff-dude Apr 24, 2025
d746034
refactor sol transfers to remove token account join, add latest quarters
jeff-dude Apr 24, 2025
734830e
also remove prices and save for downstream
jeff-dude Apr 24, 2025
5f69563
remove a few fields to use downstream instead
jeff-dude Apr 24, 2025
d39716a
simplify spl transfers call transfer macro
jeff-dude Apr 24, 2025
8b1d8e4
updated spl transfers call transfer models
jeff-dude Apr 24, 2025
19f3a75
update spl transfers macro and models
jeff-dude Apr 24, 2025
bc54382
update token22 macro and models
jeff-dude Apr 24, 2025
ec4fba8
update backfill aliases to be base_ transfers
jeff-dude Apr 24, 2025
391fa26
introduce base_ model for all backfill unions
jeff-dude Apr 24, 2025
59681a7
list columns explicitly in base_ models
jeff-dude Apr 24, 2025
9349048
materialize each transfer type model, add final joins
jeff-dude Apr 24, 2025
9d10337
update final view
jeff-dude Apr 24, 2025
ea5e3be
force token accounts to run quick
jeff-dude Apr 24, 2025
2bbfb71
remove unused blockchain column
jeff-dude Apr 24, 2025
2d22ae7
remove inner executing account col that isn't used
jeff-dude Apr 24, 2025
e19a136
remove blockchain column
jeff-dude Apr 24, 2025
0b7ea42
remove missing cols
jeff-dude Apr 24, 2025
1d44281
fix table alias bugs, add missing columns
jeff-dude Apr 25, 2025
e9ac130
add action to token22
jeff-dude Apr 25, 2025
a2cc323
Merge branch 'main' into token-accounts-full-refresh
jeff-dude Apr 25, 2025
9231c65
handle nulls in append model lookup join conditions
jeff-dude Apr 25, 2025
0a9e593
add all to schema file with unique tests
jeff-dude Apr 25, 2025
9ee5ba1
full history for token accounts
jeff-dude Apr 25, 2025
257205e
Merge branch 'main' into token-accounts-full-refresh
jeff-dude Apr 28, 2025
07e32fa
exclude all non token22
jeff-dude Apr 28, 2025
9a5b14b
refactor transfers pipeline
jeff-dude Apr 29, 2025
e1f14a8
refactor transfers pipeline
jeff-dude Apr 29, 2025
f7bcb95
Merge branch 'main' into token-accounts-full-refresh
jeff-dude Apr 29, 2025
5b11b32
fix unique key in base transfers
jeff-dude Apr 29, 2025
b27bcad
fix raydium spells
jeff-dude Apr 29, 2025
d94b77e
Merge branch 'main' into token-accounts-full-refresh
jeff-dude Apr 29, 2025
7835757
leave inner_instruction_index as-is from source
jeff-dude Apr 29, 2025
a8d6b95
include raydium v3 to match 4/5
jeff-dude Apr 29, 2025
6535f3e
force v3 to be short runtime, to match other versions
jeff-dude Apr 29, 2025
3238201
remove forced incremental on raydium
jeff-dude Apr 29, 2025
524301c
remove forced incremental from transfers
jeff-dude Apr 29, 2025
37f4a4c
Merge branch 'main' into token-accounts-full-refresh
jeff-dude Apr 30, 2025
156be81
to revert: force schema name
jeff-dude Apr 30, 2025
3772186
update to block month partitions
jeff-dude May 1, 2025
a4fadcf
filter out close events in token accounts, materialize union
jeff-dude May 1, 2025
50aa0cb
add block month and update partition on TA raw
jeff-dude May 2, 2025
f649755
less partitions on address prefix
jeff-dude May 2, 2025
8fb72b8
fix prefix, q1 2025
jeff-dude May 2, 2025
f4479d7
add valid from month, partition on month
jeff-dude May 6, 2025
c82fc22
add valid from/to month to union model
jeff-dude May 6, 2025
0def9e1
add columns to active view
jeff-dude May 6, 2025
12a43a5
filter for 2023 Q1 and use monthly partition in join for token accounts
jeff-dude May 6, 2025
7559ae9
update to Q1 2025
jeff-dude May 6, 2025
f93bfc6
try one month of transfers in 2025
jeff-dude May 8, 2025
358122b
Merge branch 'main' into token-accounts-full-refresh
jeff-dude May 9, 2025
8f53166
Merge branch 'main' into token-accounts-full-refresh
jeff-dude May 12, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion dbt_macros/dune/generate_schema_name.sql
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
{%- set default_schema = target.schema -%}
{%- if target.name == 'prod' or target.schema == 'wizard' and custom_schema_name is not none -%}

{{ custom_schema_name | trim }}
{{ default_schema }}_{{ custom_schema_name | trim }}

{%- elif target.schema.startswith("git_") -%}

Expand Down

This file was deleted.

This file was deleted.

Loading
Loading