-
Notifications
You must be signed in to change notification settings - Fork 67
do not merge: bananza mode wip #744
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
aaronsteers
wants to merge
48
commits into
main
Choose a base branch
from
aj/feat/bananza-mode-replication
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 29 commits
Commits
Show all changes
48 commits
Select commit
Hold shift + click to select a range
a663037
save bananza mode wip
aaronsteers 68e1df6
feat: Complete bananza mode lake storage implementation
devin-ai-integration[bot] d5e6713
feat: Add warehouse sizing variables and scaling analysis
devin-ai-integration[bot] 14457bc
fix: Resolve linting and type checking issues
devin-ai-integration[bot] 54069f2
feat: Complete fast lake copy implementation with warehouse scaling
devin-ai-integration[bot] 504eb39
feat: Add comprehensive timestamps and elapsed time to fast lake copy…
devin-ai-integration[bot] 9a69509
feat: Scale faker data to 10 million rows and process only purchases …
devin-ai-integration[bot] 8db85a6
fix: Add force_full_refresh=True to ensure all 10M records are processed
devin-ai-integration[bot] 65d1628
feat: Add detailed performance metrics with records/s and MB/s for ea…
devin-ai-integration[bot] 386f254
feat: Scale to 50 million records and add write_strategy=replace para…
devin-ai-integration[bot] 4d70666
feat: Configure second run with 2XLARGE warehouse and skip initial da…
devin-ai-integration[bot] 36f9fa8
fix: Add S3 eventual consistency delay and increase file descriptor l…
devin-ai-integration[bot] 158572a
fix: Use COMPUTE_WH_LARGE instead of non-existent COMPUTE_WH_2XLARGE …
devin-ai-integration[bot] da22d77
feat: Update to COMPUTE_WH_2XLARGE warehouse and add Snowflake CPU mi…
devin-ai-integration[bot] 57a3376
feat: Update warehouse configuration and add CPU minutes analysis
devin-ai-integration[bot] d1779f3
feat: Configure COMPUTE_WH_2XLARGE warehouse for 32x performance test
devin-ai-integration[bot] e0f4375
feat: Switch to co-located S3 bucket in US West 2 to match Snowflake …
devin-ai-integration[bot] 9a9a23d
fix: Use existing accessible S3 bucket ab-destiantion-iceberg-us-west-2
devin-ai-integration[bot] a1c1c0c
feat: Add unload_table_to_lake() method for arbitrary table unloads
devin-ai-integration[bot] cbbf530
refactor: Make unload_stream_to_lake() call unload_table_to_lake() to…
devin-ai-integration[bot] de339fd
fix: Shorten parameter descriptions to fix line length linting issues
devin-ai-integration[bot] 9fe4829
feat: Remove arrow-based write_dataset() and read_dataset() methods f…
devin-ai-integration[bot] 16f5226
refactor: Make unload_stream_to_lake() generic in base class, move cu…
devin-ai-integration[bot] 0f182f3
tidy up implementation
aaronsteers bc415da
add FastUnloadResultObject
aaronsteers 5471786
add ability to load from an unload result
aaronsteers 8dff561
fix: Resolve import errors and bugs preventing fast lake copy script …
devin-ai-integration[bot] c0bc120
toggle on reload, expand to 100MM sample records
aaronsteers 30664de
fix: source config was not using new constant
aaronsteers 34d4351
remove nonsense metric
aaronsteers 297674d
feat: Toggle RELOAD_INITIAL_SOURCE_DATA to False after 100MM dataset …
devin-ai-integration[bot] f6cc1ea
feat: Implement robust COPY INTO metadata capture using RESULT_SCAN()
devin-ai-integration[bot] 6b67ed1
feat: Enhance FastUnloadResult with actual record counts from RESULT_…
devin-ai-integration[bot] cf50d64
fix: Add noqa comment for necessary SQLAlchemy _mapping access
devin-ai-integration[bot] 68c5fff
feat: Add files count and manifest display to fast lake copy example …
devin-ai-integration[bot] 597c1a5
fix: Remove unnecessary f-string prefixes from static print statements
devin-ai-integration[bot] 53bb076
Merge branch 'aj/feat/bananza-mode-replication' of https://github.com…
aaronsteers 14b468c
feat: Add multi-warehouse performance analysis with timestamped S3 paths
devin-ai-integration[bot] 103ecc7
Merge branch 'aj/feat/bananza-mode-replication' of https://git-manage…
devin-ai-integration[bot] 6799539
feat: Remove cost efficiency and scaling efficiency tables from perfo…
devin-ai-integration[bot] d38423a
feat: Implement FastLoadResult class with Snowflake COPY INTO metadat…
devin-ai-integration[bot] 47aa505
fix: Update table qualification logic in Snowflake fast_unload_table …
devin-ai-integration[bot] 836d932
feat: Add FastLoadResult validation and test scripts
devin-ai-integration[bot] 5d8ae55
feat: Add debug logging to compare unload vs load file processing
devin-ai-integration[bot] 435e72d
feat: Enable debug logging with smaller dataset for load timeout debu…
devin-ai-integration[bot] 3ea679a
fix: Prepare script for 100M dataset reload after accidental deletion
devin-ai-integration[bot] 033c1aa
feat: Add destructive operation warning for RELOAD_INITIAL_SOURCE_DAT…
devin-ai-integration[bot] f53cf29
tidy (wip)
aaronsteers File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🛠️ Refactor suggestion
hasattr
check always passes – subclasses still allowed to omit an overrideCacheBase
itself definesfast_unload_table
, sohasattr(self, "fast_unload_table")
is alwaysTrue
.Would you switch to comparing the underlying function object instead?
This guarantees the override is present – wdyt?
🤖 Prompt for AI Agents