do not merge: bananza mode wip#744

Open

aaronsteers wants to merge 48 commits intomainfrom

aj/feat/bananza-mode-replication

Commits on Jul 31, 2025

save bananza mode wip
aaronsteers
committed

Commits on Aug 8, 2025

feat: Complete bananza mode lake storage implementation

devin-ai-integration[bot]
and
aaronsteers
committed
feat: Add warehouse sizing variables and scaling analysis

devin-ai-integration[bot]
and
aaronsteers
committed
fix: Resolve linting and type checking issues

devin-ai-integration[bot]
and
aaronsteers
committed
feat: Complete fast lake copy implementation with warehouse scaling

devin-ai-integration[bot]
and
aaronsteers
committed
feat: Add comprehensive timestamps and elapsed time to fast lake copy logging

devin-ai-integration[bot]
and
aaronsteers
committed
feat: Scale faker data to 10 million rows and process only purchases stream

devin-ai-integration[bot]
and
aaronsteers
committed
fix: Add force_full_refresh=True to ensure all 10M records are processed

devin-ai-integration[bot]
and
aaronsteers
committed
feat: Add detailed performance metrics with records/s and MB/s for each step

devin-ai-integration[bot]
and
aaronsteers
committed
feat: Scale to 50 million records and add write_strategy=replace parameter

devin-ai-integration[bot]
and
aaronsteers
committed
feat: Configure second run with 2XLARGE warehouse and skip initial data load

devin-ai-integration[bot]
and
aaronsteers
committed
fix: Add S3 eventual consistency delay and increase file descriptor limits

devin-ai-integration[bot]
and
aaronsteers
committed
fix: Use COMPUTE_WH_LARGE instead of non-existent COMPUTE_WH_2XLARGE warehouse

devin-ai-integration[bot]
and
aaronsteers
committed
feat: Update to COMPUTE_WH_2XLARGE warehouse and add Snowflake CPU minutes analysis

devin-ai-integration[bot]
and
aaronsteers
committed
feat: Update warehouse configuration and add CPU minutes analysis

devin-ai-integration[bot]
and
aaronsteers
committed
feat: Configure COMPUTE_WH_2XLARGE warehouse for 32x performance test

devin-ai-integration[bot]
and
aaronsteers
committed
feat: Switch to co-located S3 bucket in US West 2 to match Snowflake region

devin-ai-integration[bot]
and
aaronsteers
committed
fix: Use existing accessible S3 bucket ab-destiantion-iceberg-us-west-2

devin-ai-integration[bot]
and
aaronsteers
committed
feat: Add unload_table_to_lake() method for arbitrary table unloads

devin-ai-integration[bot]
and
aaronsteers
committed
refactor: Make unload_stream_to_lake() call unload_table_to_lake() to eliminate code duplication

devin-ai-integration[bot]
and
aaronsteers
committed
fix: Shorten parameter descriptions to fix line length linting issues

devin-ai-integration[bot]
and
aaronsteers
committed
feat: Remove arrow-based write_dataset() and read_dataset() methods from lake storage classes

devin-ai-integration[bot]
and
aaronsteers
committed
refactor: Make unload_stream_to_lake() generic in base class, move custom logic to unload_table_to_lake() in subclasses

devin-ai-integration[bot]
and
aaronsteers
committed
tidy up implementation
aaronsteers
authored
add FastUnloadResultObject
aaronsteers
authored
add ability to load from an unload result
aaronsteers
authored
fix: Resolve import errors and bugs preventing fast lake copy script execution

devin-ai-integration[bot]
and
aaronsteers
committed

Commits on Aug 12, 2025

Commits on Aug 13, 2025

Commits on Aug 17, 2025

tidy (wip)
aaronsteers
committed