Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
a663037
save bananza mode wip
aaronsteers Jul 31, 2025
68e1df6
feat: Complete bananza mode lake storage implementation
devin-ai-integration[bot] Aug 8, 2025
d5e6713
feat: Add warehouse sizing variables and scaling analysis
devin-ai-integration[bot] Aug 8, 2025
14457bc
fix: Resolve linting and type checking issues
devin-ai-integration[bot] Aug 8, 2025
54069f2
feat: Complete fast lake copy implementation with warehouse scaling
devin-ai-integration[bot] Aug 8, 2025
504eb39
feat: Add comprehensive timestamps and elapsed time to fast lake copy…
devin-ai-integration[bot] Aug 8, 2025
9a69509
feat: Scale faker data to 10 million rows and process only purchases …
devin-ai-integration[bot] Aug 8, 2025
8db85a6
fix: Add force_full_refresh=True to ensure all 10M records are processed
devin-ai-integration[bot] Aug 8, 2025
65d1628
feat: Add detailed performance metrics with records/s and MB/s for ea…
devin-ai-integration[bot] Aug 8, 2025
386f254
feat: Scale to 50 million records and add write_strategy=replace para…
devin-ai-integration[bot] Aug 8, 2025
4d70666
feat: Configure second run with 2XLARGE warehouse and skip initial da…
devin-ai-integration[bot] Aug 8, 2025
36f9fa8
fix: Add S3 eventual consistency delay and increase file descriptor l…
devin-ai-integration[bot] Aug 8, 2025
158572a
fix: Use COMPUTE_WH_LARGE instead of non-existent COMPUTE_WH_2XLARGE …
devin-ai-integration[bot] Aug 8, 2025
da22d77
feat: Update to COMPUTE_WH_2XLARGE warehouse and add Snowflake CPU mi…
devin-ai-integration[bot] Aug 8, 2025
57a3376
feat: Update warehouse configuration and add CPU minutes analysis
devin-ai-integration[bot] Aug 8, 2025
d1779f3
feat: Configure COMPUTE_WH_2XLARGE warehouse for 32x performance test
devin-ai-integration[bot] Aug 8, 2025
e0f4375
feat: Switch to co-located S3 bucket in US West 2 to match Snowflake …
devin-ai-integration[bot] Aug 8, 2025
9a9a23d
fix: Use existing accessible S3 bucket ab-destiantion-iceberg-us-west-2
devin-ai-integration[bot] Aug 8, 2025
a1c1c0c
feat: Add unload_table_to_lake() method for arbitrary table unloads
devin-ai-integration[bot] Aug 8, 2025
cbbf530
refactor: Make unload_stream_to_lake() call unload_table_to_lake() to…
devin-ai-integration[bot] Aug 8, 2025
de339fd
fix: Shorten parameter descriptions to fix line length linting issues
devin-ai-integration[bot] Aug 8, 2025
9fe4829
feat: Remove arrow-based write_dataset() and read_dataset() methods f…
devin-ai-integration[bot] Aug 8, 2025
16f5226
refactor: Make unload_stream_to_lake() generic in base class, move cu…
devin-ai-integration[bot] Aug 8, 2025
0f182f3
tidy up implementation
aaronsteers Aug 8, 2025
bc415da
add FastUnloadResultObject
aaronsteers Aug 8, 2025
5471786
add ability to load from an unload result
aaronsteers Aug 8, 2025
8dff561
fix: Resolve import errors and bugs preventing fast lake copy script …
devin-ai-integration[bot] Aug 8, 2025
c0bc120
toggle on reload, expand to 100MM sample records
aaronsteers Aug 12, 2025
30664de
fix: source config was not using new constant
aaronsteers Aug 12, 2025
34d4351
remove nonsense metric
aaronsteers Aug 12, 2025
297674d
feat: Toggle RELOAD_INITIAL_SOURCE_DATA to False after 100MM dataset …
devin-ai-integration[bot] Aug 12, 2025
f6cc1ea
feat: Implement robust COPY INTO metadata capture using RESULT_SCAN()
devin-ai-integration[bot] Aug 12, 2025
6b67ed1
feat: Enhance FastUnloadResult with actual record counts from RESULT_…
devin-ai-integration[bot] Aug 12, 2025
cf50d64
fix: Add noqa comment for necessary SQLAlchemy _mapping access
devin-ai-integration[bot] Aug 12, 2025
68c5fff
feat: Add files count and manifest display to fast lake copy example …
devin-ai-integration[bot] Aug 12, 2025
597c1a5
fix: Remove unnecessary f-string prefixes from static print statements
devin-ai-integration[bot] Aug 12, 2025
53bb076
Merge branch 'aj/feat/bananza-mode-replication' of https://github.com…
aaronsteers Aug 13, 2025
14b468c
feat: Add multi-warehouse performance analysis with timestamped S3 paths
devin-ai-integration[bot] Aug 13, 2025
103ecc7
Merge branch 'aj/feat/bananza-mode-replication' of https://git-manage…
devin-ai-integration[bot] Aug 13, 2025
6799539
feat: Remove cost efficiency and scaling efficiency tables from perfo…
devin-ai-integration[bot] Aug 13, 2025
d38423a
feat: Implement FastLoadResult class with Snowflake COPY INTO metadat…
devin-ai-integration[bot] Aug 13, 2025
47aa505
fix: Update table qualification logic in Snowflake fast_unload_table …
devin-ai-integration[bot] Aug 13, 2025
836d932
feat: Add FastLoadResult validation and test scripts
devin-ai-integration[bot] Aug 13, 2025
5d8ae55
feat: Add debug logging to compare unload vs load file processing
devin-ai-integration[bot] Aug 13, 2025
435e72d
feat: Enable debug logging with smaller dataset for load timeout debu…
devin-ai-integration[bot] Aug 13, 2025
3ea679a
fix: Prepare script for 100M dataset reload after accidental deletion
devin-ai-integration[bot] Aug 13, 2025
033c1aa
feat: Add destructive operation warning for RELOAD_INITIAL_SOURCE_DAT…
devin-ai-integration[bot] Aug 13, 2025
f53cf29
tidy (wip)
aaronsteers Aug 17, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions airbyte/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,7 @@
from airbyte.datasets import CachedDataset
from airbyte.destinations.base import Destination
from airbyte.destinations.util import get_destination
from airbyte.lakes import GCSLakeStorage, LakeStorage, S3LakeStorage
from airbyte.records import StreamRecord
from airbyte.results import ReadResult, WriteResult
from airbyte.secrets import SecretSourceEnum, get_secret
Expand All @@ -154,6 +155,7 @@
documents,
exceptions, # noqa: ICN001 # No 'exc' alias for top-level module
experimental,
lakes,
logs,
mcp,
records,
Expand All @@ -175,6 +177,7 @@
"documents",
"exceptions",
"experimental",
"lakes",
"logs",
"mcp",
"records",
Expand All @@ -195,7 +198,10 @@
"CachedDataset",
"Destination",
"DuckDBCache",
"GCSLakeStorage",
"LakeStorage",
"ReadResult",
"S3LakeStorage",
"SecretSourceEnum",
"Source",
"StreamRecord",
Expand Down
Loading