feat(dagster-backblaze): add Backblaze B2 (S3-compatible) integration#329
Draft
goanpeca wants to merge 1 commit into
Draft
feat(dagster-backblaze): add Backblaze B2 (S3-compatible) integration#329goanpeca wants to merge 1 commit into
goanpeca wants to merge 1 commit into
Conversation
Signed-off-by: Gonzalo Peña-Castellanos <goanpeca@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
Adds
dagster-backblaze, a small new integration for S3-compatible object storage, and makes the existingdagster-obstoreS3 resource work against the same class of endpoint.The Dagster ecosystem already talks to AWS S3 through
dagster-aws. Any object store that speaks the S3 API (AWS S3 itself, and other S3-compatible providers such as Backblaze B2) works with the same boto3 client once you point it at the right endpoint and hand it the right credential fields. The friction is purely ergonomic: a non-AWS S3 target means remembering an endpoint URL and supplying credentials through AWS-named fields that do not match the provider's own naming.B2Resourceremoves that friction for one such provider without forking any behavior. It subclassesdagster_aws.s3.S3Resource, pre-fills the endpoint, and accepts the provider's own credential field names, mapping them onto the boto3aws_access_key_id/aws_secret_access_keyparameters the parent already forwards. Under the hood you still get a plain boto3 S3 client, so anything that works withS3Resourceworks here.The change is intentionally minimal, additive, and follows the existing library layout (
Makefile,README.md,pyproject.toml,uv.lock, package + tests,_is_dagster_maintained()returningFalse).What is included:
libraries/dagster-backblazeexposingB2Resource, a thin typed wrapper overdagster_aws.s3.S3Resource. It defaultsendpoint_urlto an S3-compatible endpoint, accepts provider-native credential field names as aliases for the AWS-named boto3 fields, and reads matching environment variables as a fallback when a field is not set. Explicit constructor values always take precedence over environment variables, and an explicitaws_access_key_id/aws_secret_access_keystill wins over the aliased fields.get_client()mirrorsdagster_aws.s3.utils.construct_s3_clientand appends adagster-backblaze/{version}token to the botocoreuser_agent_extra(merged with the retry config so neither is lost).dagster-obstoreS3 change so its existingS3ObjectStoreresource can target any S3-compatible endpoint.regionandendpointare now threaded into the obstoreS3Configwhen set, aclient_optionspassthrough is added, and adagster-obstore/{version}user-agent token is set by default (without clobbering a caller-supplied one). The resource still works against AWS S3 unchanged whenendpointis left unset; this only makes the existing endpoint field usable end to end.Docs are updated in both READMEs to describe the S3-compatible behavior generically, naming AWS S3 first and Backblaze B2 as one example target.
How I Tested These Changes
dagster-backblazeships unit tests (dagster_backblaze_tests/test_resources.py) exercised againstmoto's mocked S3, the same waydagster-awstests its S3 code. They cover:aws_access_key_id/aws_secret_access_keyfields, including the single-field and env-var-only cases;put/getround-trip through the boto3 client returned byget_client();dagster-backblaze/{version}token being appended touser_agent_extrawhile the botocore-built prefix is preserved.Local run from
libraries/dagster-backblaze:ruff(format + check) and thetytype check are clean, matching the repository standards inCONTRIBUTING.md. Thedagster-obstorechange keeps that library's existing behavior for AWS S3 and only fills in the previously unusedendpoint/regionpath plus the user-agent token.Changelog
This is the initial release of a new library, so there is no prior
CHANGELOG.mdto append to. If you would like the new library to carry one from the start (some libraries here do, several recent new-integration PRs merged without one), I am happy to add alibraries/dagster-backblaze/CHANGELOG.mdwith a0.0.1entry in the keepachangelog format before merge. Thedagster-obstorechange is additive and backward compatible; I can add an[Unreleased]note tolibraries/dagster-obstore/CHANGELOG.mdin the same style if you prefer.