Skip to content

feat(dagster-backblaze): add Backblaze B2 (S3-compatible) integration#329

Draft
goanpeca wants to merge 1 commit into
dagster-io:mainfrom
goanpeca:b2-integration
Draft

feat(dagster-backblaze): add Backblaze B2 (S3-compatible) integration#329
goanpeca wants to merge 1 commit into
dagster-io:mainfrom
goanpeca:b2-integration

Conversation

@goanpeca

@goanpeca goanpeca commented Jul 1, 2026

Copy link
Copy Markdown

Summary & Motivation

Adds dagster-backblaze, a small new integration for S3-compatible object storage, and makes the existing dagster-obstore S3 resource work against the same class of endpoint.

The Dagster ecosystem already talks to AWS S3 through dagster-aws. Any object store that speaks the S3 API (AWS S3 itself, and other S3-compatible providers such as Backblaze B2) works with the same boto3 client once you point it at the right endpoint and hand it the right credential fields. The friction is purely ergonomic: a non-AWS S3 target means remembering an endpoint URL and supplying credentials through AWS-named fields that do not match the provider's own naming.

B2Resource removes that friction for one such provider without forking any behavior. It subclasses dagster_aws.s3.S3Resource, pre-fills the endpoint, and accepts the provider's own credential field names, mapping them onto the boto3 aws_access_key_id / aws_secret_access_key parameters the parent already forwards. Under the hood you still get a plain boto3 S3 client, so anything that works with S3Resource works here.

The change is intentionally minimal, additive, and follows the existing library layout (Makefile, README.md, pyproject.toml, uv.lock, package + tests, _is_dagster_maintained() returning False).

What is included:

  • New library libraries/dagster-backblaze exposing B2Resource, a thin typed wrapper over dagster_aws.s3.S3Resource. It defaults endpoint_url to an S3-compatible endpoint, accepts provider-native credential field names as aliases for the AWS-named boto3 fields, and reads matching environment variables as a fallback when a field is not set. Explicit constructor values always take precedence over environment variables, and an explicit aws_access_key_id / aws_secret_access_key still wins over the aliased fields. get_client() mirrors dagster_aws.s3.utils.construct_s3_client and appends a dagster-backblaze/{version} token to the botocore user_agent_extra (merged with the retry config so neither is lost).
  • A small dagster-obstore S3 change so its existing S3ObjectStore resource can target any S3-compatible endpoint. region and endpoint are now threaded into the obstore S3Config when set, a client_options passthrough is added, and a dagster-obstore/{version} user-agent token is set by default (without clobbering a caller-supplied one). The resource still works against AWS S3 unchanged when endpoint is left unset; this only makes the existing endpoint field usable end to end.

Docs are updated in both READMEs to describe the S3-compatible behavior generically, naming AWS S3 first and Backblaze B2 as one example target.

How I Tested These Changes

dagster-backblaze ships unit tests (dagster_backblaze_tests/test_resources.py) exercised against moto's mocked S3, the same way dagster-aws tests its S3 code. They cover:

  • the default endpoint value and an explicit override;
  • the endpoint environment variable overriding the default but not an explicit constructor value;
  • the provider-native credential fields mapping onto the boto3 aws_access_key_id / aws_secret_access_key fields, including the single-field and env-var-only cases;
  • explicit AWS-named credentials winning over the aliased fields;
  • a full put / get round-trip through the boto3 client returned by get_client();
  • the dagster-backblaze/{version} token being appended to user_agent_extra while the botocore-built prefix is preserved.

Local run from libraries/dagster-backblaze:

make test    # uv run pytest
make ruff    # uv run ruff check --fix && uv run ruff format
make check   # uv run ty check .

ruff (format + check) and the ty type check are clean, matching the repository standards in CONTRIBUTING.md. The dagster-obstore change keeps that library's existing behavior for AWS S3 and only fills in the previously unused endpoint / region path plus the user-agent token.

Changelog

This is the initial release of a new library, so there is no prior CHANGELOG.md to append to. If you would like the new library to carry one from the start (some libraries here do, several recent new-integration PRs merged without one), I am happy to add a libraries/dagster-backblaze/CHANGELOG.md with a 0.0.1 entry in the keepachangelog format before merge. The dagster-obstore change is additive and backward compatible; I can add an [Unreleased] note to libraries/dagster-obstore/CHANGELOG.md in the same style if you prefer.

Signed-off-by: Gonzalo Peña-Castellanos <goanpeca@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants