feat(azure_blob sink): Update crates and migrate to the new SDK #24255

joshcoughlan · 2025-11-17T18:17:06Z

This PR migrates the Azure Blob sink to the latest official Azure SDK crates and aligns our internal auth/HTTP plumbing with the new pipeline model.

Highlights:

Replace legacy crates with official SDK:
- azure_storage_blob → 0.7.x
- azure_core → 0.30.x
- Remove azure_storage_blobs/azure_storage/azure_core_for_storage
Preserve existing auth modes:
- SAS via URL stays supported
- Access key via a custom Shared Key Policy integrated into the azure_core pipeline
Add a minimal connection string parser to support SAS and Shared Key without pulling in legacy crates
Keep the rest of Vector on reqwest 0.11.26 while the Azure SDK uses reqwest 0.12.24:
- Introduce an aliased reqwest_0_12_24 dependency and inject it as the Azure transport
- Avoid rippling reqwest upgrades through non-Azure code paths

Follow-ups planned:

Per-sink proxy configuration for the Azure HTTP client (azure_blob proxy support #21314)
Possibly Support Azure Active Directory tokens (Support Azure Active Directory tokens for the Azure Blob Store Sink #24167)

Vector configuration

Examples used during testing.

SAS-based connection string:

[sinks.azure_blob]
type = "azure_blob"
inputs = ["in"]
container_name = "vector-logs"
# Example SAS (redacted); must include the leading '?'
connection_string = "BlobEndpoint=https://<account>.blob.core.windows.net;SharedAccessSignature=?sv=...&ss=b&srt=...&sp=...&se=...&st=...&spr=https&sig=...;"
compression = "gzip"
encoding.codec = "json"

Shared Key (access key) connection string:

[sinks.azure_blob]
type = "azure_blob"
inputs = ["in"]
container_name = "vector-logs"
connection_string = "DefaultEndpointsProtocol=https;AccountName=<account>;AccountKey=<base64key>;EndpointSuffix=core.windows.net"
compression = "gzip"
encoding.codec = "json"
# Uses the custom SharedKeyAuthorizationPolicy added in this PR

How did you test this PR?

Local builds for all supported targets, including cross builds in a containerized environment
Tested in a live environment to confirm functionality

Change Type

Bug fix
[ X] New feature
Non-functional (chore, refactoring, docs)
Performance

Is this a breaking change?

Yes
[X ] No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

References

Notes

This PR fixes an unreported bug introduced by Azure SDK version bump introduced regression in token refresh #23036 in which the documented SAS connection_string ("BlobEndpoint=https://mylogstorage.blob.core.windows.net/;SharedAccessSignature=generatedsastoken") doesn't work and fails with an error indicating it requires AccountName:
ERROR vector::topology::builder: Configuration error. error=Sink "testing-azure_blob": Account name missing in connection string

thomasqueirozb

Thanks for this contribution! I haven't had the time to look closely at everything yet, but I'm leaving some comments now because there is a lot to discuss.

As per your PR description it seems like we're now supporting new auth methods with this upgrade too, so it would be nice to include a changelog in this PR. Let me know if I misunderstood this.

I do have some concerns about using these new azure crates though. The azure_storage_blob crate has this warning:

Note: The azure_storage_blob crate is currently under active development and not all features may be implemented or work as intended. This crate is in beta and not suitable for Production environments. For any general feedback or usage issues, please open a GitHub issue

I see that we are using just some parts of that crate. Namely only BlobContainerClient/BlobContainerClientOptions, clients::BlobClient and models::BlockBlobClientUploadOptions.

Have you checked if they are just a port from what was being used in the old azure_storage_blobs crate or is it a full rewrite? If it is a port I'd have more confidence in using it. Either way I think our hands are tied here because MS doesn't seem to put enough effort into releasing good and stable Azure crates so anything as good as or better than the old azure_storage_blobs will do.

Cargo.toml

src/sinks/azure_common/shared_key_policy.rs

joshcoughlan · 2025-11-17T19:25:15Z

Have you checked if they are just a port from what was being used in the old azure_storage_blobs crate or is it a full rewrite? If it is a port I'd have more confidence in using it. Either way I think our hands are tied here because MS doesn't seem to put enough effort into releasing good and stable Azure crates so anything as good as or better than the old azure_storage_blobs will do.

I haven't, but I suspect it is a full rewrite since I think it is just adding support for auth from a url to the SDK. I wrote a fix for the SDK and submitted it to Microsoft, but was informed that they had fixed what I was trying to fix in a release of azure_storage_blob the day before.

Basically, clients now have a from_url method that allows you to pass a full URL with a SAS. This design closely matches our other SDKs and so it what we wanted to go with for Rust as well. Please give it a try and let us know if you have any feedback.
Azure/azure-sdk-for-rust#3329 (comment)

If I'm remembering properly, the only thing keeping SAS from working before is that the client constructor required the AD auth method. I think now it is making that credential optional, which then opened the door for auth from url.
Azure/azure-sdk-for-rust#3176

Anyway, I went about refactoring my original Vector PR to use the newly released azure_storage_blob SDK and broke it up into two PRs. I'm mainly doing all of this because we need per-sink proxy support and the legacy crates just didn't really support that so much. I have another PR that builds on this one to add proxy support: #24256

As per your PR description it seems like we're now supporting new auth methods with this upgrade too, so it would be nice to include a changelog in this PR. Let me know if I misunderstood this.

Same auth methods, SAS and Access key, are supported, but I think the implementation is different. I think this PR should help address #24167 though, so it can lead to additional auth methods.

thomasqueirozb · 2025-11-21T18:48:44Z

Same auth methods

Changelog not needed then

There are a few failing checks (and merge conflicts), I'll circle back to this once there are new commits on this branch.

fplantinga-guida · 2026-01-06T08:22:09Z

Any updates on this? @thomasqueirozb @joshcoughlan

joshcoughlan · 2026-01-07T16:53:45Z

Any updates on this? @thomasqueirozb @joshcoughlan

I'm working through this today getting the clippy whatnot passing and resolving merge conflicts. I didn't really realize there were other PRs waiting on this.

Also, full disclosure here, I'd be happy if more eyeballs were on this PR as a good amount of it is faith-based coding with AI guidance. I'm none too familiar with rust and having more knowledgeable humans involved wouldn't be a bad thing.

github-actions · 2026-01-07T20:28:43Z

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

Co-authored-by: Thomas <[email protected]>

joshcoughlan · 2026-01-08T02:11:12Z

I have read the CLA Document and I hereby sign the CLA

joshcoughlan · 2026-01-08T02:11:33Z

recheck

thomasqueirozb · 2026-01-09T17:43:45Z

Cargo.toml

 # Azure
-azure_core = { version = "0.25", default-features = false, features = ["reqwest", "hmac_openssl"], optional = true }
-azure_identity = { version = "0.25", default-features = false, features = ["reqwest"], optional = true }
+azure_core = { version = "0.30", default-features = false, features = ["reqwest", "hmac_openssl"] }


Suggested change

azure_core = { version = "0.30", default-features = false, features = ["reqwest", "hmac_openssl"] }

azure_core = { version = "0.30", default-features = false, features = ["reqwest", "hmac_openssl"], optional = true }

This should remain optional as we don't want this to be compiled in when Vector is built with minimal features

Also, looking at https://docs.rs/crate/azure_core/latest/features I'm worried that without reqwest_native_tls and maybe also without reqwest_deflate we can run into some issues...

FWIW reqwest now sets its default TLS feature to use rustls, instead of native-tls.

https://seanmonstar.com/blog/reqwest-v013-rustls-default/

Cargo.toml

src/sinks/azure_blob/integration_tests.rs

src/sinks/azure_common/config.rs

thomasqueirozb · 2026-01-09T20:45:26Z

src/sinks/azure_common/config.rs

+    // Force Azure SDK to use reqwest_0_12_24 transport to avoid affecting global reqwest
+    options.client_options.transport = Some(azure_core::http::Transport::new(std::sync::Arc::new(
+        reqwest_0_12_24::ClientBuilder::new()
+            .build()
+            .map_err(|e| format!("Failed to build reqwest_0_12_24 client: {e}"))?,
+    )));


Why are we using another version of reqwest here? I'd assume that this would work just fine if we used reqwest::ClientBuilder instead - or is there a version issue?

There is a version issue. The MS SDK uses reqwest 0.12 and trying to use 0.11 results in the following error: expected Arc<dyn HttpClient + ’static, Global>, found Arc<Client, Global> (rust-analyzer E0308). From what I understand this is due to HttpClient being implemented in the SDK and the SDK uses reqwest 0.12. I'm going pause here to reiterate that I am a rust novice and there may be better ways to do this. The most straight-forward way I could come up with to use reqwest 0.12 without changing it globally was to use a renamed dependency. Also, I did look at using reqwest 0.12 everywhere, but there would have had to have been updates made to other components.

reqwest is only being used in as a dev dependency in tests. I will bump it to use 0.12 everywhere

It looks like I'm not able to push to your branch. I made a PR here: Automattic#4

src/sinks/azure_common/shared_key_policy.rs

src/sinks/azure_common/connection_string.rs

src/sinks/azure_common/service.rs

thomasqueirozb · 2026-01-15T21:09:55Z

Hi @joshcoughlan I'm not able to get azure integration tests to pass by running ./scripts/run-integration-test.sh int azure. I suspect something is wrong with either the URL or the Authorization header

thomasqueirozb

This is almost good to go. Please merge Automattic#4 and commit these suggestions afterwards. Thanks for your work on this!

Cargo.toml

Bump reqwest to 0.12 and consolidate reqwest versions

joshcoughlan · 2026-01-20T03:08:11Z

Done, but several integration tests for other components are now failing with reqwest 0.12 everywhere.

thomasqueirozb · 2026-01-20T16:26:45Z

Done, but several integration tests for other components are now failing with reqwest 0.12 everywhere.

This is due to reqwest using http v1 and I didn't run clippy for all features locally - my bad! It was at the top of the release notes I just missed it... I'll follow up with another PR to your branch to fix this which is going to restore some of your code.

Use reqwest 0.12 and 0.11. Allow CDLA-Permissive-2.0

thomasqueirozb

Thanks!

joshcoughlan · 2026-01-29T22:30:30Z

Hi. There are some failing tests with make test-vrl in the make environment environment. I fixed a couple simple date updates with the last commit, but I'm not so sure about the others. It seems like the vrl version may need updating somewhere, but I'm not sure.

Also, if you'd rather undo the last commit and just handle that separately that's fine by me.

github-actions bot added the domain: sinks Anything related to the Vector's sinks label Nov 17, 2025

joshcoughlan marked this pull request as ready for review November 17, 2025 18:18

joshcoughlan requested a review from a team as a code owner November 17, 2025 18:18

joshcoughlan mentioned this pull request Nov 17, 2025

azure_blob proxy support #21314

Open

thomasqueirozb reviewed Nov 17, 2025

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

src/sinks/azure_common/shared_key_policy.rs Show resolved Hide resolved

pront added provider: azure Anything `azure` service provider related sink: azure_blob Anything `azure_blob` sink related labels Nov 17, 2025

thomasqueirozb added the no-changelog Changes in this PR do not need user-facing explanations in the release changelog label Nov 18, 2025

thomasqueirozb added the meta: awaiting author Pull requests that are awaiting their author. label Nov 21, 2025

This was referenced Dec 10, 2025

feat(azure_blob sink): Add proxy support #24256

Open

feat(azure_logs_ingestion sink): Initial azure_logs_ingestion sink #22912

Open

github-actions bot removed the meta: awaiting author Pull requests that are awaiting their author. label Jan 7, 2026

joshcoughlan and others added 4 commits January 7, 2026 15:31

Update Azure crates and migrate to the new SDK

a86bbbb

add a reference comment for empty headers

62136c4

Update Cargo.toml

1484192

Co-authored-by: Thomas <[email protected]>

Fix failing tests

7785983

joshcoughlan force-pushed the update_azure_crates branch from 3cd1ced to 7785983 Compare January 7, 2026 22:57

joshcoughlan changed the title ~~feat(azure_blob): Update crates and migrate to the new SDK~~ feat(azure_blob sink): Update crates and migrate to the new SDK Jan 8, 2026

thomasqueirozb reviewed Jan 9, 2026

View reviewed changes

thomasqueirozb added the meta: awaiting author Pull requests that are awaiting their author. label Jan 9, 2026

joshcoughlan added 3 commits January 14, 2026 14:25

Add optional = true to azure_core

74c0515

Add dep:azure_core to sinks-azure_blob

94af583

more changes suggested in the PR

30d4829

github-actions bot removed the meta: awaiting author Pull requests that are awaiting their author. label Jan 14, 2026

thomasqueirozb self-assigned this Jan 15, 2026

joshcoughlan and others added 2 commits January 16, 2026 13:22

Fix shared_key_policy

5faac27

Bump reqwest to 0.12 and consolidate reqwest versions

d75e21f

thomasqueirozb reviewed Jan 16, 2026

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

Cargo.toml Outdated Show resolved Hide resolved

thomasqueirozb added the meta: awaiting author Pull requests that are awaiting their author. label Jan 16, 2026

jlaundry mentioned this pull request Jan 16, 2026

fix(azure_blob sink): Re-add support for Azure token auth via environment variables #24492

Open

8 tasks

Merge pull request #4 from vectordotdev/update_azure_crates

6b631d8

Bump reqwest to 0.12 and consolidate reqwest versions

github-actions bot removed the meta: awaiting author Pull requests that are awaiting their author. label Jan 20, 2026

remove azure_identity crate, use defaults for other azure

89f3e9f

thomasqueirozb and others added 4 commits January 21, 2026 14:23

Merge remote-tracking branch 'origin/master' into update_azure_crates

2ba78cc

Separate reqwest v0.11 and v0.12 again

28d9027

Allow CDLA-Permissive-2.0

cbfbf4e

Merge pull request #5 from vectordotdev/update_azure_crates

fc8d910

Use reqwest 0.12 and 0.11. Allow CDLA-Permissive-2.0

thomasqueirozb approved these changes Jan 29, 2026

View reviewed changes

update dates in tests to 2026 so they pass

2808b22

joshcoughlan requested a review from a team as a code owner January 29, 2026 22:28

github-actions bot added the domain: external docs Anything related to Vector's external, public documentation label Jan 29, 2026

	azure_core = { version = "0.30", default-features = false, features = ["reqwest", "hmac_openssl"] }
	azure_core = { version = "0.30", default-features = false, features = ["reqwest", "hmac_openssl"], optional = true }

feat(azure_blob sink): Update crates and migrate to the new SDK #24255

Are you sure you want to change the base?

feat(azure_blob sink): Update crates and migrate to the new SDK #24255

Conversation

joshcoughlan commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Vector configuration

How did you test this PR?

Change Type

Is this a breaking change?

Does this PR include user facing changes?

References

Notes

Uh oh!

thomasqueirozb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

joshcoughlan commented Nov 17, 2025

Uh oh!

thomasqueirozb commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fplantinga-guida commented Jan 6, 2026

Uh oh!

joshcoughlan commented Jan 7, 2026

Uh oh!

github-actions bot commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joshcoughlan commented Jan 8, 2026

Uh oh!

joshcoughlan commented Jan 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thomasqueirozb commented Jan 15, 2026

Uh oh!

thomasqueirozb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

joshcoughlan commented Jan 20, 2026

Uh oh!

thomasqueirozb commented Jan 20, 2026

Uh oh!

thomasqueirozb left a comment

Choose a reason for hiding this comment

Uh oh!

joshcoughlan commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

joshcoughlan commented Nov 17, 2025 •

edited

Loading

thomasqueirozb commented Nov 21, 2025 •

edited

Loading

github-actions bot commented Jan 7, 2026 •

edited

Loading

joshcoughlan commented Jan 29, 2026 •

edited

Loading