Skip to content

feat(stdlib): add http_request function #1360

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 15 commits into
base: main
Choose a base branch
from

Conversation

benjamin-awd
Copy link

@benjamin-awd benjamin-awd commented Mar 27, 2025

Summary

This PR adds a HTTP request function, allowing users to fetch data from external sources.

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

How did you test this PR?

Added tests, otherwise currently running this in a Kubernetes environment via Helm.

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on
    our guidelines.
  • No. A maintainer will apply the "no-changelog" label to this PR. (not sure if we want to document this or not, if we follow the example of dns_lookup)

Checklist

  • Our CONTRIBUTING.md is a good starting place.
  • If this PR introduces changes to LICENSE-3rdparty.csv, please
    run dd-rust-license-tool write and commit the changes. More details here.
  • For new VRL functions, please also create a sibling PR in Vector to document the new function.

References

Closes #1359

@pront pront self-assigned this Mar 28, 2025
Copy link
Member

@pront pront left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding the Cargo.lock, the is a large number of line changes; can you revert and re-apply the cargo update only for the affected dependencies?

@benjamin-awd
Copy link
Author

benjamin-awd commented Jun 11, 2025

@pront I ran the following:

git checkout main Cargo.lock Cargo.toml

cargo add [email protected] --no-default-features --features rustls-tls
cargo add [email protected]
cargo add [email protected]

but unfortunately the lock-file changes are pretty big. I suppose it must be that reqwest a lot of dependencies, even with some default features disabled

I think it would be good to keep the other default features if possible, e.g. support for HTTP/2

@pront pront added meta: awaiting author Pull requests that are awaiting their author. no-changelog Changes in this PR do not need user-facing explanations in the release changelog labels Jun 12, 2025
@pront pront enabled auto-merge June 16, 2025 19:44
@pront
Copy link
Member

pront commented Jun 16, 2025

Oh yes, WASM builds fail: https://github.com/vectordotdev/vrl/actions/runs/15653270457/job/44203895862?pr=1360

It's possible to support it but you can also disable this function from WASM builds.

auto-merge was automatically disabled June 17, 2025 14:43

Head branch was pushed to by a user without write access

@benjamin-awd
Copy link
Author

Hey @pront, I've disabled it from the WASM build for now

@pront pront enabled auto-merge June 17, 2025 14:53
auto-merge was automatically disabled June 18, 2025 15:48

Head branch was pushed to by a user without write access

@pront pront enabled auto-merge June 18, 2025 17:43
auto-merge was automatically disabled June 19, 2025 13:55

Head branch was pushed to by a user without write access

@pront pront enabled auto-merge June 20, 2025 19:17
@pront
Copy link
Member

pront commented Jun 24, 2025

Ah, there's a license not recognized by deny: https://github.com/vectordotdev/vrl/actions/runs/15759598375/job/44502953608?pr=1360

# MPL-2.0 are added case-by-case to make sure we are in compliance. To be in
# compliance we cannot be modifying the source files.
{ allow = ["MPL-2.0"], name = "vrl", version = "*" },
{ allow = ["CDLA-Permissive-2.0"], name = "webpki-roots", version = "*" },
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to think a bit about this one. No actions needed from you @benjamin-awd. Thank you for your patience!

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood, thanks @pront!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
meta: awaiting author Pull requests that are awaiting their author. no-changelog Changes in this PR do not need user-facing explanations in the release changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add http_get function to VRL
2 participants