Support for streaming requests with `httr2`

[Aariq]

It seems that currently vcr doesn't record cassettes for requests made with httr2::req_perform_stream() or httr2::req_perform_connection(). Maybe this is a limitation of how streaming requests work—I'm not sure.

library(httr2)
vcr::use_cassette("req_perform_stream", {
  show_bytes <- function(x) {
    cat("Got ", length(x), " bytes\n", sep = "")
    TRUE
  }
  resp <- request(example_url()) |>
    req_url_path("/stream-bytes/100000") |>
    req_perform_stream(show_bytes, buffer_kb = 32)
  resp
})
#> Got 32768 bytes
#> Got 32768 bytes
#> Got 32768 bytes
#> Got 1696 bytes
#> Warning: Empty cassette (req_perform_stream) deleted; consider the following:
#>  - If an error occurred resolve that first, then check:
#>  - vcr only supports crul, httr & httr2; requests w/ curl, download.file, etc. are not supported
#>  - If you are using crul/httr/httr2, are you sure you made an HTTP request?
#> <vcr - Cassette> req_perform_stream
#>   Record method: once
#>   Serialize with: yaml
#>   Persist with: FileSystem
#>   Re-record interval (s): 
#>   Clean outdated interactions?: FALSE
#>   update_content_length_header: FALSE
#>   allow_playback_repeats: FALSE
#>   allow_unused_http_interactions: 
#>   exclusive: 
#>   preserve_exact_body_bytes: FALSE

vcr::use_cassette("req_perform_connection", {
  req <- request(example_url()) |>
    req_url_path("/stream-bytes/32768")
  resp <- req_perform_connection(req)
  
  length(resp_stream_raw(resp, kb = 16))
  length(resp_stream_raw(resp, kb = 16))
})
#> Warning: Empty cassette (req_perform_connection) deleted; consider the following:
#>  - If an error occurred resolve that first, then check:
#>  - vcr only supports crul, httr & httr2; requests w/ curl, download.file, etc. are not supported
#>  - If you are using crul/httr/httr2, are you sure you made an HTTP request?
#> <vcr - Cassette> req_perform_connection
#>   Record method: once
#>   Serialize with: yaml
#>   Persist with: FileSystem
#>   Re-record interval (s): 
#>   Clean outdated interactions?: FALSE
#>   update_content_length_header: FALSE
#>   allow_playback_repeats: FALSE
#>   allow_unused_http_interactions: 
#>   exclusive: 
#>   preserve_exact_body_bytes: FALSE

^{Created on 2025-01-21 with reprex v2.1.1}

It would be nice to either have this "just work" or get a more customized warning for streaming requests. This comes from me re-factoring code originally using curl and comments in the tests mention that original stubs were created manually, so perhaps that is an option I should explore?

[sckott]

Thanks for the issue @Aariq

I'll have a look but i'm guessing it's because the streaming fxns don't have a hook for vcr to hook into like the normal http request infra has in httr2

[sckott]

yeah if you look at code for both the parallel and stream perform functions there's no mock param which is what we use to hook into the httr2 package.

when I find some time I can see what can be done. of course PRs are welcome

[sckott]

opened an issue r-lib/httr2#651 to see if would be considered or not to add mocking ability

[sckott]

hadley +1'ed it, i'll have a go at contributing the change over there

[Aariq]

I saw that, thanks! I'll keep an eye on both issue to see if there's anything I can do.

[sckott]

@Aariq httr2 now has mocking support for the streaming functions - so now I'm working on bringing vcr support for streaming - not there yet - i'll ping you to try it out when it's ready

[Aariq]

Sweet!