Extracting hyper::Request

[ayoshi]

Is there any way / plans to allow filter to extract hyper::Reqest as is? The use case I'm aiming for: for some path/header combination filters I would like to proxy the request through into hyper::Client to another servers, modify the response and return it back (mitmproxy style).

[seanmonstar]

I figure this would probably eventually be a thing built-in. As for why it's not yet, I wanted to prevent nailing down exact details about the underlying request (like it being Request<hyper::Body>).

It is possible already to use some filters to extract the pieces you need, like warp::method(), warp::body::stream(), etc.

[ayoshi]

Yes, currently that's what I'm doing:

path::full() // Extract path
        .and(header::headers_cloned()) // Extract all headers
        .and(body::concat()) // Extract body
        .and(method::method()); // Extract method

Though, it would be nice to extract basic types eventually.
On the other note: I have to say, I'm blown away by the whole mechanics of filters, it's extremely beautiful, thank you for all your work on warp and hyper.

[howesteve]

+1 on this feature please. I just want to get the IP from the requester to implement DoS flood control and can't seem to do it as it is...

Thanks

[seanmonstar]

If anyone wants a simple drop-in filter they can use now, this would do it:

fn extract_request() -> impl Filter<Extract=(http::Request<warp::body::BodyStream>,), Error=warp::Rejection> + Copy {
    warp::method()
        .and(warp::path::full())
        .and(warp::headers::headers_cloned())
        .and(warp::body::stream())
        .map(|method: http::Method, path: warp::path::FullPath, headers: http::HeaderMap, body: warp::body::BodyStream| {
            let mut req = http::Request::builder()
                .method(method)
                .uri(path.as_str())
                .body(body)
                .expect("request builder");
            *req.headers_mut() = headers;
            req
            *req.method_mut() = method;
        
        })
}

---

We could consider adding the above filter directly in warp, if it seems significantly better than having copied that function. We'd need a name (really, a path, like warp::request::extract_full_request() or something), and docs to remind people of the same thing as in the body filters, that it consumes the body (so if eventually rejected, and another filter tries to take the body, that's an error).

[anderspitman]

@seanmonstar are those last two lines reversed? I haven't tried using this yet so can't say for sure.

[tigert1998]

@seanmonstar

Query string is not present in the code.

[jsdw]

I don't think it's possible to add whatever query string may be present to the request at present (see #686). FYI the code to extract (most of) the request currently looks like this I believe (warp ~0.2.4):

use warp::Filter;
use futures::Stream;

pub fn extract_request() -> impl Filter<Extract=(http::Request<impl Stream>,), Error=warp::Rejection> + Copy {
    warp::method()
        .and(warp::path::full())
        .and(warp::header::headers_cloned())
        .and(warp::body::stream())
        .map(|method: http::Method, path: warp::path::FullPath, headers: http::HeaderMap, body| {
            let mut req = http::Request::builder()
                .method(method)
                .uri(path.as_str())
                .body(body)
                .expect("request builder");
            { *req.headers_mut() = headers; }
            req
        })
}

BodyStream seems to be private, and so it's hard to give it a type aside from just impl Stream.

I'd definitely be in favour of a filter to do this for me, so that I don't need to import futures::Stream and don't have to worry about things like query params being missing.

[kdar]

Could a filter be provided that returns a http::Uri? I know we have host::optional() and path::full(), but seems it would be helpful just to get the full Uri (which would help this issue as well). My usecase is I have an RSS feed generator that needs the full Uri (scheme, authority, path, and query) in order to put a self referential link in the RSS: https://validator.w3.org/feed/docs/warning/MissingAtomSelfLink.html.

[danielSanchezQ]

Hi guys.
I'm working on this: https://crates.io/crates/warp-reverse-proxy, just in case it is helpful for anyone. It exposes the filter I'm using internally to extract the request data. It is not perfect but I'll be delivering features soon.

[chyvonomys]

I'm having similar situation to @kdar: I'm building response with pagination and I need to specify a full URI of next page. Host part can be figured out from headers or host filter, but it would be convenient to have Request URI filter for this.

[Vagelis-Prokopiou]

This is how I do it currently (with warp version 0.3.0):

let routes = warp::any()
        .and(warp::method())
        .and(warp::path::full())
        .and(
            // Optional query string. See https://github.com/seanmonstar/warp/issues/86
            warp::filters::query::raw()
                .or(warp::any().map(|| String::default()))
                .unify()
        )
        .and(warp::header::headers_cloned())
        .and(warp::body::bytes())
        .map(|method: hyper::http::Method, path: warp::path::FullPath, queryParams: String, headers: hyper::http::HeaderMap, body: hyper::body::Bytes| {
            let mut fullPath = path.as_str().to_string();
            if queryParams != "" { fullPath = format!("{}?{}", fullPath, queryParams); }
            let mut hyperRequest = hyper::http::Request::builder()
                .method(method)
                .uri(fullPath)
                .body(hyper::body::Body::from(body))
                .expect("Request::builder() failed");
            { *hyperRequest.headers_mut() = headers; }
            return hyperRequest;
        })
        .and_then(|hyperRequest: hyper::Request<hyper::Body>| {
            // handler signature: async fn handler(mut request: Request<Body>) -> Result<Response<Body>>
            let result = handler(hyperRequest)
                .map_err(|_e| {
                    // Todo: Add proper error mapping.
                    return warp::reject::not_found();
                }); // map_err is a future method
            return result;
        });

[johannescpk]

In case someone is looking for a filter that directly extracts http::Request, here you go

pub fn http_request() -> impl Filter<Extract = (http::Request<Bytes>,), Error = Rejection> + Copy {
    // TODO: extract `hyper::Request` instead
    // blocked by https://github.com/seanmonstar/warp/issues/139
    warp::any()
        .and(warp::method())
        .and(warp::filters::path::full())
        .and(warp::filters::query::raw())
        .and(warp::header::headers_cloned())
        .and(warp::body::bytes())
        .and_then(|method, path: FullPath, query, headers, bytes| async move {
            let uri = http::uri::Builder::new()
                .path_and_query(format!("{}?{}", path.as_str(), query))
                .build()
                .map_err(Error::from)?;

            let mut request = http::Request::builder()
                .method(method)
                .uri(uri)
                .body(bytes)
                .map_err(Error::from)?;

            *request.headers_mut() = headers;

            Ok::<http::Request<Bytes>, Rejection>(request)
        })
}

#[derive(thiserror::Error, Debug)]
pub enum Error {
    #[error(transparent)]
    Http(#[from] http::Error),
}