Collect request invocation headers in RequestCtx

[calavera]

When I trigger a new request, I would like my application to access the headers that I sent in the request invocation call. Applications that integrate with Tensorlake might want to send information in headers that are common in system integrations. There are two examples that come to my mind where this would be useful, but there might be more:

1. Integrating with external observability services: A caller sends a Trace-ID header and wants to preserve it across our applications invocations.
2. Verifying payload integrity: It's very common in integration systems like webhooks clients to verify the integrity of the payload by sharing a token between the server and the receiver as a header. Our own documentation explains this use case: https://docs.tensorlake.ai/webhooks/overview#secure-your-endpoints-with-signature-verification

Security considerations

Because every request invocation requires authentication, we should not propagate the Authentication header, at least by default. We should have a deny list of headers in the server that we don't collect, including the Authentication header. This will prevent us from storing the header in the serialized RequestCtx.

In the future, we could allow applications to explicitly indicate a list of allowed headers that the user wants to collect.

[eabatalov]

What UX is going to be there on applications side? We don't currently provide any HTTP level interface in applications.

[calavera]

What UX is going to be there on applications side? We don't currently provide any HTTP level interface in applications.

this is how I think the applications side could look like:

from svix.webhooks import Webhook

from tensorlake.applications import application, function, RequestContext

@application()
@function(secrets=["WEBHOOK_SECRET"])
def webhook_receiver(request_body: str) -> dict:
    """
    Receives and validates Tensorlake webhook notifications.

    Expects:
    - request_body: The raw webhook payload as a string
    - webhook-id: The webhook-id header value for verification
    - webhook-timestamp: The webhook-timestamp header value for verification
    - webhook-signature: The webhook-signature header value for verification
    """

    webhook_secret = os.getenv("WEBHOOK_SECRET")
    if not webhook_secret:
        raise ValueError("WEBHOOK_SECRET environment variable not set")

    ctx = RequestContext.get()
    headers = {
        "webhook-id": ctx.headers["webhook-id"],
        "webhook-timestamp": ctx.headers["webhook-timestamp"],
        "webhook-signature": ctx.headers["webhook-signature"],
    }

    # Verify the signature using Svix
    try:
        webhook = Webhook(webhook_secret)
        payload = webhook.verify(request_body, headers)
    except Exception as e:
        print(f"Signature verification failed: {str(e)}")
        return {"success": False, "error": f"Signature verification failed: {str(e)}"}

[eabatalov]

This is interesting and it might work. Integrates with existing UX smoothly.
As I understand the idea is that in some cases user doesn't control request payload in fully and parts of the request payload are sent in headers. Makes sense in this case.

[eabatalov]

I implemented HTTP request parsing in FE. It detects if application function call payload content-type header is message/http and then parses its headers and body as usual.

Here's example Server side changes needed to start forwarding full HTTP requests to FE: b35b743

We can't enable it yet because users in prod are still running older FEs without this feature. We can re-visit this in a few months and see if these FEs are not used anymore and then switch to full http request forwarding mode in Server/Frontend.