"Distilling" StaticHttp Data

[dillonkearns]

I was thinking about the idea of "distilling" optimized decoders the wrong way.

Why do we need distillation?

Here are some distillation use cases:

• Isolating sensitive data, like user data, secrets, etc., that you don't want to return in the bundled payload response
• Reducing down data that decodes a ton of data, but can be distilled down to something much more compact. For example, if you were to read an entire file, but then you only want to keep the first paragraph of that file.

Distillation wouldn't be relevant in a context like a file generator, where no resulting data is bundled into the app. For file generators, it gets the data, and then it's already just distilling things down to a single thing: a file to generate (like an RSS feed file, for example).

Distillation is a performance optimization tool and a data protection tool.

At first, I was thinking about providing some places to hook into isolating data, like taking an incoming server response for a server-side rendered page request, and building up some context where you get a user.

You might have some tokens or other sensitive data coming back from an intermediary response in a chain of StaticHttp.andThens. All of the data that came along, including just a token you fetched for example, would get included in the bundle for that page. But you shouldn't be returning that data in the page.

I could provide a specific hook for that spot where you can "isolate" the data with this Codec approach (explicitly put the data you need in here, and tell me how to build it up as JSON and then Decode it - it gives you a clean slate). But I realized, why not make this a general-purpose tool if we can, so you can use it in any context, whether you need it for performance reasons or for privacy/security.

Finding the right place to distill data

The first big thing I got wrong was that I was thinking about it as a way of modifying some JSON in-place. That leads to all sorts of consistency problems:

• What if you decode the same field two places, but you've "distilled" it in one of the path's?
• And what name do you put the modified data under?
• What if it's not a field, but a raw value?

It totally breaks down.

The second problem is that I was assuming that distillation would be an abstraction on top of an OptimizedDecoder. I realize now it should be an abstraction of StaticHttp.

OptimizedDecoders calculate what unused data can be pruned in the JSON. But StaticHttp actually prunes the data and bundles the data you depend on. For example, StaticHttp.succeed () doesn't depend on any data.

So I'm realizing now that the concept of distilling could be done as an abstraction in StaticHttp.

module StaticHttp ...

distill : Codec a -> StaticHttp.Request a -> StaticHttp.Request a

The Codec part means that you can distill the payload down to JSON, so that resulting encoded JSON is what gets included in the bundle. Then you can use the Codec to decode the data on the other side.

[dillonkearns]

My initial thoughts on implementing this:

Don't store intermediary StaticHttp responses

Add an additional argument to the request resolver function in a few places to determine whether to store the results or not. When we're running a distilled part of a StaticHttp chain, we do not want to store the results except for at the very last step. The new argument would be wired in a few places like this:

elm-pages/src/Pages/StaticHttp.elm

Line 566 in be8d145

, \appType rawResponseDict ->

elm-pages/src/Pages/StaticHttpRequest.elm

Line 13 in be8d145

= Request ( List (Secrets.Value Pages.StaticHttp.Request.Request), ApplicationType -> RequestsAndPending -> Result Error ( Dict String String, Request value ) )

Then the implementation will look similar to map, but we would wire in the new argument to tell it to skip storing the intermediary StaticHttp responses:

elm-pages/src/Pages/StaticHttp.elm

Lines 156 to 171 in be8d145

	map : (a -> b) -> Request a -> Request b
	map fn requestInfo =
	case requestInfo of
	Request ( urls, lookupFn ) ->
	Request
	( urls
	, \appType rawResponses ->
	lookupFn appType rawResponses
	|> Result.map
	(\( partiallyStripped, nextRequest ) ->
	( partiallyStripped, map fn nextRequest )
	)
	)
	Done value ->
	fn value |> Done

And then the top-level distill call will store the final result after running the encoder from the distill Codec.

Distilled result storage key

We need a unique key to distill the result. This may end up being a case where we need to just ask the user to take on the responsibility of making sure they use a unique key.

The signature to provide a unique storage key could look like this:

distill : String -> Codec a -> StaticHttp.Request a -> StaticHttp.Request a

If we had the ability to do Lisp-style meta-programming, then we could inspect the ASTs of the Codec and StaticHttp request coming in and use that to get a unique key.

There are two things that could go wrong:

• The key is not unique. Then the user would end up fetching data from the wrong key in one of the cases that has the same non-unique key.
• The key is different when run in the build vs. in the browser stage. In this case, we would end up looking up a key that won't match, and would get no data (or incorrect data). Although I think this is prevented to some degree by the fact that the user doesn't have access to non-deterministic data in places where we make StaticHttp requests, which is by design.