Tagless Final instead of hard-coded futures

[nightscape]

I've been digging through the code a little and noticed that the direct use and execution of Futures brings some inconveniences with it:

• An ExecutionContext has to be passed everywhere.
• You can't use other libraries like Monix to perform the execution.
• You can't reflect on the execution (e.g. in what sequence is stuff done).
• Tests always have to await.

I was wondering if you have considered / would consider a Final Tagless encoding instead of direct use of futures?

[OlegIlyenko]

Could you please describe in more detail what you are referring to with Final Tagless encoding?

If your mean the parametrization on an effect type (similar to fs2 and some other libraries), then it is definitely a thing to consider. It was discussed several times in past. While there are definitely some benefits to it, it also involves a fair amount of complexity for both: library implementation as well as library user. So far we tried to approach this matter carefully and with a careful analysis of the benefits it might bring compared to added complexity. At the moment library relies on standard scala Future as a representation of an effect type. So the topic is definitely open for a discussion, but I think it needs to be approached carefully and discussed in detail beforehand.

Regarding individual points

You can't use other libraries like Monix to perform the execution.

You are right about the monix Task, it needs to be converted to a Future. When it comes to streaming results (e.g. for subscription queries or batch queries), monix Observable is supported via sangria-monix when used with ExecutionScheme.Stream or ExecutionScheme.StreamExtended.

Tests always have to await.

I guess it depends on the capabilities of the testing library. For example ScalaTest natively supports async testing.

You can't reflect on the execution (e.g. in what sequence is stuff done).

I would be interested to learn more about this topic and use-cases for it. As far as I'm aware, even in case of free monad, for instance, the introspective analysis is relatively limited, at least as long a FlatMapped is involved. But I might be wrong on this one and not aware of the full scope of this matter. So I would appreciate if you could share more details and maybe some practical real-world use-cases for it.

Also, I think it is interesting to consider how DeferredValue and deferred resolvers fit into this picture. They provide a lot of flexibility when in comes to data loading.

[nightscape]

Hi @OlegIlyenko,

If your mean the parametrization on an effect type (similar to fs2 and some other libraries), then it is definitely a thing to consider

Yes, exactly. Final Tagless is related to Free Monads, but more light-weight. You don't get stack safety by default, but that only matters if you have deep recursion and then you could still use Final Tagless to build a Free structure (see e.g. here: https://softwaremill.com/free-tagless-compared-how-not-to-commit-to-monad-too-early/).

While there are definitely some benefits to it, it also involves a fair amount of complexity for both: library implementation as well as library user.

I agree that it's more effort for the library implementors, but I don't think there would be much difference for the library users (given that the required implicit handling is done well).
Complexity (as defined by Rich Hickey in his fabulous Simple Made Easy talk) should be less because you're not "complecting" program logic with execution.

So the topic is definitely open for a discussion, but I think it needs to be approached carefully and discussed in detail beforehand.

I definitely agree. At some point one would need some code fragments though to make things concrete.

I guess it depends on the capabilities of the testing library. For example ScalaTest natively supports async testing.

Right, and that point is probably the weakest argument. Debugging is a little easier when things are synchronous, but that will probably be less of an issue the more intelligent IDEs become.

I would be interested to learn more about this topic and use-cases for it.

The obvious use-case is to integrate Sangria with other libraries that use effect types other than Future
Another thing that I hope this would bring is to make some aspects of Sangria more uniform.
It should e.g. be possible to write pretty-printing, serialization and schema validation of GraphQL ASTs as different interpreters of the same Final Tagless program (as e.g. here).

As far as I'm aware, even in case of free monad, for instance, the introspective analysis is relatively limited, at least as long a FlatMapped is involved. But I might be wrong on this one and not aware of the full scope of this matter.

The introspection in that case would not happen by analyzing a Free data structure (which as you say can contain opaque FlatMaps) but by running the Final Tagless program through an analysis interpreter.

[OlegIlyenko]

Complexity (as defined by Rich Hickey in his fabulous Simple Made Easy talk) should be less because you're not "complecting" program logic with execution.

I feel that it is also important to consider the developer experience and API simplicity as well as execution semantics. Correct me if I'm wrong, but introducing this feature would imply that almost every schema definition API elements (like Schema, ObjectType, Field, DeferredResolver, etc.) will get a new type parameter. For example, for a Field it might look like this:

case class Field[Ctx, Val, F[_] : Effect](...)

In my opinion, introducing a new higher-kinded type parameter in most API elements also counts a big complexity increase for an API user. Also, consider the Field.apply which already has 4 type parameters to ensure all necessary type constraints:

def apply[Ctx, Val, Res, Out, F[_] : Effect](...) = ...

So far I tried to add type parameters only in cases where there is a very good practical justification for it.

For example, recently I was working on http4s-based HTTP server. All of the API elements are parametrized on an effect type, yet as a user I only used IO. I felt that this extra abstraction was not really necessary and was only adding complexity, cryptic compilation errors and more round trips to the docs. So I felt that it can be simplified by just baking in IO in http4s. I might be wrong on this one, and I would be interested to hear other opinion that favours parametrization on an effect type in this particular scenario. (disclaimer: I had pleasure using http4s and i <3 the lib :) this comment is just about one specific aspect of the library)

The obvious use-case is to integrate Sangria with other libraries that use effect types other than Future

I agree, I also see it as a good argument in favour of parameterizing on an effect type (or any other alternative solution that will allow effect types other than scala Future). But how does it relate to the original point: "You can't reflect on the execution"?

Another thing that I hope this would bring is to make some aspects of Sangria more uniform.
It should e.g. be possible to write pretty-printing, serialization and schema validation of GraphQL ASTs as different interpreters of the same Final Tagless program (as e.g. here).

I feel that some concepts are a bit mixed-up. Sangria provides a lot of tools for static analysis of questies and schemas which includes things like pretty printing and validation. But all of these normally do not involve actual execution (they are done beforehand).

The introspection in that case would not happen by analyzing a Free data structure (which as you say can contain opaque FlatMaps) but by running the Final Tagless program through an analysis interpreter.

Yes exactly, I also see it this way. The question I often struggle with is: in absence of ability to statically analyze Free data structure, what additional value might this interpreter provide? Don't get me wrong, I actually like the concept Free monads, but this is the same question I often ask myself when I thinking about integrating it. In particular, I'm interested in practical real-world use-cases for it in context of GraphQL query execution.

For example, one of the interesting practical application areas might be creation of an execution plan and its optimization. For instance, if you are using SQL DB to fetch all of the data, it might be very interesting to create an execution plan during the actual GraphQL execution (instead of eagerly executing it) and represent it as a Free data structure. But because of the opaque elements I find it quite hard to do. I guess this might be possible with more domain-specific variation Free structure with less opaque elements. But I'm not sure how custom interpreter might help in this respect with Free as it is defined in, let's say cats. Not to mention, that if we will look at bigger picture, the same can be achieved with a pure GraphQL query analysis For example, I believe libraries like join-monster are creating an optimized execution plan and SQL queries based on GraphQL query analysis alone (with some extra info from the GraphQL schema).

[guersam]

Another possible usecase might be interpreting a sangria query AST with fetch for efficient data fetching, e.g. parallelism, caching results for same identifiers.

It's also having a design change to support parameterized effect type like suggested here.

[DougC]

For Field it might look like this:

case class Field[Ctx, Val, F[_] : Effect](...)

In fact, in most places you would not need to declare F[_] as an effect. So it might look like this:

case class Field[F[_], Ctx, Val](...)

cats-effect provides a number of type classes that describe the needed effect more granularly, e.g. Async, Sync, Concurrent. But even then, a lot of code only needs to declare F[_] to be something like Functor (if you use map) or Monad (if you use flatMap), etc.

Your point about how this looks to the user of the library is valid and something to consider. One approach could be to have the core library be parameterized on F[_], which would be compatible with http4s and other parts of the cats-effect ecosystem, and then provide a version of the types that are specialised to Future for backwards compatibility and easier use when integrating into, say, Play or Akka Http.

[schrepfler]

I'm not sure the complexity budget is there to introduce these abstractions.
How about providing isomorphic instances in the scope to convert between the context into the other effect type you want to use? It works for interop between Scala and Twitter futures.
Then you write your API's with the effect monad and keeping your core pure but the shell can be impure (sangria), in essence, the interpreter stays simple and lets Oleg do it's magic whatever way he likes it.

[paulpdaniels]

I'd be at least interested in seeing if someone could create a spike of this approach. Maybe not even a fully functional one, but sufficient to demonstrate that a) the abstraction doesn't drastically increase the complexity of the internals (having looked through them they are already quite complex). b) That we could preserve a close approximation of the existing api surface perhaps by having as @DougC mentioned a sangria-future module.

[nightscape]

I just stumbled upon another interesting use case, optimizing the execution:
https://typelevel.org/blog/2017/12/27/optimizing-final-tagless.html
The author @LukaJCB even mentions GraphQL as possible use case 😄

[eugkhp]

Hi! Does anybody know if there were any progress on this?

[nightscape]

Hi @eugkhp, the main maintainer of this project has passed away 😢
https://www.reddit.com/r/scala/comments/bn1ia9/sangria_creator_oleg_ilyenko_has_passed_away/?rdt=49364
Some people are still keeping this project alive though.
You might also consider https://ghostdogpr.github.io/caliban/

[yanns]

We now have minimal support for effects types: #987

[eugkhp]

Oh, thanks for letting me know 😢

@yanns thanks! I will take a look and maybe contribute something we did to wrap in F