Composable event sourcing

[SemanticBeeng]

This is a design exploration of composability / orchestration patterns.

Resources as context

1. "COMPOSABLE EVENT SOURCING WITH MONADS"
   http://www.lambdadays.org/static/upload/media/1519724460390566danielkrzywickieventsourcing.pdf

2. https://stackoverflow.com/questions/51160457/event-sourcing-without-cqrs

3. https://microservices.io/patterns/data/saga.html

4. "The Application of Petri Nets to Workflow Management"
   http://wwwis.win.tue.nl/~wvdaalst/publications/p53.pdf

"State-based instead of event-based In contrast with many other process modeling techniques, the state of case can be modeled explicitly in a Petri net. Process modeling techniques ranging from informal techniques such as dataflow diagrams to formal techniques such as process algebra’s are event-based, i.e., transitions are modeled explicitly and the states between subsequent transitions are modeled implicitly. Today’s workflow management systems are typically event-based, i.e., tasks
are modeled explicitly and states between subsequent tasks are suppressed. The distinction between an event-based and a state-based description seems to be very subtle, but examples in this paper (e.g. Figure 16) show that this is of the utmost importance for workflow modeling. For example, in most
workflow management systems which abstract from states it is not possible to use the implicit OR-split."

Questions.

1. Are event sourcing and CQRS welded together in Aecor or can they be used separately?

2. Can Aecor be used to implement all use cases from "COMPOSABLE EVENT SOURCING WITH MONADS" ? (please skim through presentation to offer an initial assessment if any show stoppers exist before going deeper to prove)

3. How to implement checkpointing rolling snapshots with Aecor (could be thought of related to composability). (see https://cqrs.files.wordpress.com/2010/11/cqrs_documents.pdf for definition)

In the banking app example the assumption of all account transactions being held in memory is not realistic. Thinking that if in a real implementation the transactions would be stored in another persistent repository and the aggregate would keep the transactions not yet checkpointed. If so, then these methods would need to be implemented differently

aecor/modules/example/src/main/scala/aecor/example/account/EventsourcedAlgebra.scala

Lines 42 to 46 in be6b979

	if (account.hasProcessedTransaction(transactionId)) {
	().pure[F]
	} else {
	if (account.hasFunds(amount)) {
	append(AccountDebited(transactionId, amount))

.

If this line of thinking is agreeable with you then please share some thoughts on techniques to achieve this. Maybe to suspend the ~ rolling snapshots in a separate effect ?

update 2018/01/06 : Found "Scaling Event-Sourcing at Jet" very useful.

I have a number of real applications with non trivial such business logic it seems it would be warranted to encapsulate in separate effects (could use the other effects like Sync and MoandError for example).

Disclaimer: I know this is rough, sorry, and still working through how composition / orchestration is done in https://github.com/vpavkin/ticket-booking-aecor/blob/master/booking/src/main/scala/ru/pavkin/booking/ProcessWirings.scala (seems CQRS based).

[vpavkin]

@SemanticBeeng Since you reference my example, I'll try to address some questions of yours.

1. Are event sourcing and CQRS welded together in Aecor or can they be used separately?

For sure you can use them separately. As I show in Part 3 of the series, you can launch an eventsourced behavior without any read side. In theory, you can even read the whole state from the same aggregate, although such solution is much harder to scale properly.

Actually there's not a lot Aecor has to offer in terms of CQRS. You can just get a partitioned stream of entity events. That's it, then you do whatever you want with it.

still working through how composition / orchestration is done

Could you please elaborate on what you mean here by orchestration? There are several processes here, which are all different. More specific questions about a specific process would be much easier to answer.

Anyway, Aecor itself does NO orchestration whatsoever. The only remotely related thing it does is it distributes processing over the cluster (and only when it's possible - it requires event stream to be properly partitioned).

Overall, the way processes are defined in my example is inspired by Process Manager pattern, which is much more on choreography side (at least in this form), rather than orchestration.

I plan to dedicate Part 5 of the series to this matter, maybe it will clarify some questions as well :)

[SemanticBeeng]

"behaviors belong to the domain layer and runtime is out there in the infrastructure layer"

Key design choice indeed.

"what you mean here by orchestration?"

As per https://stackoverflow.com/a/29808740/4032515

"Orchestration employs a centralized approach for service composition."
"Choreography employs a decentralized approach for service composition."

In my initial comment above meant to focus on composition with functional programming - a programming model and business process/logic focus and less an architectural/runtime perspective.

Will follow up and improve this answer.

Examples to clarify

1. separating event sourcing from CQRS and
2. suspending the effect of event logging. Would allow us to control distributed business process/runtime state. Related is Provide a kafka streams/event stores based event log #47.

Example 1: https://github.com/dohzya/scalaio-2017-esmonad/blob/1c6c27258038f9e6af353a3d8213786209fc726b/src/test/scala/esmonad/V3Spec_3.scala#L17-L26

Example 2: http://debasishg.blogspot.com/2011/01/cqrs-with-akka-actors-and-functional.html

[notxcain]

@SemanticBeeng is there anything I can help with?

Can Aecor be used to implement all use cases from "COMPOSABLE EVENT SOURCING WITH MONADS" ? (please skim through presentation to offer an initial assessment if any show stoppers exist before going deeper to prove)

Sure, MonadActionReject is all you need for that.

[SemanticBeeng]

I think the gist of my pursuit at that time was to see how to write event sourcing based logic without being forced to side effect the events as with persistent actor based implementations.

Since asking this have learned more about Aecor and believe I understand better.

Still, it would be great if you could sketch the functionality of the example from this article in Aecor when you can afford a couple of hours. :-)