An Enterprise Service Bus is used to coordinate communications between multiple services in a service oriented architecture (SOA). ESB’s try to simplify large complex problems into smaller independent problems that can be devised, built, and maintained independently.
A large portion of an ESB is a message broker that can queue messages and distribute them to other services that need to receive them. Many brokers exist today, ActiveMQ, RabbitMQ, Apache Kafka, etc. They can be setup to support the enterprise customer’s needs by adding functionality such as persisting messages (guarantee delivery) and providing redundancy. Since they concentrate on message delivery, their performance can be tuned to provide very high message delivery rates.
Several message protocols for sending messages to these brokers exist, including Stomp, JMS, and AMQP. There are APIs for different languages to produce and consume messages to/from the brokers.
Applications can be written to communicate with the broker in both synchronous and asynchronous fashions and perform work based on the messages queued. One of the benefits of a message broker is that applications can be highly reactive by just listening on message queues. When a message arrives to the broker, the broker pushes the message to the application.
Now along comes microservices. Microservices try to solve the same problem by writing small independent applications to perform specific functions really well. Then the microservices are brought together to solve a larger problem. Typically microservices communicate together either via web-services or through message passing through a broker.
One of the benefits of microservices is that it is easier to write a bunch of smaller applications and the applications can be isolated from one another. However, one of the drawbacks is that it makes it harder to leverage common functionality that can be shared between the two. Forklift allows the developer to write microservices very quickly and benefit from having some common functionality that all services can use.
Forklift performs the the following prime duties of an ESB of:
-
An abstraction layer provided for sub-components
-
Control deployment and versioning of services
-
Marshal use of redundant services
-
Data transformation and mapping
-
Security and exception handling
The other requirements of an ESB are met by the use of a broker such as ActiveMQ. The following diagram shows how Forklift with a message broker combine to form an ESB.
Forklift is written in Java and and its consumers are to be written in Java. (can we say this will work with other JVM languages that can be loaded as a jar?) However, it can consume messages that are placed on a message queue written in any language and can use any of the standard Java communication protocols to communicate to other services.
Purposely, Forklift is built to quickly create consumers in code to perform actions based on messages. There is no XML configuration, there is no business process modeling, no developed flow tools. It is built for developers who can read/write code to implement redundant, retry capable, provable, testable, queued, message processing.
If you don’t want to read through the rest of this documentation, and instead want to start seeing Forklift in action, you can use the quickstart guide to start up the server and run it on your local box to start processing messages.
Just like a Java based application server, Forklift can be started up all on its own to do nothing. It doesn’t start processing anything until a consumer is deployed to it. Consumers are written to identify which queue/topic they will consume. A method within the class must be designated to process the messages and is run each time a new message is available. Other features such as designating methods to validate the messages can easily be added. Failed validation is handled by the Error Handler, see Standardized Exception/Error Handling.
Example:
@Queue("example")
public class MyConsumer {
@OnValidate
public List<String> validate() {
// code
return errors;
}
@OnMessage
public void handler() {
// code
}
}The way that Forklift provides easy configuration is by using a set of Java Annotations provided by the system to wrap your business logic with core code to handle all the routine things provided by Forklift. Here are a few examples of annotations that provide functionality:
-
@Queue - specified on a class given a name will begin processing any messages on the named queue and running specified methods on the messages
-
@OnMessage - specified on a method designates the method that should process every time a message is received.
-
@Message - Specified on a property of the class will try and inject the received message into the property to be used to read the message contents easily.
Forklift provides a method to extend the base life-cycle functionality using plug-ins. There are several plug-ins that have already been developed, for example, the way to have a message retry on error, or create audit logs that store an entire message that can be replayed.
Years of development experience led to Forklift being developed. Forklift needed to process messages asynchronously, but end-users wanted to know exactly what happened with each of their messages. After many discussions, the developers decided on the following minimal life-cycle processing for each message.
-
Pending - Message has been picked up from the broker and is going to passed onto the consumer.
-
Validating - Forklift will start running any @OnValidate methods to validate the message.
-
Processing - Forklift is going to run any @OnMessage methods within the consumer to process the message.
-
Complete - Forklift successfully validated and processed the message without error.
-
Invalid - Validation did not succeed successfully so the message won’t be processed.
-
Error - Validation may have succeeded but some other type of error occurred while processing the message with the consumer.
Since the life-cycle is built for the developer to be able to track processing, Forklift provides the ability to intercept each step. Components written and installed within Forklift can add the @LifeCycle annotation on methods to have targeted code run when messages reach each step. With this ability, it is quite easy to write specialized message life-cycle auditing. Plug-ins are available that intercept the life-cycle to be able to replay messages or allow for retrying a message in the event that a message errors during processing.
The purpose of a consumer is to process messages off of queues or topics from the broker. To make this easier, Forklift provides easy configuration of the consumer by annotating the consumer Java class with annotations:
-
@Queue - All messages placed on the named queue will be passed to this consumer.
-
@Topic - All messages placed on the named topic will be passed to this consumer.
-
@MultiThread - Forklift will run the specified number of consumers to process messages off the queue when needed. Helps when consumers may be longer running processes.
-
@Order - Messages are guaranteed to run in order for an exclusive consumer.
-
@RequireSystem - TBD
While making it easy to consume messages, it is also easy to configure the consumer. Properties files containing data values that can be picked up by the consumer at run-time can be deployed independently of the consumer. This provides a quick way to deploy the same consumer in different environments (such as production or test) and just change the properties for different behavior. Files with the file extension .properties placed in the deployment directory will be scanned and made available to the consumers.
@Named @Entity and other Spring configurable objects. - TBD
A lot of the base code when dealing JMS messages from Java involves marshaling the message from the message into a usable object that can then just be used. Forklift provides nice annotations that can be placed on properties to do all this marshaling for you.
-
@Config("file") - Place this annotation on a java.util.Properties object. All properties deployed properties files are made available for use. If you dont specify a file, it will infer from the field name.
-
@Config("file", "field") - Injects the specified field from the property file. If you dont specify a field, it will try to infer from the field name.
-
@Message - Placed on different property types, this will try and marshal the message data into the object.
-
@Headers - Placed on a Map of string and object, Forklift will marshal the message headers into the map.
-
@Headers(Header….) - Injects a specific header.
-
@Properties - Placed on a Map of string and object, Forklift will marshal the message properties into the map.
-
@Properties("name") - Loads a specific property "name". If you just want to marshal a few specific properties but dont need the entire object, use this. If you don’t provide a name, it will infer from the field name.
Forklift has been built to allow the use of dependency injection. Whether embedded into a Spring project or using Guice, you can inject your instantiated objects into the consumer. This does require a little setup but should be pretty straight forward for those familiar with dependency injection. Once configured, the @Inject annotation can be added to the consumer constructor or placed on fields within the consumer. For example:
@Queue("example")
public class MyConsumer {
@Inject
@Named("myClass")
MyInstantiatedClass myClass;
@OnValidate
public List<String> validate() {
// code
return errors;
}
@OnMessage
public void handler() {
myClass.myMethod();
// code
}
}
In order for the DI to work properly, it does require that the Spring BeanResolver or Guice BeanResolver are registered with Forklift, specifically a Consumer’s services. When starting up new consumers within a project like DropWizard using Guice, you’d need something like:
import com.google.inject.Injector;
import forklift.Forklift;
import forklift.consumer.Consumer;
import forklift.consumer.ConsumerService;
import forklift.consumer.ConsumerThread;
public class ForkliftStartupService {
@Inject
public ForkliftStartupService(Forklift forklift, Injector injector) {
// Configure a service to do dependency injection.
final ConsumerService forkliftInjector = new ConsumerService(new ForkliftInjector(injector));
final Consumer consumer = new Consumer(MyConsumer.class, forklift);
consumer.addServices(forkliftInjector);
final ConsumerThread thread = new ConsumerThread(consumer);
thread.start();
}
}
In helping the developer organize their code and inject code into the life-cycle, the developer can easily write message validation routines that can run before the message processing is started. Forklift provides an annotation to easily provide this functionality.
-
@OnValidate - Place this annotation on methods to validate the message. Any method that fails to validate will stop processing of the message and send the life-cycle to the Invalid state.
If a consumer errors (throwing an exception - Checked or Runtime), Forklift will automatically route the life-cycle to the Error state. This event can then be hooked to show the errors. For more information on logging and auditing see information about some of the plug-ins that have been built.
TBD
@On(step) is an annotation that can be added to a method. Its parameter is any step except Pending. When entering that step, the method will be called. This can be used for easy error handling or chaining consumers.
Example:
@On(ProcessStep.Error)
@On(ProcessStep.Invalid)
public void sendEmail() {
emailer.send(maintainer, "oops!");
}
@On(ProcessStep.Complete)
public void nextStep() {
messageQueue.send("step2", msg);
}Forklift plugins allow for additional actions to be performed at various stages in a message’s lifecycle. For more details, see the plugins documentation.
Forklift can be used as a library embedded within an application such as a SpringBoot app or DropWizard app. Just include the library and you can start lighting up Producers and Consumers. In fact a large number of projects we are aware of, just use Forklift to make it easier to Produce JMS messages.
But if you aren’t inclined to embed it within your own app and instead want to just run a set of producers and consumers within a container, Forklift provides a server that can run at a command-line or within a docker container. The server acts similarly to a Tomcat server where Consumers can be deployed into it’s deploy directory as packaged jars. Configuration property files can also be deployed for configuration values to be picked up by consumer jars. Take a look at the Quickstart Guide where we show how server can be used at the command line.
Forklift server offers a --help to see the different command line arguments that can be passed in. Another feature that was added in 3.8 was the ability to pass in some configuration via a Forklift server configuration file. It is also possible to change the logging configuration on the command line since Forklift uses logback as its logging solution.
See Command line arguments and Configuration Examples for more info.
A very popular broker in the world of Java Messaging Services (JMS) is Apache’s ActiveMQ. Forklift has been used very successfully with ActiveMQ as its primary broker. Forklift provides an out of the box connector for immediate use with ActiveMQ (located in connectors/ActiveMQ).