Orchestration using BPMN and Microservices – Good or bad Practice?
Martin Fowler recommends in his famous Microservices Article: “Smart endpoints and dumb pipes”. He states:
The microservice community favours an alternative approach: smart endpoints and dumb pipes. Applications built from microservices aim to be as decoupled and as cohesive as possible – they own their own domain logic and act more as filters in the classical Unix sense – receiving a request, applying logic as appropriate and producing a response. These are choreographed using simple RESTish protocols rather than complex protocols such as WS-Choreography or BPEL or orchestration by a central tool.
I do not agree! I think even – or maybe especially! – with microservices you have the need for orchestration. In this blog post I want to explain why. Quick side remark: I totally agree with not using WS-Choreography or BPEL – in 2015 you should definitely go for BPMN.
What are microservices?
The idea of miroservices is to build self-contained services centered around one business function including all aspects of the software (operating system and programming language, persistence, container, business logic, user interface, …). These are developed and managed by a cross-functional team and deployed independently of each other – the overall system consists not of a monolithic deployment but a bunch of microservices, which communicate with each other.
Business Processes in a microservice architecture
Let’s assume three simple steps in a typical order process:
- reserve goods
- process payment
- prepare shipment
I would expect that we have at least three services:
- Stock Service
- Payment Service
- Shipment Service
The question is how the order process is implemented. If you follow the “smat endpoints and dumb pipes” I basically see three possibilities:
- The services know about the process and do the routing accordingly. This implies that the message with the initial order is first consumed by the Stock Service to reserve goods. This service then calls the Payment Service in order to do payment. As the pipes are dumb – the services have to know which service is next.
- The service listen to a central event bus consuming messages they are interessted in. TThan the initial message is an “order placed” event, which the Stock Service is interessted in. After processing it prodocues a “goods reserved” event the “Payment Service” listenes for. And so on.
- An own “Order Service” is introduced who takes care of the overall process.
Option 1 and 2 both imply some problems to solve:
- The knowledge about the process flow is spreaded all over the various services.
- Changes in the process flow will impact different services.
- The current state of one order / process instance will be hard to find out – as various services have to be asked.
There are solutions to these problems, e.g. you could build a central datastore knowing of all events which can answer questions about the state. But my feeling is still that this smeels. A lot! One simple example: We need to implement cancellation of orders, which is only possible if the shipment is not yet prepared. Then you need to compensate payment and release the reservation. If already done. In order to do this you need to know the state of your process instance! And you need to change each and every service to implement this requirement!
So let’s have a look at option #3:
Orchestration is king!
If we build an own “Order Service” implementing the order process we do orchestration! And I want to do this – as this is one place where the whole process flow is implemented. With microservices we are free to choose technology for one service – so obviously we want to choose a BPMN process engine for orchestration (I do not want to repeat why within this blog post and hope everybody got the news that you should not hard code processes and you should not write your own engine. Repeat: Do not hard code processes or write your engine!).
This BPMN model is easy to understand and executable on engine like the Open Source BPM Platform from camunda (which is pretty lightweight and can run e.g. via docker). It helps to keep process flow out of the underlying services. A process engine keeps track of all process instances, their state and even audit information / statistics. Adding the cancellation is rather simple and compensation mechanisms exist out-of-the-box. The used Microservices are still micro and depending on the type of communication there might not be temporal coupling or the like – as we can use dump pipies to communicate between engine and service. Or we might pull tasks from the engine instead of pushing it – something I want to describe in an upcoming blog post – as it is expecially interessting in a cloud-like or polyglott environment (which might be true in microservice architectures).
Better name it “BPM” than “Orchestration”?
I can imagine that one of the main reasons Martin Fowler advised against orchestration in a microservices world is that orchestration still sounds like “the old” SOA paradigm where you have layered services calling each others via ESB-like components. But BPM is about transparency and Business-IT-Alignment. Modern BPM platforms have nothing todo with the heavy and complex BPM Suites of the last decade – so there is no reason why you shouldn’t use BPM / a process engine in your microservice architecture.