Orchestration using BPMN and Microservices – Good or bad Practice?
Please note that there is a blog post available with much more recent thoughts on this matter: https://blog.bernd-ruecker.com/why-service-collaboration-needs-choreography-and-orchestration-239c4f9700fa.
Martin Fowler recommends in his famous Microservices Article: “Smart endpoints and dumb pipes”. He states:
The microservice community favours an alternative approach: smart endpoints and dumb pipes. Applications built from microservices aim to be as decoupled and as cohesive as possible – they own their own domain logic and act more as filters in the classical Unix sense – receiving a request, applying logic as appropriate and producing a response. These are choreographed using simple RESTish protocols rather than complex protocols such as WS-Choreography or BPEL or orchestration by a central tool.
I do not agree! I think even – or maybe especially! – with microservices you have the need for orchestration. In this blog post I want to explain why. Quick side remark: I totally agree with not using WS-Choreography or BPEL – in 2015 you should definitely go for BPMN.
What are microservices?
The idea of miroservices is to build self-contained services centered around one business function including all aspects of the software (operating system and programming language, persistence, container, business logic, user interface, …). These are developed and managed by a cross-functional team and deployed independently of each other – the overall system consists not of a monolithic deployment but a bunch of microservices, which communicate with each other.
Business Processes in a microservice architecture
Let’s assume three simple steps in a typical order process:
- reserve goods
- process payment
- prepare shipment
I would expect that we have at least three services:
- Stock Service
- Payment Service
- Shipment Service
The question is how the order process is implemented. If you follow the “smat endpoints and dumb pipes” I basically see three possibilities:
- The services know about the process and do the routing accordingly. This implies that the message with the initial order is first consumed by the Stock Service to reserve goods. This service then calls the Payment Service in order to do payment. As the pipes are dumb – the services have to know which service is next.
- The service listen to a central event bus consuming messages they are interessted in. TThan the initial message is an “order placed” event, which the Stock Service is interessted in. After processing it prodocues a “goods reserved” event the “Payment Service” listenes for. And so on.
- An own “Order Service” is introduced who takes care of the overall process.
Option 1 and 2 both imply some problems to solve:
- The knowledge about the process flow is spreaded all over the various services.
- Changes in the process flow will impact different services.
- The current state of one order / process instance will be hard to find out – as various services have to be asked.
There are solutions to these problems, e.g. you could build a central datastore knowing of all events which can answer questions about the state. But my feeling is still that this smeels. A lot! One simple example: We need to implement cancellation of orders, which is only possible if the shipment is not yet prepared. Then you need to compensate payment and release the reservation. If already done. In order to do this you need to know the state of your process instance! And you need to change each and every service to implement this requirement!
So let’s have a look at option #3:
Orchestration is king!
If we build an own “Order Service” implementing the order process we do orchestration! And I want to do this – as this is one place where the whole process flow is implemented. With microservices we are free to choose technology for one service – so obviously we want to choose a BPMN process engine for orchestration (I do not want to repeat why within this blog post and hope everybody got the news that you should not hard code processes and you should not write your own engine. Repeat: Do not hard code processes or write your engine!).
Then we can model the process:
This BPMN model is easy to understand and executable on engine like the Open Source BPM Platform from camunda (which is pretty lightweight and can run e.g. via docker). It helps to keep process flow out of the underlying services. A process engine keeps track of all process instances, their state and even audit information / statistics. Adding the cancellation is rather simple and compensation mechanisms exist out-of-the-box. The used Microservices are still micro and depending on the type of communication there might not be temporal coupling or the like – as we can use dump pipies to communicate between engine and service. Or we might pull tasks from the engine instead of pushing it – something I want to describe in an upcoming blog post – as it is expecially interessting in a cloud-like or polyglott environment (which might be true in microservice architectures).
Better name it “BPM” than “Orchestration”?
I can imagine that one of the main reasons Martin Fowler advised against orchestration in a microservices world is that orchestration still sounds like “the old” SOA paradigm where you have layered services calling each others via ESB-like components. But BPM is about transparency and Business-IT-Alignment. Modern BPM platforms have nothing todo with the heavy and complex BPM Suites of the last decade – so there is no reason why you shouldn’t use BPM / a process engine in your microservice architecture.
Discussion welcome!
[…] architectures (see my blog post about BPM and Microservices): This basically intensifies the two trends […]
Hello Bernd,
I totally disagree with you on this.
You moved the essential discussion out of the focus, which has to be:
“so obviously we want to choose a BPMN process engine for orchestration (I do not want to repeat why within this blog post and hope everybody got the news that you should not hard code processes and you should not write your own engine. Repeat: Do not hard code processes or write your engine!)”
This decision is the central point!
In a Microservices architecture the core is the more or less hard coded orchestration between the services. It is implemented in each service and implicitly defined by their behavior between each other (of course with low coupling)! Only this way I have the benefits of Microservices.
Using a BPM Engine with such an Orchestration is a completely orthogonal concept. It is the way I define the behavior of my system. Using BPM Engine vs. behavior which is implemented in the overall system. I am building this way a monolith, because I have a central point of behavior definition, hence I loose the benefits of Microservices.
I agree that BPMN is a good tool for communication with the business side and can be used for better alignment. I even say it can be used in an implementation of a service. E.g. for human task management or notification sending.
BUT it should NOT be used to define the overall technical orchestration of the system!
For example the process diagram of your “Order Service” would be completely hard wired in my point of view. It would have a place order, chancel order and update payment data operation for the fronted or other services. You could have a separate BPM Engine Microservices which listens at the event bus or is called by the order service and reacts depending on the deployed BPMN and creates tasks or sends out emails to customers by creating events or calling other services. Or even triggers the 2 day order cancellation.
Practically we cannot push the whole process into one definition. We need to separate between parts of the process which always have more technical aspects such as service call sequences and more business relevant aspects such as the definition when is an automatic cancellation performed, when is an email send or a human task created.
A good decision criteria would be how often would this part of the process change to decide this part should be hard wired or configurable.
But as always, it is a use case decision!
My point is the definition of the overall process will not be defined explicitly in one service but will be distributed in the implementation between the services and the frontend.
Now we could dive into the discussion hard wire vs. BPM, but this is not my intention.
Cheers,
Conrad
Hi Conrad,
there are many good reasons for using Workflow / Process / BPM. The technical problems it solves [1] are universal and do not go away just because your build your application(s) as microservices.
🙂
Daniel
[1] http://www.enterpriseintegrationpatterns.com/ProcessManager.html
Hi Conrad.
Thank you for your comment and starting a proper discussion 🙂
“In a Microservices architecture the core is the more or less hard coded orchestration between the services. It is implemented in each service and implicitly defined by their behavior between each other (of course with low coupling)! ”
“Using a BPM Engine with such an Orchestration is a completely orthogonal concept. It is the way I define the behavior of my system. Using BPM Engine vs. behavior which is implemented in the overall system. I am building this way a monolith, because I have a central point of behavior definition, hence I loose the benefits of Microservices.”
My view is basically the opposite. With hard coding the process into multiple microservices, where you have to understand all of them to understand the overall process, you quickly build a new monlith – but not implemented by classes or components, but microservices. As in this case the services are NOT independantly changable – as the process is the central business goal the system needs to achieve. Every change affects different services.
Of course you can cut your overall process into smaller pieces and put them into different microservices, that helps to keep things changeable seperately.
But hard coding all business process aspects is really a bad idea. I can already see projects struggling to change anything in the business process – as it is close to impossible to find it in the various microservices. Same what we know today grown software – might be monoliths or might be distributed software build out of various components/services. I remember one customer where a various historic systems executed part of the overall process and comunicated via DB or message. A mess to adapt business processes to new requirements.
But you are right – that is the discussion “hard wire vs. BPM”.
“A good decision criteria would be how often would this part of the process change to decide this part should be hard wired or configurable.”
That could indeed be a driver. Another is the need for transparency/visibility in development or as well in operations.
“My point is the definition of the overall process will not be defined explicitly in one service but will be distributed in the implementation between the services and the frontend.”
The frontend is indeed a very interessting topic. But I can imagine that the process engine only controls which form is presented and get that form definition from the service that defines the process definition (compare that to what we do with embedded forms in camunda BPM: http://docs.camunda.org/latest/guides/user-guide/#task-forms-embedded-task-forms). This form now might include UI snippets from other services (e.g. a customer view panel when showing customer data).
Cheers
Bernd
I think the point is not “no orchestration” but “no orchestration by a central tool”, which is different.
When you build your “Order Service” you can use whatever tool or technology you feel is appropriate in your microservice, and orchestration is a good candidate. The point is that if you need orchestration for, let’s say, a “Supply Service”, it will be in the microservice, not in a central tool which would have all the BPM processes. In a similar way that the microservices would not have a central database.
They could have -it’s not the point- but it’s not the microservice approach.
Interesting post anyway.
Hi Geoffrey.
I agree with you – every microservice should be able to choose technology for itself. And for some of them orchestration and a workflow engine / BPM tool will be the best fit. So we are on the same page 🙂
But for the “microservice approach” (which might mean that different microservices even use different workflow engines): Even if I agree with you as well, I think in reality this will still be unwanted by most organization. It is easier to harmonize technology to some extend to handle the technology correctly and build some expertise (the smaller the company, the smaller the number of tools). So I think you often still want to leverage a known database, a known container, a known framework, a known workflow engine. And yes, this might even be a central installation of your SQL-Database or a “BPM Server”. I don’t say this is ideal – but it is what I see in a lot of companies (where “typical” developers are unable to cope with daily workload AND too much technology at the same time). But maybe that is more the question in which situation “the microservice approach” works or not 😉
Cheers
Bernd
Please note that in the meantime I learned much more about BPM & Microservices, so best read my more recent blog post on this matter: https://blog.bernd-ruecker.com/why-service-collaboration-needs-choreography-and-orchestration-239c4f9700fa