Keep informed?
Subscribe for our newsletter now!

Service Invocation and asynchronous response correlation made lightweight


As a Christmas present we got an assignment for developing a framework, which handles service invocation from processes including correlation support for asynchronous responses. And the best: We can open source it, so it will be available for everybody! Kudos to the customer (hopefully I am allowed to name them soon). In this post I will write a bit about the scope, the tackled problems and the targeted solution. An interesting side information: The framework will be able to work with different Process Engines. At the customer it is jBPM 3, but we plan to do a Proof of Concept with Activiti as well.

In this post I want to give a very easy example of a mini business process: Procurement of material. As you can see in the following BPMN the order is created by a Human, send electronically to the Supplier afterwards and then printed and archived (I needed a manual step here ;-)). After archiving them we wait for the order confirmation of the Supplier and insert the order (maybe including the scheduled delivery date) into our ERP system automatically. Here you can see the BPMN of that process:


The first problem we face: How can we communicate with the supplier and our SAP system? Or in other words: How can we do the Service Invocation? In my post about WebServices within BPMN I already wrote about WebServices being the standard way in BPMN, but often not the best or easiest. Let’s assume in our example the communication with the supplier is done via JMS and the ERP system is developed in Java EE and offers an EJB interface. If we want to use WebServices, we would have to provide a WebService facade to these technologies. Actually I am not a big fan of that. It adds much work and complexity. So we want a more Java based solution…

A second problem we need to solve in that easy example already is correlation. When we get the answer from the supplier, we have to find the right process instance, where the answer belongs to. The BPMN standard defines two correlation approaches:

  • Correlation Key: A generated key without any business meaning is created for a conversation and added to all messages. This key must be present in the response as well, so it can be easily correlated. This is actually the best and safest way of correlation, since the keys are very unique. The problem typically is, that you have to deal with interfaces not being capable of handling correlation keys, hence the response will not contain that key any more. This is why there is a need for the other approach:
  • Context-based Correlation: The response is correlated based on payload data in the response, typically business keys. In our case that could be an order reference number or the like. This is pretty handy and works even with legacy services not being capable of handling correlation keys, but is not always unique. Think of situations where you have parallel process paths both sending an order for the same reference number.

Normally you use a combination of both, depending on the capabilities of the service you want to call. But there is a small detail really interesting in that story: In our process printing and archiving the order can take a while (maybe the accountant is on holiday drinking cocktails in the Caribbean), so the response from our supplier may arrive, before the process is in the correct state. What happens to the response? Shall it be thrown away or shall it be saved until the process instance is ready? If it is saved (queued), what if no process instance ever handles that response? These are really interesting questions, and the BPMN spec doesn’t cover them at all. This means, it is product or project specific. And how is it handled there?

Actually that was a funny small research task. I asked some of my contacts (consultants, vendors and end-users) about what they think of it, how their software does it, how they solved it or if they know an open source framework doing it. And the answers were somehow disillusioning (I freely translate and summarize them):

  • For jBPM we definitely do not know an open source solution. It would be nice to have a more powerful solution in a seperate open source project!
  • Neither the BPEL nor the BPMN spec demand to save/queue messages.
  • The workaround in our BPEL projects is to design processes in a way, that they always wait for a pick (where the response is received) in parallel.
  • JMS and WS callbacks are pretty common, but inside an open source ESB I haven’t seen a lot good solutions (I’d better say none ;-)).
  • For some OSS ESB: Sadly, the integration doesn’t allow it yet.
  • If you have existing legacy systems, it always gets ugly.

After a lot of mails and some discussions I came to the conclusion, that some single vendors seem to have solved that problem somehow behind closed doors, but despite that, it is still unsolved or solved in very project specific ways. And changing the process models is not an option as well, if we take the Business-IT-Alignment story serious. Especially disappointing was, that the ESB (Enterprise Service Bus) projects or products weren’t really helpful here. In earlier days I was naive enough to think, that this is why ESBs were invented in the first place 😉 And not even the JBoss ESB as being hand in hand with jBPM in the JBoss SOA platform has a powerful answer (it is more like a very basic start). But I am not a big fan of ESBs anyway for reasons I mentioned in my WebService blog already. Basically I think they are too complicated and heavyweight, and very often you only need small parts of them to solve punctual problems. And as we can see, the really complicated problems aren’t solved anyway 😉

Hence, the conclusion was to go for an own lightweight framework, but provide it open source. It is not about Messaging, Message routing, Enterprise Integration Patterns or other stuff ESBs or framework like Spring Integration provide. It’s “just” about calling services from your process engine and processing the responses. And “just” solving the problems for this problem, not more. And remember: We want to do that in a lightweight way in Java, easy understandable for Java people.

And just to give you an impression of the scope for correlation, these are examples of Patterns in BPMN, which should be supported by such a framework:

  • Handling multiple responses in an unknown order
  • Saving responses if the process isn’t yet in the correct state
  • Handling event based gateways, were we wait for one of multiple possible responses

Or the different correlation patterns visualized as BPMN:


Additional Requirements

Beside the correlation problems we face two other requirements in that project, or let’s say goals we try to achieve with it:

  • Being Process Engine Independent: The whole solution shall not be tied to any specific Process Engine and should make as less assumptions about the environment as possible. Currently in the customers environment the JBoss SOA Platform with jBPM 3 is used, but for how long? Maybe it will be migrated to Activiti in future (see Using BPMN 2.0 Models with jBPM 3 – A migration path and more for some ideas on a migration path).
  • Productivity for Java People: At this customer (and actually in the most projects I see today) they have a lot of smart Java people but much less WebService or XSLT know how. Hence, the developers are most productive, if they can work in their known Java environment. This is why we tried to work with Java interfaces and Data Transformation being written in Java instead of using XSLT. This eliminates the need for additional tooling as well (and there is still no good open source XSLT Tooling available), you just need your IDE and code completion.

Solution: camunda fox PSI

PSI Building Blocks

PSI Building Blocks

After some discussions we named the Open Source component PSI – the Process Service Invocation Framework (unfortunately Babelfox was already taken ;-)). The image on the left shows the big building blocks of PSI on a very high level. An important component are the so called Service Adapters. The idea is to write the code required for data transformation in simple Java classes independent of the concrete process engine. For the service you use a Java Interface. The Adapter can be easily written and tested with pure Unit testing. Here a quick example of how ServiceAdapter currently look like from the camunda fox docs (but it is still subject to change):

public class StoreOrderServiceAdapter implements ServiceRequestAdapter, ServiceResponseAdapter {
  @DataObject
  private OrderDataObject order;
  @Service
  private OrderService orderService;
  @Override
  public void callService() throws Exception {
    orderService.storeOrder(order.getValue());
  }
  @Override
  public void processResponse(Order response) throws Exception {
    // lets assume the EJB call returns a Order object with filled order id or the like as response
    order.setValue(response);
  }
}

To hook them into your Process Engine environment there is the PSI Coordinator. It can be called from your processes (by small binding components providing jBPM Action Handlers or Activiti Delegates) and fulfills the following tasks:

  • Retrieve process variables from the engine and inject them into the Service Adapter
  • Writing changes to the variables after the response has arrived
  • Lookup Service Coordinates from a registry (the registry itself is pluggable)
  • Steering the correlation service for correlation support, e.g. storing correlation keys or register for responses
  • Dispatching the service call to the concrete technology like EJB or JMS (which is maybe determined by the answer from the Registry)
  • Handling responses and translate incoming events to the Process Engine by using the API from PEAL (the Process Engine Abstraction Layer)

Maybe you wonder, how we can use Java Interfaces if we talk about JMS communication? Good questions 🙂 The plan is to include the Message API, which can easily add Java Interfaces to JMS communication and even hide all the JMS hassle. I hope to give you concrete code examples soon.

As a last remark: There are even some more thoughts what the coordinator should do. At the current customer there is an own mechanism in place to handle failures of service calls. If these are technical errors (like the network is down) there is an own retry mechanism (in this case leveraging the jBPM jobs). So this is part of the solution as well, but I don’t want to bore you with all the details at once…

Roadmap

At the moment of writing this blog we already have more than just the rough picture I added here. We already have parts of the solution, but it was built too tightly coupled in the JBoss ESB and is not easily extensible. We already have more detailed architecture pictures, some pseudo-code and interfaces. So I am pretty confident, that we can make quick progress next year. The current project plan says, we should be finished at the end of march. So fingers crossed 🙂

And where can I find this Open Source project?

PSI (The Process Service Invocation Framework) is part of camunda fox. On the homepage you can find a link to the documentation as well as to the sources (but actually we haven’t yet committed anything, since till now all the package names are still customer specific and we have to change that first. We plan to move that from our internal SVN repository to the public fox repository as soon as possible).

Last but not least it would be (like always) hear your thoughts about it. So give me feedback! If you are interested in using PSI in your projects and maybe even want to participate in the development now (meaning you are maybe able to influence the development), don’t hesitate to drop me an email.

Already read?

Scientific performance benchmark of open source BPMN engines

Why BPMN is not enough

Decision Model and Notation (DMN) – the new Business Rules Standard. An introduction by example.

New Whitepaper: The Zero-Code BPM Myth

9 Responses

Leave a reply