From Push to Pull – External Tasks in BPMN processes

A process engine typically call services actively (e.g. via Java, REST or SOAP) from within a Service Task. But what if this is not possible because we cannot reach the service? Then we use a pattern we called “External Task” – which I briefly want to describe today.

Picture on the right taken from http://www.from-push-to-pull.com/projects/what-is-pull-marketing/ – thanks!

Context and problem

A couple of recent trends increased the need for this pattern, namely:

• Cloud: When running process/orchestration engines in the cloud you might not be able to reach the target service via network connections – and VPNs or Tunneling is always cumbersome. It is much easier if systems collect their work items by pullig it from the process engine.
• Polyglott Programming: When using different programming languages it gets harder to have proper service interfaces available to call. Even if REST is present in almost every programming language – provide an active REST Server is not always easy. However, calling an URL is not such a big deal.
• Microservice architectures (see my blog post about BPM and Microservices): This basically intensifies the two trends above.

Push vs. Pull

The basic idea is simple: Instead of actively calling a service (PUSH) we let the service ask for work (PULL).

The “traditional” approach using PUSH:

And the alternative using PULL:

Of course you can mix both styles within one process whenever appropriate!

External Tasks and Workers

As “External Tasks” are not not part of the BPMN 2.0 standard you can leverage extension element to configure them, e.g.:

<serviceTask id="ServiceTaskReserveGoods" name="reserve goods">
  <extensionElements>
    <camunda:properties>
      <camunda:property name="externalTaskName" value="stockService:reserveGoods"/>
      <camunda:property name="lockTime" value="PT5M" />
    </camunda:properties>
  </extensionElements>
</serviceTask>

Simple Implementation leveraging User Tasks

To implement the Extern Task Pattern on a existing camunda BPM Platform the easiest possibility is to leverage User Task behavior. Then you will create Tasks in our Task Management, but assign them to external systems (I tend to call them “worker”) instead of humans. The idea is exactly the same – you have one or more workers doing one job – and they get tasks from the task list, claim them when they start working on it and let the engine know when they are finished. You can easily scale by adding more workers. Exactly the same as with Human Task Management.

For Business-IT-Alignment reasons we want to keep a ServiceTask in the BPMN process (read: The automation icon – not the human icon). Fortunately it is pretty easy to exchange behavior for specific tasks in the camunda engine. You can find a fully working prototype with detailed technical descriptions here: https://github.com/camunda/camunda-consulting/tree/master/snippets/external-task.

Now you use the normal Task API, probably via the camunda REST API (as we talk about Polyglott, Remote, Microservices, …):

• Query Tasks for External Task Type: http://localhost:8080/engine-rest/engine/default/task/?candidateGroup=stockService:reserveGoods
• Claim Task for specific Worker: http://localhost:8080/engine-rest/engine/default/task/9f62b8d4-d465-11e4-95c6-fab420524153/claim (POST with {“userId”: “workerId”})
• Complete Task to inform engine that external task was executed successfully: http://localhost:8080/engine-rest/engine/default/task/9f62b8d4-d465-11e4-95c6-fab420524153/complete (POST with {“variables”:…})

Of course there is room for imporvement:

• Use a lock time for workers and release locks automatically afterwards, to avoid stuck processes when a worker crashes.
• Think more about transactions and rolling back a service task in this scenario.
• Provide examples or a client API for various worker technologies (e.g. Java, JavaScript, Bash, …).

Advantages

I want to quote a slide from our technical genius Daniel – it was input for an internal discussion – but shows relevant aspects pretty well:

Advantages are:

• Process Engine does not need network connection to worker.
• Availability of Process Engine and Worker is decoupled (temporal) – if a Worker is not available the engine just waits until a service task is finished somewhere in future.
• Scalability: You can easily add new workers which execute jobs.
• Technologic independence: The worker can be written in any technology you like as long as they can ask for jobs at the process engine (typically via http -> REST).

Camunda core feature?

We did discuss this already in the past and Daniel provided some pseudo-code how the External Task could be supported by core engine features.

REST API:

GET /engine-rest/external-task/foo?maxTasks=3&maxWait=4000
RESPONSE:
[
   "_embedded": {
     "external-task": [ { "id": "123", ... }, { ... }]
     "
   }
]

POST /engine-rest/external-task/foo/123/complete
PAYLOAD:
{
   "variables": {
     "myVar1": {
       "value": "{\"customerId\": \"someCustoerId\", ...}",
       "type": "Json"
     }
   }
}

He could also think of a Java API where you do not really recognize that you are building an External Task Worker. It would be as easy as implementing “normal” Service Tasks in camunda.

public static void main(String[] args) {

   CamClient client = CamClient.configure()
     .url("http://localhost:8080/engine-rest")
     .username("hi")
     .password("ho")
     .create();

   // super-reactive way of linking logic to process
   client.onExternalTask("foo:bar/asdf", new ExternalTaskDelegate() {
     public void execute(ExternalExecution ex) {

       VarialesMap vars = ex.getVariables(Arrays.asList("a", "b", "c"));
       //...
       Customer c = new Customer(...);
       ex.setVariable("myVar1", jsonValue(c))
       ex.complete();

     }
   });

   client.closeFuture()
     .get();

}

And that could be made maybe even easier in Java EE:

@Stateless
public class MyExternalTaskEjb {

   @CamExternalTask("foo:bar/asdf")
   public void doBusiness(ExternalExecution e) {
     VarialesMap vars = ex.getVariables(Arrays.asList("a", "b", "c"));
     //...
     ex.complete();
   }
}

Daniel was even half through with a protoypical implementation in the engine – which proofed that it works. See https://github.com/camunda/camunda-bpm-platform/commit/39a74c2a2c8966f22cbaa5e7e910941c45b46a25 if you are interessted in details.

Awesome – or what do you think?

Roadmap?

To be honest: There is nothing on the concrete roadmap yet. That makes it even more important that the community (that involves YOU!) give us feedback. If we see rising interesst we might implement that in the core engine! So far we are busy with implementing Decision Tables (implementing the DMN standard), see camunda BPM and rules – but as we have a growing team of really awesome people that are eager to implement new features 🙂