From Push to Pull – External Tasks in BPMN processes
A process engine typically call services actively (e.g. via Java, REST or SOAP) from within a Service Task. But what if this is not possible because we cannot reach the service? Then we use a pattern we called “External Task” – which I briefly want to describe today.
Picture on the right taken from http://www.from-push-to-pull.com/projects/what-is-pull-marketing/ – thanks!
Context and problem
A couple of recent trends increased the need for this pattern, namely:
- Cloud: When running process/orchestration engines in the cloud you might not be able to reach the target service via network connections – and VPNs or Tunneling is always cumbersome. It is much easier if systems collect their work items by pullig it from the process engine.
- Polyglott Programming: When using different programming languages it gets harder to have proper service interfaces available to call. Even if REST is present in almost every programming language – provide an active REST Server is not always easy. However, calling an URL is not such a big deal.
- Microservice architectures (see my blog post about BPM and Microservices): This basically intensifies the two trends above.
Push vs. Pull
The basic idea is simple: Instead of actively calling a service (PUSH) we let the service ask for work (PULL).
The “traditional” approach using PUSH:
And the alternative using PULL:
Of course you can mix both styles within one process whenever appropriate!
External Tasks and Workers
As “External Tasks” are not not part of the BPMN 2.0 standard you can leverage extension element to configure them, e.g.:
<serviceTask id="ServiceTaskReserveGoods" name="reserve goods"> <extensionElements> <camunda:properties> <camunda:property name="externalTaskName" value="stockService:reserveGoods"/> <camunda:property name="lockTime" value="PT5M" /> </camunda:properties> </extensionElements> </serviceTask>
Simple Implementation leveraging User Tasks
To implement the Extern Task Pattern on a existing camunda BPM Platform the easiest possibility is to leverage User Task behavior. Then you will create Tasks in our Task Management, but assign them to external systems (I tend to call them “worker”) instead of humans. The idea is exactly the same – you have one or more workers doing one job – and they get tasks from the task list, claim them when they start working on it and let the engine know when they are finished. You can easily scale by adding more workers. Exactly the same as with Human Task Management.
For Business-IT-Alignment reasons we want to keep a ServiceTask in the BPMN process (read: The automation icon – not the human icon). Fortunately it is pretty easy to exchange behavior for specific tasks in the camunda engine. You can find a fully working prototype with detailed technical descriptions here: https://github.com/camunda/camunda-consulting/tree/master/snippets/external-task.
Now you use the normal Task API, probably via the camunda REST API (as we talk about Polyglott, Remote, Microservices, …):
- Query Tasks for External Task Type: http://localhost:8080/engine-rest/engine/default/task/?candidateGroup=stockService:reserveGoods
- Claim Task for specific Worker: http://localhost:8080/engine-rest/engine/default/task/9f62b8d4-d465-11e4-95c6-fab420524153/claim (POST with {“userId”: “workerId”})
- Complete Task to inform engine that external task was executed successfully: http://localhost:8080/engine-rest/engine/default/task/9f62b8d4-d465-11e4-95c6-fab420524153/complete (POST with {“variables”:…})
Of course there is room for imporvement:
- Use a lock time for workers and release locks automatically afterwards, to avoid stuck processes when a worker crashes.
- Think more about transactions and rolling back a service task in this scenario.
- Provide examples or a client API for various worker technologies (e.g. Java, JavaScript, Bash, …).
Advantages
I want to quote a slide from our technical genius Daniel – it was input for an internal discussion – but shows relevant aspects pretty well:
Advantages are:
- Process Engine does not need network connection to worker.
- Availability of Process Engine and Worker is decoupled (temporal) – if a Worker is not available the engine just waits until a service task is finished somewhere in future.
- Scalability: You can easily add new workers which execute jobs.
- Technologic independence: The worker can be written in any technology you like as long as they can ask for jobs at the process engine (typically via http -> REST).
Camunda core feature?
We did discuss this already in the past and Daniel provided some pseudo-code how the External Task could be supported by core engine features.
REST API:
GET /engine-rest/external-task/foo?maxTasks=3&maxWait=4000 RESPONSE: [ "_embedded": { "external-task": [ { "id": "123", ... }, { ... }] " } ] POST /engine-rest/external-task/foo/123/complete PAYLOAD: { "variables": { "myVar1": { "value": "{\"customerId\": \"someCustoerId\", ...}", "type": "Json" } } }
He could also think of a Java API where you do not really recognize that you are building an External Task Worker. It would be as easy as implementing “normal” Service Tasks in camunda.
public static void main(String[] args) { CamClient client = CamClient.configure() .url("http://localhost:8080/engine-rest") .username("hi") .password("ho") .create(); // super-reactive way of linking logic to process client.onExternalTask("foo:bar/asdf", new ExternalTaskDelegate() { public void execute(ExternalExecution ex) { VarialesMap vars = ex.getVariables(Arrays.asList("a", "b", "c")); //... Customer c = new Customer(...); ex.setVariable("myVar1", jsonValue(c)) ex.complete(); } }); client.closeFuture() .get(); }
And that could be made maybe even easier in Java EE:
@Stateless public class MyExternalTaskEjb { @CamExternalTask("foo:bar/asdf") public void doBusiness(ExternalExecution e) { VarialesMap vars = ex.getVariables(Arrays.asList("a", "b", "c")); //... ex.complete(); } }
Daniel was even half through with a protoypical implementation in the engine – which proofed that it works. See https://github.com/camunda/camunda-bpm-platform/commit/39a74c2a2c8966f22cbaa5e7e910941c45b46a25 if you are interessted in details.
Awesome – or what do you think?
Roadmap?
To be honest: There is nothing on the concrete roadmap yet. That makes it even more important that the community (that involves YOU!) give us feedback. If we see rising interesst we might implement that in the core engine! So far we are busy with implementing Decision Tables (implementing the DMN standard), see camunda BPM and rules – but as we have a growing team of really awesome people that are eager to implement new features 🙂
greate feature , best fit for microservices!
Hi Bernd,
for me this sounds like a very important feature.
Also the described approach sounds very feasible for me.
But I think, the reason why people use CamundaBPM and in general middleware stuff, is to focus on business logic and not on infrastructure coding.
This means, until this faeture is part of the CamundaBPM core solution, all will use a classic approach with an ESB (BPMN engine calls ESB, ESB takes care about the messaging).
But be sure, if such a feature is available in Camunda, I would also love to get rid off the ESB stuff 😉
Cheers,
Matthias
P.S. for me it sounds also more important as DMN 😉 For the DMN mission, I would call another microservice which should solve the decision probelm…
Hi Matthias.
We already did a prototype for this pattern (included in the core engine) last week on our hack days (including a nice monitoring). But we did not yet decide if and when we integrate it into the product (and in what scope). If you present a proper business case for this feature that can always influence decisions 🙂
Cheers
Bernd
[…] another improvement based on customer requests. You can see more of the technical details here and here: instead of a system task calling an external task asynchronously then wait for a response, this […]