Scientific performance benchmark of open source BPMN engines
In May 2016, a group of authors from the universities of Stuttgart (Germany) and Lugano (Switzerland) has conducted a profound performance benchmark of three open source BPMN process engines, Camunda being one of them.
As the authors state in their introduction:
“This work proposes the first microbenchmark for WfMSs that can execute BPMN 2.0 workflows. To this end, we focus on studying the performance impact of well-known workflow patterns expressed in BPMN 2.0 with respect to three open source WfMSs. We executed all the experiments under a reliable environment and produced a set of meaningful metrics.”
Besides Camunda, two other well-known engines have been benchmarked. Unfortunately the other vendors did not allow to be named in the paper, which could have something to do with one of the conclusions the authors draw:
“In general, we may conclude that Camunda performed better and more stable for all metrics when compared with WfMS A and WfMS B.”
When comparing Camunda specifically to “WfMS A”, the authors make some interesting statements:
“WfMS A and Camunda share many architectural similarities because Camunda was originally a fork of WfMS A. Still their behaviour is not identical and leads to some interesting points. Camunda kept the duration values low for all the workflow patterns, but for [SEQ] and [EXC] WfMS A executed slightly better. However, we note large differences in the duration values for [EXT], [PAR], and [MIX], that indicate an impact of parallelism on the performance of the WfMS A, and increased resource utilization. The parallelism does not seem to have much impact on Camunda, as it remained relatively stable in all tests.”
Since we have conducted many improvements of the core engine during the last three years, I was not entirely surprised to read this.
There are many more interesting insights in this paper, and I think it is a valuable contribution to the scientific BPM community. You can order it online, but the authors have also allowed us to post a free download link here:
Leave a reply