The need for a process mining evaluation framework in research and practice: Position paper (original) (raw)

Towards an Evaluation Framework for Process Mining Systems

Although there has been a lot of progress in developing process mining algorithms in recent years, no effort has been put in developing a common means of assessing the quality of the models discovered by these algorithms. In this paper, we outline elements of an evaluation framework that is intended to enable (a) process mining researchers to compare the performance of their algorithms, and (b) end users to evaluate the validity of their process mining results. Furthermore, we describe two possible approaches to evaluate a discovered model (i) using existing comparison metrics that have been developed by the process mining research community, and (ii) based on the so-called k-fold-cross validation known from the machine learning community. To illustrate the application of these two approaches, we compared a set of models discovered by different algorithms based on a simple example log.

Towards an evaluation framework for process mining algorithms

Reactivity of Solids, 2007

Although there has been a lot of progress in developing pro- cess mining algorithms in recent years, no eort has been put in devel- oping a common means of assessing the quality of the models discovered by these algorithms. In this paper, we outline elements of an evaluation framework that is intended to enable (a) process mining researchers to compare

Definition and Validation of Process Mining Use Cases

Process mining is an emerging topic in the BPM marketplace. Recently , several (commercial) software solutions have become available. Due to the lack of an evaluation framework, it is very difficult for potential users to assess the strengths and weaknesses of these process mining tools. As the first step towards such an evaluation framework, we developed a set of process mining use cases and validated these use cases by means of expert interviews and a survey. We present the list of use cases and discuss the insights from our empirical validation. These use cases will then form the basis for a detailed evaluation of current process mining tools on the market.

Development of the Process Mining Discipline

It is exciting to see the spectacular developments in process mining since I started to work on this in the late 1990-ties. Many of the techniques we developed 15-20 years ago have become standard functionality in today's process mining tools. Therefore, it is good to view current and future developments in this historical context. This chapter starts with a brief summary of the history of process mining showing how ideas from academia got adopted in commercial tools. This provides the basis to talk about the expanding scope of process mining, both in terms of applications and in terms of functionalities supported. Despite the rapid development of the process mining discipline, there are still several challenges. Some of these challenges are new, but there are also several challenges that have been around for a while and still need to be addressed urgently. This requires the concerted action of process mining users, technology providers, and scientists. Adoption of traditional process mining techniques Process mining started in the late nineties when I had a sabbatical and was working for one year at the University of Colorado in Boulder (USA). Before, I was mostly focusing on concurrency theory, discrete event simulation, and workflow management. We had built our own simulation engines (e.g., ExSpect) and workflow management systems. Although our research was well-received and influential, I was disappointed by the average quality of process models and the impact process models had on reality. In both simulation studies and workflow implementations, the real processes often turned out to be very different from what was modeled by the people involved. As a result, workflow and simulation projects often failed. Therefore, I decided to focus on the analysis of processes through event data [1]. Around the turn of the century, we developed the first process discovery algorithms [2]. The Alpha algorithm was the first algorithm able to learn concurrent process models from event data and still provide formal guarantees. However, at the time, little event data were available and the assumptions made by the first algorithms were unrealistic. People working on data mining and machine learning were (and perhaps still are) not interested in process analysis. Therefore, it was not easy to convince other researchers to work on this. Nevertheless, for me, it was crystal clear that process mining would become a crucial ingredient of any process management or process improvement initiative. In the period that followed, I stopped working on the traditional business process management topics and fully focused on process mining. It is interesting to see that concepts such as conformance checking, organizational process mining, decision mining, token animation, time prediction, etc. were already developed and implemented 15 years ago [2]. These capabilities are still considered to be cutting-edge and not supported by most of the commercial process mining tools.

A framework for comparing process mining algorithms

2011 IEEE GCC Conference and Exhibition (GCC), 2011

There are many process mining algorithms with different theoretical foundations and aims, raising the question of how to choose the best for a particular situation.

Process Mining: A Database of Applications (2016)

2016

A database containing the most comprehensive list of process mining applications (#1 worldwide) in different industries, including banking, insurance, telco, manufacturing, retail, healthcare, government. The research was presented at the CIS IEEE Task Force on Process Mining annual meeting, during 14th International Conference on BPM in Rio de Janeiro. • Scope: cases studies from 100+ companies, and 6 industries, between 2005-2016, • Target audience: executives, decision makers, consultants and researchers

From Zero to Hero: A Process Mining Tutorial

Lecture Notes in Computer Science, 2017

Process mining is an emerging area that synergically combines model-based and data-oriented analysis techniques to obtain useful insights on how business processes are executed within an organization. This tutorial aims at providing an introduction to the key analysis techniques in process mining that allow decision makers to discover process models from data, compare expected and actual behaviors, and enrich models with key information about the actual process executions. In addition, the tutorial will present concrete tools and will provide practical skills for applying process mining in a variety of application domains, including the one of software development.

Process Mining Put into Context

IEEE Internet Computing, 2012

Process mining techniques help organizations discover and analyze business processes based on raw event data. The recently released "Process Mining Manifesto" presents guiding principles and challenges for process mining. Here, the authors summarize the manifesto's main points and argue that analysts should take into account the context in which events occur when analyzing processes.

PM 2 : a Process Mining Project Methodology

Process mining aims to transform event data recorded in information systems into knowledge of an organisation's business processes. The results of process mining analysis can be used to improve process performance or compliance to rules and regulations. However, applying process mining in practice is not trivial. In this paper we introduce PM 2 , a methodology to guide the execution of process mining projects. We successfully applied PM 2 during a case study within IBM, a multinational technology corporation, where we identified potential process improvements for one of their purchasing processes.

Process Mining: A 360 Degree Overview

Process mining enables organizations to uncover their actual processes, provide insights, diagnose problems, and automatically trigger corrective actions. Process mining is an emerging scientific discipline positioned at the intersection between process science and data science. The combination of process modeling and analysis with the event data present in today's information systems provides new means to tackle compliance and performance problems. This chapter provides an overview of the field of process mining introducing the different types of process mining (e.g., process discovery and conformance checking) and the basic ingredients, i.e., process models and event data. To prepare for later chapters, event logs are introduced in detail (including pointers to standards for event data such as XES and OCEL). Moreover, a brief overview of process mining applications and software is given.