python/0_data_loading.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predictive Process Monitoring - Data Loading \\& Exploration\n",
    "\n",
    "This notebook is part of the [starter package](https://github.com/fmannhardt/starter-predictive-process-monitoring) for predictive process monitoring. It contains examples for data loading from event logs for the purpose of developing and applying predictive process monitoring techniques."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "The following Python libraries are used, please refer to the installation instructions to prepare your environment:\n",
    "\n",
    "* [PM4Py](https://pm4py.fit.fraunhofer.de/)\n",
    "* [Pandas](https://pandas.pydata.org/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import pm4py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Event Log \n",
    "\n",
    "We are using a publicly available event log called Sepsis Cases as an example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CSV File\n",
    "Event logs can be loaded with Pandas from CSV files and are then converted to an event log by PM4Py by specifying columns for the three main requirements of an event log: case identifier, activity identifier, and timestamps. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sepsis = pd.read_csv(\"../data/sepsis.csv\", sep=';')\n",
    "sepsis_log = pm4py.format_dataframe(sepsis, case_id='case_id', activity_key='activity', timestamp_key='timestamp')\n",
    "sepsis_log = pm4py.convert_to_event_log(sepsis_log)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ignore the Pandas warning which is due to some internal issue in PM4Py and let us check if the import succeeded:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(sepsis_log)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### XES File\n",
    "\n",
    "Alternatively, PM4PY can also load the event log directly from a file in the standardized XES file format for event log."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.request import urlretrieve\n",
    "import os\n",
    "\n",
    "# download from 4tu.nl\n",
    "urlretrieve('https://data.4tu.nl/file/33632f3c-5c48-40cf-8d8f-2db57f5a6ce7/643dccf2-985a-459e-835c-a82bce1c0339', 'sepsis0.xes.gz')\n",
    "sepsis_log = pm4py.read_xes('sepsis0.xes.gz')\n",
    "os.unlink('sepsis0.xes.gz') # clean up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(sepsis_log)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Exploration\n",
    "\n",
    "It is a good idea to perform some data exploration to investigate the properties of the event log. Please note that there are much more capabilities in ProM. So, do not limit yourself to using PM4Py."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# number of distinct trace variants\n",
    "len(pm4py.get_variants_as_tuples(sepsis_log))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# how does the process start\n",
    "pm4py.get_start_activities(sepsis_log)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# how does the process end\n",
    "pm4py.get_end_activities(sepsis_log)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In newer PM4Py version the default format of an event log is a Pandas DataFrame (sepsis_log) but we can also look at a log as a list of lists containing all attributes as dictionary (sepsis_event_log)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sepsis_log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pm4py.objects.conversion.log import converter as log_conversion\n",
    "sepsis_event_log = log_conversion.apply(sepsis_log, {}, log_conversion.TO_EVENT_LOG)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sepsis_event_log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A typical 'view' on the log is to project events on the event label (concept:name in XES):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pm4py.objects.log.util.log import project_traces\n",
    "def print_nth(log, index):\n",
    "    print(str(project_traces(log, 'concept:name')[index]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print_nth(sepsis_event_log, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print_nth(sepsis_event_log, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us look at the directly-follows graph of the whole event log:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfg, start_activities, end_activities = pm4py.discover_dfg(sepsis_log)\n",
    "pm4py.view_dfg(dfg, start_activities, end_activities)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A quite messy process. There are countless filtering options in PM4Py that allow you to focus on a subset of the data. For example, what happens if we remove the very frequently repeating activities: Leucocytes, CRP, LacticAcid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sepsis_log_filtered = pm4py.filter_event_attribute_values(sepsis_log, \n",
    "                                    attribute_key = 'concept:name', # special column for the activity name always added by PM4Py\n",
    "                                    values = ['LacticAcid', 'CRP', 'Leucocytes'], \n",
    "                                    level = 'event',    # we want to keep all traces and modify events\n",
    "                                    retain = False)     # remove matching events"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfg, start_activities, end_activities = pm4py.discover_dfg(sepsis_log_filtered)\n",
    "pm4py.view_dfg(dfg, start_activities, end_activities)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A bit more comprehensible but more filtering or pre-processing could be performed, explore the PM4Py documentation on what is possible:\n",
    "\n",
    "https://processintelligence.solutions/static/api/2.7.11/api.html#filtering-pm4py-filtering\n",
    "\n",
    "You can also look at the trace or event attributes in an event log to generate suitable subsets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pm4py.get_event_attributes(sepsis_log)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The set of values of a specific event attribute can be extracted. A 'nan' indicates that there are many events without such attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pm4py.get_event_attribute_values(sepsis_log, 'Diagnose')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "28aff1567d8aae5536826c1be921f2ff2e204808293d43dc67bdcb73bd29110e"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}