Data set properties

We are using mobile communication data of a specific format as a concrete example to discuss and show the properties of sequential structural and temporal data. It was recorded to provide a wide-ranging quality analysis of the underlying network infrastructure. The data is collected by a fleet of specialized cars equipped with roof-mounted antennas and multiple android smartphones. The mobile phones run test sequences, which consist of automatically calling a phone within one of the other cars, establishing a connection, playing a voice chat of 1 minute and finally closing the connection. This whole process is monitored, recorded and consecutively analyzed by a system which focuses on various key performance indicators (KPIs) and statistics, radio frequency (RF) values and the successful completion of the key processing steps of the respective protocol state machines of UMTS, LTE and GSM. Since the cars are moving while all of this takes place, the recorded data also contains switches between different transmission technologies (e.g. the sequence may start in GSM, switches then to LTE, and finishes in UMTS). The resulting data already allows for drawing simple conclusions, e.g. on whether communication sequences have been successful at all (i.e. the relevant KPI and RF values did not show negative deviations, and all relevant protocol states have been completed successfully), whether they dropped in the middle (i.e. important protocol states have not been completed), or whether they have failed for other reasons.

Those failed sequences are then manually analyzed to determine their reason of failure and potentially even its cause. This process is called failure classification, allowing to assign a specific failure class to a failed sequence. Doing this manually by looking at many hundred log file entries is a tedious, time-consuming and expensive task. By using statistics over the KPIs, RF values and the protocol states for rule-based approaches, this can partially be automated, specifically for failures with simpler behavioral patterns. A more versatile machine learning approach could however help solving this problem better, potentially allowing to cover less clearly defined failure classes, while also adding some flexibility when trying to find new failure classes, caused by changes in the communication technology backbone architecture, e.g. with the upcoming 5G [61] technology.

For our analyses we collected two different data sets: the MFC data set, containing manually labeled and unlabeled failure class samples, and the AFC data set, containing failure class samples which are automatically labeled by a rule-based approach, as well as unlabeled samples. Table 1 shows some of their most important characteristics. Each contained sample represents a call sequence, which utilizes at least the GSM, UMTS or LTE protocol, following its respective specification and call phases, which is reflected in the logged events and the respective timestamps. Instead of using all the recorded events though, we are mainly interested in those that are relevant for the potential failure classes. Hence we are using filters based on rules that have been defined by experts with deep domain knowledge, removing all events that contain redundant or irrelevant information. As a result we obtain a final set of base events, which together with their different states (e.g. different reasons for a location update reject), and in combination with the respectively utilized protocol, leads to overall 335 different event identifiers.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Data set statistics: Number of events e_#, of failure sub classes c_# and of samples s_#.

https://doi.org/10.1371/journal.pone.0228434.t001

Both data sets will serve different purposes in this paper, because the details of the different labeling approaches used for both data sets are expected to impact the AFC evaluation performance negatively. The filtered events available in both data sets are selected to cover all important call phases and event sequences potentially relevant for a proper class prediction. As such they are theoretically sufficient to properly reproduce the MFC labels. However, they are insufficient to achieve a similar performance for the AFC data, where also data of additional events, as well as relevant KPI and RF data has been used for defining the classes, none of which we have access to in our structural-temporal data. As a result of these restrictions Table 1 shows that the average number of events per sequence is much smaller for AFC data, and its variance is much larger, reflecting a larger class variance and making a proper discrimination harder. Additionally, the AFC data contains 8 very similar variations of the Core Network Failure class, which are harder to discriminate as well. Due to these shortcomings of the AFC data set, the smaller MFC data set is more relevant for our purpose, as its labels provide a better ground truth. Since this is a problem in the AFC data set, we will restrict our analyses on the AFC data to those which are expected to provide valuable insights on this data set only, namely how well the proposed feature spaces and learning methods can still reproduce the AFC labels under these conditions, and—given its larger number of samples—how much of a performance improvement we can expect when increasing the training data size.

We will now discuss the format of the contained sequences. Each sample in our data sets consists of the bi-directional communication sequence between a caller and a callee. We denote the caller as the MOC client (mobile originated call) and the callee as the MTC client (mobile terminated call). The samples contain highly structured sequential data (order of events) with highly relevant temporal components (inter-event durations), all of which are semantically relevant for discriminating the failure classes, e.g. reflecting call phases being incomplete or too long, or reflecting an anomalous order of events. Table 2 shows an exemplary event sequence of a successful call, starting in LTE and proceeding and ending in UMTS. This example also introduces the new timestamp format t_s.rel, denoting sequence-relative timestamps, defined for a sequence s as t_s.rel(e_i) = t(e_i) − t(e₀) (or t_s.rel(s)_i = t(s)_i − t(s)₀), i.e. the timestamp of the first sequence event is set to 0.0, and all other timestamps are reset relative to this value. In the example the MOC client is set up until t_s.rel = 12.232, then the MTC client is set up until t_s.rel = 14.231. Once the clients are connected at t_s.rel = 30.103 the call takes place, before they are finally disconnected again. The abbreviations represent the following event types: extended service (ES) request, security mode (SM) command and complete, connection management service (CMS) request, radio bearer (RB) setup and extended service (ES) request.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Example sequence of successful bi-directional mobile communication.

https://doi.org/10.1371/journal.pone.0228434.t002

Failure classes

Before analyzing how the concrete sequence properties are utilized in the feature spaces, we first need to discuss the existing failure classes and their structural and temporal properties in more detail, to provide a better practical background of the problem domain. Table 1 contains an overview and provides relevant class statistics for the selected, sufficiently sized failure sub-classes of the listed main failure classes. It also contains details about the samples of insufficiently sized failure classes, grouped together in the set of Other Failures, as well as the additional number of unlabeled samples per data set.

Base processing

All of the following feature spaces require an identical base processing, for which we need a second timestamp format, enabling the consideration of relative time-dependencies (i.e. local delays): the event-relative timestamps t_e.rel, defined as t_e.rel(e_i) = t_s.rel(e_i) − t_s.rel(e_i−1) (or t_e.rel(s)_i = t_s.rel(s)_i − t_s.rel(s)_i−1) for a given sequence s.

One objective for our proposed feature spaces is, that they represent multi-client behavior within the sequential data. This is relevant for our use case, because some failures can be caused by erroneous behavior on the MOC side of the call, while others are caused by problems on the MTC side—or even by problems on both sides, individually or combined. To reflect those structural properties, we need to create the sub-sequences s_MOC and s_MTC to contain exclusively the events of MOC and MTC respectively, with |s_2C| = |s_MOC| + |s_MTC|. These representations allow the feature spaces to omit events of the respective opposite client. Table 2 shows this behavior exemplarily for the MOC-events at t_s.rel = 5.192 and t_s.rel = 12.232. They are interrupted by three MTC events in the s_2C sequence, but are represented consecutively in the s_MOC sequence. As a result, a sequence s can be described by a triple of sequences s_2C, s_MOC and s_MTC. If not stated otherwise, we use s synonymously with s_2C to denote the complete 2-client communication. As such the example in Table 2 effectively illustrates an s_2C sequence. This definition is extended to the whole set of sequences. Where previously we denoted all sequences s in a set S as s ∈ S, we can now also denote the sets of sequences of each different representation, i.e. s_2C ∈ S_2C, s_MOC ∈ S_MOC and s_MTC ∈ S_MTC. This also allows extending the definition of t_s.rel to those representations, in that t_s.rel(s_2C) denotes the vector of the sequence-relative timestamps of s_2C, and t_s.rel(s_MOC) and t_s.rel(s_MTC) denote those of the client-wise sequence representations. The same holds for t_e.rel. Note that t_s.rel is set to 0.0 for the first event of each sequence representation respectively (i.e. t_s.rel(s)_i = t(s)_i − t(s)₀ for s ∈ {s_MOC, s_MTC}), to allow for a better comparability of sequences of the same client type.

Using an exemplary set of event identifiers E = {′A′, ′B′, ′C′, ′D′} allows creating two artificial example sequences x₁ and x₂ and their timestamps in the two formats, as shown in Table 3. These will be used in the next sections to illustrate various aspects of the different feature spaces.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Two artificial communication sequences x₁ and x₂.

https://doi.org/10.1371/journal.pone.0228434.t003

Γ_qT features

Γ_qT features are based on the s ∈ S_2C feature vectors, essentially representing the most common type of data representation used in the related work of process mining. We extend this representation additionally by quantized temporal features. The idea is to get a sequential representation of the event identifiers, which is specifically suited for classifiers used in process mining, but to additionally include temporal information. To achieve this, we start with the event indices in s. Consecutive events with a temporal distance closer than a predefined minimum interval θ_mi maintain their current position in the event index sequence. If consecutive events exceed θ_mi, an additional empty event e_∅ is inserted, reflecting the larger interval between those consecutive events. This is repeated until the next event is reached. As a result, smaller values of θ_mi introduce more events e_∅ and lead to larger, sparser feature vectors, while larger values of θ_mi introduce less empty events, thereby decreasing the feature vector length while increasing the density—until a completely dense feature vector is achieved, containing no empty events e_∅ at all. Through manual analysis of the temporal properties of the analyzed data, a value of θ_mi = 5.0s is selected as a compromise capable of filling large temporal gaps occurring in the data (which often represent overly long durations between two protocol states) while not increasing the overall feature vector length too much in less relevant regions of the sequence. For θ_mi = 5.0s, the resulting Γ_qT feature vector for sequence x₁ in Table 3 is thus Γ_qT(x₁) = [′D′, ′A′, ′B′, ′C′, e_∅, e_∅, ′A′], s.t. the large t_e.rel(′A′) is represented by two instances of e_∅. Choosing θ_mi > 60.0s allows eliminating all occurrences of e_∅, which is identical to the original sequence of events, without the additional temporal information provided by the e_∅ inserted in the sequence. When using Γ_qT in this way, we denote it as Γ_qT*, which allows highlighting the performance differences when using both types of feature representations with competing classifiers of the process mining domain.

Γ_T features

To create the set of temporal features Γ_T, we use the set S_2C. The idea of Γ_T is to create a feature space which projects properties of the ith occurrence of each event type in a sequence s onto the same dimension. The projected property is the timestamp t_s.rel of the respective event, allowing to compare it with the timestamps of the ith occurrence of the same event type of other sequences. We start by calculating the occurrence frequency f(e, s) for each event type e ∈ E in each sequence s ∈ S_2C. Then we define its maximum value as m_e = max(f(e, s)), ∀e ∈ E, ∀s ∈ S_2C, calculating what is the most frequent occurrence of event type e in any sequence. Furthermore a function κ(e, s, i) is required, returning the ith occurrence of e in s. For example the simple sequence x₁ in Table 3 has two occurrences of the event type ‘A’. As such, κ(′A′, x₁, 2) returns e₅ = ′A′, i.e. the 5th event in s is the 2nd occurrence of ‘A’ in s, with t_s.rel(κ(′A′, x₁, 2)) = 18.075. Now a function ts(s, e, m_e) can be defined, providing a vector of these timestamps t_s.rel of all occurrences of a selected event in a sequence, and 0.0 otherwise: with Concatenating the resulting vectors of ts(s, e, m_e) for a sorted list of all e ∈ E via the concatenate()-function yields the final feature vector for a sequence s, which has for all samples the same length of ∑m_e, ∀e ∈ E. A small example should make this better understandable. Table 3 shows two simple sequences {x₁, x₂} = X, for which the maximum frequencies m_e = max(f(e, x)), ∀x ∈ X of each e ∈ {′A′, ′B′, ′C′, ′D′} are m_A = 2, m_B = 3, m_C = 1, m_D = 1. As a result the feature vectors have the following format: for s ∈ {x₁, x₂}, resulting in the following final feature vectors: and

To obtain an optional binary representation of Γ_T, values of τ(s, e, i) larger than a defined threshold can be set to 1, and to 0 otherwise.

Γ_S features

The structural Γ_S features are event n-gram features, similar to the previously used token n-gram features, and as such representing a feature representation commonly used in the related works of sequence classification. Therefore we extract for all s_2C, s_MOC and s_MTC all event n-grams and index their sorted list, spanning the final feature space Γ_S—including the client-specific n-grams of s_MOC and s_MTC. We denote an event n-gram of a sequence feature vector s via its vector indices in interval notation (similar to the one used in Matlab or numpy), s.t. s_[i,i+n) denotes the event n-gram from position i (inclusive) to position i + n (exclusive). The Γ_S feature vector of sequence sample s is then defined via the binary occurrence of the respective event n-gram within s. When using the examples x₁ and x₂ from Table 3, the sorted list of n-grams for n = 3 is [‘DAB’, ‘ABC’, ‘BCA’, ‘DBB’, ‘BBC’, ‘BCB’, ‘CBA’], resulting in the final Γ_S feature vector Γ_S(x₁) = [1, 1, 1, 0, 0, 0, 0].

Γ_S+T features

One base hypothesis of this paper is that the classification performance can be increased by using a complementary structural and temporal feature space. For the structural-temporal Γ_S+T feature space we treat the feature vectors of Γ_S and Γ_T as equivalent. Because of its binary format, Γ_S already produces qualitative feature vectors, but Γ_T produces quantitative feature vectors. If we binarize its values, we create a qualitative representation, which we can simply concatenate with the Γ_S feature vector. This is used here to provide a baseline complementary feature space, before defining the more complex complementary feature space Γ_ST.

Γ_ST features

For the structural-temporal Γ_ST feature space we will first need an analysis of the representative capabilities we specifically want to achieve with this feature space. As such, we will start this section with an analysis of some feature requirements, before explaining how these requirements are met by creating the data representation via structural-temporal δ − n matching and the use of model sequences.

Defining the structural temporal features.

The Γ_ST feature space is based on the idea of representing structural and temporal properties of the respective sequences. In this paragraph we will discuss, how to achieve this by using n-grams and model sequences in structural-temporal matching procedure, with a focus on the feature properties of context and position. We define the context of each event by the size of the n-grams and the parameter of positional variance δ, and the positional properties by the actual matching procedure. The idea of this procedure is to match each n-gram of each s ∈ S with the n-grams of each s^M ∈ sorted(S^M), requiring identical positions of both matching n-grams within the two sequences—and then loosen this requirement via the parameter δ. The result of this matching for each s ∈ S is a vector of its structural similarity to each of the model sequences s^M, for each of its sequence representations s_2C, s_MOC and s_MTC. Specifically the additional matches on the and are relevant for failures, as they allow detecting event chains of a single client, automatically crossing the gap caused by interfering events of the other client. We achieve this by a structural δ-n matching function, denoted as , where denotes a vector of length |s^M|, which is initialized as a zero-vector. By iterating over all indices, this vector is populated as follows:

∀(i + j) ≥ 1 and ∀(i + n + j) ≤ |s^M| + 1, with i ∈ [1, |s^M| − n + 1], j ∈ [−δ, δ] and δ ≥ 0. denotes here the element-wise incrementation of vector by a one-vector of length n. Here we are using the indexing method introduced for Γ_S to denote individual event n-grams, s.t. s_[i,i+n) denotes the event n-gram of the events [e_i, …, e_i+n−1] in the sequence s. As such we essentially compare s_[i,i+n) with , and allow a positional variance in the model sequence by defining j over the range of [−δ, δ].

Using the exemplary sequence x₁ and x₂ of Table 3, with x₁ as and x₂ as model sequence the structural δ-n matching with δ = 1 and n = 1 yields the following matching results for the respective values of i and j

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

resulting in the vector

Since we are not only interested in the structural properties of our data, we will now extend by integrating the event-relative timestamps t_e.rel as temporal properties, obtaining the final structural-temporal projection function Φ. The idea is to calculate the absolute differences of the t_e.rel of the structurally matching events of s and s^M. This is done by modifying the previously used incrementation function int(), giving rise to the final definition of Φ: again ∀(i + j) ≥ 1 and ∀(i + n + j) ≤ |s^M| + 1, with i ∈ [1, |s^M| − n + 1], j ∈ [−δ, δ] and δ ≥ 0. The function defines a vector by calculating the absolute element-wise difference between two vectors and , which is in this case the temporal difference of the respective matching events of s and s^M, as illustrated in the previous example. As this can result in multiple matches per event in ŝ^M, we finally average each field in ŝ^M by its number of matches. Applying this final formulation to the previous example sequences and their event-relative timestamps and yields the final feature vector

Since Φ is defined over a single s^M, it has to be executed for all s^M ∈ sorted(S^M), and the resulting vectors ŝ^M have to be concatenated via the concatenate()-function, giving rise to the final definition of Γ_ST: The final feature vector Γ_ST(s, S^M, δ, n) has the length , and contains the structural-temporal δ − n matches of the sequence s and all model sequences S^M. The O-complexity of this whole process is linear, as creating the feature space by indexing the model sequences S^M is done in linear time, and projecting a single sequence s onto S_M is linear to the number of model sequences |S_M|, as it requires matching the n-grams of each s with those of every s^M ∈ S^M.

Model class detection and prediction

In our use case, the classes of sequence behavior are defined by the different ways communication sequences between both clients can fail. When samples of failed sequences of a new data campaign need to be classified, we could assume a hypothetical scenario in which no new failure classes are found in the new campaign, i.e. all potential failure classes have already been seen before. However, this is not true in practice, where new types of previously unseen failures occur indeed. This is also true in our data sets, with the consequence that only a limited number of failures classes have a sufficient size to properly evaluate supervised classification models with them. We denote such classes as model classes, or MC. Sequences of Other Failures, i.e. of insufficiently large failure classes are denoted as non model classes, or ¬MC.

This motivates the design of our system, consisting of two major components, which allow to detect whether a new sample is potentially a MC sample (and not a sample of ¬MC), and if that is the case, to predict the respective model class. Accordingly, those steps are called the model class detection (MCD) and the model class prediction (MCP). For the MCP a classifier is trained in a multi class approach, learning to discriminate only samples of the MC, but not the ¬MC. For the MCD multiple classifiers are trained, one for each MC. Each of those classifiers is trained in a two class approach, learning to discriminate the respective MC against samples of ¬MC. After training the MCP and MCD classifiers, we can predict the failure class of a new sample by predicting a MC with the MCP classifier, and then using the MCD classifier trained for this MC to confirm or reject this prediction.

These predictions are obtained by applying their respective prediction functions to the feature vector Γ(s) of a test sample s, using one of the previously defined feature spaces, i.e. Γ ∈ {Γ_qT, Γ_T, Γ_S, Γ_S+T, Γ_ST}. The function for the model class prediction classifier is F_MCP, with and with y_MCP ∈ {MC₁, …, MC_k}, the set of all k model class labels. The prediction function for the model class detector is F_MCD(MC_i), obtaining the prediction of the classifier trained for MC_i via with y_MCD ∈ {MC_i, ¬MC}. As one can see, F_MCP returns one of the MC-labels, and F_MCD(MC_i) returns the label of the class MC_i, or ¬MC.

Combined classification system

We are combining the prediction functions F_MCP and F_MCD by using two confidence ratings as a way to ensure a higher confidence in the predictions of the combined (MCP and MCD) classifier, as this helps improving the classification precision of our approach, which is crucial in the practical application. These confidence ratings produce the binary results High and Low. They can be logically combined and can be interpreted either as providing support for the prediction of the MCP (High confidence) or objecting against its prediction (Low confidence).

Since the output of F_MCD is limited to a single MC_i and ¬MC, it is used as the first confidence rating for y_MCP, answering the question of whether sample s really belongs to the already predicted y_MCP (i.e. a specific MC_i) or whether it belongs to ¬MC. For this purpose, the confidence rating function Θ_MCD(Γ(s)) ∈ {High, Low} is defined as:

To obtain further confidence on the classification result of F_MCP, we define an additional confidence rating Θ_db. It uses the decision boundary of the MCP classifier of the predicted class. The idea behind Θ_db is to calibrate the decision boundary of each MCP classifier towards more conservative values, requiring a sample with y_MCP = MC_i to cross a stricter decision boundary for this MC_i to obtain a High confidence for Θ_db, which reduces the false positive rate and increases the precision. As it is defined over the decision scores, it can be applied to any classifier which provides access to its decision scores or probabilities. For this purpose we access the prediction score of the MCP via the function D_MCP(Γ(s)). We also require the existing bias b of the MCP classifier of y_MCP, and a parameter θ_DB ≥ 0 to define the new bias b_db = b + (D_⌀ − b) · θ_db, with D_⌀ representing the mean of those decision scores of the training samples that have been correctly classified by the MCP. As such, the parameter θ_db gives rise to the following definition of the confidence rating Θ_db:

For example in the two-class definition of the SVM the decision function is F_MCP(Γ(s)) = sign(w^TΓ(s) + b), with w being the weight vector of the trained model. Here we achieve a positive prediction if w^TΓ(s) + b > 0. The respective function to access the score is then D_MCP(Γ(s)) = w^TΓ(s) + b.

θ_db can be changed dynamically during model selection, which allows for a calibration of the model precision, similar to other methods of false positive calibration as used in e.g. [59]. The confidence ratings are then processed together with the prediction y_MCP to produce the final predicted label F_combined(Γ(s)) ∈ {MC₁, …, MC_k, ¬MC}, defined as follows:

The effect of the confidence ratings and the consequently created High-confidence prediction subset on the applied evaluation metrics will be elaborated in the next section. Now that the MCDs and MCPs are trained and calibrated, we can apply the system to classify unlabeled sequences, as illustrated in Fig 2. As just described, the final prediction F_combined(Γ(s)) of this combined classifier for a given sequence s only provides the label of the predicted MC_i if Θ_MCD(Γ(s)) = High and Θ_db(Γ(s), θ_db) = High, and ¬MC otherwise. This, together with the still accessible Θ-confidence ratings allows for an effective way of increasing the system classification precision, thereby discriminating reliable and unreliable predictions, which is highly relevant in the practical application.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Application of the detection and prediction system.

https://doi.org/10.1371/journal.pone.0228434.g002

Learning methods

While unsupervised or semi-supervised methods have shown to produce good results on textual and structural data and could be relevant to our problem of model class detection, supervised methods are regularly outperforming them and are the preferred solution, if labeled training data is available. As discussed in Section Related Work, Decision Trees, Markov Classifiers and LSTM are learning methods widely used in the domain of process mining, while MLP and SVM are more widely used on non-sequential data representations of sequential data. For these reasons we are conducting our evaluations on those methods, to provide a broad picture of the classification performances achievable on the discussed sequential (Γ_qT), non-sequential (Γ_T, Γ_S, Γ_S+T) and semi-sequential (Γ_ST) feature representations. We also include the classification performance using k-nearest neighbors to provide an additional baseline. As some of the feature spaces are designed with specific learning methods in mind, they are only evaluated on those learning methods. In our experiments we actually tested all types of features with all learning methods, but achieved suboptimal results on non-suitable learning methods. For those reasons the respective results are omitted. All of the models below have been chosen using standard cross-validation based model selection.

Evaluation metrics

The general system application of reliably detecting and predicting known amidst unknown sequence classes, and its concrete practical application in failure classification enforces a focus on two primary objectives: (1) to obtain reliable predictions (2) for as many MC samples as possible. Precision, recall and the F1 score capture those aspects. For the evaluation of the MCP and the MCD classifiers they are calculated for multiple cross validation repetitions, s.t. we chose to calculate their unweighted class mean (denoted with the keyword macro), because we already configured the sampling procedure to produce similarly sized model classes. Whenever the ¬MC class participated in the evaluation (in MCD and combined classifiers) we excluded it from the calculation of the classwise mean values, because our focus is on the correct detection of MC samples—and not on the correct detection of ¬MC samples. As the set of ¬MC samples is much larger than each individual MC, this would otherwise lead to overly optimistic evaluation results. The formula below shows this approach exemplarily for the macro precision, where the ¬MC are implicitly contained in the calculation of the precision of each MC, but are not used as a primary class:

By combining MCP and MCD classifiers and applying confidence ratings, we effectively create a filter, allowing us to focus solely on the created High-confidence subset of the predictions, which is designed to contain only those samples and labels which truly are of a model class MC and are correctly classified as such, thereby fulfilling objectives (1) and (2). Due to this mixture of multi-class (MCP) and two-class (MCD) predictions, we also have to adapt the metrics in the utilized confusion matrix, as described in Table 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Metrics of the confusion matrix.

https://doi.org/10.1371/journal.pone.0228434.t004

Based on these values precision and recall are defined as usual, with precision = TP/P and recall = TP/(TP + FN). However, to correctly address objective (2) we have to calculate an additional effective recall by considering all existing MC samples, not only those in the High-confidence subset. Therefore we define the effective recall as the recall of correctly predicted samples of MC over the sum of samples in all MC, i.e. . Together with the precision over the High-confidence subset, this metric allows for a conclusive analysis of the overall system classification performance, as provided by the combined classification processing.

Experiments on MFC data

To be able to apply the selected learning methods, we have to assure sufficiently sized failure classes. To obtain any model classes at all, we sample only from classes with a minimum size of 15 sequences. To improve the interpretability of our classification results we opted for similarly sized failure classes, which we achieved by limiting the size of each failure class to a maximum of 25 sequences. As such, the number of samples per MC used for the evaluation, s_#MC, is 15 < s_#MC ≤ 25. To simulate the complete system, we also need members of the Other Failures class, of which we used all 86 available sequences, i.e. s_#¬MC = 86. In the MCD evaluation this allows highlighting the detection purpose of the method, as the ratio of samples of MC_i to ¬MC is approximately 1: 4. In the combined evaluation the ratio of all MC samples to ¬MC samples is approximately 1: 1, which helps in the interpretation of the classification results. For defining the Γ_ST features we used all 6,077 labeled and unlabeled MFC samples as model sequences S^M. As previously described this use of the unlabeled sequences helps to extract additional information about the behavior of the projected sequence. To further reduce the high dimensionality of the resulting Γ_ST feature space, we additionally applied a dimensionality reduction, which utilizes the redundancy between the S_2C and S_MOC, and the S_2C and S_MTC feature vectors, once they are projected with the structural δ − n matching of . This resulted in a feature space of 294,435 dimensions. We also conducted an additional analysis on the number of dimensions actually relevant for a sufficient data representation, which could motivate further dimensionality reduction steps. This analysis is summarized in the Appendix.

For all evaluations we used 20 times random sampling, each with a 5-fold cross validation. For a proper evaluation we made sure all conducted comparisons between classifiers and feature spaces were done on the same respective samplings. For the event n-gram sizes of the Γ_S, Γ_S+T and Γ_ST feature spaces we used a fixed n-gram size of n = 2, which yielded the overall best results. We also tested using multiple values of n simultaneously, as e.g. described in [65]. However, the performance increase was only minimal. As a convention, all results are listed as the mean and standard deviation in percent.

Evaluation of the individual MCP and MCD classifiers.

The purpose of the first set of experiments is to find the best performing learning method for all feature spaces, s.t. we can restrict the further experiments to this learning method, allowing to focus on the feature space analyses. Since Γ_ST is further parametrized by δ, we start with analyzing the impact of different values of δ on the MCP classification performance of Γ_ST using the SVM classifier. To obtain more reliable parameter values, the calibration is conducted using solely the MCP, and not the combined MCP-MCD system. The mean sample length in the MFC data set is 44.11 events, with a standard deviation of 13.51 events. Since δ encodes the positional variance of the matched n-grams within the projected sequence, it does not make sense to increase its size beyond a value of δ = 60, at which the complete average sequence length is covered. Since we expect a high importance of a similar positioning of the matched n-grams, we expect better results for smaller values of δ. The left plot in Fig 3 shows the results for δ ∈ [0, 60]. As expected we achieve the best results with a value of δ ≤ 10. For that reason we focused stronger on the range of δ ∈ [0, 10], which is illustrated in the right plot in Fig 3, allowing to further reduce the selection of an optimal value down to δ = 5. As this value allows a positional variance of ±5 events on S^M, it can additionally be explained by the circumstance that event sub-sequences, which are crucial to the protocol, like the call setup (also illustrated in Table 2) take around 10 events in the s_2C representation, requiring any sequence to match the contained events.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. MCP evaluation on MFC data for Γ_ST: F1 scores (macro) in% for ranges of δ ∈ [0, 60] (left) and δ ∈ [0, 10] (right).

https://doi.org/10.1371/journal.pone.0228434.g003

Now that we know a proper setting of δ we can conduct a comparative evaluation of all MCP classifiers on the MFC data set, using all feature spaces. The results are shown in Table 5. Of those learning methods which are widely used in the field of process mining, and which have been applied on Γ_qT and Γ_qT*, the decision tree showed the overall best performance, specifically on Γ_qT*, i.e. the original sequence representation. However, both the Markov and the LSTM classifier achieve an improved performance on the temporally enriched Γ_qT feature space. When compared to the performance on the other feature spaces though, all of those approaches commonly used in the field of process mining are clearly outperformed by the other feature spaces and learning methods. While this was to be expected, given that the process mining approach, and specifically deep learning approaches like LSTM usually require much larger training data sets, also the lack of the additionally included concrete temporal information is highly relevant, since they are even outperformed by Γ_T, which only contains strongly reduced structural information about the data. Hence the SVM classifier performs best on all feature spaces, outperforming the otherwise widely used MLP, as well as the KNN approach. For those reasons we are using it for the remaining experiments. The results of the SVM classifier also show, that in the optimal scenario in which all MC are known, good results can already be achieved without using the proposed θ_db system calibration.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Individual evaluation results of MCP and MCD on MFC data (%).

https://doi.org/10.1371/journal.pone.0228434.t005

For the MCD evaluation only the SVM classifier is evaluated. This decision was taken as it shows the most robust behavior in the MCP evaluation. The results are shown at the bottom of Table 5. Obviously discriminating the MC and ¬MC is harder than separating the MC in the MCP setting, which is to be expected, as the ¬MC samples are very heterogenous. However, using semi-supervised learning via a One-Class SVM to model each MC_i against the ¬MC performed even worse. Since all other learning methods also performed much worse, their results have been omitted to maintain a proper readability of this section. Of all feature spaces Γ_S+T performs best, while the specifically crafted Γ_ST feature space is slightly outperformed by all other feature spaces. While one might think that Γ_ST does not look that promising yet, the combined classifier evaluation will show, that it performs better than its competitors in the final system layout, when the effective recall becomes relevant.

Evaluation of the combined classifiers.

Now that we established some understanding of the performance of the individual MCD and MCP classifiers, we will now evaluate for each qualified feature space its combined classification performance, also integrating the previously described confidence ratings. We do this to find the feature space which has the highest precision, at the highest possible effective recall, which is highly relevant for an effective system in the practical application. Achieving high precision predictions means that we can trust the results to be correctly classified and to not contain any samples of ¬MC falsely being classified as an MC sample. And getting the high effective recall means, we get this predictive behavior for a larger portion of the MC samples that are actually contained in the test set. This aspect is illustrated in Fig 4, which shows the percentage of High-confidence samples of MC to all samples of MC in the test set, under a shifting parameter θ_db ∈ [0, 1.0]. The more θ_db is increased, the less samples of MC are actually contained in the High-confidence subset, reducing the potential effective recall.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Percentage of MC samples in the High-confidence set, for θ_db ∈ [0, 1.0], for Γ_T, Γ_S, Γ_S+T and Γ_ST.

https://doi.org/10.1371/journal.pone.0228434.g004

For comparing the combined classification performance of the different feature spaces with the SVM classifier, we need to select values of θ_db representing practically relevant values of precision and recall, which are similar for the respective feature spaces. Fig 5 contains the precision, recall and the effective recall of the classification results for θ_db ∈ [0, 1.0]. The precision starts to reach 100% for most classifiers at θ_db = 0.8. At this value also the recall reaches the maximum of 100%. When looking at the concrete values of mean and standard deviation, shown in Table 6, we see that the recall can not be further increased, and that the corresponding precision can be selected s.t. it is around 93% for Γ_ST, Γ_S+T and Γ_S. Since further increasing the precision would not further increase the recall, and 93% is already a reasonable system precision, we will use this value and the respective settings of θ_db for the further analyses. Thus we are using θ_db = 0.8 for Γ_S and Γ_S+T, and θ_db = 0.9 for Γ_ST. In this respect, the performance of Γ_T was not sufficiently high to achieve similar values of precision and recall, which is why we used θ_db = 1.1 there, achieving relatively close values for further analyses.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Combined classification results on MFC data: Precision, recall, effective recall (from top to bottom) in % for θ_db ∈ [0, 1.0], for Γ_T, Γ_S, Γ_S+T and Γ_ST.

https://doi.org/10.1371/journal.pone.0228434.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Combined evaluation results on MFC data (in %).

https://doi.org/10.1371/journal.pone.0228434.t006

Now we need to evaluate which feature space offers the best effective recall, i.e. the fraction of MC samples that can be recovered from the set of test samples, with those high values of precision and recall previously described. At their respective values of θ_db, Γ_S has the lowest effective recall of 25.05%, followed by Γ_S+T with 26.27%, and Γ_ST with an effective recall of 31.79%. Thus Γ_ST produces a precision performance similar to both Γ_S+T and Γ_S, while achieving a 5.5% higher effective recall than Γ_S+T, and a 6.7% higher effective recall than Γ_S. This means, we can get for 31.79% of the MC samples in the test set a correct prediction with 93.11% precision and 100% recall when using Γ_ST, compared to 26.27% effective recall, 93.51% precision and 100% recall when using Γ_S+T, and even worse when using Γ_S. These results are highly relevant in practice, as they effectively allow filtering near-certain from uncertain predictions with a very high precision. These results are also highlighting the effectiveness of both combined feature spaces, with a significant advantage for the Γ_ST feature space. In that regard both structural-temporal feature spaces Γ_S+T and Γ_ST outperform Γ_S and Γ_T: Whereas Γ_T has a relatively good effective recall, but a relatively low precision, Γ_S has an acceptable precision, but a low effective recall. This renders both feature spaces less practically relevant than their combined counter parts, highlighting the relevance of combined structural-temporal feature spaces.

The dimensions most relevant for the respective classification results in this use case were security handshake events, followed by the existence of events representing a successful response to the most relevant key protocol states, like a successful radio bearer setup. As we saw in the MCP evaluation, the temporal features are also relevant and utilized in both combined feature spaces. The Γ_ST feature space also has an advantage here, as its S^M-based feature space allows locating the concrete positions and structural-temporal properties of the relevant events within the sequence, which are in the evaluation often identified as responses occurring too late in the sequence, or security mode negotiations at anomalous sequence positions. All of this can then be used to obtain deeper insights into the data, which can help manual analysts to limit the number of causes for this specific failure class.

Due to their potential in combining classifiers of different features spaces, we also evaluate the classification performance of an ensemble method [66], namely the ensemble classifier , which could potentially further optimize precision and effective recall. It predicts the combined classification results by using a majority voting over the predicted labels of Γ_S, Γ_S+T and Γ_ST. Table 6 shows its results when using their default trained models of θ_db = 0.0, and for their optimized decision boundary models, using θ_db = 0.8 for Γ_S and Γ_S+T, and θ_db = 0.9 for Γ_ST respectively. Γ_T has not been used due to its lower performance. When using the default models at θ_db = 0.0, the ensemble classifier in fact achieves the best precision at the cost of the effective recall, an effect similar to the trade-off of θ_db. For the optimized models of θ_db ∈ {0.8, 0.9} the ensemble classifier achieved worse results though.

Significance analysis.

To further substantiate the results of the previous section, we conduct significance tests on both of our theoretical hypotheses, namely that the combined feature spaces Γ_S+T and Γ_ST outperform the base feature spaces Γ_T and Γ_S in terms of effective recall at similar precision (Hypothesis H^A), and that the more complex combined feature space Γ_ST outperforms the simpler combined feature space Γ_S+T under the same premises (Hypothesis H^B). For the formulation of the hypotheses we denote θ_er as the minimal effective recall.

For hypothesis H^A the null hypothesis is defined as follows: When using Γ_S or Γ_T for achieving a test set precision mean of 93%, a fraction of p₀ samplings have an effective recall ≥ θ_er. The alternative hypothesis is then defined as follows: When using Γ_ST or Γ_S+T for achieving a test set precision mean of 93%, a fraction of samplings have an effective recall ≥θ_er. Now we can formulate the question for hypothesis H^A: Is there sufficient evidence at the α = 0.05 level to conclude that the effective recall for the high precision classification performance is increased, when using one of the combined feature spaces Γ_ST or Γ_S+T instead of one of the individual feature spaces Γ_T or Γ_S? And at which minimal effective recall θ_er does this hold? The results for the minimal θ_er, at which we can reject in favor of (i.e. above which p ≤ α always holds for the resulting p-values) are shown in Table 7, for each pair of base and combined feature space, as calculated on the same sampling that have also been used for the previous combined MFC evaluation. We can see that can be rejected for Γ_T for values of θ_er ≥ 5%, i.e. for nearly all values of θ_er, excluding those which do not occur in the combined feature spaces due to their generally higher effective recall. For Γ_S, can be rejected for θ_er ≥ 9% for Γ_ST, and for θ_er ≥ 14% for Γ_S+T. This means Γ_ST is better for a larger number of samplings, while Γ_S+T starts outperforming Γ_S later—both of which is also relevant for hypothesis H^B.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Values of minimal θ_er to reject hypotheses and at α = 0.05.

https://doi.org/10.1371/journal.pone.0228434.t007

For hypothesis H^B the null hypothesis is defined as follows: When using Γ_S+T for achieving a test set precision mean of  93%, a fraction of p₀ samplings have an effective recall ≥θ_er. The alternative hypothesis is then defined analogous: When using Γ_ST for achieving a test set precision mean of  93%, a fraction of samplings have an effective recall ≥θ_er. The question for hypothesis H^B is then: Is there sufficient evidence at the α = 0.05 level to conclude that the effective recall for the high precision classification performance is increased, when using the complex combined feature space Γ_ST instead of the simpler combined feature space Γ_S+T? And at which minimal effective recall θ_er does this hold? As shown in Table 7, can be rejected for all values of θ_er ≥ 30%, showing that Γ_ST indeed outperforms Γ_S+T, a fact that is further strengthened by the performance advantage of Γ_ST over Γ_S+T, as shown for hypothesis H^A.

We will now further elaborate hypothesis H^B by analyzing the distribution of precision and effective recall, when using either feature space on the same test sets. The results of this performance variance analysis are shown in Fig 6. As previously stated, and shown in Table 6, we conducted the significance tests on SVM classification models calibrated for an average precision ≥ 93%. As shown in the left plot of Fig 6, the models of both Γ_S+T and Γ_ST show similar performance distributions, with a slight advantage for Γ_S+T, due to its slightly higher average precision of 93.51%, compared to the 93.11% of Γ_ST. However, as the right plot shows, the effective recall on the same test sets is much balanced towards Γ_ST, clearly supporting hypothesis H^B. As a result these analyses support our conclusion that Γ_ST is the most advantageous feature space in the discussed use case.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Precision (left) and effective recall (right) of Γ_ST against Γ_S+T(in %).

https://doi.org/10.1371/journal.pone.0228434.g006

Experiments on AFC data

Due to the already discussed shortcomings of the AFC data set properties, we are only interested to see, whether the MCP are capable of discriminating the model classes of the AFC data set at all, and how much of a performance improvement we can expect with a larger training data set, which is possible only on the AFC data set. Similar to the class size restrictions described for the MFC evaluation, we have to ensure sufficiently large as well as similarly sized failure classes. To reflect smaller and larger training data sets, we evaluate two different setups. The first setup is defined with comparability to the MFC evaluations in mind. Hence we use s_#MC = 25, resulting in the 13 sufficiently sized failure classes listed in Table 1. In the second setup, selecting s_#MC = 100 allows for a larger training data set, resulting in 4 sufficiently sized failure classes. For defining the Γ_ST features we used all 3,264 sequences as model sequences S^M, resulting in a feature space of 144,938 dimensions after redundancy-based dimensionality reduction.

Table 8 shows the results of the MCP evaluations on the AFC data set. Due to the differences in the MFC and the AFC data, we expected a worse classification performance than on the MFC data, which indeed occurs. However, for the larger sets of training data with s_#MC = 100 the results are largely improved, which shows, that the AFC data set still contains a sufficient number of discriminative features to enable classification. This also documents the potential for an equally increased classification performance on the MFC data, once more class-wise training data is there available as well—which also applied to the general use case of similar classification problems.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. Individual evaluation results of MCP (SVM) on AFC data (in %).

https://doi.org/10.1371/journal.pone.0228434.t008

Relevant dimension estimation

Due to the projection on the set of model sequences S^M, the resulting Γ_ST feature space can become very large. To estimate the potential for the application of a dimensionality reduction approach, we conducted additional experiments using Γ_ST and the model class predictors (MCP) on the MFC data, to achieve an empirical estimation of the number of relevant dimensions, similar to the analyses conducted in [62]. However, the sparsity of the Γ_ST feature space, together with the limited size of the analyzed data sets makes it hard to analyze the full set of dimensions with a sufficient statistical robustness. To achieve statistically more robust results, the number of base dimensions of Γ_ST was reduced to those 5.000 dimensions with the largest variances, selected from a filtered set of those dimensions that had a density > 70%. For those dimensions a PCA analysis was conducted repetitively, to extract the k largest PCA components in a range of 1 to 1.000. We then projected the train and test samples onto the dimensions defined by those k largest PCA components to achieve a feature representation with a reduced dimensionality. This allowed us to train and test a linear classifier on each of those projected sets, obtaining the results for the largest 100 and 1.000 PCA dimensions respectively, as shown in Fig 7.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Results over the first 100 (top) and 1.000 (bottom) dimensions of the largest PCA components using a condensed set of Γ_ST features.

https://doi.org/10.1371/journal.pone.0228434.g007

As can be seen, the performance slowly increases, as long as additional dimensions are added to the utilized feature space. Although the variance is still relatively high, we already achieve a convergence at about 100 dimension. The variance starts to decrease from 400 dimensions on. This means, that for this set of filtered dimensions, a reduction to the 400 largest PCA dimensions can be achieved without loosing much of the precision of the base feature set. Due to the restrictions mentioned above, the results achieved on this limited set of dimensions do not represent the full set of features. This also explains, why the results are lower than those presented in Table 5. Despite these restrictions, the results allow establishing the hypothesis that a much lower number of dimensions may be sufficient when using the Γ_ST feature space. This hypothesis needs to be tested on larger data sets in the future.