\section{Evaluation}\label{sectEvaluation}

The goal of this evaluation is to measure and quantify the performance of the implementation of the IVML reasoner, in particular in comparison to the initial version that we used as starting point for the revision. Therefore, we aim here at a practical and illustrative comparison in the sense of a relaxed technical experiment rather than a fully-fledged technical experiment. However, the technical measurement support that we employ would allow for an experiment in a controlled technical environment. 

The practical perspective on the experiment that we take here allows us to measure the reasoner in some form of application setup, i.e., we consider variant setups including different Java versions (Java 8 and Java 9) and different operating systems (Windows 7, Windows 10, Linux) for the standalone variant of EASy-Producer. This allows us to discuss the impact of Java versions and operating systems. In contrast, for a strict technical experiment, we would have to ensure that, e.g., only absolutely required services are running or a user interface that may influence the measurements is not present. Typically, such strict requirements exclude using Windows as operating system. We focus here exclusively on response time and leave other potential interesting performance dimensions like memory usage to future evaluations. 

We present in Section \ref{sectEvaluationSetup} the setup of this evaluation. As the involved IVML models must be ordered for the presentation of the results, we discuss in Section \ref{sectModelComplexity} a pragmatic ranking based on the model complexity. Finally,  in Section \ref{sectEvaluationResults}, we present and discuss the results.

\subsection{Setup}\label{sectEvaluationSetup}

In this section, we present the setup of the experiment in terms of subjects (Section \ref{sectEvalSetupSubjects}), treatments (Section \ref{sectEvalSetupTreatments}), data collection (Section \ref{sectEvalSetupDataCollection}), experimental procedure (Section \ref{sectEvalSetupProcedure}), and analysis (Section \ref{sectEvalSetupAnalysis}).

\subsubsection{Subjects}\label{sectEvalSetupSubjects}

The subjects in this evaluation are two versions of the EASy-Producer SSE reasoner, namely: 
\begin{itemize}
  \item The original reasoner implementation that acted as basis for the revision. This base version\footnote{\label{reasonerBaseVersion}Git hash 6a00aa9c5aaa37ddb3d490d36c7e9a037e792656} is part of EASy-Producer release 1.1.0, i.e., we will call this original implementation \emph{reasoner v1.1.0}.
  \item The revised reasoner implementation using the algorithms described in this document. This version\footnote{\label{reasonerActualVersion}Git hash e0b09aec719d5387255e2b979d90708f05364e75. This version rather similar to EASy-Producer release 1.2.0, but includes some further improvements.} is part of EASy-Producer release 1.3.0, i.e., we will call it \emph{reasoner v1.3.0}.
\end{itemize}

\subsubsection{Treatments}\label{sectEvalSetupTreatments}

Several test cases of EASy-Producer involve reasoning, namely 
\begin{itemize}
\item the EASy-Producer test suites for the SSE reasoner (based on the \IVML{ReasonerCore} test suite).
\item the EASy-Producer test suite for the VIL runtime extension involving the reasoner to detect runtime deviations and to validate runtime changes in the configuration before executing them.
\item scenario test cases for the BMWi ScaleLog\footnote{These test cases are not publicly available as they contain propretary knowledge of the industrial partner in the ScaleLog project.} project.  
\item the EASy-Producer scenario test cases including several (historical) model variants from FP7 QualiMaster \cite{EichelbergerQinSizonenko+16}. 
\item an extended set of scenario test cases derived from the largest QualiMaster models in the EASy-Producer scenario test cases. These tests were specifically defined for this experiment as the gap in model size between the QualiMaster models and the other models was too big. The QualiMaster model consists of several imported projects, various user-defined types, a topological structure for defining Big Data processing pipelines and 16 configured pipelines. This corresponds roughly to 20.000 individual variables. In the set of test cases, we created models with the same type definitions but systematically varied the number of pipelines, i.e., created projected models with one pipeline, two pipelines etc. All these models contain only the required linked variables such as algorithms or data sources so that the respective model is structurally valid and its configuration is consistent.
\end{itemize}
We use these test cases as experimental treatments, although they involve test dependencies such as jUnit. While some of the test cases rely on programmed models (in terms of the IVML object model), most of the test cases specify the model in terms of IVML, i.e., for a more realistic setup and require for execution the IVML parser as well as dependent Eclipse libraries. As we focus on the reasoning time, the actual creation of the reasoning model shall not affect the results. 

It is important to note that the code for reasoner v1.1.0 does not include several test cases that have been created for v1.3.0. For this experiment, we enable for v1.1.0 as many test cases as possible, i.e., we manually ported back\footnote{\label{fnPatch}Patch is available from \MISSING{XXX}.} the v1.3.0 test cases into v1.1.0 and, if required, either adjusted the expected test result accordingly or, in extreme cases, disable test cases that cannot be handled by the v1.1.0 reasoner (or the underlying IVML implementation). For this reason, the treatment sets differ in terms of specific tests, while most of the imporant small and large models are the same. We believe that this is acceptable for an illustrative experiment. 

\subsubsection{Data Collection}\label{sectEvalSetupDataCollection}

In the test cases mentioned above, we employ the generic measurement data collector of EASy-Producer. The collector stores key-value pairs representing measured (real) values. A measurement  carries additional information to identify the test cases, including a tag (used to indicate the reasoning mode), the name of the involved IVML model, the name of the executing test case as well as the intra-experiment repetition count, i.e., the repetition of the same reasoning run within the test case. By default, the collector automatically accounts for (wall) response time, which can help validating more detailed time measures collected during the execution of a treatment. Collected values are stored along with the additional information when the data collection for a test case is finished. As default storage format we use tab-separated values, i.e., a column-table format, as this can easily be read by statistic frameworks such as R as well as Microsoft Excel. We use Excel in particular for gaining a quick overview of the collected information.

As specific measurements for this evaluation, we include 
\begin{itemize}
 \item the default statistics of the SSE reasoner, i.e., the time consumed by constraint translation (translation time), the time used for evaluating constraints (evaluation time), the time used for reasoning (including translation and constraint evaluation time), the number of failed constraints, and the number of re-evaluated constraints.
 \item  statistics model about the underlying model as well as the complexity measure (cf. Section \ref{sectModelComplexity}) realized by EASy-Producer.
\end{itemize}

However, EASy-Producer release v1.1.0 does not contain the generic data collector and several measures, so, along with the test cases, we ported the related code from v1.3.0 back\footref{fnPatch} into v1.1.0. 

\subsubsection{Experimental Procedure}\label{sectEvalSetupProcedure}

During the experiment, we execute each of the test suites in Section \ref{sectEvalSetupTreatments}. Due to their use in continuous integration, each test suite runs in an own JVM instance. To compensate delayed just-in-time (JIT) compilation, we include a specific ramp-up test in the experimental runs to warm up the JVM.  For most test cases, reasoning over a simple representative model including a compound type, a collection over that type and a quantor constraint over the container variable seems to be sufficient. For the QualiMaster models, we added as ramp-up run a full run of one of the largest models without accounting for reasoning time. If the test cases include artifact instantiation trough VIL, we disable the instantiation phase.

However, pilot experiments showed that still significant differences between the first runs of a test case and subsequent runs may occur. Thus, within each suite, we repeat the reasoning functionality of each test case 10 times on a fresh configuration. These 10 (here intuitively chosen) repetitions make up the intra-experiment repetitions. In particular, the repetitions allow for (later) excluding  warmup runs as well as for basic descriptive statistics such as confidence intervals \cite{GeorgesBuytaertEeckhout07}. 

On a given machine/device, we first perform an initial run and then the experimental runs. During the initial run, we execute the test suites on the target device to validate that all tests are passed successfully. The measurements of this run are stored separately. For taking the experimental measures, we repeate the execution of the test suites 5 times (inter-experiment repetition) with 5 seconds pause between two subsequent runs.

Executing the test suites is realized as an ANT script, because collecting all dependencies for the standalone version of EASy-Producer is not trivial. For this task, we reuse the respective part of the ANT build mechanism from the continuous integration. ANT supports executing jUnit test suites based on a classpath constructed from the dependencies, so executing the test suites described in Section \ref{sectEvalSetupTreatments} is rather straightforward. The inclusion of the ramp-up tests, the intra-experiment and inter-experiment repetitions as well as the waiting time are configured in the ANT script and passed to the EASy-Producer test suites via environment parameters. For separating the initial and the experimental runs, the ANT script defines two specific tasks. In turn, the ANT script itself can be used as build action in the continuous integration, i.e., for collecting performance readings on the continuous integration server, e.g., for detecting performance degradation.

We execute the experiment on the following devices:
\begin{enumerate}
  \renewcommand{\theenumi}{D\arabic{enumi}}
  \item\label{oldLaptopId} Retired office computer used for developing reasoner v1.3.0: Dell Latitude 6430u laptop with Intel core i7-3367U (2 physical cores at 2.00GHz), Windows 7 version 6.1.761 SP 1 and Oracle JDK 1.8.0\_66 64 bit. Besides Maven 3.2.3, ANT 1.10.3 with Maven-ANT-tasks 2.1.3, an Eclipse Mars2 release 4.5.2 are installed.
  \item\label{newLaptopId} Actual office laptop: Dell Latitude 7490 laptop with Intel core i7-8650 U processor (4 physical cores at 1.9 GHz), 32 GBytes RAM with Windows 10 Professional (10.0.17134) and Open JDK 1.8.0.201 64 bit as well 10.0.2 64 bit. The installation here relies on OpenJDK instead of Oracle JDK due to licensing issues. Besides Maven 3.2.3, ANT 1.10.3 with Maven-ANT-tasks 2.1.3, an Eclipse Oxygen 3a release 4.7.3a are installed. 
  \item\label{jenkinsId} Our continuous integration server, a Ubuntu Linux 16.4.5 LTS VM with 4 GBytes RAM and OpenJDK 8 64bit. The measurement script is integrated as a manual build action that prevents any other continuous build action running at the same time.
  \item\label{piId} A Raspberry Pi 3 by vendor element14 hosting an 8 GB SanDisk class-4 SD (a device used in \cite{KnocheEichelberger18}) with Raspbian Stretch Lite version November 2018, Linux Kernel 4.14 and Oracle JDK 1.8.\TBD{65} ARM. % and, as alternative for some experiments, Oracle JDK 1.8.0\_201 ARM.
\end{enumerate}
For both windows machines, we terminate all programs that are not required for the execution of the tests (leaving the virus scanner in operation as usual during development). 

\subsubsection{Analysis}\label{sectEvalSetupAnalysis}

After collecting the runtime measurements, we execute an R script which combines the results of all test executions for one complete run, calculates statistical measures and draws summarizing diagrams. To analyze the initial runs, the script reads the individual files created for the test suites from Section \ref{sectEvalSetupTreatments}, removes the first three runs per test case intra-experiment repetition, calculates average, mean, variance and confidence interval per test case (cf. \cite{BulejHorkTuma17} for validity) and join all results into one large table. Although the inter-experiment repetitions can be seen as individual experiments, they are executed in direct sequence on the same machine. Thus, we also follow \cite{BulejHorkTuma17} here and create in this case a statistical summary table over all runs resulting in a similar large table.

Finally, zhe script produces various plots, all relating runtime (reasoning time, translation time or evaluation time) and model complexity (cf. Section \ref{sectModelComplexity}). These plots are intended to provide an overview on the reasoning time for different modes, i.e., to discuss how the reasoning time is composed and some indicating the deviations across all series of test suite execution.

\subsection{Model Ranking and Complexity}\label{sectModelComplexity}

When evaluating the reasoning capabilities of variability models, such as feature models, typically a measure is employed to characterize the model. Measures include the number of features \cite{Benavides2006AFS}, the number of constraints \cite{Benavides2006AFS} or combinations, e.g., constraints per feature or the constraint ratio \cite{Mendoca08}, graph width \cite{PohlStrickerPohl13} or more complex approaches \cite{StuikysDamasevicius09}. 

However, just relating the number of variables to the number of constraints in an IVML model can be misleading. As long as only top-level variables and Boolean constraints as in some generated models are used, a measure based on the number of variables and constraints may lead to a usual complexity classification. However, at a glance, models with compounds, containers and quantor constraints then are underestimated as nested variables are not considered and iterating container expressions are just counted like a basic Boolean constraint (regardless of its number of terms).

For this evaluation, we combine the basic idea of counting variables vs. constraints with the approach used in the McCabe complexity metric, i.e., we also count the nested elements and weight them according to their perceived complexity. For the weights, we 'calibrate' the calculation based on our test models. However, it is important to note that we do not claim that the weights are universal and hold for all kinds of IVML models or even the approach is universal for all kinds of variability models. The only aim is to obtain an objective (hopefully fair) criterion to rank IVML models in order to support the visualization and the comparison of the reasoning time for several (different) models.

The complexity metric applied here consists of four parts, the
\begin{enumerate}
\item measure of the structure of a given model element $e$, denoted $cpx_v(e)$.
\item measure of the constraints of a certain model element $e$, denoted by $cpx_c(e)$, which, in turn, is based on the 
\item measure of a given (constraint, default value) expression $e$, denoted as $cpx_e(e)$ .
\item weighting $w_{cpx}(e)$ for a model element, constraint or expression $e$.
\end{enumerate}
%
For a given IVML configuration $cfg$, we calculate then
%
$$cpx(cfg) = cpx_v(cfg) + cpx_c(cfg)$$ 
%
as a combination of structure and constraints.

Due to the nested structure of IVML models, most of the formulas are recursive. Within these formulas, the weighting function $w_{cpx}(e)$ used as in additive fashion if the argument is an usual (user-defined) type. If $w_{cpx}(e)$ refers to a meta-type such as \IVML{Expr} denoting all expression elements, then $w_{cpx}(e)$ is used in multiplicative manner to allow for disabling the related structure or constraint calculation. 

In the sections below, explain now the formulas for the four parts. In this course, we introduce some special functions for IVML model elements, in particular to query the meta types of IVML elements. These functions have not been used so far and that will only be used within this section.

\subsubsection{Structure Measure}

The measure of the structure $cpx_v(e)$ calculates a weighted sum of the (nested) variables in a given configuration. We rely here on configuration variables, as a configuration contains all available variables including nested ones as defined by assignment constraints for compounds or containers. 
%When applying $cpx_v(e)$ to a configuration, the sum of the measures for all variables is calculated. 

The measure of an individual variable consists of two additive components: 
\begin{enumerate}
\item  The weight for the respective type of the variable to allow for weighting complex types differently than basic types. 
\item The (recursive) sum of $cpx_v(e)$ over all variables nested nested in $e$. This sum is weighted by the IVML meta type \IVML{Var} to be able to disable considering nested structures.
\end{enumerate}
%
$$
   cpx_v(e) = \begin{cases} 
       \sum_{v\in vars(e)}cpx_v(e) & \text{if } isConfiguration(e)\\
       w_{cpx}(type(e)) +  & \\
       \ w_{cpx}(\IVML{Var}) \cdot \sum_{n\in vars(e)}cpx_v(n) & \text{if } isVariable(e)\\
       0 & \text{else}
       \end{cases}
$$
\subsubsection{Constraint Measure}
%
The measure of the constraints $cpx_c(e)$ considers all types of model elements that may contain constraints and calculates the recursive sum of the measures for these constraints. In more details, for a configuration we consider the contained configuration variables as well as all top-level elements in the underlying project. For a
%
\begin{itemize}
  \item configuration variable we build the recursive sum over all nested variables and add the measures of the default value expressions. Here we apply the weighting $w_{cpx}(\IVML{Expr})$ for the IVML meta type \IVML{Expr} denoting all expressions to allow disabling the measure for default value expressions.
  \item project we calculate the sum of $cpx_c(e)$ for all model elements, in particular top-level eval blocks, assignment blocks, and constraints as well as operation definitions. For now we do not take the import structure of projects into account.
  \item eval-block we sum up the measures for all nested constraints and (potentially nested) recursive eval blocks. 
  \item annotation assignment we sum the measures for all nested constraints and (potentially nested) recursive annotation assignments. The declared variables are already considered by $cpx_v(e)$.
  \item user-defined operation (identified via $isOpDef(e)$), we take the expression measure for the function defining the operation into account. The expression measure $cpx_e(e)$ will be introduced in Section \ref{sectExpMeasureExpr}. 
  \item constraint, we combine an additive weight (to enable counting constraints) with the weighted measure for the meta type \IVML{expr} and the expression measure of the constraint (to be able to exclude the measure for the constraint expression $expr(e)$). The expression measure $cpx_e(e)$ will be introduced in Section \ref{sectExpMeasureExpr}. 
\end{itemize}
%
$$
   cpx_c(e) = \begin{cases} 
       \sum_{v\in vars(e)}cpx_c(v) + cpx_c(project(e)) & \text{if } isConfiguration(e)\\
       \sum_{v\in vars(v)}cpx_c(e) \\
       \text{ } + w_{cpx}(\IVML{Expr}) \cdot cpx_e(default(e)) & \text{if } isVariable(e)\\
       \sum_{f\in elements(e)}cpx_c(f)  & \text{if } isProject(e)\\
       \sum_{x\in constraints(e)~\cup~evals(e)} cpx_c(x) & \text{if } isEval(e)\\ 
       \sum_{x\in constraints(e)~\cup~assignments(e)} cpx_c(x) & \text{if } isAssignment(e)\\
       cpx_e(function(e)) & \text{if } isOpDef(e)\\
       w_{cpx}(e) + w_{cpx}(\IVML{Expr}) \cdot cpx_e(expr(e)) & \text{if } isConstraint(e)\\
       0 & \text{else}
       \end{cases}
$$
\subsubsection{Expression Measure}\label{sectExpMeasureExpr}
%
The measure for an expression is calculated based on the expression tree, i.e., we add the individual weight for each tree node $w_{cpx}(e)$\footnote{Here, expression (meta) types are used in an additive manner.} to the (recursive) sum over the respective expression sub-trees (argument expressions, contained expressions, value expressions). A tree node can have various types that we enumerate as cases here. Most of the operations and functions that can be used in an IVML expression are internally represented as a call node. For a call node, we summarize the measures of all arguments, e.g., for the plus operation the measures of the left and right operand expression. Parentheses, container iterators, let-expressions and accessors consist of a single expression that makes up the respective measure. For an if-then-else expression, we sum up the expressions constituting the condition, the then-part as well as the else-part. For (compound or container) initializers as well as for expression blocks, we calculate the sum of the measures of the contained expressions.
%
$$
    cpx_{e}(e) = w_{cpx}(e) + \begin{cases}
       \sum_{a\in args(e)}c_{cpx}(a) & \text{if } isCall(e)\\
       c_{cpx}(expr(e)) & \text{if } isParenthesis(e)\\
       c_{cpx}(expr(e)) & \text{if } isLet(e)\\
       c_{cpx}(expr(e)) & \text{if } isIter(e)\\
       c_{cpx}(expr(e)) & \text{if } isAccessor(e)\\
       c_{cpx}(expr(e)) + c_{cpx}(then(e)) + & \\
       \text{ }c_{cpx}(else(e)) & \text{if } isIfThen(e)\\
       \sum_{e\in expressions(e)}c_{cpx}(e) & \text{if } isInitializer(e)\\ % compound + container
       \sum_{e\in expressions(e)}c_{cpx}(e) & \text{if } isBlock(e)\\ 
       0 & \text{else}
    \end{cases}
$$
%
\subsubsection{Weighting Function}
%
The weighting function decides about the relative weight of individual elements and whether parts of the structure shall be considered at all. 

According to our experience, traditional measures such as the number of variables, the number of constraints, their sum or ratio leads to a wrong impression of the expected 'complexity' of complex IVML models. This is due to the fact that neither nested variables, nor types or constraint forms are considered appropriately, also as they mostly do not exist in such forms in traditional variability modeling. 

Based on our impression, compound and container types as well as quantor or iterator expressions lead to a higher perceived complexity, while accessing the actual value of a variable and of a constraint (often used in default value expressions) does not really increase complexity. For this experiment, we defined the weighting function as follows (not claiming to present a universal complexity measure for IVML models here), that we calibrated by sample runs over the subjects. 
%
$$
    w_{cpx}(e) = \begin{cases}
       1 & \text{if } isType(e)\\
       2 & \text{if } isCompound(e)~\vee~isContainer(e)\\
       1 & \text{if } \IVML{Var}~\vee~\IVML{Expr}\\
       1 & \text{if } isCall(e)~\vee~isLet(e)\\
       1 & \text{if } isParenthesis(e)~\vee~isAccessor(e)\\
       1 & \text{if } isIfThen(e) \\
       1 & \text{if } isInitializer(e)~\vee~isIsBlock(e)\\
       1 & \text{if } e = \IVMLself{}\\
       5 & \text{if } isIter(e)\\
       0.25 & \text{if } isVariableUse(e)~\vee~isConstant(e)\\
       0 & \text{else}
    \end{cases}
$$

For short, compounds and containers are taken into account by a double weight compared with all other types. As argued above, we consider nested variables as well as the contents of expressions. Most of the constraint tree nodes are weighted by 1 including access to the actual instance of a compound (\IVMLself{}), except for container iterator operations that we weight by a higher and access to the value of decision variables and constants that we weight by a lower value.

%as overall measure, because ratio-based measures as in the more homogeneous feature case mentioned above do not seem to correctly reflect the expected 'complexity' of IVML models.
%
The measurement approach defined in this section also allows calculating some traditional measures, e.g., the number of top-level variables or the number of constraints. Just for demonstration purposes, this can be achieved using a different weighting function. Here, we disable measuring nested variables and expressions, and enable counting constraints and remaining variables by setting the type weight for $cpx_v(e)$ and the constraint weight for $cpx_c(e)$ both to 1. All remaining weights, in particular for constraint expressions are set to 0.
%
$$
    w_{cpx}(e) = \begin{cases}
       1 & \text{if } isType(e)\\ %count variables equally
       0 & \text{if } \IVML{Var}~\vee~\IVML{Expr}\\
       1 & \text{if } isConstraint(e)\\ % in particular 0 for Expr
       0 & \text{else} % scope out \IVML{Variable}
    \end{cases}
$$
For a given configuration $cfg$, the number of (top-level) variables can now be determined by $cpx_v(cfg)$, the number of constraints by $cpx_c(cfg)$, the sum of both by $cpx(cfg)$ and the constraint ratio by $cpx_c(cfg)/cpx_v(cfg)$.
%
\subsection{Results}\label{sectEvaluationResults}

In this section, we present the collected and aggregated results of running the IVML reasoner in two versions on different devices. We start with a descriptive summary of the results (Section \ref{sectEvaluationResultsDescriptive}) and continue with the results for the individual setups.

\subsubsection{Descriptive Summary}\label{sectEvaluationResultsDescriptive}

\newcommand\centerCell[1]{\multicolumn{1}{|c|}{#1}}
\begin{table*}[ht]
%\begin{adjustbox}{angle=90}
\begin{adjustbox}{center}
\begin{tabular}{|c|c|c|c||r|r|r|r|r|r|r||r|}
\hline
\textbf{id} & \textbf{de-}      & \textbf{ver-} &                       & \textbf{test}                    & \multicolumn{2}{|c|}{\textbf{reasoning}} & \multicolumn{2}{|c|}{\textbf{constraints}} & \multicolumn{2}{|c||}{\textbf{evaluations}} & \textbf{total} \\
                  & \textbf{vice}     & \textbf{sion}  & \textbf{JDK} & \centerCell{\textbf{[\#]}} & \multicolumn{2}{|c|}{\textbf{time [ms]}} & \multicolumn{2}{|c|}{\textbf{[\#]}} &
\multicolumn{2}{|c||}{\textbf{[\#]}} & \textbf{time}\\
                  &                          &                       &                       &                                            & \textbf{avg} & \textbf{max}                     & \textbf{avg} & \textbf{max} & 
\textbf{avg} & \textbf{max} & \textbf{[min]}\\
\hline
1 & \ref{oldLaptopId}   & 1.1.0 & 1.8.0   & 400 & 17 & 670       & 723   & 24110   & 1092 & 46469 & 29\\
2 & \ref{oldLaptopId}   & 1.3.0 & 1.8.0   & 433 & 11 & 390       & 53     & 6000     & 603   & 23696 & 46\\
3 & \ref{newLaptopId} & 1.3.0 & 1.8.0* & 433 & 14 & 502       & 53     & 6000     & 603   & 23696 & 55\\
4 & \ref{newLaptopId} & 1.3.0 & 10*      & 433 & 15  & 493      & 53     & 6000     & 603   & 23696 & 55\\
5 & \ref{jenkinsId}        & 1.3.0 & 1.8.0   & 427 & 13  & 480      & 53     & 6000     & 494  & 19466 & 104\\
6 & \ref{piId}                & 1.1.0 & 1.8.0   & 400 & 548 & 28613 & 723   & 24110   & 1092 & 46477 & 366\\
7 & \ref{piId}                & 1.3.0 & 1.8.0   & 433 & 255 & 6117   & 53     & 6000     & 815   & 23708 &\\
\hline
\end{tabular}
\end{adjustbox}
\caption{Descriptive summary of experiments (* = OpenJDK).}
\label{tab:experimentsDescSummary}
%\end{adjustbox}
\end{table*}

Table \ref{tab:experimentsDescSummary} summarizes for the initial run the number of tests, the measured reasoning time, the created number of constraints, the required constraint evaluations. For providing an indication of the overall time consumption of the experiment, we indicate the total execution time for the 5 repeated executions of the testsuites. We list number of tests, constraints and evaluations although they shall be the same for the same treatments, because in the pre-experiments the numbers indicated that due to minor bugs the experiments were done over different models. We do not denote the minimum reasoning time, number of constraints or evaluations, respectively, the minimum number in all cases is 0 due to IVML tests without constraints. 

On the original device \ref{oldLaptopId}, we execute both versions of the reasoner on the same JDK (id 1 and 2). On the newer \ref{newLaptopId} we continue with the more recent reasoner version in a more modern (id 3, 4). In this setup, we use two different JDKs, Java 8 to enable a comparison with \ref{oldLaptopId}, and Java 10 to give an outlook on more recent JVMs. On the continuous integration machine (\ref{jenkinsId}, id 5) we just measure reasoner version 1.3.0 (corresponding to the most recent commit at that time). Finally, on the Pi device \ref{piId} we measure both reasoner versions on the same JDK (id 6 and 7).

As mentioned in Section \ref{sectEvalSetupTreatments}, the treatments differ between the reasoner versions (id 1/2 as well as 6/7) as not all tests could be ported back successfully. Moreover, the number of tests on Jenkins (\ref{jenkinsId}, id 5) differs, as some tests involving VIL are disabled there due to technical issues. Except for the Pi \ref{piId}, the average reasoning time is rather similar across the devices and JDKs. However, according to the maximum reasoning time, reasoner version 1.3.0 performs faster than version 1.1.0. It is important mentioning that reasoner version 1.3.0 is both, more IVML complete and better tuned for performance as discussed in Section \ref{sectPerformance}. In contrast, reasoner version 1.1.0 even creates significantly more (including accidentally ineffective) constraints and, thus, requires more re-evaluations, i.e., consumes more reasoning time. This can also be identified in terms of constraints initially stored by the reasoner in the constraint base as well as the number of re-evaluations. Jenkins (\ref{jenkinsId}, id 5) appears to be a bit faster at lower re-evaluations as some reasoning tests are disabled.

Regarding the total execution time, id 1 may appear as an outlier or a measurement error. However, we observed this behavior several times. As besides pure reasoning experiments also IVML and VIL tests are executed (\TBD{pi}), we attribute this difference to the differences in the number/complexity of treatments, possibly also to changes in the IVML/VIL code base of EASy-producer. However, comparing id 2 and id 3 it seems that an OpenJDK on Windows 10 behaves sigificantly worse than Oracle JDK on a rather old Windows 7 installation. This impression was confirmed by the average reasoning time when repeating the experiments (maximum reasoning time 336 ms for id 1 and 485 ms for id 3). OpenJDK 10 seems to be slightly faster (with an even smaller difference in the repeating run). Although we executed the experiments on Jenkins (id 5) in a separate build task prevening parallel builds, the overall execution time is almost twice as high as on a laptop. Potential reasons may be due to the setup (Jenkins running in a web container with Apache as frontend, regular checks for commits for 85 build tasks, an almost full virtual hard drive, virtualization overhead in particular for I/O operation when loading IVML and VIL models, etc.).

Based on \cite{KnocheEichelberger18}, we expected an average performance drop of factor 10 for the Pi experiments (id 6 and 7). \TBD{discuss}

In summary, reasoner version 1.3.0 is more IVML complete and faster than reasoner version 1.1.0. However, we cannot confirm the huge differences reported in Section \ref{sectEvalSetupTreatments}. To identify the reason, we also executed the experiments in Eclipse (cf. Section \ref{sectEvalSetupProcedure}). When executing the ANT measurement script in terms of an external tool configuration, we did not notice significant timing differences. However, during development, one typically executes a jUnit test individually using the Eclipse jUnit test execution feature. Running the tests suites as JUnit tests in Eclipse doubled the average reasoning time for version 1.1.0 on \ref{oldLaptopId}. The reasoning time for individual models raised on \ref{oldLaptopId} even by more than factor 5. Moreover, working in a  mobile setting, i.e., executing the tests on an unplugged laptop, may have further increased the reasoning time leading to a wrong impression of the overall performance gain.

\subsubsection{Individual Setups}

Four groups: reasoning core tests, scenario tests, incremental reasoning, runtime/instance reasoning

Figure \ref{fig:old-1_10-jdk8}:
\begin{itemize}
  \item Fluctuations due To Windows, UI?
  \item Translation/evaluation time low, less constraints
  \item Sum of translation/evaluation time does not lead to full reasoning time. Additional activities, see Section \ref{sectPerformance} in particular result handling.
\end{itemize}

Figure \ref{fig:old-1_30-jdk8}:
\begin{itemize}
    \item requires less time
    \item produces less constraints, less re-evaluations (depends on model)
    \item incremental slightly faster
    \item instance: memory allocation in reasoning time, no translation/evaluation time
    \item all reasoning/evaluation 1:1, sum up to reasoning time
\end{itemize}

Figure \ref{fig:new-1_30-ojdk8}:
\begin{itemize}
    \item new computer, windows, OpenJDK 8 slower
    \item less big spikes (different installation)
\end{itemize}

Figure \ref{fig:new-1_30-ojdk10}:
\begin{itemize}
    \item even slower, but even less spikes
\end{itemize}

Figure \ref{fig:jenkins-1_30-ojdk10}:
\begin{itemize}
    \item \textbf{data missing?}
\end{itemize}

Figure \ref{fig:pi-1_10-jdk8}:
\begin{itemize}
    \item factor 60 slower
    \item evaluation speed
    \item no time difference for larger models
    \item as version 1.1.0, 200+200 vs 30.000? output problems
    \item \textbf{data missing?}
\end{itemize}

Figure \ref{fig:pi-1_30-jdk8}:
\begin{itemize}
    \item less spiky than windows and jenkins
    \item no time difference for larger models
    \item factor 13 slower - ICPE paper)
    \item times sum up
    \item \textbf{data missing?}
\end{itemize}

\newcommand\evalPDFscale[0]{0.67}
\begin{figure}[!htb]
  \centering
  \includegraphics[scale=\evalPDFscale]{figures/benchmark-results-old-1_1_0-jdk1_8-20190308_reasoningTime_all.pdf}
  \hfill
  \includegraphics[scale=\evalPDFscale]{figures/benchmark-results-old-1_1_0-jdk1_8-20190308_translationTime_all.pdf}
  \hfill
  \includegraphics[scale=\evalPDFscale]{figures/benchmark-results-old-1_1_0-jdk1_8-20190308_evaluationTime_all.pdf}
  \caption{Reasoner 1.10 on the Dell Latitude 6430u using Oracle JDK 1.8.}\label{fig:old-1_10-jdk8}
\end{figure}

\clearpage

\begin{figure}[!htb]
  \centering
  \includegraphics[scale=\evalPDFscale]{figures/benchmark-results-old-1_3_0-jdk1_8-20190308_reasoningTime_all.pdf}
  \hfill
  \includegraphics[scale=\evalPDFscale]{figures/benchmark-results-old-1_3_0-jdk1_8-20190308_translationTime_all.pdf}
  \hfill
  \includegraphics[scale=\evalPDFscale]{figures/benchmark-results-old-1_3_0-jdk1_8-20190308_evaluationTime_all.pdf}
  \caption{Reasoner 1.30 on the Dell Latitude 6430u using Oracle JDK 1.8.}\label{fig:old-1_30-jdk8}
\end{figure}

\clearpage

\begin{figure}[!htb]
  \centering
  \includegraphics[scale=\evalPDFscale]{figures/benchmark-results-new-1_3_0-ojdk1_8-20190308_reasoningTime_all.pdf}
  \hfill
  \includegraphics[scale=\evalPDFscale]{figures/benchmark-results-new-1_3_0-ojdk1_8-20190308_translationTime_all.pdf}
  \hfill
  \includegraphics[scale=\evalPDFscale]{figures/benchmark-results-new-1_3_0-ojdk1_8-20190308_evaluationTime_all.pdf}
  \caption{Reasoner 1.30 on the Dell Latitude 7490 using Open JDK 1.8.}\label{fig:new-1_30-ojdk8}
\end{figure}

\clearpage

\begin{figure}[!htb]
  \centering
  \includegraphics[scale=\evalPDFscale]{figures/benchmark-results-new-1_3_0-ojdk10-20190308_reasoningTime_all.pdf}
  \hfill
  \includegraphics[scale=\evalPDFscale]{figures/benchmark-results-new-1_3_0-ojdk10-20190308_translationTime_all.pdf}
  \hfill
  \includegraphics[scale=\evalPDFscale]{figures/benchmark-results-new-1_3_0-ojdk10-20190308_evaluationTime_all.pdf}
  \caption{Reasoner 1.30 on the Dell Latitude 7490 using Open JDK 10.}\label{fig:new-1_30-ojdk10}
\end{figure}

\clearpage

\begin{figure}[!htb]
  \centering
  \includegraphics[scale=\evalPDFscale]{figures/benchmark-results-jenkins-1_3_0-jdk1_8-20190308_reasoningTime_all.pdf}
  \hfill
  \includegraphics[scale=\evalPDFscale]{figures/benchmark-results-jenkins-1_3_0-jdk1_8-20190308_translationTime_all.pdf}
  \hfill
  \includegraphics[scale=\evalPDFscale]{figures/benchmark-results-jenkins-1_3_0-jdk1_8-20190308_evaluationTime_all.pdf}
  \caption{Reasoner 1.30 on Jenkins Ubuntu Open JDK 1.8.}\label{fig:jenkins-1_30-ojdk10}
\end{figure}

\clearpage

\begin{figure}[!htb]
  \centering
  \includegraphics[scale=\evalPDFscale]{figures/benchmark-results-pi-1_1_0-jdk1_8-20190308_reasoningTime_all.pdf}
  \hfill
  \includegraphics[scale=\evalPDFscale]{figures/benchmark-results-pi-1_1_0-jdk1_8-20190308_translationTime_all.pdf}
  \hfill
  \includegraphics[scale=\evalPDFscale]{figures/benchmark-results-pi-1_1_0-jdk1_8-20190308_evaluationTime_all.pdf}
  \caption{Reasoner 1.10 on a Pi 3 using Oracle JDK 1.8.}\label{fig:pi-1_10-jdk8}
\end{figure}

\clearpage

\begin{figure}[!htb]
  \centering
  \includegraphics[scale=\evalPDFscale]{figures/benchmark-results-pi-1_3_0-jdk1_8-20190219_reasoningTime_all.pdf}
  \hfill
  \includegraphics[scale=\evalPDFscale]{figures/benchmark-results-pi-1_3_0-jdk1_8-20190219_translationTime_all.pdf}
  \hfill
  \includegraphics[scale=\evalPDFscale]{figures/benchmark-results-pi-1_3_0-jdk1_8-20190219_evaluationTime_all.pdf}
  \caption{Reasoner 1.30 on a Pi 3 using Oracle JDK 1.8. \TBD{still old}}\label{fig:pi-1_30-jdk8}
\end{figure}

\clearpage


\TBD{Potential topics:

\begin{itemize}
    \item Artificial models of some size/variable/constraint ratio. Christian had a generator for that and Roman typically did some evaluations using these models. Compare to known results where available.
    \item QualiMaster full, incremental, runtime reasoning (full, no defaults, no frozen, reuse constraint base). Runtime vs. full reasoning seems to behave rather linearly, at around 23 constraints evaluated per ms. Compare to known results where available.
    \item All test cases involving reasoning as an overview map.
    \item Eclipse 4.7/Java9 runs tests with asserts!!!
\end{itemize}

}
