Index: /reasoner/evaluation.tex =================================================================== --- /reasoner/evaluation.tex (revision 267) +++ /reasoner/evaluation.tex (revision 268) @@ -16,5 +16,5 @@ \begin{itemize} \item The original reasoner implementation that acted as basis for the revision. This base version\footnote{\label{reasonerBaseVersion}Git hash 6a00aa9c5aaa37ddb3d490d36c7e9a037e792656} is part of EASy-Producer release 1.1.0, i.e., we will call this original implementation \emph{reasoner v1.1.0}. - \item The revised reasoner implementation using the algorithms described in this document. This version\footnote{\label{reasonerActualVersion}Git hash d2967af5385799f24c8fbcfacac0c396835f44ec. This version rather similar to EASy-Producer release 1.2.0, but includes some further improvements.} is part of EASy-Producer release 1.3.0, i.e., we will call it \emph{reasoner v1.3.0}. + \item The revised reasoner implementation using the algorithms described in this document. This version\footnote{\label{reasonerActualVersion}Git hash e0b09aec719d5387255e2b979d90708f05364e75. This version rather similar to EASy-Producer release 1.2.0, but includes some further improvements.} is part of EASy-Producer release 1.3.0, i.e., we will call it \emph{reasoner v1.3.0}. \end{itemize} @@ -31,5 +31,5 @@ We use these test cases as experimental treatments, although they involve test dependencies such as jUnit. While some of the test cases rely on programmed models (in terms of the IVML object model), most of the test cases specify the model in terms of IVML, i.e., for a more realistic setup and require for execution the IVML parser as well as dependent Eclipse libraries. As we focus on the reasoning time, the actual creation of the reasoning model shall not affect the results. -It is important to note that the code for reasoner v1.1.0 does not include several test cases that have been created for v1.3.0. For this experiment, we enable for v1.1.0 as many test cases as possible, i.e., we manually patched back\footnote{\label{fnPatch}Patch is available from \MISSING{XXX}.} the v1.3.0 test cases into v1.1.0 and, if required, either adjusted the expected test result accordingly or, in extreme cases, disable test cases that cannot be handled by the v1.1.0 reasoner (or the underlying IVML implementation). For this reason, the treatment sets differ in terms of specific tests, while most of the imporant small and large models are the same. We believe that this is acceptable for an illustrative experiment. +It is important to note that the code for reasoner v1.1.0 does not include several test cases that have been created for v1.3.0. For this experiment, we enable for v1.1.0 as many test cases as possible, i.e., we manually ported back\footnote{\label{fnPatch}Patch is available from \MISSING{XXX}.} the v1.3.0 test cases into v1.1.0 and, if required, either adjusted the expected test result accordingly or, in extreme cases, disable test cases that cannot be handled by the v1.1.0 reasoner (or the underlying IVML implementation). For this reason, the treatment sets differ in terms of specific tests, while most of the imporant small and large models are the same. We believe that this is acceptable for an illustrative experiment. \subsubsection{Data Collection}\label{sectEvalSetupDataCollection} @@ -43,5 +43,5 @@ \end{itemize} -However, EASy-Producer release v1.1.0 does not contain the generic data collector and several measures, so, along with the test cases, we patched the related code from v1.3.0 back\footref{fnPatch} into v1.1.0. +However, EASy-Producer release v1.1.0 does not contain the generic data collector and several measures, so, along with the test cases, we ported the related code from v1.3.0 back\footref{fnPatch} into v1.1.0. \subsubsection{Experimental Procedure}\label{sectEvalSetupProcedure} @@ -59,7 +59,7 @@ \renewcommand{\theenumi}{D\arabic{enumi}} \item\label{oldLaptopId} Retired office computer used for developing reasoner v1.3.0: Dell Latitude 6430u laptop with Intel core i7-3367U (2 physical cores at 2.00GHz), Windows 7 version 6.1.761 SP 1 and Oracle JDK 1.8.0\_66 64 bit. Besides Maven 3.2.3, ANT 1.10.3 with Maven-ANT-tasks 2.1.3, an Eclipse Mars2 release 4.5.2 are installed. - \item\label{newLaptopId} Actual office laptop: Dell Latitude 7490 laptop with Intel core i7-8650 U processor (4 physical cores at 1.9 GHz), 32 GBytes RAM with Windows 10 Professional (10.0.17134) and Open JDK 10.0.2 64 bit. Besides Maven 3.2.3, ANT 1.10.3 with Maven-ANT-tasks 2.1.3, an Eclipse Oxygen 3a release 4.7.3a are installed. + \item\label{newLaptopId} Actual office laptop: Dell Latitude 7490 laptop with Intel core i7-8650 U processor (4 physical cores at 1.9 GHz), 32 GBytes RAM with Windows 10 Professional (10.0.17134) and Open JDK 1.8.0.201 64 bit as well 10.0.2 64 bit. The installation here relies on OpenJDK instead of Oracle JDK due to licensing issues. Besides Maven 3.2.3, ANT 1.10.3 with Maven-ANT-tasks 2.1.3, an Eclipse Oxygen 3a release 4.7.3a are installed. \item\label{jenkinsId} Our continuous integration server, a Ubuntu Linux 16.4.5 LTS VM with 4 GBytes RAM and OpenJDK 8 64bit. The measurement script is integrated as a manual build action that prevents any other continuous build action running at the same time. - \item\label{piId} A Raspberry Pi 3 by vendor element14 hosting an 8 GB SanDisk class-4 SD (a device used in \cite{KnocheEichelberger18}) with Raspbian Stretch Lite version November 2018, Linux Kernel 4.14 and Oracle JDK 1.8.\TBD{65} ARM and, as alternative for some experiments, Oracle JDK 1.8.0\_201 ARM. + \item\label{piId} A Raspberry Pi 3 by vendor element14 hosting an 8 GB SanDisk class-4 SD (a device used in \cite{KnocheEichelberger18}) with Raspbian Stretch Lite version November 2018, Linux Kernel 4.14 and Oracle JDK 1.8.\TBD{65} ARM. % and, as alternative for some experiments, Oracle JDK 1.8.0\_201 ARM. \end{enumerate} For both windows machines, we terminate all programs that are not required for the execution of the tests (leaving the virus scanner in operation as usual during development). @@ -203,25 +203,30 @@ \subsection{Results}\label{sectEvaluationResults} +In this section, we present the collected and aggregated results of running the IVML reasoner in two versions on different devices. We start with a descriptive summary of the results (Section \ref{sectEvaluationResultsDescriptive}) and continue with the results for the individual setups. + +\subsubsection{Descriptive Summary}\label{sectEvaluationResultsDescriptive} + \newcommand\centerCell[1]{\multicolumn{1}{|c|}{#1}} \begin{table*}[ht] %\begin{adjustbox}{angle=90} -\centering -\begin{tabular}{|c|c|c|r|r|r|r|r|r|r|} +\begin{adjustbox}{center} +\begin{tabular}{|c|c|c|c||r|r|r|r|r|r|r||r|} \hline - & & & \textbf{tests} & \multicolumn{2}{|c|}{\textbf{reasoning}} & \multicolumn{2}{|c|}{\textbf{constraints}} & \multicolumn{2}{|c|}{\textbf{evaluations}} \\ -\textbf{device} & \textbf{ver} & \textbf{JDK} & \centerCell{\textbf{[\#]}} & \multicolumn{2}{|c|}{\textbf{time [ms]}} & \multicolumn{2}{|c|}{\textbf{[\#]}} & -\multicolumn{2}{|c|}{\textbf{[\#]}}\\ - & & & & \textbf{avg} & \textbf{max} & \textbf{avg} & \textbf{max} & -\textbf{avg} & \textbf{max}\\ +\textbf{id} & \textbf{de-} & \textbf{ver-} & & \textbf{test} & \multicolumn{2}{|c|}{\textbf{reasoning}} & \multicolumn{2}{|c|}{\textbf{constraints}} & \multicolumn{2}{|c||}{\textbf{evaluations}} & \textbf{total} \\ + & \textbf{vice} & \textbf{sion} & \textbf{JDK} & \centerCell{\textbf{[\#]}} & \multicolumn{2}{|c|}{\textbf{time [ms]}} & \multicolumn{2}{|c|}{\textbf{[\#]}} & +\multicolumn{2}{|c||}{\textbf{[\#]}} & \textbf{time}\\ + & & & & & \textbf{avg} & \textbf{max} & \textbf{avg} & \textbf{max} & +\textbf{avg} & \textbf{max} & \textbf{[min]}\\ \hline -\ref{oldLaptopId} & 1.1.0 & 1.8.0 & 400 & 16 & 655 & 723 & 24110 & 1092 & 46469 \\ -\ref{oldLaptopId} & 1.3.0 & 1.8.0 & 433 & 11 & 399 & 53 & 6000 & 603 & 23696\\ -\ref{newLaptopId} & 1.3.0 & 1.8.0* & 433 & 13 & 424 & 53 & 6000 & 603 & 23696\\ -\ref{newLaptopId} & 1.3.0 & 10* & 433 & 15 & 484 & 53 & 6000 & 603 & 23696\\ -\ref{jenkinsId} & 1.3.0 & 1.8.0 & 427 & 19 & 562 & 53 & 6000 & 709 & 23703\\ -\ref{piId} & 1.1.0 & 1.8.0 & 400 & 958 & 28651 & 978 & 24110 & 1679 & 46487\\ -\ref{piId} & 1.3.0 & 1.8.0 & 433 & 255 & 6117 & 53 & 6000 & 815 & 23708\\ +1 & \ref{oldLaptopId} & 1.1.0 & 1.8.0 & 400 & 17 & 670 & 723 & 24110 & 1092 & 46469 & 29\\ +2 & \ref{oldLaptopId} & 1.3.0 & 1.8.0 & 433 & 11 & 390 & 53 & 6000 & 603 & 23696 & 46\\ +3 & \ref{newLaptopId} & 1.3.0 & 1.8.0* & 433 & 14 & 502 & 53 & 6000 & 603 & 23696 & 55\\ +4 & \ref{newLaptopId} & 1.3.0 & 10* & 433 & 15 & 493 & 53 & 6000 & 603 & 23696 & 55\\ +5 & \ref{jenkinsId} & 1.3.0 & 1.8.0 & 427 & 13 & 480 & 53 & 6000 & 494 & 19466 & 104\\ +6 & \ref{piId} & 1.1.0 & 1.8.0 & 400 & 958 & 28651 & 978 & 24110 & 1679 & 46487 &\\ +7 & \ref{piId} & 1.3.0 & 1.8.0 & 433 & 255 & 6117 & 53 & 6000 & 815 & 23708 &\\ \hline \end{tabular} +\end{adjustbox} \caption{Descriptive summary of experiments (* = OpenJDK).} \label{tab:experimentsDescSummary} @@ -229,15 +234,19 @@ \end{table*} +Table \ref{tab:experimentsDescSummary} summarizes for the initial run the number of tests, the measured reasoning time, the created number of constraints, the required constraint evaluations. For providing an indication of the overall time consumption of the experiment, we indicate the total execution time for the 5 repeated executions of the testsuites. We do not denote the minimum reasoning time, number of constraints or evaluations, respectively, the minimum number in all cases is 0 due to IVML tests without constraints. + +On the original device \ref{oldLaptopId}, we execute both versions of the reasoner on the same JDK (id 1 and 2). On the newer \ref{newLaptopId} we continue with the more recent reasoner version in a more modern (id 3, 4). In this setup, we use two different JDKs, Java 8 to enable a comparison with \ref{oldLaptopId}, and Java 10 to give an outlook on more recent JVMs. On the continuous integration machine (\ref{jenkinsId}, id 5) we just measure reasoner version 1.3.0 (corresponding to the most recent commit at that time). Finally, on the Pi device \ref{piId} we measure both reasoner versions on the same JDK (id 6 and 7). + +As mentioned in Section \ref{sectEvalSetupTreatments}, the treatments differ between the reasoner versions (id 1/2 as well as 6/7) as not all tests could be ported back successfully. Moreover, the number of tests on Jenkins (\ref{jenkinsId}, id 5) differs, as some tests involving VIL are disabled there due to technical issues. Except for the Pi \ref{piId}, the average reasoning time is rather similar across the devices and JDKs. However, according to the maximum reasoning time, reasoner version 1.3.0 performs faster than version 1.1.0. It is important mentioning that reasoner version 1.3.0 is both, more IVML complete and better tuned for performance as discussed in Section \ref{sectPerformance}. In contrast, reasoner version 1.1.0 even creates significantly more (including accidentally ineffective) constraints and, thus, requires more re-evaluations, i.e., consumes more reasoning time. This can also be identified in terms of constraints initially stored by the reasoner in the constraint base as well as the number of re-evaluations. Jenkins (\ref{jenkinsId}, id 5) appears to be a bit faster at lower re-evaluations as some reasoning tests are disabled. + +Regarding the total execution time, id 1 may appear as an outlier or a measurement error. However, we observed this behavior several times. As besides pure reasoning experiments also IVML and VIL tests are executed (\TBD{pi}), we attribute this difference to the differences in the number/complexity of treatments, possibly also to changes in the IVML/VIL code base of EASy-producer. However, comparing id 2 and id 3 it seems that an OpenJDK on Windows 10 behaves sigificantly worse than Oracle JDK on a rather old Windows 7 installation. This impression was confirmed by the average reasoning time when repeating the experiments (maximum reasoning time 336 ms for id 1 and 485 ms for id 3). OpenJDK 10 seems to be slightly faster (with an even smaller difference in the repeating run). Although we executed the experiments on Jenkins (id 5) in a separate build task prevening parallel builds, the overall execution time is almost twice as high as on a laptop. Potential reasons may be due to the setup (Jenkins running in a web container with Apache as frontend, regular checks for commits for 85 build tasks, an almost full virtual hard drive, virtualization overhead in particular for I/O operation when loading IVML and VIL models, etc.). + +Based on \cite{KnocheEichelberger18}, we expected an average performance drop of factor 10 for the Pi experiments (id 6 and 7). \TBD{discuss} + +In summary, reasoner version 1.3.0 is more IVML complete and faster than reasoner version 1.1.0. However, we cannot confirm the huge differences reported in Section \ref{sectEvalSetupTreatments}. To identify the reason, we also executed the experiments in Eclipse (cf. Section \ref{sectEvalSetupProcedure}). When executing the ANT measurement script in terms of an external tool configuration, we did not notice significant timing differences. However, during development, one typically executes a jUnit test individually using the Eclipse jUnit test execution feature. Running the tests suites as JUnit tests in Eclipse doubled the average reasoning time for version 1.1.0 on \ref{oldLaptopId}. The reasoning time for individual models raised on \ref{oldLaptopId} even by more than factor 5. Moreover, working in a mobile setting, i.e., executing the tests on an unplugged laptop, may have further increased the reasoning time leading to a wrong impression of the overall performance gain. + +\subsubsection{Individual Setups} + Four groups: reasoning core tests, scenario tests, incremental reasoning, runtime/instance reasoning - -Table \ref{tab:experimentsDescSummary}: -\begin{itemize} - \item min always 0, average of averages, similar max but left out digits - \item some tests on Jenkins are disabled - \item reasoner 1.3.0 is faster and produces less constraints/re-evaluations although being more complete. - \item reasoner 1.1.0 produces ineffective and less complex constraints that can be evaluated faster - \item Win 10 is slower, also Open JDK 10. - \item Seems that models look structurally different on Linux than on Windows. Different model complexity and constraint evaluation measures. Strange, because VIL leads to same code for QualiMaster. Reason unclear. For creating the figures, replaced constraint metrics by windows values. -\end{itemize} Figure \ref{fig:old-1_10-jdk8}: Index: /reasoner/measures/log.txt =================================================================== --- /reasoner/measures/log.txt (revision 267) +++ /reasoner/measures/log.txt (revision 268) @@ -1,107 +1,68 @@ -[1] "processing folder W:/offlineFiles/EASy-concepts/reasoner/measures/benchmark-results-jenkins-1_3_0-20190215" +[1] "processing folder W:/offlineFiles/EASy-concepts/reasoner/measures/benchmark-results-jenkins-1_3_0-20190308" [1] "#tests;427" -[1] "reasoning time;0;18.8758503401361;562.142857142857" -[1] "translation time;0;8.05720976915356;280.285714285714" -[1] "evaluation time;0;6.41889706702353;278.857142857143" +[1] "reasoning time;0;13.3681833389093;480.428571428571" +[1] "translation time;0;5.60321177651388;210.714285714286" +[1] "evaluation time;0;4.99013047842088;212.857142857143" [1] "constraints count;0;53.3629976580796;6000" -[1] "reevaluation count;0;709.311140849783;23703.2857142857" -[1] "#tests;514" -[1] "reasoning time;0;46.4212293866963;559.142857142857" -[1] "translation time;0;19.5075597554197;280.285714285714" -[1] "evaluation time;0;15.6086946451732;277.285714285714" -[1] "constraints count;0;44.6050583657588;6000" -[1] "reevaluation count;0;1712.07329071706;23703.2857142857" -[1] "processing folder W:/offlineFiles/EASy-concepts/reasoner/measures/benchmark-results-new-1_3_0-ojdk1_8-20190215" +[1] "reevaluation count;0;494.383405821345;19466.5714285714" +[1] "#tests;447" +[1] "reasoning time;0;12.4123042505593;391.314285714286" +[1] "translation time;0;5.2651326302333;174.428571428571" +[1] "evaluation time;0;4.48766379034835;173.057142857143" +[1] "constraints count;0;51.3064876957494;6000" +[1] "reevaluation count;0;473.795877277085;19468.4" +[1] "processing folder W:/offlineFiles/EASy-concepts/reasoner/measures/benchmark-results-new-1.3-ojdk1.8-20190308" [1] "#tests;433" -[1] "reasoning time;0;13.4028373474101;424" -[1] "translation time;0;5.78983833718245;205.857142857143" -[1] "evaluation time;0;4.33058396568789;170.285714285714" +[1] "reasoning time;0;14.6421972946222;502.142857142857" +[1] "translation time;0;6.29874628835368;230.142857142857" +[1] "evaluation time;0;4.78356977895084;208.285714285714" [1] "constraints count;0;52.6235565819861;6000" -[1] "reevaluation count;0;603.311448366876;23696" -[1] "#tests;517" -[1] "reasoning time;0;28.8461545546652;443.771428571429" -[1] "translation time;0;12.4963894261767;200.542857142857" -[1] "evaluation time;0;8.96901538178134;176.2" -[1] "constraints count;0;44.2959381044487;6000" -[1] "reevaluation count;0;1236.59463940315;23696" -[1] "processing folder W:/offlineFiles/EASy-concepts/reasoner/measures/benchmark-results-new-1_3_0-ojdk10-20190215" +[1] "reevaluation count;0;603.30435499835;23696.5714285714" +[1] "#tests;450" +[1] "reasoning time;0;14.4322010582011;485.885714285714" +[1] "translation time;0;6.19645502645503;224.971428571429" +[1] "evaluation time;0;4.67887830687831;199.142857142857" +[1] "constraints count;0;50.9177777777778;6000" +[1] "reevaluation count;0;581.779746031746;23693.5714285714" +[1] "processing folder W:/offlineFiles/EASy-concepts/reasoner/measures/benchmark-results-new-1.3-ojdk10-20190308" [1] "#tests;433" -[1] "reasoning time;0;14.8669306059606;484.428571428571" -[1] "translation time;0;6.47476080501485;235.857142857143" -[1] "evaluation time;0;4.78389970306829;202.142857142857" +[1] "reasoning time;0;14.8512042230287;492.857142857143" +[1] "translation time;0;6.28686902012537;217.142857142857" +[1] "evaluation time;0;4.89211481359287;203.857142857143" [1] "constraints count;0;52.6235565819861;6000" -[1] "reevaluation count;0;603.316397228637;23696.5714285714" -[1] "#tests;538" -[1] "reasoning time;0;34.9081209063551;479.314285714286" -[1] "translation time;0;15.116887944769;225.228571428571" -[1] "evaluation time;0;10.7190254912374;205.4" -[1] "constraints count;0;42.6078066914498;6000" -[1] "reevaluation count;0;1360.5263409453;23697" -[1] "processing folder W:/offlineFiles/EASy-concepts/reasoner/measures/benchmark-results-new-eclipse-1_3_0-ojdk10-20190215" -[1] "#tests;433" -[1] "reasoning time;0;15.2083470801716;506.571428571429" -[1] "translation time;0;6.34147146156384;223.428571428571" -[1] "evaluation time;0;5.02062025734081;206.285714285714" -[1] "constraints count;0;52.6235565819861;6000" -[1] "reevaluation count;0;603.332233586275;23697.5714285714" -[1] "#tests;525" -[1] "reasoning time;0;32.8600453514739;485.628571428571" -[1] "translation time;0;14.1727074829932;210.228571428571" -[1] "evaluation time;0;10.1196371882086;208.628571428571" -[1] "constraints count;0;43.6914285714286;6000" -[1] "reevaluation count;0;1255.07419501134;23697.5714285714" -[1] "processing folder W:/offlineFiles/EASy-concepts/reasoner/measures/benchmark-results-old-1_1_0-jdk1_8-20190215" +[1] "reevaluation count;0;603.324645331574;23693.5714285714" +[1] "#tests;452" +[1] "reasoning time;0;14.2645701643489;481.371428571429" +[1] "translation time;0;6.1114412136536;222.028571428571" +[1] "evaluation time;0;4.64415297092288;199.542857142857" +[1] "constraints count;0;50.7256637168142;6000" +[1] "reevaluation count;0;579.44608091024;23693.5714285714" +[1] "processing folder W:/offlineFiles/EASy-concepts/reasoner/measures/benchmark-results-old-1.1.0-jdk1.8-20190308" [1] "#tests;400" -[1] "reasoning time;0;16.40125;655.142857142857" -[1] "translation time;0;1.25892857142857;182.857142857143" -[1] "evaluation time;0;10.9328571428571;548" +[1] "reasoning time;0;16.8614285714286;670.857142857143" +[1] "translation time;0;0.78125;22.7142857142857" +[1] "evaluation time;0;11.2844642857143;562.285714285714" [1] "constraints count;0;723.18125;24110" [1] "reevaluation count;0;1092.48946428571;46469.1428571429" -[1] "#tests;477" -[1] "reasoning time;0;39.1217929519816;667.257142857143" -[1] "translation time;0;1.60014974543276;48.2285714285714" -[1] "evaluation time;0;26.4839572726365;561.628571428571" -[1] "constraints count;0;1714.75576519916;24110" -[1] "reevaluation count;0;2612.6485474693;46469.1428571429" -[1] "processing folder W:/offlineFiles/EASy-concepts/reasoner/measures/benchmark-results-old-1_3_0-jdk1_8-20190215" +[1] "#tests;423" +[1] "reasoning time;0;16.0985984464708;679.771428571429" +[1] "translation time;0;0.846420128335022;45.8857142857143" +[1] "evaluation time;0;10.7499493414387;563.714285714286" +[1] "constraints count;0;685.332151300236;24110" +[1] "reevaluation count;0;1035.12715298886;46469.1428571429" +[1] "processing folder W:/offlineFiles/EASy-concepts/reasoner/measures/benchmark-results-old-1.3.0-jdk1.8-20190308" [1] "#tests;433" -[1] "reasoning time;0;11.7713625866051;399.142857142857" -[1] "translation time;0;4.74249422632794;171.285714285714" -[1] "evaluation time;0;4.11382382052128;188.285714285714" +[1] "reasoning time;0;11.6585285384362;390" +[1] "translation time;0;4.83404816892115;171.571428571429" +[1] "evaluation time;0;4.15357967667437;195.428571428571" [1] "constraints count;0;52.6235565819861;6000" [1] "reevaluation count;0;603.293632464533;23695.4285714286" -[1] "#tests;542" -[1] "reasoning time;0;27.0017527675277;373.742857142857" -[1] "translation time;0;11.5333684765419;207.571428571429" -[1] "evaluation time;0;8.86414953435249;171.828571428571" -[1] "constraints count;0;42.2952029520295;6000" -[1] "reevaluation count;0;1393.59093305219;23695.4285714286" -[1] "processing folder W:/offlineFiles/EASy-concepts/reasoner/measures/benchmark-results-old-eclipse-1_1_0-jdk1_8-20190215" -[1] "#tests;400" -[1] "reasoning time;0;16.6328571428571;663" -[1] "translation time;0;0.728392857142857;21.2857142857143" -[1] "evaluation time;0;11.4808928571429;561.714285714286" -[1] "constraints count;0;723.18125;24110" -[1] "reevaluation count;0;1092.57232142857;46470" -[1] "#tests;482" -[1] "reasoning time;0;43.6953764078245;755.571428571429" -[1] "translation time;0;1.8002618059672;16.5714285714286" -[1] "evaluation time;0;29.8965372456036;637.714285714286" -[1] "constraints count;0;1855.79356846473;24110" -[1] "reevaluation count;0;2839.77845287493;46470" -[1] "processing folder W:/offlineFiles/EASy-concepts/reasoner/measures/benchmark-results-old-eclipse-1_3_0-jdk1_8-20190215" -[1] "#tests;433" -[1] "reasoning time;0;11.8201913559881;468.428571428571" -[1] "translation time;0;4.69993401517651;163" -[1] "evaluation time;0;4.32151105245793;272.571428571429" -[1] "constraints count;0;52.6235565819861;6000" -[1] "reevaluation count;0;603.267568459254;23701.1428571429" -[1] "#tests;538" -[1] "reasoning time;0;26.5948884758364;394.085714285714" -[1] "translation time;0;11.0238228004957;163.742857142857" -[1] "evaluation time;0;8.91158169587538;197.085714285714" -[1] "constraints count;0;42.631970260223;6000" -[1] "reevaluation count;0;1357.47769516729;23701.1428571429" -[1] "processing folder W:/offlineFiles/EASy-concepts/reasoner/measures/benchmark-results-pi-1_1_0-jdk1_8-20190219" +[1] "#tests;446" +[1] "reasoning time;0;11.32955370489;366.428571428571" +[1] "translation time;0;4.64997864616699;162.028571428571" +[1] "evaluation time;0;3.97361733931241;165.971428571429" +[1] "constraints count;0;51.3139013452915;6000" +[1] "reevaluation count;0;586.722293401666;23695.4285714286" +[1] "processing folder W:/offlineFiles/EASy-concepts/reasoner/measures/benchmark-results-pi-1.1-jdk1.8-20190308" [1] "#tests;400" [1] "reasoning time;0;958.052678571429;28651.7142857143" @@ -110,11 +71,11 @@ [1] "constraints count;0;977.62375;24110" [1] "reevaluation count;0;1679.00803571429;46487.8571428571" -[1] "#tests;478" -[1] "reasoning time;0.0285714285714286;2790.09086471409;28689.8571428571" -[1] "translation time;0;38.8088563458856;279.428571428571" -[1] "evaluation time;0;2433.89591552102;26623.8571428571" -[1] "constraints count;0;2659.4550209205;24110" -[1] "reevaluation count;0;4696.50678421996;46492.7142857143" -[1] "processing folder W:/offlineFiles/EASy-concepts/reasoner/measures/benchmark-results-pi-1_3_0-jdk1_8-20190219" +[1] "#tests;416" +[1] "reasoning time;0.0285714285714286;920.879407051282;28460.2857142857" +[1] "translation time;0;13.8444482600733;242.2" +[1] "evaluation time;0;798.885966117216;26417.3428571429" +[1] "constraints count;0;941.034855769231;24110" +[1] "reevaluation count;0;1615.95147664835;46487.8571428571" +[1] "processing folder W:/offlineFiles/EASy-concepts/reasoner/measures/benchmark-results-pi-1.3-jdk1.8-20190308" [1] "#tests;433" [1] "reasoning time;0;254.985015946332;6117.28571428571" @@ -123,8 +84,8 @@ [1] "constraints count;0;52.6235565819861;6000" [1] "reevaluation count;0;815.268228307489;23708.2857142857" -[1] "#tests;544" -[1] "reasoning time;0.0571428571428571;671.583193277311;6285.85714285714" -[1] "translation time;0;276.622168242297;3041.85714285714" -[1] "evaluation time;0;199.457269782913;2626.71428571429" -[1] "constraints count;0;42.1764705882353;6000" -[1] "reevaluation count;0;2032.48671218487;23708.2857142857" +[1] "#tests;454" +[1] "reasoning time;0.0571428571428571;244.176431718062;6107.45714285714" +[1] "translation time;0;100.193953220055;2782.71428571429" +[1] "evaluation time;0;74.8047566603734;2528.48571428571" +[1] "constraints count;0;50.5374449339207;6000" +[1] "reevaluation count;0;779.189175582127;23702.2285714286" Index: /reasoner/reasoner.bib =================================================================== --- /reasoner/reasoner.bib (revision 267) +++ /reasoner/reasoner.bib (revision 268) @@ -211,5 +211,5 @@ } -@thesis{Mendoca08, +@phdthesis{Mendoca08, author = "M. Mendonça", title = {{Efficient Reasoning Techniques for Large Scale Feature Models}}, @@ -527,5 +527,5 @@ _address = {New York, NY, USA}, keywords = {Raspberry pi, performance benchmark, replicability, single-board computer}, - +} @article{GeorgesBuytaertEeckhout07,