Index: /reasoner/conclusion.tex =================================================================== --- /reasoner/conclusion.tex (revision 272) +++ /reasoner/conclusion.tex (revision 273) @@ -2,2 +2,9 @@ \TBD{We are better than before, we are now more IVML-complete, but not reasoning-complete. Future, integrate with SMT or other solvers for left-over reasoning problems or configuration completion/instance creation (potential student work).} + +\begin{itemize} + \item model optimizations (see Section \ref{sectEvaluationResultsDescriptive}, Table \ref{experimentsDescSummary}) + \item execute default value assignments upon creation and throw away + \item type cache + \item quantification unrolling +\end{itemize} Index: /reasoner/evaluation.tex =================================================================== --- /reasoner/evaluation.tex (revision 272) +++ /reasoner/evaluation.tex (revision 273) @@ -234,11 +234,13 @@ \end{table*} -Table \ref{tab:experimentsDescSummary} summarizes for the initial run the number of tests, the measured reasoning time, the created number of constraints, the required constraint evaluations. For providing an indication of the overall time consumption of the experiment, we indicate the total execution time for the 5 repeated executions of the testsuites. We list number of tests, constraints and evaluations although they shall be the same for the same treatments, because in the pre-experiments the numbers indicated that due to minor bugs the experiments were done over different models. We do not denote the minimum reasoning time, number of constraints or evaluations, respectively, the minimum number in all cases is 0 due to IVML tests without constraints. - -On the original device \ref{oldLaptopId}, we execute both versions of the reasoner on the same JDK (id 1 and 2). On the newer \ref{newLaptopId} we continue with the more recent reasoner version in a more modern (id 3, 4). In this setup, we use two different JDKs, Java 8 to enable a comparison with \ref{oldLaptopId}, and Java 10 to give an outlook on more recent JVMs. On the continuous integration machine (\ref{jenkinsId}, id 5) we just measure reasoner version 1.3.0 (corresponding to the most recent commit at that time). Finally, on the Pi device \ref{piId} we measure both reasoner versions on the same JDK (id 6 and 7). - -As mentioned in Section \ref{sectEvalSetupTreatments}, the treatments differ between the reasoner versions (id 1/2 as well as 6/7) as not all tests could be ported back successfully. Moreover, the number of tests on Jenkins (\ref{jenkinsId}, id 5) differs, as some tests involving VIL are disabled there due to technical issues. Except for the Pi \ref{piId}, the average reasoning time is rather similar across the devices and JDKs. However, according to the maximum reasoning time, reasoner version 1.3.0 performs faster than version 1.1.0. It is important mentioning that reasoner version 1.3.0 is both, more IVML complete and better tuned for performance as discussed in Section \ref{sectPerformance}. In contrast, reasoner version 1.1.0 even creates significantly more (including accidentally ineffective) constraints and, thus, requires more re-evaluations, i.e., consumes more reasoning time. This can also be identified in terms of constraints initially stored by the reasoner in the constraint base as well as the number of re-evaluations. Jenkins (\ref{jenkinsId}, id 5) appears to be a bit faster at lower re-evaluations as some reasoning tests are disabled. - -Regarding the total execution time, id 1 may appear as an outlier or a measurement error. However, we observed this behavior several times. As besides pure reasoning experiments also IVML and VIL tests are executed (\TBD{pi}), we attribute this difference to the differences in the number/complexity of treatments, possibly also to changes in the IVML/VIL code base of EASy-producer. However, comparing id 2 and id 3 it seems that an OpenJDK on Windows 10 behaves sigificantly worse than Oracle JDK on a rather old Windows 7 installation. This impression was confirmed by the average reasoning time when repeating the experiments (maximum reasoning time 336 ms for id 1 and 485 ms for id 3). OpenJDK 10 seems to be slightly faster (with an even smaller difference in the repeating run). Although we executed the experiments on Jenkins (id 5) in a separate build task prevening parallel builds, the overall execution time is almost twice as high as on a laptop. Potential reasons may be due to the setup (Jenkins running in a web container with Apache as frontend, regular checks for commits for 85 build tasks, an almost full virtual hard drive, virtualization overhead in particular for I/O operation when loading IVML and VIL models, etc.). +Table \ref{tab:experimentsDescSummary} summarizes for the initial run the number of tests, the measured reasoning time, the created number of constraints, and the performed constraint evaluations. As an indication of the overall time consumption of the experiment, we list also the total execution time for the 5 repeated executions of the testsuites. At a glance, the number of of tests, constraints and evaluations shall be the same for the all treatments and might appear to be irrelevant here. However, as the treatments differ (cf. Section \ref{sectEvalSetupTreatments}), we list also these numbers. In particular, in the pre-experiments these numbers helped us to identify minor bugs in experiments that accidentally caused the evaluation of same rather than different models in the extended QualiMaster cases. We do not show here the minimum reasoning time, number of constraints or evaluations, respectively, as the minimum number is usually 0 due to IVML test cases without constraints. + +On the original device \ref{oldLaptopId}, we execute both versions of the reasoner on the same JDK (id 1 and 2). On the newer laptop \ref{newLaptopId} we continue with the more recent reasoner version in a more modern (id 3, 4). In this setup, we use two different JDKs, Java 8 to enable a comparison with \ref{oldLaptopId}, and Java 10 to give an outlook on more recent JVMs. On the continuous integration machine (\ref{jenkinsId}, id 5) we just measure reasoner version 1.3.0 corresponding to the most recent commit\footref{reasonerActualVersion} at that time. Finally, on the Pi device \ref{piId} we measure both reasoner versions on the same JDK (id 6 and 7). + +As mentioned in Section \ref{sectEvalSetupTreatments}, the treatments differ between the reasoner versions (id 1/2 as well as 6/7) as not all tests could be ported back successfully. Moreover, the number of tests on Jenkins (\ref{jenkinsId}, id 5) differs, as some tests involving VIL are disabled there due to technical issues. Except for the Pi \ref{piId}, the average reasoning time is rather similar across the devices and JDKs. However, according to the maximum reasoning time, reasoner version 1.3.0 operates faster than version 1.1.0. It is important mentioning that reasoner version 1.3.0 is both, more IVML complete and better tuned for performance as discussed in Section \ref{sectPerformance}. Moreover, reasoner version 1.1.0 even creates significantly more (including accidentally ineffective) constraints and, thus, performs more re-evaluations, i.e., consumes more reasoning time. This can also be identified in terms of constraints in the constraint base as well as the number of re-evaluations. Jenkins (\ref{jenkinsId}, id 5) appears to be a bit faster at lower re-evaluations, which is due to some disabled test cases. + +Regarding the total execution time, id 1 vs. 2 may appear as an outlier. However, we observed this behavior several times. Moreover, the difference between the reasoner versions is also evident on the Pi device in id 6 and 7. As besides pure reasoning experiments also IVML and VIL tests are executed, we attribute the significant increrase in overal experimentation time to the differences in the number/complexity of treatments, possibly also to changes in the IVML/VIL code base of EASy-Producer. It is important to note that complex and long-running VIL instantiations such as the code generation in the QualiMaster models are disabled by the experiment script. + +A comparison of id 2 and id 3 suggests that an OpenJDK on Windows 10 behaves worse than Oracle JDK on a rather old Windows 7 installation in this experiment. This impression is confirmed by the average reasoning time in the repeated experiments (maximum reasoning time 336 ms for id 1 and 485 ms for id 3). OpenJDK 10 seems to be slightly faster (with an even smaller difference in the repeated experiments). Although we executed the experiments on Jenkins (id 5) in a separate build task prevening parallel builds, the overall execution time is almost twice as high as on a laptop. This is in contrast to the reasoning times, which are pretty similar to the other experiments for reasoner version 1.3.0, i.e., again reading the IVML model or running some VIL instantiations may cause the high overall execution time. Potential reasons may also be due to the setup, i.e., limited main memory size of 4 GBytes, an almost full virtual hard drive, virtualization overhead in particular for I/O operation when loading IVML and VIL models, etc. Based on \cite{KnocheEichelberger18}, we expected an average performance drop of factor 10 for the Pi experiments (id 6 and 7). \TBD{discuss}