Index: /reasoner/conclusion.tex =================================================================== --- /reasoner/conclusion.tex (revision 275) +++ /reasoner/conclusion.tex (revision 276) @@ -7,12 +7,13 @@ This report is a fundamental building brick in the direction of reasoning for complex non-Boolean variability models. It shows that on actual devices, reasoning for such models, here exemplified for IVML, is possible fulfilling both, a complete coverage of the required modeling concepts as well as the performance. For rather large conventional models, the proposed approach even operates on a Raspberry Pi 3 in less than 50ms. Larger and more complex models, such as the QualiMaster models deserve more computational power and can currently be analyzed in around 500 ms full reasoning, 150 ms incremental reasoning and 50 ms reasoning with re-used constraint base. -Future work will require reasoning capabilities in the embedded and IoT field, and, thus, either requires as faster reasoning approach, a native implementation of the presented approach or a performance optimization. Even for a native version, performance optimization is required. In this report, we already identified several opportunities: +Future work will require reasoning capabilities in the embedded and internet-of-things (IoT) area, and, thus, either requires as faster reasoning approach, a native implementation of the presented approach or significant performance optimizations. In this report, we already identified several opportunities. It is important that the related modifications are applied, tested and analyzed for their respective performance gain individually. \begin{itemize} - \item Immediate evaluation of default value constraints and storing only initially failing assignment constraints in the constraint base. - \item IVML type caching allowing for - \item quantification unrolling - \item model optimizations (see Section \ref{sectEvaluationResultsDescriptive}, Table \ref{experimentsDescSummary}) + \item Analyze constraints for their properties with respect to the actual reasoning mode before translating them. In most cases, this may blur the code or require a traversal of the constraint syntax tree, i.e., it may easily lead to an adverse effect to reasoning time. + \item Immediate evaluation of default value constraints and storing only initially failing assignment constraints in the constraint base. Although this appears to be a rather simple idea, an initial experiment showed that it neither works for deferred default values nor for (nested) default values of constraint variables, the latter due to side effects of immediate constraint re-scheduling. + \item Integrate a type-based cache with the variable mapping to avoid translating complex types from scratch over and over again. While this may lead to a slow down for smaller models, it may lead to a reduction of the translation time in general. + \item Unrolling quantified expressions so that only simple constraints (with IVML operations) are added to the constraint base. This is an opportunity implying many tradeoffs. While unrolling will enlarge the constraint base, it may also ease the integration of further reasoning steps that do not support quantification. Unrolling will increase the size of the constraint base and, thus, require more memory, but as only simple constraints are evaluated, only failing sub-constraints instead of the quantified constraint must be re-scheduled. However, additional tracking is needed to correctly evaluate (embedded) quantors, e.g., in case of existential quantification. Moreover, this idea increases the complexity of the re-scheduling, as sub-constraints must be added or removed when container and container element values change. + \item Several operations of the underlying IVML model may require a critical revision with respect to performance. As these operations are frequently called, they may have an hidden effect on the performance of the reasoner. Probably also some leaf node types of the Constraint tree might be merged wit Model elements, e.g., variable use vs.~ variable declaration; \end{itemize} -Moreover, advanced reasoning capabilities are of high interest for our work, i.e., integrating the presented reasoner as a fast initial step in a reasoning chain with other reasoning-complete approaches, such as SMT-solvers is also a possible future direction. +Moreover, advanced reasoning capabilities are of high interest for our work, i.e., integrating the presented reasoner as a fast initial step in a reasoning chain with other reasoning-complete approaches, such as SMT-solvers is also a possible future direction. However, this requires an approach that copes with IVML operations and may require, quantification unrolling. Index: /reasoner/evaluation.tex =================================================================== --- /reasoner/evaluation.tex (revision 275) +++ /reasoner/evaluation.tex (revision 276) @@ -265,5 +265,5 @@ At a glance, the number of of tests, constraints and evaluations shall be the same for the all treatments and might appear to be irrelevant here. However, as the treatments differ (cf. Section \ref{sectEvalSetupTreatments}), it is important indicating also these numbers. In particular, in the pre-experiments the readings helped us identifying minor bugs in the experiments that accidentally caused the evaluation of same rather than different models in the extended QualiMaster cases. In more details, the \emph{treatments differ} between the reasoner versions (id 1/2 as well as 6/7) as not all tests could be ported back. Moreover, the number of tests on Jenkins (\ref{jenkinsId}, id 5) differs, as some tests involving VIL are disabled there due to technical and memory issues. Except for the Pi \ref{piId}, the average reasoning time is rather similar across the devices and JDKs. However, according to the maximum reasoning time, reasoner version 1.3.0 operates faster than version 1.1.0. It is important mentioning that reasoner version 1.3.0 is both, more IVML complete and better tuned for performance as discussed in Section \ref{sectPerformance}. Moreover, reasoner version 1.1.0 even creates significantly more (including accidentally ineffective) constraints and, thus, performs more re-evaluations, i.e., consumes more reasoning time. This can also be identified in terms of constraints in the constraint base as well as the number of re-evaluations. Jenkins (\ref{jenkinsId}, id 5) appears to be a bit faster at lower re-evaluations, which is due to some disabled test cases. -Regarding the \emph{total execution time}, id 1 vs.~2 may appear as an outlier. However, we observed this behavior several times. Moreover, the difference between the reasoner versions is also evident on the Pi device in id 6 and 7. As besides pure reasoning experiments also IVML and VIL tests are executed, we attribute the significant increase in overal experimentation time to the differences in the number/complexity of treatments, possibly also to changes in the IVML/VIL code base of EASy-Producer. It is important to note that complex and long-running VIL instantiations such as the code generation in the QualiMaster models are disabled by the experiment script. +Regarding the \emph{total execution time}, id 1 vs.~2 may appear as an outlier. However, we observed this behavior several times. Moreover, the difference between the reasoner versions is also evident on the Pi device in id 6 and 7. As besides pure reasoning experiments also IVML and VIL tests are executed, we attribute the significant increase in overal experimentation time to the differences in the number/complexity of treatments, the deferred but still executed output of failing constraints (in particular for large generated models that are supposed to faile), but possibly also to changes in the IVML/VIL code base of EASy-Producer. It is important to note that complex and long-running VIL instantiations such as the code generation in the QualiMaster models are disabled by the experiment script. A comparison of id 2 and id 3 suggests that \emph{OpenJDK} on Windows 10 behaves worse than \emph{Oracle JDK} on a rather old Windows 7 installation in this experiment. This impression is confirmed by the average reasoning time in the repeated experiments (maximum reasoning time 336 ms for id 1 and 485 ms for id 3). OpenJDK 10 seems to be slightly faster (with an even smaller difference in the repeated experiments). Although we executed the experiments on Jenkins (id 5) in a separate build task prevening parallel builds, the overall execution time is almost twice as high as on a laptop. This is in contrast to the reasoning times, which are pretty similar to the other experiments for reasoner version 1.3.0, i.e., again reading the IVML model or running some VIL instantiations may cause the high overall execution time. Potential reasons may also be due to the setup, i.e., limited main memory size of 4 GBytes, an almost full virtual hard drive, virtualization overhead in particular for I/O operation when loading IVML and VIL models, etc. Index: /reasoner/reasoner.bib =================================================================== --- /reasoner/reasoner.bib (revision 275) +++ /reasoner/reasoner.bib (revision 276) @@ -105,5 +105,5 @@ author = {Kollia, Ilianna and Glimm, Birte and Horrocks, Ian}, title = {{SPARQL Query Answering over OWL Ontologies}}, - booktitle = {Proceedings of the Extended Semantic Web Conference on The Semantic Web: Research and Applications (EWSC'11) - Vol. Part I}, + booktitle = {Extended Semantic Web Conference on The Semantic Web: Research and Applications (EWSC'11) - Vol. Part I}, _series = {ESWC'11}, year = {2011}, @@ -121,5 +121,5 @@ title = {{Structure of the Dresden OCL Toolkit}}, author = "B. Demuth, S. Löcher, St. Zschaler", - booktitle = "International Fujaba Days, MDA with UML and Rule-based Object Manipulation", + booktitle = "Intl. Fujaba Days, MDA with UML and Rule-based Object Manipulation", year = 2004 } @@ -158,5 +158,5 @@ author={Holger Eichelberger and Aike Sass and Klaus Schmid}, title={From Reproducibility Problems to Improvements: A journey}, -booktitle={Proceedings of the Symposium on Software Performance (SSP'16)}, +booktitle={Symposium on Software Performance (SSP'16)}, number={4}, journal={Softwaretechnik-Trends}, @@ -420,5 +420,5 @@ author = {Mendonca, Marcilio and Wasowski, Andrzej and Czarnecki, Krzysztof}, title = {{SAT-based Analysis of Feature Models is Easy}}, - booktitle = {Proceedings of the Intl. Software Product Line Conference (SPLC '09)}, + booktitle = {Intl. Software Product Line Conference (SPLC '09)}, _series = {SPLC '09}, year = {2009}, @@ -454,5 +454,5 @@ author = {Benavides, David and Trinidad, Pablo and Ruiz-Cort{\'e}s, Antonio}, title = {Automated Reasoning on Feature Models}, - booktitle = {Proceedings of the Intl. Conference on Advanced Information Systems Engineering (CAiSE'05)}, + booktitle = {Intl. Conference on Advanced Information Systems Engineering (CAiSE'05)}, _series = {CAiSE'05}, year = {2005}, @@ -471,5 +471,5 @@ author = {Mendonca, Marcilio and Branco, Moises and Cowan, Donald}, title = {{S.P.L.O.T.: Software Product Lines Online Tools}}, - booktitle = {Proceedings of the Conference on Object Oriented Programming Systems Languages and Applications (OOPSLA '09)}, + booktitle = {Conf. on Object Oriented Programming Systems Languages and Applications (OOPSLA '09)}, year = {2009}, isbn = {978-1-60558-768-4}, @@ -489,5 +489,5 @@ title = {{Flexible System-level Monitoring of Heterogeneous Big Data Streaming Systems}}, year = 2018, - booktitle = "Proceedings of the Euromicro Conference on Software Engineering and Advanced Applications (SEAA'18)", + booktitle = "Euromicro Conference on Software Engineering and Advanced Applications (SEAA'18)", pages = "289--292", } @@ -496,5 +496,5 @@ author = {Bulej, Lubom\'{\i}r and Hork\'{y}, Vojtech and T\r{u}ma, Petr}, title = {Do We Teach Useful Statistics for Performance Evaluation?}, - booktitle = {Proceedings of the ACM/SPEC on International Conference on Performance Engineering Companion (ICPE '17), Companion}, + booktitle = {Intl. Conference on Performance Engineering Companion (ICPE '17), Companion}, _series = {ICPE '17 Companion}, year = {2017}, @@ -514,5 +514,5 @@ author = {Knoche, Holger and Eichelberger, Holger}, title = {Using the Raspberry Pi and Docker for Replicable Performance Experiments: Experience Paper}, - booktitle = {Proceedings of the ACM/SPEC International Conference on Performance Engineering (ICPE'18)}, + booktitle = {Intl. Conference on Performance Engineering (ICPE'18)}, _series = {ICPE '18}, year = {2018}, Index: /reasoner/reasoner.tex =================================================================== --- /reasoner/reasoner.tex (revision 275) +++ /reasoner/reasoner.tex (revision 276) @@ -102,6 +102,6 @@ \bibliography{reasoner} +\clearpage \begin{appendices} -\pagebreak \input{appendixTests} \end{appendices}