# Modernizing a Platform's Technical Dependencies - an oktoflow (horror) Story
July 2024, updated September 2024

## Introduction

oktoflow was for a long time on Java 11/13. During an industrial workshop, a participant indicated that these Java versions are rather outdated and asked why we do not migrate. Our answer at that time was that we cannot predict whether this will work and how long it will take ("hoping for the best but expecting the worst", Forever Young by Alphaville). For [IIP-Ecosphere](https://www.iip-ecosphere.de/), the project in which oktoflow was born, such an endeavor would not have been possible, in particular at the time of the workshop, which took place shortly before public fairs.

After performing a migration to Java 17 and in a subsequent crazy step to Java 21, we know now why our answer and decision to postpone the migration was justified. This story summarizes the obstacles and experiences we made on oktoflow's way to Java 17/21.

## Background

oktoflow is based on a large number of libraries and as a software platform - at least how we understand this term - one of the main responsibilities is to integrate the abilities of existing solutions to allow users to create and run (Industry 4.0) applications more easily. So each of the libraries oktoflow is based on, i.e., its technical dependencies, may work in a recent version with Java 17/21 or not and may even determine whether we may still be compliant with Java 11/13. Individual dependencies may be outdated, imply security issues etc. Thus, although a migration to more recent versions may be beneficial, success is not guaranteed and may force subsequent actions and lead to restrictions, which are finally "decided" by others. 

One of the core libraries of oktoflow is the variability modeling toolset [EASy-Producer](https://github.com/SSEHUB/EASyProducer) with its dependencies [Eclipse](https://www.eclipse.org/) and [xText](https://eclipse.dev/Xtext/), while other important players are [Spring Cloud Stream](https://spring.io/projects/spring-cloud-stream) and [Eclipse BaSyx](https://projects.eclipse.org/projects/dt.basyx). Before we can think about a migration of oktoflow, we have to create a migration plan involving all potentially critical components. After that we may migrate EASY-Producer/oktoflow components starting at the lowest architectural layers.

## EASY-Producer (JDK 17)

At a glance, [EASy-Producer](https://github.com/SSEHUB/EASyProducer) worked well on a Eclipse 2024-03, e.g., the variability modeling language IVML and its xText based editors behaved as expected. However, dynamic parts of EASy-Producer, which rely on Java annotations and Java reflection, failed in initial tests with Java 17. These were, in particular, the tests for the instantiation languages VIL and VTL, which are essential to the oktoflow code generation. 
  * One reason was that Java 17 does not allow (empty) arrays as **initial values for annotations**. These values must now be compile-time constant, which is not the case for arrays and it seems that there is no workaround possible in the JVM. This is easy to overlook as the Java Language Specification states that the Java compiler can just ignore such erroneous annotations even without emitting a warning. In fact, parts of the type systems of VIL and VTL failed for this reason and required a refactoring of complex annotations into simpler, individual ones so that array-based annotation values become attributes of their own (then optional) annotations.
  * Moreover, in certain cases, individual (covariant) methods acting as VIL/VTL operations were not recognized correctly by our runtime reflection/bytecode analysis. In bytecode, covariantly overriden methods are represented by two methods, a bridge method (emulating the overriding with the more generic return type) and a method with the covariant return type. Here, the Java compiler changed the **sequence of operations in bytecode**, placing the bridge method before the implementing covariant method, while we accidentally took the first occurring method (ignoring the second) leading to wrong result/incompatible types in the tests. Filtering out such bridge methods fixed this issue.
  * Besides deeply hidden language problems, Eclipse changed some years ago its **build process**, from ANT to Maven/[Tycho](https://tycho.eclipseprojects.io/doc/latest/index.html). Until this migration, we kept the old ANT scripts of the EASy-Producer build process running. In the last years, this caused massive headaches when upgrading to a new Eclipse version and the automatically creating builds for multiple operating systems. This was exaberated, as the multi-OS support (Eclipse delta pack) disappeared/deprecated and the related Eclipse tooling (the "director") required manually collecting all required dependencies. Finally, and sadly, in the last years, our pre-packaged builds only worked for Windows. Hence, we decided to migrate the build process of EASy-Producer to Maven/Tycho, hoping that future changes of JDK and Eclipse are much easier to handle. However, the actual versions of Tycho at the time of migration forced us to Maven 3.9.6 and JDK 17. While we were able to migrate the build process for all EASy-Producer components, we ran into trouble when automatically building the update site and the pre-packaged Eclipse applications. One reason here was that the flow among the involved Tycho build plugins is not documented very well (or we did not find a good documentation): Examples for Tycho typcially focus on simple plugins and a full Maven module build, which does not comply with our incremental CI build approach. While we were able to build an update site, the following steps always failed stating that a dependency is missing. This message was meant to be literally, because just the contents of the update site was not made properly available by a precursor Tycho plugin. In more details, Tycho changes per plugin the timestamps in the names of the processed files while not updating the update site's index files, i.e., the files are there but cannot be found through the index files. Our workaround just copies the update site contents into the workspace of the director plugin. It's worth noting that the naming conventions of Tycho forced us to a renaming of most of the Maven components of EASy-Producer leading to some further detours.
  * Although now EASy-producer was working again, we were not able to build our **xText**-based languages. As we did our last upgrade of EASy-Producer several years ago and the workflow format of xText changed in the mean time (again) without migration guideline. However, at that point in time, we missed the helpful [guide of Jan Köhnlein](https://www.typefox.io/blog/xtexts-new-generator-migration/). Instead, we roughly relied of the hint in a forum post that it's helpful to create a new language project skipping the existing project. As a more helpful variation for us, to create a new project with the new xText version and to use the workflow as blueprint for the migration. The first trial failed as we aimed at taking over the previous rather detailed workflow specification and obviously missed various settings. Using the new default workflow components instead, which allow for a more concise workflow, and just tweaking some settings came to our rescue here. However, now the tests for the re-generated language components failed, on the one side as some xText interfaces were not met anymore, on the other side as xText applies new naming conventions. Fixing interfaces in several files and moving some files of older languages into the desired packages brought back the xText build flows.
  * Running EASy-Producer in a runtime Eclipse launch configuration showed, that around ten plugins were missing for several reasons. Fortunately, we found them for download and placing them into the `dropins` folder resolved that issue.
  * Although the xText languages may have worked at their core functionality, then the **language editors integrated** with EASy-Producer failed in the Eclipse Runtime environment with `ClassNotFoundExceptions`, although the classes were definitively there and accessible. Also the pre-packaged EASy-Producer versions (based on Eclipse 2024-03) failed for classes in `log4j` that are not even known to Google search. After a number of wrong hypothesis, it seems that the Eclipse bundle class loaders silently stop loading classes at the first issue, e.g., a problematic activator caused by missing actions during the xText upgrade. We finally resolved the editor class loading issues, which re-enables the editors in all of our Eclipse variants. The `log4j` issues remained, so we just hoped for the next version of Eclipse but did not terminate our migration efforts.  
  * Finally, (usual) modifications of the check specification format of **checkstyle** were needed to get also the codebase of EASy-Producer to Java 17.

## FP7 QualiMaster (JDK 17)
    
Besides oktoflow, the most complex test case of oktoflow is [FP7 QualiMaster](https://cordis.europa.eu/project/id/619525), which we still maintain for this purpose. Here, outdated dependencies and renaming towards the new Tycho-build (renamed) EASy-Producer components resolved many issues. Although upgrading QualiMaster's build flow took some time, it encouraged us to start the migration of oktoflow.

## oktoflow (JDK 17)

After EASy-Producer, we started the component-wise migration of oktoflow. 
  * Of course, also here the **checkstyle** format caused initial build issues. This was fixed quickly based on the experience from the EASy-Producer migration.
  * One announced surprise was that **Java 17 only supports TLS v1.3** and removed older TLS versions for security reasons. This immediately broke the regression tests of our Asset Administration Shell integration for BaSyx (v1.3). Fortunately, the interfaces of BaSyx allow for specifying the TLS protocol, which allowed us to fix the issue within the AAS abstraction/BaSyx integration.
  * oktoflow implements several Maven plugins, e.g., for Python syntax checking/tests, extended dependency management, encompassing in-Maven invocation and variablity instantiation/code generation/testing. Due to the upgrade of Maven in EASy-Producer and the extended Tycho artifacts in deployments there, we were forced to upgrade the **oktoflow build process** to the same Maven version. However, now Maven dependencies, in particular Google Guice collided with another version of Guice in the dependencies of Eclipse in EASy-Producer, preventing all instantiations/code generations from execution. We just decided to separate Maven from the required EASy-Producer executions, initially through an own class loader (which failed for still unknown reasons), finally by running the instantiation in an own JVM process.
  * Before switching to version 0.7.1-SNAPSHOT, the local builds on Windows looked fine, even for the rather complex platform examples. However, when **building oktoflow from scratch** without any components already deployed to our Maven repository, further issues occurred. 
    * In our case, it became evident that **Maven now passes system properties** from profiles to build plugins, which breaks some assumptions of our Maven invoker plugin, thus crashing all complex build flows that require further invokations of Maven, in particular all application code generations. Further, the specification of **final names of Maven assemblies** became more strict, preventing us to name some core assemblies as desired and, in turn, caused failures in upstream tests. As soon as the underlying reason is obvious, both issues are not complicated to fix, the first one requiring a change of our invoker, the second a more compliant use of assembly descriptors.
    * **HiveMQ community server and HiveMQ client collide** in a data structure library dependency, again causing Google Guice to fail. For now, we just force HiveMQ client to the more recent version of that library.
    * Some Java execution tests failed as (now) required **Java module command line options** were not in place. This affected the execution of test brokers (can be fixed by code generation) and subsequent Java processes in testing (handled now implicitly by the oktoflow Maven test plugin).
    * Other Java tests failed as **mockito** and the underlying bytecode manipultion were not compliant with JDK 17. Fortunately, only some components were effected, e.g., device management and influx connector, so that we fixed the issue by upgraded dependencies there locally.
    * Our continuous integration (CI) server is for legacy reasons installed with **multiple versions of Java, Maven and Python**. As we still did not containerize our build environments for EASy-Producer, QualiMaster and oktoflow, which would prevent malicious installation interdependencies, we added mechanisms to our plugins allowing to specify/detect the required versions and to modify the execution build paths. This re-enabled the tests also on our CI/Linux.

As a result, oktoflow was ready for JDK 17 (except for automated container building, which is still on the task list) and EASy-Producer just failed in Eclipse-related issues. This encouraged (not to say forced) us to immediately try the next upgrade.

## Eclipse 2024-06 and JDK 21

Local change of the JDK and running the EASy-Producer test suites showed that it can be executed and tested on JDK 21. So we continued the migration for that JDK:

* As envisioned, the Eclipse migration here was mainly changing the Eclipse repository URLs for Maven Tycho and adjusting the dependency versions constituting the Eclipse target platform during Maven Tycho builds. However, as a continuation of an issue above, it is surprising that still **Eclipse components are missing in Maven** (like xText UI or codetemplates are either not deployed to Maven or outdated there), changed their names (BouncyCastle gprov) or that our product specification now requires different Eclipse components for pre-prepackaged applications. In this context it is worth noting that for this update, no additional `dropin` bundles were needed to start EASy-Producer through an Eclipse launch configuration. 

* However, when running pre-packaged versions of EASy-Producer, two errors **affected a successful RCP/PDE start**. An [older issue](https://www.eclipse.org/forums/index.php/t/1106943/) that `org.osgi.service.component` cannot be resolved (in a pre-packaged RCP/PDE build) - which does not seem to harm, but is distracting if you try to solve issue by issue. A second, more serious issue indicates that the Eclipse workspace cannot be opened or is already closed. Akin to the issue above, class loading stops or at least affects higher levels of EASy-Producer components. We did not experience that behavior in previous versions and it does not occur in Eclipse launch configurations. May be the change of the OSGI framework in Eclipse over time or a more recent static initialization involving workspace access in JDT causes that issue. Ultimately we understood that this issue was caused by our code, as we loaded and registered classes that reference Eclipse JDE, which, in turn, tries to access the Eclipse workspace, too early, i.e., already during OSGI service startup. Now, a new mechanism of EASy-Producer allows to postpone this class loading until the EASy-Producer app is completely loaded.

* Discrepancies between the included functionality of installed bundles from an update site, Eclipse launch configuration and the pre-built RCP app caused **missing functionality** and thus further effort. One cause for these discrepancies is that *RCP installations* are defined in terms of a product specification file (which may contain may more features and bundles), *update site* are built based on selected features/plugins and their transitive dependencies in a category file and *launch configurations* are defined in terms of enabled plugins (if not all installed plugins are used). Differences among these three specifications lead to different observable behavior. In particular, contributions to the Eclipse help system require bundles that do not appear to be necessary for an normal installation (as Eclipse usually ships with these bundles), while RCP installations need them explicitly stated. In our case, the help window first failed with an error message and, after adding the first dependencies, came up showing a webserver/JSP error, which can be resolved by adding missing dependencies (thanks to [this discussion](https://www.eclipse.org/forums/index.php/t/149680/) and similar posts). Moreover, in our case, the RCP start (triggered through application and startup code) did not execute the applications bundle activator, causing missing functionality through missing registrations. Here it helps to explicitly call these registrations from the RCP application startup code.

* High-level EASy-Producer tests failed as they tried to add an xText nature to a new project, which, in turn, requires all dependencies to the xText nature set up. Initially, the symptom was on **classes for fragment hosts not found by Tycho**, which we partially could mitigate through `extraRequirements` of the target platform configuration (thanks to [this discussion](https://bugs.eclipse.org/bugs/show_bug.cgi?id=436617#c11)). However, we ultimately failed getting the dependency to the native SWT right, which contains some classes transitively required by the xText nature. For now, we just exclude the xText nature from the tests, re-enabling the tests.

* The **pre-packaged RCP builds did not contain a JRE** although enabled in the product specification. While this is working in a product export of Eclipse, Maven Tycho needs the information in different form. Initially, we tried to manually copy the JREs for Windows, Linux and Mac into the "jre" folder of the RCP build, which is mentioned by some sites and works on Windows, but fails, depending on the JRE packaging, on Linux and Mac. Finally, [the justj documentation](https://eclipse.dev/justj/?page=documentation) helped us to get the RCP builds running, now, akin to Eclipse/Tycho with a simple specification of the JDK version. At first trial, the Windows RCP failed, but this was just due when starting it in the folder where it has been built - the execution file paths became too long for Windows.

* During the migration, one particular test failed, which, since our migration to JDK 17, validates that **annotations** required by VIL/VTL are present and carry the requested information. However, we were a bit lazy and just emitted annotations through their own mechanisms to a file and diffed information collected for JDK 17 with those produced by JDK 21. Sadly, the sequence of the emitted annotation attributes/methods changed between the JDKs (compiler, reflection, etc.) so that now the test failed and required explicit sorting and output code to run independently of the JDK version. No further bytecode/reflection issued occurred during this step. 

* Akin to the upgrade of **bytecode manipulation frameworks** for mokito for JDK 17, we now hat to upgrade these dependencies to JDK 21. Fortunately, these were just local changes, although a global upgrade of the bytecode manipulation would have even worked with Spring (cloud streams).

Finally, we upgraded the **CI tasks** to JDK 21, which confirms that the platform runs on that JDK.
    
## Summary
    
At the end of this story, we can state, that we successfully migrated oktoflow to Java 17 and then to Java 21, also through the help of a large set of regression test cases from functional unit tests up to complex integration tests. While many hidden issues can clearly be attributed to our work, the core obstacles are caused by decisions of component maintainers of our dependencies, which, as far as we can see, also struggled with such decisions over time. While improvements (e.g., TLS) and cleanups (also interface removals as they now happen in Java) are justified and announced, other decisions are still mysteries to us. For the symptoms they cause, it is initially rather difficult to find a root cause - the inspections, detours, hypotheses and (various unintended) solutions lead to significant effort and frustration. Thus, it would be very interesting to better understand the decision processes leading to such changes and also how deeply collateral damage is being discussed upfront.

## Additional information

This is just information that we always forget to note down or is difficult to find. Might be it's helpful for others. Further entries may follow.

* Eclipse launcher application parameter for classloader debugging: `-Dorg.eclipse.osgi/debug/loader=true -Dorg.eclipse.osgi/debug=true  -Dorg.slf4j.simpleLogger.defaultLogLevel=debug`
* [Justj documentation](https://eclipse.dev/justj/?page=documentation)