# oktoflow platform: App development

This document summarizes guidelines on how to realize oktoflow applications. Further, we collect service-relization rules that apply when writing own code.

## Ways to implement an oktoflow application

There are three ways to create an own application. These summaries are currently a rough sketch. More details and links to tutorials will follow. There is a further document on [versioning](versioning.md) or hints on [debugging](debugApps.md).

### Using the management UI: 

- [Install](../INSTALL.md) the platform
- Start the platform
- Open the management UI 
- Enter the application configuration (in particular data types, services, service mesh, application). 
- For the specific application, let the platform create an application code template. 
- Import the template into your IDE.
- Complement the application classes with your specific code.
- Test your services with the generated data input files and unit tests (may require a takeover from the application code template)
- Upload a ZIP of your service implementation into the platform and let it integrate the app
- Create a deployment plan
- Start the app via the platform using the deployment plan

### Based on an existing example:

- Download an existing all-in-one application, e.g., [examples.python](../../examples/examples.python) or one matching described in the [examples collection](../examples/examples.md). For `examples.python` we provide a [source package](https://https://projects.sse.uni-hildesheim.de/oktoflow/examples.python-src.zip).
- Modify the configuration model in `src/main/easy`, details for the individual element are linked above
- Run the code generation `mvn install` (may lead to compile or test errors as the code may not fit the application)
- Change the application classes to include your specific code (or take the application code template in `target/gen/xx/templates` to implement new services using your code)
- Test your services with the generated data input files and unit tests (may require a takeover from the application code template)
- Build and test the whole application with `mvn install` 

If you want to do this with different examples, please note that they must have individual Maven artifacts as otherwise implementation parts are incorrectly loaded across examples. As copying examples is not the usual approach, the adjustments must be done manually. In more details,
- In `TechnicalSetup.ivml` the artifact name in `sharedArtifact` must be unique for all examples. In a platform managed configuration, only a single such artifact exists for all applications. However, all-in-one examples mimick a platform instance and, thus, need a unique name here.
- In the application`s IVML, the `name` and `id` must be unique for all all-in-one examples as this makes up the name of the application Maven artifact.
- In `pom.xml` 
  - the `artifactId` must be unique and replace the respective artifact id in `AllServices.ivml`. Please pay attention to the `groupId` which in the examples typically ends with `.apps`, which must be correct in both, `pom.xml` and `Services.ivml`.
  - the dependency pointing to the `sharedArtifact` must be replaced (as renamed above)
  - if stated, the dependency the generated application must be replaced (as renamed above)
  - the application executions (Java execution plugin `exec-maven-plugin`) must be adjusted so that they match path and artifact of the application (as renamed above)

If you then change the model, please remove all classes that could have taken over from the original app code template that would not be valid anymore as they rely on different tyes, e.g., those in `src/test/iip/connectivity`. After building the app interfaces, you may find them with `mvn -P App install`.

### Based on a separated model [impl.model](../../tools/impl.model) and implementation project:

- Obtain a copy of [impl.model](../../tools/impl.model)
- Modify the platform configuration in `src/main/easy`
- Run the build process with `mvn install`
- Obtain the application code template from `target/gen/*/templates`
- Continue with the template as stated above for "using the management UI".

Please consider the typical/expected [project structures](projects.md). A collection of build commands can be found in the [build guideline](../BUILDING.md).

Apps can be tested in automated/distributed fashion using the [Platform Evaluation and Testing Environment (PETE)](../../tests/test.environment).

# Testing

oktoflow support various testing levels. We just briefly summarize them here.

- Testing the connectivity of an individual connector: From the generated app code templates, take over the generated connectivity test in `src/test/java/iip/connectivity` as well as the respective `exec-maven-plugin` entries from the generated `pom.xml`. Please note that for authenticated connectors, also the identity store must be up to date. [Simulations and virtual twins](twinning.md) may support this step.
- Testing individual services through (Java/Python) unit tests: Generic tests per service are generated into the app code templates. The tests read data from service-specific data files, load the service into a feasible environment and feed the service instance with the data. Please note that we generate only data file templates, i.e., you must provide actual testing data. Further, there are currently no meaningful asserts as we do not required support service validity expressions in the model.
- Testing individual services in the service environment: Packaging a services with resources, unpacking the service and then running it through the respective service environment may reveal deployment issues. However, running a service in a simulated service-only Spring Cloud Stream environment is not tivial. For this purpose, the generated app code templates contain respective tests, that you may take over into your implementation project.
- Testing an individual connector as a service: Works similar to testing individual services, but requires a connector to be mocked, i.e., `mock=true` in the `AllServices.ivml`.
- Testing applications through mocking: Feed the connectors with mocking data (as for testing individual services/connector) but letting the data run through the entire application. For this purpose, applications can be started as Java applications. Please note that all runtime prerequisites like Python installation and packages must be met. The generated `pom.xml` contains respective entires (the `id` is the application `id`) in `exec-maven-plugin`.
- Testing the application with real data in a simulated/real environment.
- Testing the deployment and execution of (mocked) applications using the Platform Evaluation and Testing Environment (PETE).

The levels are discussed in more details in the [platform handbook](../PlatformHandbook.pdf).

# Service Realization Rules and Considerations

Realizing an application within the conventions and assumptions of frameworks or an entire platform is not trivial, in particular if conventions and assumptions are not documented. The [platform handbook](../PlatformHandbook.pdf) details the architectural constraints for the platform. While the constraints apply to the whole platform, in particular also to the realization of platform compnents, in this section, we operationalize the constraints in terms of specific rules for realization of application services. To remind about the rules, we place clearly visible comments into the generated skeletons in the application code templates. 

- A service is a **long-running function** that may continuously be fed with data. A service starts when its status is set by the service management of the platform to `STARTING`. The responsibility of the service is to do then all preparations for starting, e.g., resource allocation, and to set itself into status `RUNNING` (or `FAILED` for good reasons). Akin to the startup, at any point in time, the platform may request a change to `STOPPING`, giving the service the opportunity to stop processing and to release resources gracefully. The responsibility of the service is to go to `STOPPED` when the cleanup is done. In particular, this means that global code of a service implementation as well as constructors need to be implemented carefully to not execute unintended long running parts. While variable/attribute initializations are usually fine, e.g., loading an AI model or checking for the presence of a GPU causing an exception in certain cases may prevent successful loading of the service even prior to the execution of the application. This, in turn, may affect application tests assuming that services start up in a certain time; in other words, test data may already be ingested until your service becomes ready.
- **Don't block.** All service operations are expected to be executed as fast as possible, in particular service management operations. While there may be operations that take a certain amount of time, e.g., starting a JVM or a complex Python script within a service, longer running, resource consuming (blocking) operations are not permitted, e.g., reading a file and waiting for some reactions of the service. Such blocking operations may interfere with other services or the service control and leave the impression of timeouts, which may cause the respective service to enter the `FAILED` state.
- While a service is running, it is **kept in memory** (to conceptually run forever) by the platform to process data, i.e., it shall be ready to receive data and, depending on its configuration, produce output in synchronous or asynchronous manner. In other words, a service may keep its resources, AI models, etc. in memory. In particular on resource constrained devices, it makes sense to keep an eye on the overall memory consumption and to dynamically allocate/release resources that are not frequently used.
- A service **never returns/ingests "the non-existing object"**, e.g., `null` in Java or `None` in Python. Synchronous services always return a value upon their invocation, while asynchronous services may ingest any number of return values at any time. This specific requirement needs to be taken into account during application design, e.g., for synchronous services to indicate through a flag that the data item is invalid or for subsequent services of an asynchronous service not to expect when data arrives. 
- **Take care for the executability**, i.e., do not kill your service/the executing JVM. Further, catch/handle exceptions properly. The reason for this is that the service execution sets the `FAILED` state on a service that unexpectedly dies. This implies that you must also keep an eye on the used libraries whether they cause such problems (process termination, throwing undocument runtime exceptions) and that you as service developer are responsible for handling such issues in the scope of your service. For Python services, the service execution environment tries to catch unexpected exceptions on the methods of the service interfaces as far as possible. Moreover, Python libraries sometimes overlap and prevent execution, e.g. tensorflow and tensorflow-lite. In this case, the ability of the Python integration can be used allowing the definition of individual service packages or virtual environments to separate conflicting libraries.
- **Most service state change activities are handled for you** when you use the generated default implementations as basis (as prepared by the code skeletons in the application code templates). In most cases, the existing/generated service frame allows you to hook into this process, e.g., through start/stop methods that you can override, e.g., to place service initialiation/cleanup code into the right lifecycle phase.
- **Services are classes**. This is no big deal for Java as every language unit is a class. However, it may be a surprise for Python, as there are also Python scripts without class definitions. The Python service environment requires that a service is a class and that your operations are correctly declared with *self* parameters. The generated code tries to give you hints about the expected instances for input and output in terms of Python type annotations. The build processes execute Python source code checks based on *pyflakes* to notify at least about syntax errors.
- **Services shall be re-entrant**, i.e., further input data may arrive while your service is processing. While synchronous services may be throttled by the service execution engine, for asynchronous services there is no external visible correlation between input and output data. In other words, for an asynchronous service, the service developer is responsible for handling parallel input data, e.g., by a carefully designed service state or a stateless service. It is also important to consider that the application design may prevent parallel inputs, but from the service side no assumptions on this can be made. 
- Some connectors have a **sampling time** or an **ingestion time** that typically, if not specified, disables functionality (e.g., without sampling time no polling happens) or goes for a rather fast time ingestion interval (e.g., 1ms on the file connector) which might be too fast for your application/services. Please consider specifying/adjusting these values to be in line with the planned data processing of your app.
- Please note further that **Java services may employ Python functions**, e.g., by executing a script. Here, the class requirement for Python does not apply, as the Java service calls Python as a whole (in contrast to Python services that are hooked into the Python service environment). However, it is important that such Python functions are packaged correctly at compile/packaging time and unpackaged at runtime. For compile time, a Maven assembly descriptor shall package all Python scripts and their required resources into a ZIP file and deploy it to the Maven repository (classifier python, type zip) along with accompanying Java code. The Java code shall use the `ProcessSupport` class shipped with the Java service environment, which cares for correct unpacking and execution of Python scripts. The POM of the utilizing service must declare both, the Java and the Python artifact (classifier python, type zip) as dependencies.
- Do not make assumptions about **locations of file resources**, e.g., images or AI models. A service will be packaged together with other services of the same application into a service artifact. The layout of the artifact depends on the capabilities and conventions of the service execution engine. Currently, the platform  employs Spring Cloud Stream with two packaging strategies as discussed in the platform handbook, and you as a service developer do not know which artifact/packaging layout will ultimately created (also considering future alternative service execution engines or their evolution). Please also consider that paths into your home directory, to your desktop or operating system specific paths may not be existing on target devices. Thus, the following rules apply:
  - For **Java services**, rely on the `ResourceLoader` of the support layer of the platform rather than plain resource loading via classloaders or other mechanisms. The `ResourceLoader` which is designed to cope with this situation in a systematic manner and it is supposed to be set up correctly by the executing service environment. If for some reason needed, you may introduce your own (local) resource resolver.
  - For **Python services**, rely only on files placed relative to your service implementation. These files may be packed and shipped with your implementation or be downloaded from some source. Upon startup, your service is being unpackaged (along with the Python service environment and the Python application interface classes) into an own temporary directory as working directory for the Python process. This directory is used as process working directory for the Python process, which then has access to all Python modules and resources in that folder. However, you shall not make any assumptions on where this directory will be located or how it will be named. Python Scripts used by Java services are treated similarly.

  From a more general perspective, **for any other resource**, e.g., the temporary folder, do not make any assumptions about their location. The basic locations may differ among operating system installations (`/tmp` on Linux, somewhere in the user profile on Windows) and finally the platform configuration may decide about the directory to be used. Java, as well as Python offer programmatic access to the temporary folder, e.g., via `de.iip_ecosphere.platform.support.FileUtils` in Java or `tempfile.gettempdir()` in Python.
- Be as **generic as possible**, so that your service can be migrated to a generic platform-service, i.e., no unnecessary assumptions, instance creation via service methods rather than constructors [see e.g. examples.python](../../examples/examples.python/src/main/python/services/ExamplePythonService.py), generic data copy methods instead of fixed in/out data assignments [see e.g. examples.python](../../examples/examples.python/src/main/python/services/ExamplePythonService.py). 
- Rely on **default service implementations** where possible. For each service interface, the platform/application instantiation generates a service interface and a default implementation. For Java, interface and default implementation are separated, for Python interface and implementation are defined by the same class. Such a default implementation contains code for handling the service parameters as well as for the asynchronous data ingestors . Due to type safety, both aspects, parameters and ingestors cannot be realized in a generic manner. Although it is not difficult to implement both aspects manually, it is also a tedious task. Moreover, it is a common programming error to accidentially forget adjusting the parameters or ingestors when your application configuration is changed, e.g., when parameters are added/removed or multiple ingestors are needed due to multiple output streams. Inconsistent service implementations easily lead to long and tedious debugging tasks, while just a parameter declarator is missing, a parameter name is wrong or the service implementation expected to receive just a single ingestor. To ease the work of service developers and to keep up with changing models, we recommend to rely on generated default service implementations and the application code templates.
- Do not **delete code parts of the skeletons** in application templates (of course, comments may be removed). The generated code complies with the actual rules of the platform, e.g., to automatically register services instances or to rely on certain constructor signatures. Carelessly removing methods or global code may render your services unusuable and may lead to rather long debugging sessions to figure out why a service is not operating at all. 
- Service output to the console is typically subject to **logging**. The logging target may depend on the active service execution engine, e.g., for Spring Cloud Stream, a temporary folder per service is created in which files capture standard output or standard error stream, respectively. It is important to mention that depending on the use of Python for services, the logging target may differ. Further, the logging target for Python services may differ; command-line based integrations log to the error stream while network-based integrations may use both streams.
- Be careful with the **amount of logging**. Although helpful during development and debugging, logging may be your enemy during production, where overdone logging can easily impact performance fill up log files on the file system. Consider adjusting the log level when a service becomes ready for production deployment or use conditional logging, e.g., boolean "constants" in Java (although this requires more partially ugly code). Similarly, consider this for used libraries as they may carry out unintended logging activities on their own. 
- In addition to logging, further service output can be used for **debugging**, e.g., the generated (optional) logging or tracing of messages. Here, logging is more intended for local debugging while tracing (messages via the transport layer) can be helpful for remote debugging. To support remote debugging, the platform contains a transport message logger, which allows for receiving, emitting, storing and basic filtering of status, trace and monitoring messages. The kind of logging can be configured per service in the application configuration.
- As an app configuration may apply type substitutions, i.e., specified service types may be substituted by refined types without modifying the service implementation, services shall work with **generic type operations and type interfaces** rather than concrete data types. These operations, defined in the respective (generated) service base classes, include type factory methods (to be used instead of type constructors) and generic data instance copy methods (as then the actual type of input and output type may differ from the base types and data files may not be taken over).