Package de.iip_ecosphere.platform.services.environment

Class AbstractPythonProcessService

java.lang.Object
de.iip_ecosphere.platform.services.environment.AbstractService
de.iip_ecosphere.platform.services.environment.AbstractRunnablesService
de.iip_ecosphere.platform.services.environment.AbstractPythonProcessService
All Implemented Interfaces:
GenericMultiTypeService, ParameterConfigurerProvider, Service, ServiceBase
Direct Known Subclasses:
PythonAsyncProcessService, PythonSyncProcessService
public abstract class AbstractPythonProcessService extends AbstractRunnablesService implements GenericMultiTypeService
Generic command-line-based Python integration for multiple data types. Considers environment mappings if the arguments indicate that the Python service shall be executed in a conda environment.
Author:
Holger Eichelberger, SSE

  • Field Details

    • TYPE_SEPARATOR_CHAR

      public static final char TYPE_SEPARATOR_CHAR
      See Also:

    • home

      private File home

    • pythonArgs

      private List<String> pythonArgs

    • locationKey

      private String locationKey

    • transportChannel

      private String transportChannel

    • impl

      private GenericMultiTypeServiceImpl impl

    • paramConfigurers

      private Map<String,ParameterConfigurer<?>> paramConfigurers

    • callbacks

      private Map<String,de.iip_ecosphere.platform.transport.connectors.ReceptionCallback<?>> callbacks

    • averageResponseTime

      private String averageResponseTime

    • enableFileDeletion

      private boolean enableFileDeletion

  • Constructor Details

    • AbstractPythonProcessService

      public AbstractPythonProcessService(String serviceId, InputStream ymlFile)
      Creates a service from a service id and a YAML artifact.
      Parameters:
      serviceId - the service id
      ymlFile - the YML file containing the YAML artifact with the service descriptor

    • AbstractPythonProcessService

      public AbstractPythonProcessService(YamlService yaml)
      Creates a service from YAML information.
      Parameters:
      yaml - the service information as read from YAML. By default, the Python executable is "ServiceEnvironment.py", which can be overridden by YamlProcess.getExecutable(). YamlProcess.getHomePath() is set to the home path where the executable was extracted to. Further, YamlProcess.getCmdArg() are taken over if given.

  • Method Details

    • initializeFrom

      protected void initializeFrom(YamlService yaml)
      Description copied from class: AbstractService
      Does further initialization of this instance from the given YAML information.
      Overrides:
      initializeFrom in class AbstractService
      Parameters:
      yaml - the service information as read from YAML

    • unbuffer

      protected boolean unbuffer()
      Returns whether the python process shall be unbuffered on the console streams (-u).
      Returns:
      true for unbuffer (default), false for buffered console streams

    • customizePythonArgs

      protected void customizePythonArgs(List<String> pythonArgs)
      Allows to customize the Python command line arguments taken from the yaml spec.
      Parameters:
      pythonArgs - the arguments to customize

    • enableFileDeletion

      public void enableFileDeletion(boolean enableFileDeletion)
      Enables or deletes file deletion. By default, Python files are delete upon end of the process.
      Parameters:
      enableFileDeletion - enables deletion

    • isFileDeletionEnabled

      public boolean isFileDeletionEnabled()
      Returns whether file deletion is enabled. By default, Python files are delete upon end of the process.
      Returns:
      true if enabled, false else

    • getHome

      protected File getHome()
      Returns the home directory.
      Returns:
      the home directory

    • getPythonModule

      private static String getPythonModule(String module, YamlService yaml, File homePath)
      Returns the name of the Python module. The default (if not explicitly given) is "ServiceEnvironment.py".
      Parameters:
      module - the module name, may be empty or null
      yaml - the YAML service deployment information
      homePath - optional home path to check for the module first if not given
      Returns:
      the Python module name

    • startExecutableByName

      protected boolean startExecutableByName()
      Returns whether the Python executable shall be started by name or by full path.
      Returns:
      false for starting by full path returned by getPythonExecutable()

    • scanInputStream

      protected String scanInputStream(Process proc, AbstractPythonProcessService.InputHandler handler)
      Scans the input stream of the given process for return data.
      Parameters:
      proc - the Python process
      handler - the input handler
      Returns:
      the first line if returnFirst, else null
      See Also:

    • createScanInputThread

      protected Thread createScanInputThread(Process proc, AbstractPythonProcessService.InputHandler handler)
      Creates a thread calling scanInputStream(Process, InputHandler).
      Parameters:
      proc - the Python process
      handler - the input handler
      Returns:
      the thread

    • handleResult

      protected <O> void handleResult(Class<O> cls, String data, String typeName)
      Handles a received processing result and ingests it back asynchronously.
      Type Parameters:
      O - the data type
      Parameters:
      cls - the data type class
      data - the serialized data
      typeName - the data type name as specified in the configuration model

    • toJson

      protected String toJson(Map<String,String> reconfValues)
      Turns a map of string values into JSON.
      Parameters:
      reconfValues - the values to turn into JSON
      Returns:
      the JSON representation

    • toPythonDataCls

      private static String toPythonDataCls(Class<?> cls)
      Turns a Java data class name to a Python data class name.
      Parameters:
      cls - the class
      Returns:
      the simple Python data class name

    • getTypeSubstitutionsJson

      protected String getTypeSubstitutionsJson()
      Returns the type mappings as JSON.
      Returns:
      the type mappings

    • createAndCustomizeProcess

      protected Process createAndCustomizeProcess(String data, Map<String,String> reconfValues) throws ExecutionException
      Creates and customizes the process.
      Parameters:
      data - the data to be directly handed to the process via command line
      reconfValues - values to (re)configure the service environment, may be null for none
      Returns:
      the process
      Throws:
      ExecutionException - if the process cannot be created

    • getLocationKey

      protected String getLocationKey()
      Returns the location key for access into InstalledDependenciesSetup.
      Returns:
      the key, may be null for fallback

    • getPythonExecutable

      protected File getPythonExecutable()
      Returns the Python executable, either via InstalledDependenciesSetup and getLocationKey() or as fallback via PythonUtils.getPythonExecutable().
      Returns:
      the Python executable

    • handleErrorStream

      protected void handleErrorStream(InputStream err)
      Handles the error stream upon process creation.
      Parameters:
      err - the process error stream

    • getLogger

      protected static de.iip_ecosphere.platform.support.logging.Logger getLogger()
      Returns the logger.
      Returns:
      the logger

    • registerInputTypeTranslator

      public <I> void registerInputTypeTranslator(Class<I> inCls, String inTypeName, de.iip_ecosphere.platform.transport.serialization.TypeTranslator<I,String> inTrans)
      Description copied from interface: GenericMultiTypeService
      Adds an input type translator.
      Specified by:
      registerInputTypeTranslator in interface GenericMultiTypeService
      Type Parameters:
      I - the input data type
      Parameters:
      inCls - the class representing the input type
      inTypeName - symbolic name of inCls, e.g. from configuration model
      inTrans - the input data type translator
      See Also:

    • getInTypeInfo

      public GenericMultiTypeServiceImpl.InTypeInfo<?> getInTypeInfo(String inTypeName)
      Returns the input type information object for the given symbolic type name.
      Parameters:
      inTypeName - the symbolic type name
      Returns:
      the information object or null if none was registered

    • registerInOutRelation

      public void registerInOutRelation(String inTypeName, String outTypeName)
      Description copied from interface: GenericMultiTypeService
      Explicitly registers the relation between two type names.
      Specified by:
      registerInOutRelation in interface GenericMultiTypeService
      Parameters:
      inTypeName - symbolic name of an input type, e.g. from configuration model
      outTypeName - symbolic name of an output type, e.g. from configuration model
      See Also:

    • getOutTypeName

      public String getOutTypeName(String inTypeName)
      Description copied from interface: GenericMultiTypeService
      Returns a registered output type name for a given input type name.
      Specified by:
      getOutTypeName in interface GenericMultiTypeService
      Parameters:
      inTypeName - the symbolic input type name
      Returns:
      the related symbolic output type name, may be null if there is none
      See Also:

    • getOutTypeInfo

      public GenericMultiTypeServiceImpl.OutTypeInfo<?> getOutTypeInfo(String outTypeName)
      Returns the output type information object for the given symbolic type name.
      Parameters:
      outTypeName - the symbolic type name
      Returns:
      the information object or null if none was registered

    • registerOutputTypeTranslator

      public <O> void registerOutputTypeTranslator(Class<O> outCls, String outTypeName, de.iip_ecosphere.platform.transport.serialization.TypeTranslator<String,O> outTrans)
      Description copied from interface: GenericMultiTypeService
      Adds an output type translator.
      Specified by:
      registerOutputTypeTranslator in interface GenericMultiTypeService
      Type Parameters:
      O - the output data type
      Parameters:
      outCls - the class representing the input type
      outTypeName - symbolic name of outCls, e.g. from configuration model
      outTrans - the output data type translator
      See Also:

    • attachIngestor

      public <O> void attachIngestor(Class<O> outCls, String outTypeName, DataIngestor<O> ingestor)
      Description copied from interface: GenericMultiTypeService
      Attaches an asynchronous result data ingestor.
      Specified by:
      attachIngestor in interface GenericMultiTypeService
      Type Parameters:
      O - the output data type
      Parameters:
      outCls - the class representing the type
      outTypeName - symbolic name of outCls, e.g. from configuration model
      ingestor - the ingestor instance

    • compose

      protected static String compose(String typeName, String data)
      Composes a symbolic type name with the string-serialized data.
      Parameters:
      typeName - the type name
      data - the data
      Returns:
      the composed String

    • getParameterConfigurer

      public ParameterConfigurer<?> getParameterConfigurer(String paramName)
      Description copied from interface: ParameterConfigurerProvider
      Returns the parameter configurer for a given parameter. We need this generic approach to ease code generation and initial setting of parameter values.
      Specified by:
      getParameterConfigurer in interface ParameterConfigurerProvider
      Specified by:
      getParameterConfigurer in interface Service
      Parameters:
      paramName - the name of the parameter
      Returns:
      the associated parameter configurer or null if there is none

    • getParameterNames

      public Set<String> getParameterNames()
      Description copied from interface: ParameterConfigurerProvider
      The set of parameter names.
      Specified by:
      getParameterNames in interface ParameterConfigurerProvider
      Specified by:
      getParameterNames in interface Service
      Returns:
      the parameter names, may be null for none

    • addParameterConfigurer

      public void addParameterConfigurer(Consumer<Map<String,ParameterConfigurer<?>>> paramConsumer)
      Adds parameter configurers via a consumer.
      Parameters:
      paramConsumer - the consumer, user AbstractService.addConfigurer(Map, String, Class, TypeTranslator, ValueConfigurer) and related methods in there

    • start

      protected ServiceState start() throws ExecutionException
      Description copied from class: AbstractService
      Starts the service and the background process.
      Overrides:
      start in class AbstractService
      Returns:
      the state to transition to, may be null for none
      Throws:
      ExecutionException - if starting the process fails

    • stop

      protected ServiceState stop()
      Description copied from class: AbstractService
      Stops the service and the background process.
      Overrides:
      stop in class AbstractRunnablesService
      Returns:
      the state to transition to, may be null for none

    • establishServerListener

      private void establishServerListener(String typeName, String serverChannel) throws IOException
      Establishes a server listener for internal server-client communication via Transport.
      Parameters:
      typeName - the name of the type for Java-Python communication
      serverChannel - the name of the server channel to send initial request to
      Throws:
      IOException - if sending/receiving messages fails

    • establishClientListener

      private void establishClientListener(String typeName, String serverChannel) throws IOException
      Establishes a client listener for internal server-client communication via Transport.
      Parameters:
      typeName - the name of the type for Java-Python communication
      serverChannel - the name of the server channel to send initial request to
      Throws:
      IOException - if sending/receiving messages fails

    • getAvgResponseTime

      public long getAvgResponseTime()
      Returns the average response time for the execution in Python (without transport).
      Returns:
      the average response time (in nanoseconds)