ProActive Workflows & Scheduling — User Guide

API documentation

Scheduler REST    Scheduler GraphQL   Scheduler CLI   Scheduler Java   Workflow Creation Java   Python Client   Jupyter Kernel .NET Client

The following terms are used throughout the documentation:

In order to use Scheduler for executing various computations, one needs to write the execution definition also known as the Workflow definition. A workflow definition is an XML file that adheres to XML schema for ProActive Workflows.

It specifies a number of XML tags for specifying execution steps, their sequence and dependencies. Each execution step corresponds to a task which is the smallest unit of execution that can be performed on a computation resources (ProActive Node). There are several types of tasks which caters different use cases.

A Native Task is a ProActive Workflow task which main execution is defined as a command to run.

Native tasks are the simplest type of Task, where a user provides a command with a list of parameters.

Using native tasks, one can easily reuse existing applications and embed them in a workflow.

Once the executable is wrapped as a task you can easily leverage some of the workflow constructs to run your executable commands in parallel.

You can find an example of such integration in this XML workflow or you can also build one yourself using the Workflow Studio.

Native application by nature can be tied to a given operating system so if you have heterogeneous nodes at your disposal, you might need to select a suitable node to run your native task. This can be achieved using selection script.

A Script Task is a ProActive Workflow task which main execution is defined as a script.

ProActive Workflows supports tasks in many scripting languages. The currently supported dynamic languages or backends are Groovy, Jython, Python, JRuby, Javascript, Scala, Powershell, VBScript, R, Perl, PHP, Bash, Any Unix Shell, Windows CMD, Dockerfile, Docker Compose and Kubernetes.

As seen in the list above, Script Tasks can be used to run native operating system commands by providing a script written in bash, ksh, windows cmd, Powershell, etc…​ Simply set the language attribute to bash, shell, or cmd and type your command(s) in the workflow. In this sense, it can replace in most cases a Native Task.

You can find an example of a script task in this XML workflow or you can also build one yourself using the Workflow Studio. The quickstart tutorial relies on script tasks and is a nice introduction to writing workflows.

Scripts can also be used to decorate any kind of task (Script, Java or Native) with specific actions, as described in the additional scripts section.

It is thus possible to combine various scripting languages in a single Task. For example a pre script could be written in bash to transfer some files from a remote file system, while the main script will be written in python to process those files.

Finally, a Script Task may or may not return a result materialized by the result script binding.

A workflow can execute Java classes thanks to Java Tasks. In terms of XML definition, a Java Task consists of a fully-qualified class name along with parameters:

The provided class must extend the JavaExecutable abstract class and implement the execute method.

Any parameter must be defined as public attributes. For example, the above WaitAndPrint class contains the following attributes definitions:

A parameter conversion is performed automatically by the JavaExecutable super class. If this automatic conversion is not suitable, it is possible to override the init method.

Finally, several utility methods are provided by JavaExecutable and should be used inside execute. A good example is getOut which allows writing some textual output to the workflow task or getLocalSpace which allows access to the task execution directory.

The complete code for the WaitAndPrint class is available below:

To run a workflow, the user submits it to the ProActive Scheduler. It will be possible to choose the values of all Job Variables when submitting the workflow (see section Job Variables). A verification is performed to ensure the well-formedness of the workflow. A verification will also be performed to ensure that all variables are valid according to their model definition (see section Variable Model). Next, a job is created and inserted into the pending queue and waits until free resources become available. Once the required resources are provided by the ProActive Resource Manager, the job is started. Finally, once the job is finished, it goes to the queue of finished jobs where its result can be retrieved.

You can submit a workflow to the Scheduler using the Workflow Studio, the Scheduler Web Interface or command line tools. For advanced users we also expose REST and JAVA APIs.

It is possible to retrieve multiple logs from a job, these logs can either be:

Unless your account belongs to an administrator group, you can only see the logs of a job that you own.

Once a job or a task is terminated, it is possible to get its result. Unless you belong to the administrator group, you can only get the result of the job that you own. Results can be retrieved using the Scheduler Web Interface or the command line tools.

The quickstart tutorial on try.activeeon.com shows how to build a simple workflow using script tasks.

Below is an example of a workflow created with the Studio:

In the left part, are illustrated the General Parameters of the workflow with the following information:

In the right part, we see the graph of dependent tasks composing the workflow. Each task can be selected to show the task specific attributes and defintions.

Finally, above the workflow graph, we see the various palettes which can be used to drag & drop sample task definitions or complex constructs.

The following chapters describe the three main type of tasks which can be defined inside ProActive Workflows.

The GUI interaction with the Catalog can be done in two places: the Studio and Catalog Portal. The portals follow the concepts of the Catalog: Workflows are stored inside buckets, a workflow has some metadata and can have several revisions.

The Catalog view in the Studio or in the Catalog Portal allows to browse all the workflows in the Catalog grouped by buckets and to list all the revisions of a workflow along with their commit message.

Inside the Studio, there is a Catalog menu from which a user can directly interact with the Catalog to import or publish Workflows.

Additionally, the Palette of the Studio lists the user’s favourite buckets to allow easy import of workflows from the Catalog with a simple Drag & Drop.

Dependencies can be set between tasks in a TaskFlow job. It provides a way to execute your tasks in a specified order, but also to forward result of an ancestor task to its children as parameter. Dependency between tasks is then both a temporal dependency and a data dependency.

Dependencies between tasks can be added either in ProActive Workflow Studio or simply in workflow XML as shown below:

The replication allows the execution of multiple tasks in parallel when only one task is defined and the number of tasks to run could change.

The branch construct provides the ability to choose between two alternative task flows, with the possibility to merge back to a common flow.

The loop provides the ability to repeat a set of tasks.

Workflows often relies on task blocks. Task blocks are defined by pairs of start and end tags.

Task blocks are very similar to the parenthesis of most programming languages: anonymous and nested start/end tags. The only difference is that a parenthesis is a syntactical information, whereas task blocks are semantic.

The role of task blocks is to restrain the expressiveness of the system so that a workflow can be statically checked and validated. A treatment that can be looped or iterated will be isolated in a well-defined task block.

Variables can come at hand while developing ProActive applications. Propagate data between tasks, customize jobs, and store useful debug information are a few examples of how variables can ease development. ProActive has basically two groups of variables:

In order to immediately distinguish advanced and hidden variables, we present advanced variables in orange and hidden variables in light grey in our portals.

Here is an example with an Object’s details in the Catalog Portal

In addition to variables, another key/value structure can be accessed inside a script: Generic Information.

Generic information semantics differ from Job variables semantics in the following way:

Example of a generic information definition:

For example:

Some generic information can also be configured globally on the ProActive server. These global generic information will then apply to all submitted workflows or to certain categories of workflows (e.g. workflows with a given name). Global generic information are job-level only.

See Configure Global Variables and Generic Information section to understand how global generic information can be configured on the ProActive server.

The following table describes all available generic information:

Another way to propagate data from a task to another is the use of the result script binding. Anywhere in a parent task (usually at the end) you can set a value to a reserved variable named result. This value will be available in the child tasks i.e which are depending on the parent task. If a task has one or several parents, results will always be an array.

Assuming that we have two tasks task1 and task2 written in Groovy:

and task3 that depends on tasks task1 and task2, then, you can access result values defined by the parents as follows:

For nearly all script languages, the results variable contains a list of TaskResult java object. In order to access the result value, the value() method of this object must be called. Example for Python/Jython:

Job Results are composed of two elements: Result List and Result Map. Each task can contribute to both. Once the job is finished, the user can download/visualize Job Results from the Scheduler portal as shown in screenshot below. The label in Result List corresponds to the task name that generated the value.

To perform a control flow action such as if, replicate or loop, a Control Flow Script is executed on the ProActive node. This script takes the result of the task as input; meaning a Java object if it was a Java or Script task, or nothing if it was a native task.

The script is executed on the ProActive node, just after the task’s executable. If the executable is a Java Executable and returns a result, the variable result will be set in the script’s environment so that dynamic decisions can be taken with the task’s result as input. Native Java objects can be used in a Groovy script.

Similarly to how parameters are passed through the result variable to the script, the script needs to define variables specific to each action to determine what action the script will lead to.

The assigned value needs be a strictly positive integer.

The assigned value needs to be the string value if or else.

The assigned value needs to be a boolean.

Failure to set the required variables or to provide a valid control flow script will not be treated gracefully and will result in the failure of the task.

When Control Flow actions such as replicate or loop are performed, some tasks are replicated. To be able to identify replicated tasks uniquely, each replicated task has an iteration index, replication index, and a tag. In addition to help to identify uniquely, these tags are useful to filter tasks by iterations for example.

An Embarrassingly Parallel problem is a problem that is easy to split into smaller independent tasks. With ProActive Scheduler you can tackle this type of problem with the Replicate construct.

The Advanced workflows is an example of an embarrassingly parallel problem where the computation is easily distributed across ProActive Nodes.

MPI is often used in the area of parallel computing. The Scheduler integrates with MPI with the concept of multi node tasks. This particular task will acquire several nodes and will expose these nodes to the MPI environment.

Applications built with MPI are often executed using the mpirun command. It accepts several parameters to choose how many CPUs and how many machines will be used. One particular parameter is the machine file that is built by the Scheduler when using multi node tasks. Here is how an MPI application invocation would look like when executed with the Scheduler:

The variable $variables_PA_NODESFILE contains the path to a machine file created by the Scheduler that will be similar to:

Meaning that myApplication will use 2 CPUs on compute-host and 1 CPU on another-host. You can also use $variables_PA_NODESNUMBER to retrieve the number of acquired nodes.

You can find an example of a native task in this XML workflow or you can also build one yourself using the Workflow Studio.

To achieve the best performance for MPI applications, nodes can be selected taking into account their network topology. One might want to select nodes that are as close to each other as possible to obtain better network performance or nodes on different hosts to split the I/O load. Refer to Topology Types for more details.

Some workflow tasks require the prior installation of specific software tools and libraries that are necessary to properly execute these tasks. ProActive offers two methods to enable the installation of such tools and libraries:

A fork environment is a new forked Java Virtual Machine (JVM) which is started exclusively to execute a task. For example, this task may need to use some third-party libraries, so it can be executed properly. In this example, the list of path elements to be added when starting a new JVM will be specified inside the Additional Classpath field. Starting a new JVM means that the task inside it will run in a dedicated, configurable environment.

Another functionality is the possibility to define pre and post scripts. For a given task (Java, Script or Native task), it is possible to launch a script before and after its execution. This possibility can be useful to copy files to a node, or clean a directory before or after task execution, for example.

This is a way to separate from business code the preparation of execution environment and its cleaning. A good example is a script that removes files from a specific directory once the task is finished.

When workflow tasks are executed inside a ProActive Node, they run by default under the system account used to start the Node.

The RunAsMe mode, also called user impersonation, allows to start a workflow Task either:

The RunAsMe mode is defined in the XML representation of a Task:

In the ProActive Studio, the RunAsMe mode is available in the Task Properties Fork Environment section.

In order to impersonate a user, RunAsMe can use one of the following methods:

The above modes are defined by the ProActive server administrator when ProActive Nodes are deployed. It is possible to define for each Node a different RunAsMe mode. The underlying operating system need to be configured appropriately in order to impersonate a task.

See RunAsMe section in the administration guide for further information.

The default mode for RunAsMe is to impersonate as the current user with the user login and password. Accordingly, the user’s login and password inside ProActive must match the login and password of the user in the target machine’s operating system.

It is possible though to impersonate through a different user, using a different password, or even using a different impersonation method as the one configured by default.

This possibility is given by adding Generic Information described in the below table to the workflow task. Each of these generic information can be defined both at task-level or job-level. When at job-level, it will apply to all tasks of this workflow with RunAsMe enabled. Other tasks without RunAsMe enabled will ignore these generic information.

To create a multi-nodes execution (often used for MPI applications), you need to add a parallel environment to the task definition. Parallel environment describes how many nodes are needed for a task execution as well as where these nodes should be located. Thus, in addition to the number of ProActive Nodes, you can specify the nodes network topology, e.g the set of nodes within some latency threshold, nodes on the same host and many more.

Here is the list of topologies we support:

For instance, if a user wants to run 4 processes in parallel inside the scope of one task and in the same compute host, then she/he should specify that in the Multi-Nodes Execution section as shown in the following screenshot.

In the Job xml schema, the multi-nodes execution is available at the parallel tag

It’s possible to have errors when your Job is executed. ProActive Scheduler provides several mechanisms to deal with exceptional situations in your Jobs. If a Task contains errors during Job execution, the ProActive Scheduler automatically restarts this Task. The number of automatic restarts is defined by the Workflow owner by assigning an integer value to the counter Number of Automatic Restarts. The user also defines Where he wants that the scheduler restarts In-Error tasks, i.e., on the same ProActive Node or on another ProActive Node.

If automatic restarts does not fix errors, the Scheduler applies one of the following Task error management policies, i.e., the one defined by the user during Workflow creation:

Each In-Error Task inside the Job transits to Faulty and its successor Tasks are executed.

Successor Tasks of each In-Error Task are Pending until errors are fixed. The Job is in In-Error status. The user can restart In-Error Tasks as many times as he wishes.

Successor Tasks of each In-Error Task are paused until errors are fixed. The Job is paused. The user can restart In-Error Tasks as many times as he wishes.

The Job is canceled with all Tasks.

Error handling policy can be set globally on the job level, and also can be specified (overridden) individually for each Task.

The ProActive Scheduler allows to schedule Workflows with the Job Planner. Find more information in the Job Planner User Guide.

ProActive Scheduler supports the execution of time-based tasks or cron tasks. The users can assign a cron expression to the loop variable of a Loop in a ProActive Job. All tasks which belong that Loop will be executed iteratively and indefinitely. The start time of the next iteration is the next point of time specified by the cron expression.

Using the Scheduler Web Interface, it is possible to access the display of the ProActive Node running a given task. First remote visualization must be activated via the Scheduler configuration files (set novnc.enabled to true).

The task that you want to visualize must output a special string using the following format: PA_REMOTE_CONNECTION;JOB_ID;TASK_ID;vnc;HOSTNAME:PORT where JOB_ID must be the current job id, TASK_ID the current task id and HOSTNAME:PORT the hostname and port where the VNC server runs.

It is the task’s responsibility to print the string. Here is a sample script that starts a Xvnc server and runs a graphical application. It can be used in a native task for instance.

In the Scheduler Web Interface, select the running job and use the Preview tab to enable remote visualization.

When a task has IN_ERROR status, a right click on it allows the user to select Mark as finished and resume. This will set the task status as FINISHED and then resume job execution: this task won’t be executed again, subsequent tasks will execute as if the task executed correctly. This allows the user to manually perform actions to the IN_ERROR task, without cancelling the whole job execution.

If the task has not IN_ERROR status, the Mark as finished and resume option will be in grey, meaning that the action is disabled.

Add a "NOTIFICATION_EVENTS" Generic information in the workflow level, to list events on which a user wants to be notified. List is comma-separated and should be taken from(events are not case sensitive):

Scheduler can send email to the users when one of the above events is received.

If a user wants to be notified by all these events, the "NOTIFICATION_EVENTS" Generic information should have "All" (not case sensitive) as its value.

Notifications will only be sent for jobs whose generic information (workflow level) contains user’s email address(es) under the "EMAIL" key, if multiple email addresses need to be used, they should be separated by comma. Example:

To enable this functionality, the configuration need to be done following the steps introduced in Get Notification on Job Events Configuration.

In ProActive Studio, the SubmitJobNoWait and SubmitJobAndWait templates under the Controls menu allow you to submit a workflow from another workflow using the schedulerapi binding.

These templates accept the single parameter:

By default, these templates do not pass any parameter to the submitted workflow (using only default values provided by the workflow).

It is possible to modify this behavior by changing the SubmitJobNoWait or SubmitJobAndWait implementation.

For example, the following line can be replaced:

by

To provide the firstName and lastName variables parameters to the submitted workflows.

The SubmitJobNoWait or SubmitJobAndWait templates behavior is different in the sense that the latter will wait until the submitted workflow terminates.

In case an error occurs in the chained jobs, a user may want to escalate this error from the submitted (child) job to the parent job by throwing an exception. It is possible to integrate this behavior in the SubmitJobAndWait implementation using the schedulerapi binding as described in the code below.

Web Validations and Email Validations are two Tasks available in the Studio’s control-notification Bucket.
When the Workflow’s execution flow reaches one of these Tasks, the Job is paused, emails are sent if necessary and a Validation is created in the Notification Portal. From there, the Job remains in pause until an authorized user or user group performs an action on the Validation.

A Web Validation defines:

Email Validations define an extra field called Emails which holds a comma separated list of emails.
Each provided email will be notified that a Validation is pending. The email contains a link to the Notification Portal where the user can perform an action on the Validation.

Authorized users may perform three types of actions on a Validation:

All actions are final for each user. A user cannot modify his response. Once a Validation receives a Yes or No action, the Validation is considered processed and no future actions can be done by any user.

A label is a property that can be attached to a job, and changed dynamically by end-users. The Labels give the possibility to tag Jobs, group them and manipulate them by filtering based on these custom labels. These Labels act as a custom status for the Jobs. For example, a company can label jobs which are having errors to be reviewed by operators (label "to-be-reviewed"), and later mark them as "reviewed" when done. Job labels can be used for this specific purpose, but for any other purpose.

Before being able to use labels for Jobs, an Administrator has to define the authorized label list, see Manage job labels. If you are not an Admin of your platform, and want to use Labels, ask your administrator to add a label list.

It’s possible to add a label on a job in Workflow Execution The user can set a label on a job by clicking on Job Actions button of the job and then click on Edit label. The user can select any of the labels available in the sub-menu list. It’s possible as well to remove a label from a job by selecting Job Actions button of the job and then click on Remove label.

The user can set a label as well on a job from Scheduler Portal. It’s possible to select one or more jobs and then, from the right-click drop-down menu, the user can select Edit label and chose from the available labels. The user has the option to remove a label from a job or a list of jobs by selecting Remove label from the same drop-down menu.

Server Data Spaces have two types of storage on the host where the ProActive server is running:

By default, these spaces are linked to folders in the data directory of the ProActive Scheduler host:

But it can be changed in PROACTIVE_HOME/config/scheduler/settings.ini.

See Data Spaces configuration for further details on how to configure the Global and User spaces.

The Local Space is a Node Data Space created dynamically when a Task is executed.

The Local Space allows to:

These two types of transfers must be declared in the Task definition in order to occur (transfer is not implicit). You need to define which files have to be transferred from a data space to computing nodes by using input files and reciprocally from computing nodes to a data space by using output files:

Upon the execution of the previous example, ProActive Scheduler automatically transfers all files to a ProActive Node where the task will be executed. The files are put in a special place called Local Space on the ProActive Node. It corresponds to the working directory of a task when the task is in fork mode (default). From a task point of view, files can be accessed normally like shown below:

Then, based on the previous task definition, all the files whose the name starts with titi (e.g titi, titi2, etc.) produced by task1 will be transferred back automatically to Global Space by the Scheduler once the task is finished.

Files can be also transferred from/to User Space by specifying transferFromUserSpace/transferToUserSpace.

Before running your jobs you should first upload your files into either Global or User Space. To do it you can export folders linked to these spaces by one of the standard protocols like FTP, NFS, or use available web interfaces like Pydio.

It is possible to use wildcards in transfer directives, such as:

More information of pattern syntax can be found in the FileSystem class JDK documentation.

In addition to transferring to the Task Local Space, it is also possible to transfer files to the Node host Cache Space.

From the Node host point of view, the Cache Space is unique and shared for all tasks and all ProActive Nodes deployed on the host.

Transfers to the cache are not concurrent, only a single task can transfer to the cache at the same time. I.e if multiple tasks are run in parallel on the same hosts and they all transfer to the cache, these transfers will be executed sequentially.

If a file declared for the transfer is newer than the existing version in the cache, the file will be updated.

In order to transfer files to the cache, simply use the cache transfer directives inside your job:

The files will not be transferred to the Local Space of the task, so it will not be possible to access these files from the current directory.

In order to access these files, use the cachespace variable which contains the location of the cache, for example in groovy:

There are no transfer directives to transfer output files from the cache. Output files transfers are only possible from the Local Space.

Files in the cache space are automatically deleted after a given period. The default invalidation period is two weeks, which means that files older than two weeks will be automatically deleted.

In order to change this period, the ProActive Node must be started with the property node.dataspace.cache.invalidation.period (in milliseconds).

The property node.dataspace.cachedir can also be used to control the cache location on the node.

This section gathers the available scripts engines and the operating system where they run.

The Bash and Shell script engines allows to run linux shell scripts inside ProActive Tasks. The main difference between the two script engines is that the Shell script engine must use the Shebang instruction to define the Shell in use inside the script.

Both engines features the following behaviors:

Example bash script (the shebang notation is omitted):

Example shell script:

Writing Bash or Shell scripts in ProActive Workflows is rather straightforward, but a few considerations need to be taken into account.

The automatic binding mechanism works differently as other scripting language: it makes use of environment variables.

For example, suppose that the workflow variable foo is defined. This workflow variable can be accessed in the script engine through a specific environment variable created by the ProActive Scheduler:

The Script Bindings Reference chapter shows how various bindings can be used in bash or shell script engines.

In chapter Script variables, we explain how a given task script can modify the variable map and propagate its modifications inside the workflow. This does not apply to shell script engines, i.e. a shell script engine cannot modify the variables map directly.

For example, suppose the initial value of the foo variable is "Old Value" and Task_A contains the script:

And Task_B a child of Task_A contains:

Then, Task_B will display

We can see that the execution of Task_A had no impact on the foo variable.

In general, when a modification of a workflow variable is needed, other script engines should be used. In ProActive Catalog, an example groovy post script called update_variables_from_file allows to propagate a variable modification from a shell main Task script using a temporary file:

Task_A main bash script:

Task_A grovy post script (update_variables_from_file)

Task_B a child of Task_A:

Displays:

Windows Cmd script engine behaves similarly as bash script engine, but for Microsoft Windows Operating systems.

It features the following behaviors:

Example variable usage:

VBScript engine behaves similarly as Windows CMD but using visual basic syntax instead of batch. It uses the cscript.exe command to execute vbs scripts.

It features the following behaviors:

Example variable usage:

Any error occurring during the script execution does not imply necessarily that the workflow task running the script will fail. This is due to the cscript command not immediately exiting upon error.

In order to report a failure to the task when running a VB script, the following instruction can be added in the beginning of the script:

Example:

Javascript engine allows running Javascript scripts inside ProActive Tasks. This engine is based on Nashorn Javascript engine. Nashorn Javascript engine provides Rhino compatibility mode which is described in the following documentation.

Example of Javascript ProActive Task which modifies Job Variable:

Example of Javascript ProActive Task which modifies Job Variable containing JSON object:

Example of Javascript ProActive Task which returns JSON object as a result:

We support both Jython and Python Script Engines. Jython is an implementation of the Python programming language designed to run on the Java platform, Jython programs use Java classes instead of Python modules. The advantage of using our Jython Script Engine is that you do not need to do any installation. It includes some modules in the standard Python programming language distribution, but lacking the modules implemented originally in C. Besides, the libraries such as numpy, matplotlib, pandas, etc. are not supported by Jython. And the libraries which depends on numpy such as TensorFlow, PyTorch and Keras etc. are not supported neither.

In order to support native Python, we provide also a Python Script Engine. To use the Python Script Engine, the 'Language' field should be put to 'cpython'. By using Python Script Engine, you can personalize the Python version that you want to use. Since there are many different versions of Python (mainly Python2 and Python3) which are not compatible, ProActive supports all the Python versions (Python2, Python3, etc). By default, the Python used to execute the script is the default Python version on your machine. In order to use another Python version to execute the task, it is required to add a 'PYTHON_COMMAND' Generic Information. Its value should contain the symbolic or absolute path to the desired python command to run (for example 'python3' or '/usr/local/opt/python3/bin/python3.6'). If the Generic Information is put at task level this version of Python will be only used for this task, if it is put in the job level this version of Python will be used for all the tasks in this job.

For every tasks which use the native python script engine:

Here is a workflow example (in xml format) about a simple Python task:

A jython script engine execution runs in the same Java process as the Task execution. A cpython script engine execution runs inside a separate python process.

ProActive R script engine is based on the Java R Interface. In order to use the R script engine inside a ProActive Node (container which executes a workflow Task), the following prerequisites are needed:

The ProActive R script engine works on both Linux and Windows.

Here is an example of R script:

The following paragraphs describes the R script language specific syntaxes.

The progress variable is set as follows (notice the leading dot):

In contrary to other languages such as groovy or jruby, the parent tasks results (results variable) is accessed directly:

Variable affectation can be done via:

Access to dataspaces variables is similar to other languages:

Some internal R types (such as lists, vectors, strings) are automatically converted when stored as a result or in the workflow variable map, but other types such as data.table are not automatically converted. Conversion for these types should be done manually, for example using json serialization or an output file.

Java objects such as fork environment variable, scheduler, userspace or globalspace APIs are not available in R.

ProActive PowerShell script engine is based on jni4net to call the Powershell API from Java.

It requires that Powershell 2.0 Engine and .NET Framework 3.5 are installed on the relevant machines.

An example of Powershell script:

In contrary to other languages such as groovy or jruby, the parent tasks results (results variable) is accessed directly:

Variable affectation can be done via:

Internal PowerShell types such as Dates are automatically serialized to an internal format which can be understood by another powershell task, for example in the following two tasks:

Task1:

Task2:

The second task is able to automatically use the Date object received from the first task.

When an internal PowerShell type needs to be used by another language than PowerShell, a manual conversion such as json must be performed.

The Perl script engine features the following behaviors:

In that sense, the Perl script engine behaves similarly as the Bash or Cmd script engines.

Please refer to Script Bindings Reference for the complete list of bindings accessible through environment variables.

Inside Perl, you can access environment variables using the %ENV hash.

Next examples clarify how script engine bindings can be accessed inside Perl scripts.

A workflow variable can be accessed using system environment variables:

Similarly, the result of a parent task:

Or another script binding (for example, USERSPACE):

The PHP script engine features the following behaviors:

Inside PHP, you can access environment variables using the getenv() function.

Next examples clarify how script engine bindings can be accessed inside PHP scripts. Please refer to Script Bindings Reference for the complete list of bindings accessible through environment variables.

A workflow variable can be accessed using system environment variables:

Similarly, the result of a parent task:

Or another script binding (for example, USERSPACE):

The execution of a Docker_Compose task requires the installation of both Docker and Docker Compose on the host machine of the ProActive Node. Please refer to the official Docker documentation to see how to install Docker and Docker Compose.

A Docker_Compose task expects the content of a Docker Compose file to be implemented inside the Script editor of the Task Implementation section. You can find out how to write Docker Compose files in the official Docker Compose documentation.

To get started, we present below the content of the Script editor of a simple example of a Docker_Compose task.

The above example describes a container which is called 'helloworld'. That container is created from a busybox image, which will run the command 'echo "Hello ProActive"'

The Docker_Compose task allows to set parameters to the docker-compose tool with regard to the docker-compose CLI reference.

It supports general parameters as well as commands options (we currently only support options for the up command). You can specify these options by supplying a space character separated list in the generic informations.

The two latter generic information will be used to generate the following command:

If splitting by space is prohibitive you can specify the split regex in the generic informations with the key docker-compose-options-split-regex. If you supply e.g. "!SPLIT!" as value, then your docker-compose-up-options will need to look like this: "--option1!SPLIT!--option2".

A Bucket is a collection of ProActive Objects and in particular ProActive Workflows that can be shared between users.

When the ProActive Scheduler is first started, it contains already a predefined set of buckets, such as:

Listing the Catalog for existing Buckets is done using the following HTTP request (protocol, port and hostname needs to be adapted accordingly):

It is also possible to list all objects inside a particular Bucket using its bucketName:

Then, we can fetch a particular object based on its name:

Every bucket is identified by a unique bucket name. The bucket naming should match the following rules:

Each Catalog object is identified by a unique name. The object naming should respect the following rules:

The Catalog provides versioning of objects. You can list all the revisions of a particular object using the HTTP request:

The revision of objects is identified by its commit_time. Commit time specifies the time when the object’s version was added in the Catalog. You can fetch an object detailed metadata information using its name and commit_time:

When an object revision is added to the catalog, depending on its type, it may be parsed by the catalog service to extract some meaningful information.

For example, from a ProActive Workflow, the workflow name, project_name, description, documentation, Generic Information and Workflow variables are extracted.

These information are used in the various portals which interacts with the catalog such as the Workflow Execution Portal.

The Catalog Portal provides the capability to store objects like Workflows, Calendars, Scripts, etc. Powerful versioning together with full access control (RBAC) is supported, and users can easily share Workflows, templates, and other objects between teams, and various environments (Dev, Test, Staging, Prod).

The Notification Portal allows users to subscribe, be notified and react when different types of events occur during Job execution, on the state of the ProActive Scheduler or when automating and scheduling jobs with the Job Planner.

The Notification Portal is also where users can perform actions on Validations when they are authorized to visualize them. Learn more at the Validation’s dedicated section.