Script Engine

The Script Engine is not really an engine itself but rather a framework for engines that relies on logic written in a Script rather than JAVA. For example, a Data Scientist may develop a complex logic utilizing the powers of Python and GPUs that doesn’t translate to JAVA well. An engine can be built on the Script Engine pattern to incorporate that into NEO.

Script Engine is broken into two parts - the IXM Engine running on any NODE such as IXM/Integ/Grid Node and the Script Node. The IXM Engine’s job is to read data from the OLTP database, create CSV files for that data, pass the data to the Script Node, call the script on the Script Node, wait for the script to complete, take the results in the form of a CSV file on the script node back from the Script Node, and write the results back into the OLTP. The Script Node's job is to be able to accept the input from NEO, run the script, and write the output file back for NEO to pick up.

Platform supplies two classes to help support the interaction between NEO and external processes.

• java

  • Used for running external processes in the OS. Formalizes certain concepts, such as the exchange of flat files with the external process.

• java –

  • Used to execute a remote process on another machine via a REST API provided by tools/rest-facade.

Script Runner utilizes RESTFacadeRunner. The RESTFacade is running on the Script Node. The configuration on the Script Node is kept in /home/centos/rest-facade. The configuration identifies things like the paths, expected input files, output files etc.

ScriptEngine, when calling RestFasaceRunne,r must know the IP, port, and path. For example, we run this with Telstra and the Field Service module, and it hits http://172.31.17.159:3000 with path /long-term-optimization-engine.

On the script node, a wrapper phyton script is added to make it easier for Script Engine to control the read and writing of the files. This is handled by a call to /home/centos/rest-facade/LTOWrapper.sh. The config then tells this script the name of the specific script for the engine. The wrapper can identify paths to copy files, log the files that are being transferred, or even include keys for libraries such as Gurobi.