Working with Machine Learning Pipelines

Machine learning (ML) is a digital tool used to make predictions. Data scientists train a model using a data set to prepare a machine learning algorithm with a machine learning model; this model can then make predictions based on learned patterns in the data.

An ML pipeline is an end-to-end framework that directs data flows into and out of a machine learning model. In other words, the pipeline allows users to get data in and out of NEO for the purpose of delegating decisions to a machine learning model. An ML pipeline adapts between ONE's inputs and the ML model's inputs.

Users can view and edit the ML pipeline for any ML plug point.

Complete the following steps to view or edit an ML pipeline:

  1. Log in to the ONE system.

  2. Click Menu/Favs > Tools > Machine Learning > Machine Learning Plug Point.
    The Machine Learning Plug Point screen appears.

    images/download/attachments/131840250/image2022-3-30_10-30-16-version-1-modificationdate-1686580196000-api-v2.png



  3. Click the View button for the ML plug point whose pipeline you wish to view or edit. For this example, we selected Compute Stop Level ETA.
    The Compute Stop Level ETA screen appears with the existing pipelines displayed along with their statuses.

    A green icon with a checkmark indicates the pipeline is valid. A red icon with an exclamation point indicates the pipeline is invalid


    images/download/attachments/131840250/image2022-4-22_9-56-46-version-1-modificationdate-1686580196000-api-v2.png

  4. Click the View Pipeline icon for the plug point you wish to view.

    images/download/attachments/131840250/image2022-3-30_10-35-27-version-1-modificationdate-1686580196000-api-v2.png

    The ML Pipeline Editor screen appears in a new tab with an ML Plug Point Entry and an ML Plug Point Return displayed, as well as any other nodes that have been added to the pipeline.

    images/download/attachments/131840250/image2022-4-22_8-57-5-version-1-modificationdate-1686580196000-api-v2.png

    There are four types of nodes available.

    1. Source Node: All pipelines must have a source node. This node injects the data into the pipeline (the input of data).

    2. Sink Node: All pipelines must have a sink node. This node extracts the resultant data from the pipeline (the output of data).

    3. Script Node: This node performs a user-defined action on the data received from the previous node(s) in the pipeline.

      1. The script must contain "def executeNode(inputs):", which works as a hook point to call this script.

      2. User can read the file(s) uploaded in the 'File Manager' using the path './{filenameWithExtension}'.

    4. Prediction Node: This node takes the input data and uses the selected trained machine learning model to make the prediction.

  5. Click the Node List icon.

    images/download/attachments/131840250/image2022-3-30_10-44-26-version-1-modificationdate-1686580196000-api-v2.png



    The Node List slideout appears.

    images/download/attachments/131840250/image2022-3-30_10-43-46-version-1-modificationdate-1686580196000-api-v2.png



Adding a ML Prediction Node

  1. Click the + (plus sign) icon for the node for ML Prediction.
    The new node appears on the pipeline editor screen.

    images/download/attachments/131840250/image2022-3-30_10-47-39-version-1-modificationdate-1686580196000-api-v2.png



  2. For each node, click the icon with three vertical dots and select Properties to configure that node.
    The Node Properties popup window displays. These fields vary according to the node type chosen.

    images/download/attachments/131840250/image2022-3-30_10-51-57-version-1-modificationdate-1686580196000-api-v2.png



  3. Complete the following fields. Fields with an asterisk ( * ) are required. Note that these fields vary according to the node type.

    Tab

    Field

    Description


    * Node Name

    Enter a name for the node.


    * ML Model

    Click the search icon to select a machine learning (ML) model.

    Inputs

    * Port Name

    Enter an input port name.

    * Port Type

    Select an input type from the dropdown list.

    Outputs

    * Port Name

    Enter an output port name.

    * Port Type

    Select an output type from the dropdown list.

  4. Click the Save button.
    The ML Pipeline Editor screen reappears with the node properties updated.

    images/download/attachments/131840250/image2022-4-4_9-14-20-version-1-modificationdate-1686580196000-api-v2.png



  5. Repeat until all desired ML Prediction nodes are added.

Adding a Script Node

  1. From the Node List slideout, click the + (plus) icon beside Script.
    A new Script Node is added.

    images/download/attachments/131840250/image2022-4-22_9-13-4-version-1-modificationdate-1686580196000-api-v2.png



  2. On the new script node, click the icon with three vertical dots and select Properties to configure that node.
    The Node Properties popup window appears.

    images/download/attachments/131840250/image2022-4-22_9-15-10-version-1-modificationdate-1686580196000-api-v2.png



  3. Complete the following fields. Fields with an asterisk ( * ) are required.

    Field

    Description

    * Node Name

    Enter a name for this node.

    Is Join

    Select to indicate this node acts as a merging point for multiple arcs. This node will be executed even if any of the input arcs(s) are executed successfully.

    * Language

    Select the programming language from the dropdown list.

    * Script

    Click the pencil icon to edit the script. See the next step for more information.

    Input tab (in Ports section)

    Click the plus and minus icons to add and remove input ports.

    Output tab (in Ports section)

    Click the plus and minus icons to add and remove output ports.

  4. If desired, click the pencil icon in the Script field, and the Edit Script popup window displays. Edit the current code and click Save when finished.

    images/download/attachments/131840250/image2022-4-22_9-27-57-version-1-modificationdate-1686580196000-api-v2.png



  5. Repeat the process until all script nodes are added.

  6. If a file needs to be uploaded for a script node, click the file folder icon in the top right.

    images/download/attachments/131840250/image2022-4-22_9-32-22-version-1-modificationdate-1686580196000-api-v2.png



    The File Manager slideout displays.

    images/download/attachments/131840250/image2022-4-1_4-27-16-version-1-modificationdate-1686580196000-api-v2.png



Testing the ML Pipeline

  1. Once all nodes are added, click the icon in the top right corner to enter values to run a test of the ML pipeline.

    images/download/attachments/131840250/image2022-4-1_4-16-14-version-1-modificationdate-1686580196000-api-v2.png



    The Test Run Pipeline slideout displays.

    images/download/attachments/131840250/image2022-4-1_4-17-24-version-1-modificationdate-1686580196000-api-v2.png



  2. Enter test data for any field desired.

  3. Click the Run Test button.
    The test results appear on the ML Pipeline Editor page. Note that the results shown will depend on the test data used.

    images/download/attachments/131840250/image2022-4-1_4-20-18-version-1-modificationdate-1686580196000-api-v2.png



  4. In the Execution Sequence column in the Node Logs section, click any node to see the detailed results from the test on that node.
    The Node Inputs, Node Outputs, and Node Log display.

    images/download/attachments/131840250/image2022-4-1_4-24-10-version-1-modificationdate-1686580196000-api-v2.png



  5. Click OK to return to the ML Pipeline Editor screen.

  6. Click the magnifying glass icons on the bottom left to zoom in and out for the desired view of the pipeline.

  7. Click the lock icon to lock or unlock the current pipeline view.

  8. Click the four arrows icon images/download/attachments/131840250/image2022-4-1_4-31-11-version-1-modificationdate-1686580196000-api-v2.png to view the entire pipeline.

  9. Click the auto-arrange icon to automatically arrange the nodes for a more organized view.

    images/download/attachments/131840250/image2022-4-4_9-16-14-version-1-modificationdate-1686580196000-api-v2.png



  10. A Minimap is displayed at the bottom right corner of the pipeline. The Minimap displays the entire pipeline, enabling the user to easily navigate to the part of the pipeline they want to view and to see which part of the pipeline they are currently viewing. This is especially helpful when working with large pipelines.

    images/download/attachments/131840250/image2022-4-22_9-38-27-version-1-modificationdate-1686580196000-api-v2.png



  11. Click the Save button to save the pipeline.
    The ML plug point screen displays with the pipeline updated.