Message Queues, Messages and Polling

The One Network Platform can receive files from an external system using many different protocols and formats. Possible message sources include:

• HTTP/S

• Polling the Filesystem

• Polling an external SFTP site

• Direct submission from another Platform instance (aka "P2PI" or "Platform-to-Platform-Integration", a process that occurs over HTTPS)

Processing of messages is done asynchronously. This means clients can drop a large message, get an immediate response when the message is delivered, and then check on the processing status later. To enable asynchronous processing without loss of data in the event of a power outage or other event, Platform persists all inbound Messages in Message Queues. The Messages are stored in the database, and the queue guarantees the order of processing is guaranteed to be first-in-first-out.

In this tutorial section, we will create an Inbox queue to allow a Book CSV file to be dropped and processed. We will setup file polling to assign the file to an interface and queue when it is created.

To begin our exercise, let's assume we have a legacy Enterprise system that contains a catalog of books. It is capable of producing a CSV file containing the details of any book added to that catalog each night, and dropping them into a directory. We will setup polling in Platform to watch for that file to be dropped and add it to a message queue.

Let's first setup the MessageSource. A MessageSource identifies a location from which Messages can be acquired. We will give it the following properties:

• We will name it "BookstoreFileDrop" (this allows us to easily identify it later)

• We will use the Type "File"

• We will specify in the Config that the source directory for the files is C:/PLT/inbox/bookstore

Create a new file in the Bookstore_dataset/data directory called "MessageSources.csv" with the following contents:

#* Name,Enterprise Name,* Source Type,* Config
BookstoreFileDrop,,File,{ srcDir: 'C:/PLT/inbox/bookstore' }    

Now, submit the file using the interface PLT.MessageSource_IB, version 1.0, and username BookstoreAdmin. (This is an interface that is available as part of the Platform core, thus the "PLT" prefix.) You should get confirmation of successful submission.

Next, let's create a MessageQueue. This is the actual queue into which the Message will be placed after it's picked up from the filesystem. We will give it the following properties:

• We will name it "inbox/Bookstore" (a good practice is to prefix inbound queues with "inbox" and outbound with "outbox")

• We will indicate its Classification as "Inbox", meaning it processes inbound messages

Create a new file in the Bookstore_dataset/data directory called "MessageQueues.csv" with the following contents:

#* Name,Enterprise Name,Classification,"Exclusive Consumer (0 or 1 (0=false, 1=true))","Paused (0 or 1 (0=false, 1=true))"
inbox/Bookstore,,Inbox,1,0    

Submit this file using the interface PLT.MessageQueue_IB, version 1.0. (Please keep in mind that the MessageQueue has no relationship to the "C:/PLT/inbox/bookstore" message source yet.)

Once this is loaded, we can actually log in and see this MessageQueue. Log in to NEO as InstanceAdminUser. You will see a menu item called Message Queues under Integraton menu - open that page then provide required filter "Message Created After" date value(Select Today).

You may actually have several queues, but at a minimum you should see the new inbox/Bookstore queue. We can use this page to view pending and processed messages, pause processing, or reprocess old messages.

Finally, we need to set up polling. By creating a MessageSourcePoll, we can tell Platform that "when a file comes into a directory whose name matches a certain pattern, put it in a particular queue and process it with a particular interface." Specifically, we'll set ours up as follows:

• We will name it "BookLoad", just for reference purposes

• We will tell it the directory to poll by referencing the "BookstoreFileDrop" MessageSource we created earlier

• We will grab all Files matching ending in "Book.csv"

• When we find a matching file, we will put it in the inbox/Bookstore queue

• When we find a matching file, we will process it with ZBKS.BookLoad version 1.0

Create a new file in the Bookstore_dataset/data directory called "MessageSourcePoll.csv" with the following contents:

#* Name,Enterprise Name,Message Source Name,Message Source Enterprise Name,Group Name,"Precedence (#,###)",Include Expr,Exclude Expr,Inbound Queue Name,Inbound Queue Enterprise Name,Inbound Interface,Inbound Interface Version
BookLoad,,BookstoreFileDrop,,BookstoreGroup,0,.*Book\.csv,,inbox/Bookstore,,ZBKS.BookLoad,1.0

Submit this file using the interface PLT.MessageSourcePoll_IB, version 1.0

Now that we have created our MessageSource, MessageSourcePoll and MessageQueue, we're ready to start dropping files. Let's create a new file Book.csv somewhere other than the polled directory with the following contents:

#* Title,Published Date (MM/dd/yyyy),"Rating (#,#.#####)"
The Mythical Man Month,3/01/1975,8   

Now copy it to the polled directory. Navigate to C:/PLT/inbox/bookstore in a Windows explorer. (Platform should have created it for you by now!) Copy/paste your Book.csv file into this directory.

Within a few seconds, Platform should pick up the file, after which it should disappear:

Now if we return to the Message Queues page, we should see that 1 file was Processed from the inbox/Bookstore queue, and a new message was created in the outbox/Bookstore/success queue (indicating that processing was 100% successful).

But what if there are processing errors? Try deleting "The Mythical Man Month" from Book.csv and dropping the file again. This time, after the file is picked up, you will see a new queue outbox/Bookstore/error with a new message in it. Click the queue name to see the error message, then click Download to get the details of the error.

This should help you to understand the basics of message queues and file polling. Here are a few additional troubleshooting tips:

• If your file isn't being picked up, try turning on DEBUG logging for polling. Right-click on any Java class in the Outline tree, choose the Logging menu, Shortcuts, and File Enqueue/Dequeue -> DEBUG. This will print more detailed logs to the console explaining which directories are polled, which files are considered as candidates, etc.

• When polling for large files, a best practice is for the client to copy the file using one name, and then rename it after the copy is complete. This helps ensure the file isn't picked up halfway through the copy operation. For example, you might ask your clients to copy with an extension of .csv and then rename to .csv.ready when done. Then you can setup your polling on .csv.ready only.

• If the order in which messages is picked up is important, you can use the "Group Name" and "Precedence" attribute of MessageSourcePoll to identify groups of interdependent files, and the order in which they should be polled.