rtndf (for rtn data flow) is a set of pipeline processing elements (PPEs) that can be used to create streaming data flow processing pipelines. rtndf uses Manifold as its underlying network infrastructure. PPEs can include machine learning elements based on TensorFlow or any other ML engine.
PPEs can be written in Python and C++ currently. Python is fine where other libraries are doing any heavy processing and makes it quick and easy to add new PPEs. C++ is better when heavier custom processing is required.
The Pipeline Processing Elements (PPEs)
- uvccam. UVC-based USB webcams can be used to generate MJPEG video streams using this element.
- picam. A Raspberry PI fitted with a PiCamera can be used to generate an MJPEG video stream using this script.
- audio. This element streams PCM audio.
- imu. Streams data from an IMU (e.g. Raspberry Pi fitted with SenseHat).
- sensehat. Streams all sensor data from a Raspberry Pi with SenseHat.
- sensors. Streams sensor data from selected environmental sensors.
- imageproc. A video pipeline element that uses TensorFlow to perform simple image processing – processing a half-sized image with a 3 x 3 Laplacian.
- modet. A video pipeline element that implements a motion detector that draws a box around objects in motion.
- recognize. A video pipeline element that tries to recognize objects in the stream using Inception-v3.
- facerec. A video pipeline element that tries to recognize faces in the stream using OpenFace (https://cmusatyalab.github.io/openface/).
- speechdecode. An audio pipeline processor that uses CMU Sphinx to decode words and phrases in the stream.
- avview. This script can be used to display video streams and play audio streams.
- imuview. An OpenGL PPE that can display being streamed by the imu PPE.
- sensorview. Display sensor data generated by sensehat and sensors.
PPE Message structure
All pipeline messages are encoded using JSON. The basic message includes standard fields from rtndfdefs.py, a metadata entry and the data fields.
See Python/rtndf/metadefs/rtndfdefs.py for more information.
All messages include a standard header that include the following fields:
* TIMESTAMP - the time at which the message was generated (in seconds, float). * DEVICEID - a string indicating the unique ID of the source of the message. * TOPIC - a string indicating the topic to which this message belongs. * TYPE - the data of data in the Data field. * METADATA - this is a JSON object that contains entries from PPEs that have added information to the message.
The PPEs can insert extra metadata based on their processing into the METADATA field. These are described in the JSON defs file for the PPE in the rtndf/metadefs directory.
The Data field contains the actual data part of the message, if there is one. The format of this field depends on the type of data. The formats for this field are described in the Python/rtndf/datadefs directory.