Remote Telemetry Data Acquisition

A full stack remote data acquisition project monitoring temperature every 5 seconds and using a WiFi link, the remote telemetry unit (RTU) connects to an Internet based data acquisition server (could be MQTT or HTTPS) to register the unit id, temperature and the time of the transmissions from the RTU.

An Internet facing API is available to retrieve and consume services based on the registered data and make it available for monitoring systems like mobile applications to get the current temperature and an oversight of how it changed looking back in time, or be alerted when temperatures exceed defined settings.

Remote Telemetry Data Aquisition Diagram

Data acquisition

Temperature measurements A MCP9808 high precision digital temperature sensor from the USA based company Microchip. It operates between -40 and 125 °C with an accuracy of ±0.5°C

Data processing and dispatch

Microcontroller An ESP8266 is a WiFi based chip that has built in support for TCP/IP with impressive micro controller ability, made by the Chinese based Espressif Systems.

Online logging

Internet server service An online HTTPS API (or MQTT) has an Internet facing data logging interface that listens for inbound telemetry transmissions from the remote data collection units.

Data storage

Data storage A database registers and stores the incoming data via stored procedures and indexes data for efficient and effective on demand retrieval.

Data Processing

Internet server service Periodic tasks like calculating the average values per day. week, month and year are done by processes running on the telemetry receiving server.

Data Consumers

Data consumers Data consumer Example Outbound data feed processes, data consumer management as well as alarm and alert push notifications are dealt with via real-time outbound server processes.

External Data Processing

External Data Processing Secured unprocessed and processed APIs are made available for external systems to consume and use the data for own and internal application usage.

Application Feeds

Application Feeds Special compact preprocessed feeds of data are available to be consumed by portable and mobile wireless devices.

ESP8266

ESP8266 Small footprint WiFi module with programmable MCU

MCP9808

MCP9808 Compact precision temperature measurements

OLED Display

OLED Display A small low power OLED display was added to the remote telemetry collector to enable insight into whether it could and was connected to WiFi as well as whether the temperature sensor was operational.
Basically a pacifier for the installer.

Remote Telemetry Schematic

OLED Display

Notes:

Everything runs of a 3.3 Volt power source. If you want to run it of a LIPO cell, you will want to put a voltage regulator inline.

Provision is made for an optional OLED display. The MCP9808 chip should not be mounted close to the ESP8266 as this can affect the correctness of the temperature readings.

Pin4 and Pin5 of the programming connector need to be shorted out in order to enable flashing of the ESP8266 module.

Ideally each chip should have a 100pF capacitor located as close as possible to the chip between VDD and GND.

Source code:

C++ source code for the ESP8266. It should be self explanatory how it works, however do make sure you have all the used libraries available. Easiest way to get up and running will be via the Arduino IDE. Replace the WiFi specifications for your own as well.

Remote Telemetry Reception

The section above is about reading data sensors, formatting the data into JSON data stream and dispatching it to a remote server. This is about receiving data from remote telemetry sensors.

Internet facing services There are many ways to develop APIs like web services frameworks and SOAP, however these are much to complex and cumbersome for this purpose. Considering we already have a firewall and surrounding processes monitoring inbound traffic as well as using HTTPS as transport medium, a single class with an Internet aware interface will do just fine.

Inbound data handler and decoder class

Simple straight forward JSON data telemetry receiver and decoder. The complete class can be downloaded here.

telemetry receiver and decoder How it works: when requests to the API URI are received by the server, they are routed to this class for initial processing. The data stream from the request is read in (note the web server is monitoring for max data length and buffer overflow), checked if it is JSON data and then decoded according to the telemetry data construct (ENUM). Should the data decode correctly it is then offered to the database service for storage in the database. The response back to the sending party if a good telemetry data packet was received is an 'OK' for success or a 'NOK' for a failure to register the data.

Public Sub ProcessRequest(context As HttpContext) Implements IHttpHandler.ProcessRequest
            context.Response.Clear()
            Try
                '// Check for 'signature' in header of request, if found continue else report IP to interactive FW and drop request
                '// ....
                Dim strJson As String = New StreamReader(context.Request.InputStream).ReadToEnd()
                If Not String.IsNullOrEmpty(strJson) Then
                    Dim objMeas As MeasureInfo = JsonConvert.DeserializeObject(Of MeasureInfo)(strJson)
                    If DataService.PutLoggerData(objMeas.mac, objMeas.ip, objMeas.temp) Then
                        context.Response.Write("Ok")
                    Else
                        context.Response.Write("Nok")
                    End If
                Else
                    context.Response.Write("No Data")
                End If
                context.Response.ContentType = "text/plain"
                context.Response.Cache.SetCacheability(HttpCacheability.NoCache)
                context.Response.Cache.SetAllowResponseInBrowserHistory(False)
                context.Response.Flush()
                context.Response.Close()
            Catch ex As Exception
                context.Response.Flush()
                context.Response.Close()
            End Try
        End Sub

Data processing and forwarding

Secure server processes As the data receiver class is potentially connected directly to the Internet, isolating it from processes deeper in the systems is a smart idea. In order the get the checked and verified data into the database another (data) class that utilizes data base stored procedures is implemented. This prevents any direct access to the database from Internet facing processes. This is the call to

DataService.PutLoggerData(objMeas.mac, objMeas.ip, objMeas.temp)

Public Shared Function PutLoggerData(mac As String, ip As String, value As StringAs Boolean
        Dim ok As Boolean = False
        Try
            Dim ConnectionString As String = ConfigurationManager.ConnectionStrings("YOUR_DB_CONNECTIONSTRING").ConnectionString
            Using Connection As SqlConnection = New SqlConnection(ConnectionString)
 
                Using Command As SqlCommand = New SqlCommand("put_logger", Connection)
                    Command.CommandTimeout = 2
                    Command.Parameters.AddWithValue("@mac", mac)
                    Command.Parameters.AddWithValue("@ip", ip)
                    Command.Parameters.AddWithValue("@value", value)
                    Command.CommandType = CommandType.StoredProcedure
                    If Connection.State = ConnectionState.Closed Then
                        Connection.Open()
                    End If
                    Command.ExecuteNonQuery()
                    ok = True
                End Using
            End Using
        Catch ex As Exception
            Errorhandler.ErrorHandler(0, ex.ToString)
        End Try
        Return ok
    End Function

Database Storage

Database To store the telemetry data, at this point two things are needed. A database table to store the information and the stored procedure to put the data into the data table. In this instance a MS SQL backend server is used located on the local LAN network.

Data table creation script

Set ANSI_NULLS On
GO
Set QUOTED_IDENTIFIER On
GO
CREATE TABLE [dbo].[logger](
	[id] [bigint] IDENTITY(1,1) Not NULL,
	[mac] [nvarchar](17) Not NULL,
	[ip] [nvarchar](39) Not NULL,
	[productid] [int] Not NULL,
	[value] [nvarchar](50) Not NULL,
	[date] [datetime] Not NULL
) ON [PRIMARY]
GO
ALTER TABLE [dbo].[logger] ADD
CONSTRAINT [DF_logger_productid]  Default ((0)) For [productid] GO ALTER TABLE [dbo].[logger] ADD
CONSTRAINT [DF_logger_date]  Default (getutcdate()) For [date] GO
 

Stored procedure generation script

Set ANSI_NULLS On
GO
Set QUOTED_IDENTIFIER On
GO
CREATE PROCEDURE [dbo].[put_logger]
	(
	@mac nvarchar(50),
	@ip nvarchar(50),
	@value nvarchar(50)
	)
AS
BEGIN
Set NOCOUNT On;
INSERT INTO logger
                         (mac, ip, value)
VALUES(@mac,@ip,@value)
End

Data example

Database Example of logged data as stored in the data table on the database.

The 'id' column is the primary key with a self incrementing number. This way we can guarantee that if the numbers are consecutive, the database is intact and no data has been removed.

The date is populated with the UCT date time of the data record entry into the database. If it is important to register the time the data value was actually measured, then that will need to be sent from the remote telemetry unit.

The MAC and IP addresses serve to identify from which unit the data transmissions origionated from.

The product id, not used at present, is to associate the measured reading to a particular product, location, whatever.

As time goes by and more data is collected other processes can start using the historic data, like daily, weekly, monthly and yearly averages with the ability to compare current values to historic values, analyse the differences and determine trends.

Processed telemetry data output

Telemetry output Making the received and stored telemetry data available for use by external parties like mobile apps, web services are used again, but this time instead of inbound data streams they are going to be outbound. Examples of output data:

  • Get current data value.
  • Get last ten measurements.
  • Get average for time of day for week
  • Get average for today
  • Get trend (up or down)
  • Get chart data for period
  • Setup and subscribe to push notifications
  • .....

Live api pull example

View actual last 10 received telemetry values

Data consumer raw output API

0   
mac "80:7D:3A:6E:68:55"
ip  "172.20.22.180"
productid   0
value   "27.38"
Date    "2022-08-20T10:44:28.303"
1   
mac "80:7D:3A:6E:68:55"
ip  "172.20.22.180"
productid   0
value   "27.37"
Date    "2022-08-20T10:44:14.35"
2   
mac "80:7D:3A:6E:68:55"
ip  "172.20.22.180"
productid   0
value   "27.31"
Date    "2022-08-20T10:44:00.62"
3   
mac "80:7D:3A:6E:68:55"
ip  "172.20.22.180"
productid   0
value   "27.30"
Date    "2022-08-20T10:43:46.9"
Actual 30 day temperature data.

Real world application

Near real time track and trace system for vehicles using location and data telemetry for monitoring and management. This system uses GPS (Global Positioning System) for location detection, an accelerometer for measuring gravitational forces and has an interface to the vehicle CAN bus. Telemetry dispatches are determined by predefined conditions such as distance, change in rotation angle, accelerometer values and time. When any of the dispatch conditions are met, telemetry is dispatched via cellular networks to a telemetry endpoint server. Here the data is checked, decoded and processed before sending it to a data base server for storage and later processing.

Processed telemetry overview

Telemetry trip reporting Collected data is processed and stored, from here reporting and analytics can be undertaken to produce information about the whereabouts of the vehicle as well as fuel use, speed limit adherence, max and average speed for a trip, the amount of CO2 produced and how long the trip took from star address to stop address. From this overview it is possible to drill down on various options, like where exactly were the speed limits exceeded, or where was the sharp corner that registered .57G. The option to view the trip on the map showing where and how the vehicle drove from point A (start trip) to point B (end trip)

Telemetry trip reporting Advantages
  • The distance the vehicle traveled determined by the odometer at the start and end of the trip, using GPS telemetry point locater coordinates to determine distance is not accurate enough.
  • Fuel consumption. Accurate fuel consumption from the vehicle motor management system ensures correct fuel readings as these can vary per trip depending on weather conditions, terrain, and the (heavy) foot of the driver.
  • Time management. How long it took to get from point A to point B. It could be there was a traffic jam or roadworks, making it difficult to predict when a vehicle will arrive at it's destination. Real time telemetry allow insight into where the present location of the vehicle is as well as its speed.
  • Register points of interest, like speeding in a sharp corner (the G-forces), Abnormal acceleration (pulling away with screeching tires) or standing hard on the brakes to avoid an accident and last but not least registering an accident as it happens.

Telemetry mapping overview scope

Telemetry units mapping scope

Fast and effective overview of all telemetry units with easy identity and physical location projected on a map. The ability to select a unit and see its trip progress from where it started the current trip. Also having the ability to show all transmission points for said unit for the complete day to date.

Telemetry trip points

Telemetry trip points

Show interactive trip point transmissions. These are colour coded and can have icons. The colours indicate whether the situation is normal and within all set boundaries (green), if boundary settings are reached or slightly higher (orange) or if boundaries have been exceeded (red), like speeding more than 5 kilometers above the set legal limit.

Telemetry data details

Telemetry data details

Each received telemetry data packet can contain a host of information and is determined by the use and value of each data item in the telemetry message constructor. In the case of the above vehicle a after market product called ECOdrive* has been installed in the vehicle. The purpose of the ECOdrive is to monitor driver driving practices and assist in saving fuel and on maintenance, like aggressive throttling or tailing a forerunner. It is a very smart device that saves of fuel use, extends tire life and saves on maintenance. It also allows for insight in the motor management system of the vehicle and can supply a wealth of information on the actual state of the vehicle.

* I work for this company and built this system based on Microsoft Server, SQL server, IIS, ASP.NET, JSript and C#.