Serializing data from IoT nodes

This is a followup to my previous post about my home automation/IoT system. I introduced some of the hardware used to build sensors and now I will look at an architecture for sending data from sensor nodes to a receiver. First I will explain the initial attempt at sending data from nodes and why that was destined to fail. Next we will look at an architecture for data serialization and quick node prototyping.

 

Doomed for failure

The first attempt at sending data was quite simplistic with data structures being sent through the air. The base structure looked like this:

typedef struct {
 unsigned char type;
 unsigned char seq_no;
} branly_report_t;

From this struct, holding a message type and a sequence number, I defined others for different kinds of sensor data, eg. temperature:

typedef struct {
 branly_report_t br;
 int temp;
} temp_report_t;

sensor battery voltage:

typedef struct {
 branly_report_t br;
 unsigned char vcc;
} battery_report_t;

plant moisture:

typedef struct {
 branly_report_t br;
 int temperature;
 unsigned int moisture;
 unsigned char alarm;
} plant_report_t;

and so on. Whenever I thought of a new kind of node I would implement a data structure for the data being transmitted. The receiver would look at the sequence number to skip duplicate transmissions and, most importantly, the type field to know what kind of data was being sent.

 

This is a horrible way of doing things.

 

Each new node type being built would require a copy paste job on both the node side (copy a previous implementation and make changes) and one on the receiver side (again, copy previous implementation and make changes). The receiver would of course require knowledge of all different branly_report_t types.

An actual architecture

The solution? Data serialization. Taking a step back makes one realize that we have do not want system with nodes reporting temperature, other nodes reporting plant moisture and so on. We want a system with nodes transmitting data. There is quite a difference between the two you see. Frankly we do not even care what kind of data nodes transmit until we want to act on it or look at it. Discussing this problem with a colleague I came up with the following model of nodes and contacts.

  1. A node has an identification number and one or more contacts.
  2. Each contact has an index number, 0..N.
  3. A contact has a predefined type that everybody agrees on.
  4. A contact may define a fixed reporting interval.
  5. A contact is readable and possibly writable.
  6. A contact may be violated, eg. a battery contact is violated if the battery voltage drops below a defined limit.

Any transmission from the node would be a reduced to a message with an updated contact property (contact identification number, contact value and flags for violations). A writable contact (eg. an RGB light) would receive messages with the same kind of data. Lots and lots of contact types can be defined and adding more further down the road is not as much of a hassle as the copy-and-past jobs mentioned earlier. Even more important, a node can report any data we want it to. My garage node could report inside and outside temperatures as well as the light intensity level (needed to know if I forgot to turn off the lights at night) without me cooking up some strange data structure only implemented once in one single node.

An example node

Before diving into how this idea of data serialization is implemented, let us look at an example. The Arduino node sketch below uses the RFM69 ISM radio to send data and has two contacts; battery voltage (reported every 1h 30min) and temperature (reported every 15min).

// Contact value getters
long temperatureValue(bool setValue, long value);
long batteryValue(bool setValue, long value);

RFM69 radio;
BranlyNode node(&radio, HW_VERSION, SW_VERSION);
BranlyContact battery(1, kTypeVoltage, batteryValue, k1Hour+2*k15Minutes);
BranlyContact temperature(2, kTypeTemperature, temperatureValue, k15Minutes);

long batteryValue(bool setValue, long value)
{
 (void) setValue; (void) value; // Contact is not writeable
 return 2940; // Sample data
}

long temperatureValue(bool setValue, long value)
{
 (void) setValue; (void) value; // Contact is not writeable
 return 245; // Fixedpoint, 24.5deg (sample data)
}

void setup()
{
 radio.initialize(NODE_FREQ, NWK_ID, NODE_ID);
 radio.sleep();
 radio.encrypt(CRYPT_KEY);
 battery.setLowerThreshold(2400); // Contact violated if VCC drops below 2.4V
}

void loop()
{
 node.run();
}

Yup, it does not take more code than that to create the full node software stack. Define the node, its contacts with contact getters (and setters if contact is writeable), define reporting intervals and call node.run();

Nodes

A node has, besides its contacts, version fields for hardware and software and an internal state (refer to BranlyNode.cpp). Upon boot the node step though its mCurState state variable until it has synchronized with the receiver. It will go through the following steps, requiring an ACK from the receiver in each step:

  1. send a ping packet and wait for a response from the receiver.
  2. send a hello packet with HW/SW versions.
  3. send a complete list of its contacts.
  4. send a complete contact report.
  5. wait until the next reporting interval and send a report on that specific contact

The receiver requires no configuration in order to receive data from a node. Instead, the node will tell the receiver what kind of data it will transmit.

Contacts

A contact (see BranlyContact.cpp) has an index and type as mentioned as well as a getter function and optional reporting interval. Contacts may be “enqueued” meaning its report will be transmitted on the next call to node.run. A button contact will typically be enqueued from an interrupt handler.

The radio protocol

The radio protocol has beed designed to be minimalistic to save power (critics may say the ping packet is not strictly necessary) and the heavy lifting is done in BranlyProtocol.cpp.

Each contact in the contact list (see buildContactListPacket) occupies a single byte, 1 bit for writeable, 3 bits for index and 4 bits for type (currently limiting the number of contacts to 8 per node and contact types to 16 in all, probably needs to be increased).

In the contact report (see buildContactReportPacket), contacts are encoded in 2-5 bytes with a flag field (for tracking violations), a size field telling how many bytes are needed to encode the current contact value and the actual value (1-4 bytes).

Lastly, contact value reports (see buildContactValuePacket) also uses 2-5 bytes and are sent in the same manner as contact reports.

The receiver

The receiver reads frames tom the BranlyPi modem running this receiver sketch. On the host side, the receiver is written in python and posts the received data to an Emoncms installation. The beauty of it all is that the receiver can create the Emoncms feed for each contact for a node on the fly based on the contact list packet.

 

 

Closing remarks

The current architecture has really enabled me to quickly prototype new nodes without changing a single line of code in the receiver and very few lines of code are needed to create a new RF node. On the receiver side, names need to be added for nodes and contacts enabling automatic bridging between RF node updates and, say, MQTT. Martin Harizanov is working on something similar and it will be exciting to see what he has come up with. If you know other implementations on this topic, please let me know in the comments below.

 

Meet the Branly IoT platform

These days it seems everybody is tinkering with their own IoT project, I am no different.

It started almost two years ago when I, while looking to build RF nodes, stumbled across the blogs of Nathan ChantrellMartin HarizanovJean-Claude Wippler and Felix Rusu. These guys had built exactly what I was looking for and Nathan even offered the gerbers needed to order Tiny328 PCBs. Tiny328 is an Arduino based RF node fitted with the RFM69C ISM radio. My experience with soldering was limited and my experience with designing PCBs was none, I am a software guy after all. It turned out that reflow soldering PCBs with 0603 components in a hot plate is quite easy (and fun!) once you get the hang of it. Google is your friend.

My fist baby steps in designing PCBs was an Arduino based modem with the RFP69 radio and though hole components, the BranlyPi v2 (v1 was without an external oscillator which caused the ATMega’s buadrate to be off leaving the modem and the RaspberryPi unable to talk).

It was superseeded by the v4 (don’t ask about the v3) which was surface mounted components all the way. Schematics and gerbers are here if you want to build your own.

With all that hardware available it was time to write some software and make two Tiny328s talk. The “hello world” was LowPowerLab‘s “Struct send” and “Struct receive” examples. I soon attached a DS18B20 temperature probe to one of the nodes for some more fun (sic!) data. Software for the BranlyPi was written as well as a python script running on the RaspberryPi listening to the BranlyPi modem.

In all, I had the following setup:

  1. A node fills in data into a struct holding a type field. The types can be “temperature”, “battery voltage measurement” and so on. The struct is transmitted to the BranlyPi modem.
  2. The BranlyPi modem receives the struct and formats in into a textual representation that is sent to the RaspberryPi over UART, eg.
    T:node id:rssi value:temperature

    and

    B:node id:rssi value:battery voltage
  3. The gateway script on the RaspberryPi reads the line above from the UART and posts it to a local Emoncms installation.

The node was placed in my garage with the probe outside. I had (sort of) built my own RF thermometer. Cool! An identical node (with a different node id 🙂 was built and placed indoor. Next, how about building one of those moisture measuring plant nodes? Or one that could tell me if I had forgotten to turn off the lights in the garage at night? I had a feeling the RF network would grow.

So far all is well from a technical point of view. The current software architecture would however need a complete rewrite to be useful, more of that later. Source code and hardware docs on Github.

“Branly” you might ask? Édouard Branly was one of the early pioneers in wireless telegraphy.

My wifi ghost lamp

Sometime ago all wifi hobbyist projects where quite expensive. USB Wifi dongles could be found for a handfull or dollars apiece but something easily connectible to a micro controller was also “connected” with a hefty price tag. Then the ESP8266 surfaced…

Having just purchased a Pac Man ghost LED lamp on the Black Friday sale I quickly saw an application for the ESP8266. Nothing novel, nothing that hadn’t been done 100 times before but something fun.

 

Make the ghost green
A warranty waiting to be voided

The software part

Following in the MQTT footsteps of previous projects I decided against running a server on the ESP8266 listening to yet another protocol implementation. Instead the ghost would listen to an MQTT topic and light up accordingly. I also wanted it to be able to run preconfigured light shows. The ESP8266 runs tuanpmt’s MQTT implementation and listens to the topic ghost/led sending anything it receives to the Arduino which responds to the following commands:

  • #RRGGBB – set all LEDs to specified color
  • :nnRRGGBB – set LED nn to specified color
  • – update the LED ring with all LEDs set with the :nnRRGGBB commands
  • p01 – run “light show” 01
  • * – turn all LEDs off

The “anything it receives” means I can add functionbality on the Arduino without touching the firmware on the ESP8266.

The hardware part

The hardware consists of the following:

  • ESP8266-01 board
  • Arduino Nano
  • 16 LED Neopixel Ring
  • Level shifter
  • 3.3V regulator with buddy capacitors
  • 1A USB charger

I purchased an Adafruit Neopixel ring with the lovely WS2812 LEDs. To save some time I took an Arduino I had lying around to control the LEDs using the Adafruit library. The two systems talks over UART. The Arduino runs at 5V (the level at which it can control the Neopixel ring) and the ESP8266 run at 3.3V which means a level shifter needs to be placed between the two. You can find level shifters almost for free on eBay. The wiring is quite simple. The USB charger provides 5V for the Arduino, the Neopixel ring and the high side side of the level shifter. The regulator provides 3.3V for the ESP8266 and the low side of the level shifter. I threw everything together on a prototyping board, nothing will be visibe anyhow 🙂

Once wired up the ghost can be tested with the command

% mosquitto_pub -t "ghost/led" -m "#00ff00"

The hard part

The board sits on nylon spacers to be glued to the lamp base
The board sits on nylon spacers to be glued to the lamp base

Now for the hard part, mounting the hardware inside the ghost. The ghost case is made up of three parts. The dome, the base and a bottom plate where the electronics is attached. The bottom plate is attached to the base using screws and the base is glued to the dome. This leaves the screws on the inside meaning the ghost cannot be cracked open without causing permanent damage. Using a fine saw and a drill I removed the bottom plate, sanded the edges and rinsed the dome in water ro remove the plastic dust. The electronics is mounted on spacers on the bottom plate and I used silicone to reattach it to the dome. It stays in place and reopening should not be that hard.

More software

Controlling the ghost via the command line would not attract much venture capital so I set out to make an iPhone app for controlling the ghost. NKOColorPickerView and MQTTKit fit the bill and resulted in this neat little app I installed on my son’s iPad.

Make the ghost green
Make the ghost green

He was somewhat impressed and have changed the ghost color about three times to date which was in line with my expectations. I do have other plans for the wifi ghost, but that’s for another post.

Code available on Github

Final thoughts

In the time between finishing this hack and actually writing about it, Arduino on the ESP8266 has matured phenomenally so today I would skip the AVR. Actually I could replace all the hardware with the WifiPixels.

Gallery

Another Raspberry Pi powered Macintosh Classic

The Macintosh Classic is somewhat special to me as it was my first computer I used for other things than just games. Not suprisingly perhaps as the line games was somewhat limited although there where classics like Dark Castle, Apache Strike, Empire and Deja Vu. That aside, I have had a Mac Classic II in my study for years reminding me of where my career in IT started and it was time to pour some life into the old machine (click for hires images or see the gallery at the end).

Hello (again and again)
Hello (again and again)

The obvious choice was to use a Raspberry Pi. Placing Pis in old macs is by no mean a new idea but I wanted to make something different and above all make something as sturdy as the old mac. The newly born Mac should be able to ride the bus as my old Classic did at times meaning I could not just put loose hardware into the box. Things needed to be fastened. I set out with the following specification:

  • Raspberry Pi
  • TFT screen
  • Speaker
  • Internal USB hub
  • External connectors for USB and ethernet
  • Working programmer’s key and reset button
  • Working floppy drive ejector motor
  • External 12V power supply

I had found this 8 inch TFT screen on eBay (update June 5th 2017, I have found a much cheaper variant on AliExpress that by the looks of it is identical) but as you can see the frame has no mounting support. Adding the cabling and driver board with its adapter boards sums up to quite a mess. How do we mount this nicely inside the Classic? Plexiglass to the rescue! I cut out two sheets of plexiglass and placed the TFT screen between them. Glued piexes of plexiglass on the back sheet keeps the TFT screen from falling out. The different boards are placed on spacers mounted on the back side plexiglass sheet. The front and back sheets are held together usings screws.

An 8" HDMI display package
An 8″ HDMI display package

The final part was mounting the “screen module” inside the Classic. As I had thrown the old CRT screen out, it was only a matter of drilling the right holes in the plexiglass screen module and mount it the same manner the original screen was.

Mounting the display package
Mounting the display package

Next was the mounting of the Raspberry Pi. For this I reused the hard drive bay where I mounted a piece of plexiglass holding the Pi on spacers.

Pi, power and speaker
Pi, power and speaker

I wanted to bring back the yawning like sound of a Macintosh ejecting a floppy disk. The idea was to have the Mac automatically eject an inserted floppy with a delay. So how did the old Macs detect the precense of a floppy disk?
There are a number of micro switches sitting in the front of the floppy drive. These will be pressed (or not) when a floppy is inserted into the drive. From right to left they will tell us

  1. Is there a floppy present?
  2. Is is write protected?
  3. Is is a single sided or double sided floppy?
Adding blue and white wires for floppy presence detection
Adding blue and white wires for floppy presence detection

<side note>Floppy disks back in the days had a write protect switch on them consisting of a small piece of plastic that would open or close a hole. An open hole indicated that the floppy disk could be written to. Another hole in the floppy disk indicated if data would be written on one or both sides of the disc. Switches 2 and 3 above would be not pressed if the corresponding hole was present on the floppy disk.</side note>

For this project, I only cared about switch #1. Deciding the floppy drive would never see real action again, I disconnected the switch from the rest of the floppy drive PCB by severing the traces. Soldering wires to the switch and attaching them to the Raspbery Pi GPIO header, the Pi could now sense the precense of a floppy disk. Next was the ejector motor. It runs on 12V (as the TFT screen) and I purchased a relay on eBay that the Raspberry Pi could control.

On the left hand side of the old compact Macs was the programmer’s key and the reset button. The former would enter the debugger built into the computer. I wanted to connect these to the Raspberry Pi so once again I severed some traces. On the the motherboard this time.

For power, I purchased a 12V to 5V converter with quad USB output on eBay. This together with the relay was mounted on a sheet of plexiglass that was mounted on spacers in the back of the computer. The speaker was mounted in a large hole i drilled in (you guessed it) a piece of plexiglass mounted on (guessed it again?) spacers. As the sound quality on the original Raspberry Pi was quite poor I added a USB sound card (also from eBay). I desoldered the microphone connector on the Mac’s mother board and replaced it with a power jack that I connected to the 12V/5V converter. An old Western Digital USB disk power supply provies the 12V needed.

Last but not least, I added two external USB ports and an ethernet port. Both use passthrough cables found on eBay (search for “USB 2.0 A panel mount extension cable 25cm” and “ethernet panel mount extension cable 25cm“. They are mounted on the last piece of plexiglass that is glued to the inside of the case. I reused the holes where the original 220V power cable connector and power switch where located.

Backside USB and ethernet connectors
Backside USB and ethernet connectors

A simple python script checks the programmer’s key, the reset button and the floppy detection switch and controls the eject motor. Pressing one of the keys will play the lovely old Macintosh Quadra chime. A long press will shut the Raspberry Pi down.

Floppy ejection was a bit tricky. The eject motor is a simple motor and not a servo meaning you cannot tell it to goto a position and back like those servos you have been playing around with using your Arduino. The motor needs to return to (roughly) its original position or it will be impossible to insert a floppy again. I have no idea how this was accomplished back in the days but this simple algorithm (based in parts of my recollection of the old floppy sound) did the trick

  1. Start motor
  2. Wait for lost floppy presence
  3. Wait for 1 second
  4. Stop motor

I am quite pleased with how this mod turned out. There is nothing loose inside the case that can fall over, get tangled up and cause shorts. By accident, the Mac was drop tested from a height of one meter. It survived, nothing came loose.

I have some future plans for HW modifications including a touch screen, replacing the clicking mechanical relay with a transistor and I should add a fuse to the 12V line for safety.

Most of all, I need to add software to make the old Classic actually do something. But that is for a later post.

Update 26th of June 2016, the python script running the show is now on Github.
Update 5th of June 2017, added AliExpress link to cheaper display and eBay links to USB/ethernet passthrough cables.


Gallery

A wifi OLED display for my wife

Somehow, the cord to the external temperature probe got ripped the out of the temperature display in the bedroom. My wife insisted on us purchasing a new thermometer but as an engineer I immedietaly saw the opportunity to build hardware and write software. Spending lots of time and resources on “engineering joy” is not something one can do at work but in one’s sparetime the sky is (almost) the limit.

So instead of spending about a dozen Euros on a new thermometer I designed and built a PCB and wrote code which was a lot more fun and educational. And way more expensive 🙂 The result is the Wife OLED that displays the current outside temperature as well as the forecasted temperature in 8 hours. While the morning might be cold, it is nice to know what the temperature will be at lunchtime.

WifeOLED
Another ESP8266 MQTT OLED display, on the warmest day this summer!

As the OLED display would be always on and close to a power outlet I choose a wifi solution and went for the the ESP8266, more specifically the ESP-03 module for its small size. The OLED was found on eBay, is a mere 0.91″ large and has an active area of 128×32 pixels. The device runs from a USB charger and an SPX3819 LDO provides the 3.3V the ESP and OLED needs. I designed a PCB in Eagle and had DirtyPBCs.com produce it. As the PCB is quite constrained on size there is no full FTDI-style connector. The RX/TX pins and ground are the only ones available for programming and a small tactile switch is used to get the ESP8266 into download mode.l

Looking at the software side, it turned out most of the work had been done by Nathan Chantrell (and others). He used a 128×64 pixel OLED display and the driver seemed to originate from the Arduino OLED driver by Adafruit. As I had chosen a 128×32 pixel display the driver needed some work. Also the available fonts where too small to be readable from across the room so I rendered Ubuntu Condensed in 28px and 32px. The larger font will be used for displaying the current temperature and the smaller one for the forecasted temperature. The device subscribes to the MQTT topics `home/temperature/outside` and `home/temperature/forecast` and displays whatever arrives on those topics.

The current outside temperature comes from my setup with a Tiny328 node equipped with a DS18B20 probe and a Raspberry Pi gateway (a future post). The forecasted temperature is pulled from the open data feed of the Swedish Meteorological and Hydrological Institute (SMHI) using the script get-weather-smhi.py.

The HW schematics (and gerbers!) as well as the SW is found on GitHub.

A note one year later: The OLED display will start to deteriorate after about half a year. It will still be readable but the pixels get “worn out” and will not be as bright. I would not recommend soldering the display to the PCB for this reason. Nowadays these displays retail at €2.50 on eBay so a replacement now and then is acceptable.