For my (home) automation and other planned projects I am looking for simple means of communication between various device nodes, sensors and actuators. These will mostly be based on small microcontrollers (like Atmel's 8-bit AVR, ARM Cortex or similar MCU's) with a small amount of memory. More details on these MCU-driven nodes (or should I say 'IoT' devices) will follow as I start digging into the details of these microcontrollers and do some designing and building along the way. But for now I want to have a simple communication protocol for tiny devices with as little as a few KB of memory, and connected through different wired or RF links.
Note: I will try to avoid fashionable words or abbreviations like IoT and Internet of Things as much as possible - I promise...maybe... ;o)
There are many software and hardware solutions on the market, that could probably fulfill some, if not most, of my home automation needs. But that may come at a high price, and/or be limited to certain means of communication, like only wireless or only through I2C, RS485 or KNX, etc. And I may need specific form factors to fit in a small space, for instance. To have the flexibility to pick and choose the right means of communication, based on the location of the device or sensor node, the proximity of a power source, the wiring, walls blocking a signal, etc., I need a solution that can indiscriminately communicate over wire, both RS485 and 230V power lines, or an RF frequency, including even LoRa (more on that maybe in a future blog post). For specific sensor nodes I will also use 1-Wire, but only local from a sensor to a controlling node or hub. Besides, what’s the fun in just buying stuff, when I can have a lot of fun with my oscilloscope, soldering iron and firmware programming tools?
Over the coming months, and probably for the next year or so, I will describe the steps I take to design and build the various components, both hardware and software. Along the way I may wander off from time to time and discuss somewhat related topics that come along, like PCB design, software tools and other topics that attract my attention (or get me distracted).
The making of ...
Today I will go in to some of the thinking I have done on a simple protocol to communicate over any (unreliable) serial link, wired and optionally also wireless. But before I do that, let’s have a look at the “bigger picture” of building a stack from the hardware to the application interfaces to use the sensor information and control the nodes. One of the challenges you face is to build a protocol stack that is both flexible and modular, and at the same time very efficient in terms of memory and CPU consumption. It should also work on small 8-bit MCU’s with as little as 4K or 8K of Flash memory (some even have less memory) and no more than 1K or 2K of RAM, again some devices have even less RAM.
One of the candidates that could somewhat suit my needs is the RadioHead protocol (offspring of the VirtualWire protocol). But that is a little bit too complex for basic sensor communication and would not easily be ported to a tiny 8-by MCU with only a few KB of memory. For the design of my protocol stack I have been inspired by various existing protocols while keeping it simple and easy to handle by small or even tiny 8-bit MCU’s.
My first attempt at visualizing the desired protocol stack is shown in the diagram below.
Let’s have a closer look, starting at the bottom with the black layer. Each type of communication device has its own hardware (and often built-in firmware) to deal with the specifics of the ‘physical/electrical’ signal level. Can be quite simple, in the case of RS485, or very complex, with LoRa for instance.
The blue layer on top of the bottom layer is the low level driver for each of these communication devices that takes care of communicating with the hardware, handling interrupts, and bus or collision aspects. The latter could be part of the hardware/firmware in some cases, and must be dealt with in software in other cases. Some of these hardware/firmware devices do some kind of error checking, but we will not solely rely on that in our protocol stack, as not every device/interface is on the same level. The combined efforts of the black and blue layer result in a more or less universal interface towards the layer above it: streams of bytes (packets as they are usually called) to be sent down the line or received from the (virtual) wire.
The actual error detection of packets and some of the handling of it is done at the Serial Driver level (orange layer). Also we detect missing or incomplete communication flows as far as an acknowledgement of received packets is concerned. So when we send a packet, we expect an acknowledgement (within a certain period of time) of reception, a response to the command, or an error response, if the packet didn’t make it in one peace. Any other type of response is passed on to the next higher layer, like an error to indicate a command could not be executed. As we are dealing with very different types of sensors and devices, we have to come up with a scheme that works well both for a simple temperature sensor and a complex alarm system for instance.
The more complex aspects of handling the communication with the nodes, including the command-reply sequences and whether or not a response is actually expected (as opposed to just a simple acknowledgement), is partially handled at the master and slave library level and partially on the application level that uses these libraries. Almost all communications over any type of (virtual) serial bus involves a master that is sending some command or data and a slave that is responding to the command or dealing with the received data. The master is always the initiator of the communication, so nodes must switch roles in certain scenarios.
There are basically three types of communications that somewhat overlap in functionality: a basic serial/bus type protocol we are designing right now, targeted at simple communication between sensors/actuators and intermediate nodes; a mesh type protocol (usually wireless) that uses a network of nodes to communicate via any node to other nodes; and a fully routable protocol that can support internet connectivity, TCP based. Over time, we will start looking at all these topics, as need arises.
The bus master
A key concept of any form of serial bus communications is the master/slave concept. Well know examples of serial bus types are for example I2C, 1-Wire, LIN, CAN or RS485. At any given moment in time, only one device can talk on the bus, otherwise there would be a collision. All devices can listen at the same time and they receive the exact same information. By using a form of addressing in the data sent, the device that is addressed knows it is meant for him. Let’s look at a simple example: A temperature sensor is constantly measuring the temperature, and may log values every x seconds or minutes, but in the serial bus world it can never send the temperature value(s) on its own, as no one may be listening at that moment and it could cause an unwanted collision on the serial bus if another temperature sensor would start sending its data at the same time. One of the basic characteristics of (most types of) busses is that only one device can ‘speak’ at any given moment in time. Otherwise the signals would interfere. The bus master is the one that controls who get's to speak when. By sending a message to a device on the bus that somehow indicates that the device should send an answer, the bus master knows that something will be put on the bus next. It will start listening right after the message is sent. As a result, many implementations don't allow direct slave-to-slave communications, although the bus would allow it if a suitable protocol can be used.
The 'bus master/slave' concept does not support communication from sensors that must send a signal on their own, like a switch or an alarm, without waiting for a bus master to ask for their status. Of course, I can think of some kind of polling mechanism for not to critical input nodes. However, pressing a button and having to wait a full second or more for the lights to come on, can be quite annoying. I will deal with that kind of event driven communication later. Baby steps, bay steps... ;o)
Finally, we will put a very fashionable API layer on top of that to provide a standardized means of dealing with the rest of the world, like the application(s) that provide the logic of switching things on or off based on some sensor input or other external event like time of day. We could add that to the core Master/Slave routines, but it would make it more complex and less flexible, not to mention increase the memory footprint. Of course we have to be careful not to ‘over modularize’ as that would have negative impact on the CPU and memory footprint as well.
Before we actually start designing things, let's define some ground rules and requirements for the simple serial communications protocol (or SSCP).
- Suitable for small 8-bit MCU's Since I intend to use the smallest and cheapest types of microcontrollers (I stick with Atmel AVR for that), low memory footprint is essential.
- Simple and robust error handling Arguably the most important part of any protocol is the error handling and (optionally) correction. Straight forward error-free communications are not a big issue in most cases, it's the error checking, correcting, retry, synchronization and other actions needed for a reliable communication across an unreliable wire (e.g. RF signal with lots of interference or very weak at longer distances). To keep things simple, the basic communication layers focus on error checking and some minor correction. Things like retry are left to the higher levels in the stack.
- Physical interface independent protocol The protocol must be the same on all types of interface, at least from the API level This allows me to use the same software on devices with different interfaces. For instance, a sensor can have an RF interface in one situation and an RS485 interface in another place. These sensors are then driven from the same software, only the lower communication libraries must be replaced.
- Targeted at short command/response type messages The protocol is designed for sending simple commands with parameters, and responses. Large datasets are not supported and neither is any kind of streaming.
Let's take it one step further and make some key design decisions to fulfill the requirements.
- Packets are limited to a maximum of 32 bytes of actual data (the payload), excluding any control bytes, so we can deal with memory limits, and line speed and reliability constraints. Given most home automation scenarios, that should not pose any limitation on functionality. One of the reasons behind this is directly related to RAM constraints on the smaller Atmel AVR8 MCU’s, which don’t allow allocation of large buffers for storing received data of variable length. The calling routine (through the API layer) can specify a shorter packet length if it is expecting only a short response message. If it is not known how long the packet we are going to receive will be, we can use a length of 0 in the calling routine. But I am getting ahead of myself; we will deal later with these kind of details.
- Every packet gets a sequence number to be able to detect missing packets. We will logically use 8 bits for that, giving us a 256 packet window. How we send binary data like this is explained in the next design decisions.
- Every packet has an address field to be able to uniquely identify the receiving side of the message. This is important in any bus-like structure as everyone on the bus is listening in and we must have some means to indicate who this message is intended for. We will use 16-bits for the address field (8 should be enough for each individual bus, but I like to plan ahead…). Some values will be reserved for specific use cases. The address ranges 0000h to 00FFh and FF00h to FFFFh are reserved for broadcasts and other specific need we will work out later. An example could be that addresses in the range of FF00h to FFFEh are reserved for broadcasts to specific types of devices (.e.g. FF01h could be a broadcast to all lights and FF02h to all temperature sensors). A device can have more than one type code and act on each of these targeted broadcasts. It is probably a good idea to uniquely identify every single node by assigning each one a unique number made of the bus number (first byte) and node number on that bus (2nd byte). Again, we will deal with these details later. For now it is sufficient to determine that we use a 16-bit address and recognize three types of broadcasts that nodes could react upon (generic, type and logical bus specific).
- Every packet has a length byte to define the size of message payload (can be between 0 and 32). The receiving side should do a basic sanity check on the size parameter (it can never be larger than the maximum length specified from the calling routine).
- Messages are sent as a sequence of nibbles (4 bits) by splitting every byte in its two hexadecimal digits and adding the reverse of that nibble in the high 4 bits of the byte to send. This serves two purposes: it as a simple means of error checking since only a very limited set of values can be sent or received and every byte sent has a simple checksum attached to it in the high nibble. An added benefit is that simple 8-bit MCU’s like the Atmel ones I am going to use can deal with this kind of bit fiddling with just a few simple instructions.
- Every packet starts with a special code that uniquely defines the start of the message. Let’s call it STX (Start of Transmission) and ends with a similar code we call ETX (End of Transmission). We will use the ASCII standard for these control characters. An added benefit of the combination of design decision 5 and 6 is that we don’t need a special ‘escape’ character – like DLE – to deal with issues like a message byte having the same value as the STX or ETX code. A bus-specific driver can choose to send these STX and ETX codes multiple times (usually 2 but no more than 3) to deal with line reliability issues. The receiving side should just ignore them once they received a correct STX or ETX byte. The STX/ETX handling is therefor done at the specific bus-driver level and not passed to the serial protocol library.
- We will only use a very simple checksum (CRC) mechanism for the entire message since every byte (except the start and end codes) already include a simple kind of check. I decided to just XOR every byte and put that in the 8-bit CRC position in the message.
- Broadcast type of messages will not be responded to by the receiving parties (that would become a mess on the serial bus). Instead they are transmitted multiple times with the same sequence number. Since the number of retransmits we need to increase the reliability is related to the type of (virtual) wire, this will be dealt with on the bus-specific driver level (the blue layer). The maximum retransmit is currently determined at 3 times in total (although we should normally send it just 2 times and start sending it 3 times only if we have determined that the signals don’t come across. Note that if we need 100% reliability on broadcast messages, we should fall back to sending the message to every node individually and get a confirmation response (ACK).
- One thing we have to deal with is the type of sensors that signal some kind of external event we should be able to react upon (like a door opening). Since we are connecting over some kind of serial bus, we should somehow deal with situations where multiple devices are transmitting at the same time. This calls for some kind of priority scheme and a random delay retry mechanism. But I will start dealing with that at a later stage. Note that in some cases the bus driver/firmware may solve bus conflicts on its own.
This may all be a little bit confusing, so let’s build a message stream that applies these rules. A complete packet looks like this (logical view).
As defined, every message starts with a special code (STX) to uniquely identify the beginning of the message. This allows a receiver to simple wait for STX to come along and ignore anything else in the meantime. Every byte, except the STX and ETX bytes are encoded (split in two) according to design decision number 5.
We will use simple ASCII for some of the special control codes, including some codes that are part of the commands or response. ASCII codes of interest to us are:
STX 02h Start of Transmission ETX 03h End of Transmission ENQ 05h General request for information (e.g. temperature of a temperature sensor) ACK 06h Acknowledge – response code to return OK to the master NAK 15h Not acknowledge – response code to return not-OK to the master ESC 1Bh Terminate a previous action/command
These are just some initial thoughts on the subject. Other commands and responses will be tailored to the specific needs of the type of nodes involved. We will deal with that later.
An example can help understand all this even better. Let’s say we want to set a light to a specific level of brightness with the following command sequence:
Let's decompose this message:
- The message starts with an address word of 16-bits (0804h in our example), which brings on another topic: we use Big-Endian convention for multi-byte values – another design decision.
- Next byte is the sequence number. It’s our first message, so we use 00h.
- Then comes the length of the data (the actual payload of the message): 02h.
- The following two bytes are the (fictitious) command (10h) and parameter (80h) of the message. 10h is supposed to be a ‘set light to a specific brightness’ command, and 80h is the brightness level (meaning 50%) in this example.
When we look at the binary sequence that is ultimately transmitted, it will be like this (assuming this bus driver sends STX/ETX codes twice).
As can be seen from this example, every byte in the original message is split in two bytes where every byte in the ‘over the wire’ message holds a nibble of the original message, complemented by a high nibble that is the reverse (bit wise) of the low nibble. This gives quite some overhead on the bus, but since speed is not our primary concern, we shouldn’t care too much about that. Robust messaging with simple error detection and handling are the primary design goals, combined with a small footprint, in terms of CPU, RAM and Flash memory. And with this protocol we can meet the design goals.
You may have noticed that the layer separation from the first diagram isn't always strictly followed. This is partly due to the fact that some details on the implementation is still not definite, but also due to resource limitations imposed by small 8-bit MCU’s I will use (that may require some form of optimization/efficient coding that is not in line with a fully layered and modular approach).
Any smart IoT network should have some kind of mesh/auto-discover mechanism in place for easy installation and configuration of new nodes. Although I haven’t given that much thought yet, I would say that this low-level protocol doesn’t stand in the way of an intelligent mesh network. But we will see as things progress…
In the next article we will look at the flow for processing a message.