MQTT for beginners

Although there are ample ready made programs for the ESP8266 to connect to
MQTT, I still get questions of people on how to do it. So I will explain it step by step here. If you already know exactly on how to do this, you may just as well skip this post, although there might be some things of a more general interest towards the end.

For those who really have no idea what MQTT actually is, it is a lightweight protocol for machine to machine communication. It consists of a topic and a message (also called payload) that is used often in iot and homeautomation projects. In short, and pulling it a bit in absurdum, with MQTT your refrigerator can talk to your oven and your coffee-maker, provided these are all ‘smart devices. However, they do not talk directly to eachother, they do that via a ‘broker’. A broker is a machine that keeps track of all the MQTT traffic and knows what messages need to go to what device. That means that when your fridge
wants to talk to the coffee maker, there is no need for the oven to listen to that message.

So first of all you need an MQTT broker. That could be a public broker or your own broker that you run locally (for instance on a Raspberry Pi). Mosquitto is a popular broker to run yourself (check here how). If you do not want to run your own broker you have options like test.mosquitto.org, CloudMQTT, HiveMQ, MQTT Dashboard (though that uses Hive), RabbitMQ and io.adafruit . To get up and running fast, test.mosquitto.org is the easiest. However, it is a completely open test broker. There is no security on it whatsoever so it is adviseable to change to a private or password protected broker as soon as you have your mqtt up and running.

For the ESP8266 program that we will be making, there will be 4 important credentials that we need. Those are the server, the user, the password and the port. Mind you that “test.mosquitto.org” does not require a user log-in and also when you are running your own broker you do not need (but can set) a log-in, but I will introduce these two variables in the program for when you decide you need/want a log in.

The ESP8266 using the MQTT protocol is considered a ‘client’ of the ‘broker’. To let the ESP8266 talk to the broker there is very popular library called PubSubClient. It is called ‘PubSub’ because it allows the ESP to ‘publish’ MQTT messages to the broker and to ‘Subscribe’ to MQTT messages of the broker. Do you remember I said the oven des not need to listen to the fridge talking to the coffeemaker? Well that is what the ‘subscription’ is for: The specific ESP8266 is only getting messages that it ‘subscribed’ to. The library can be installed via Arduino IDE library manager.


The code

Like in all programs, we begin by including the libraries that are needed for all the functionality we want/need. We need the ESP8266WiFi library, for the ESP8266 to connect to a WiFi network, and the PubSubClient library, for the ESP8266 to connect to a MQTT broker and publish/subscribe messages in topics.

#include <PubSubClient.h>
#include <ESP8266WiFi.h>
#include <WiFiClient.h

Next, we will declare the global variables for our connections. We need the WiFi credentials, to connect to the WiFi network.

constchar* ssid = "YourWiFiSSID";  constchar* password = "YourWiFiPassword";

Next we need the information of the MQTT server. As explained earlier, you will need the server address, the port, the username and the password, but for our test we can leave the user and password empty. In most MQTT brokers, the port is 1883

const char* mqttServer = "test.mosquitto.com"; 
const int mqttPort = 1883; 
const char* mqttUser = "YourMQTTUsername"; 
const char* mqttPassword = "YourMQTTPassword";

Next we will declare two ‘objects’ (like with all libraries): the WiFiClient object establishes a connection to a specific IP and port. The PubSubClient receives input of the WiFiClient that we just created

We will keep the PubSubclient object simple with just one argument, but it can receive more arguments. For that you may want to check the the API documentation.

WiFiClient espClient;
PubSubClient client(espClient);

In the setup function we connect to the WiFi network and for convenience we print the progress to the serial monitor

Serial.begin(115200);
WiFi.begin(ssid, password);
while(WiFi.status()
!= WL_CONNECTED) {

delay(500);
Serial.print("Connecting to WiFi..");
}
Serial.println("Connected to the WiFi network");

Then we set the address and the port of the MQTT server. We do this with the setServer method of the PubSubClient object.

client.setServer(mqttServer, mqttPort);

We are almost there, but we still need to tell the program what to do with incoming MQTT messages. For that we use the setCallback method of the PubSub object to set a function that is executed when a MQTT message is received. Traditionally this function is called ‘callback’, but you can give it any other name

client.setCallback(callback);

Now, we need the Setup function to connect to the MQTT server. Preferably we do this in a loop till we get succes.

while
(!client.connected()) {
Serial.println("Connecting to MQTT...");
if (client.connect("ESP8266Client", mqttUser, mqttPassword )) {
Serial.println("connected");
}else{
Serial.print("failed with state ");
Serial.print(client.state());
delay(2000);
}
}

The setup function is a good place to publish our first MQTT message to tell the world we are there. Kind of an “Hello World” to let the broker know we are on line. We do this with the publish method, that takes a topic and a message as arguments. As a topic we choose “test/esp8266”, and for a message “Hello ESP world”.

client.publish("test/esp8266","Hello ESP world");

Leaves us to subscribe to a topic to let the broker know what messages from other publishers we like top receive. We do that with the subscribe method, with the topic name as argument. We can take any topic we desire, but for now lets subscribe to the same topic as we publish.

client.subscribe("test/esp8266");

The full setup function now looks like this:

voidsetup()
{
  Serial.begin(115200);
  WiFi.begin(ssid,password);
  while(WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.println("Connecting to WiFi..");
  }
  Serial.println("Connected to the WiFi network");
  client.setServer(mqttServer, mqttPort);
  client.setCallback(callback);
  while(!client.connected()) {
    Serial.println("Connecting to MQTT...");
    if(client.connect("ESP8266Client", mqttUser, mqttPassword )) {
      Serial.println("connected");
    }else{
      Serial.print("failed with state ");
      Serial.print(client.state());
      delay(2000);
    }
  }
  client.publish("test/esp8266", "Hello ESP world");
  client.subscribe("test/esp8266");
}

The Topic

This would be a good moment to say some words about the format of the topic. In its smallest form, the topic only needs to be one character long.
Traditionally though, topics have a hierarchy that is defined by backslashes. A topic for instance could be “home\garden\lawn\sprinkler”. This hierarchie allows
subscription on several levels. You could for instance subscribe to the topic “home\#” to get all the messages that deal with ‘home’. You could subscribe to the topic “home\garden\#” to get all the messages that have to do with your garden etc. etc. Often you will see a topic starting with a backslash, like
“\home\garden\lawn\sprinkler” There is absolutely no gain in starting a topic with a backslash.

The callback function

The callback function receives the MQTT messages (topic + payload) that our ESP PubSub client has subscribed*) to. The arguments of the callback function are the name of the topic, the payload and the length of that payload.

As illustration, we will setup the function to print
the topic to the serial port, and then print the received payload.
Since we also have the length of the payload as argument of the
function, this can be easily done in a loop.

voidcallback(char* topic, byte* payload, unsigned intlength)
{

  Serial.print("Message arrived in topic: ");
  Serial.println(topic);
  Serial.print("Message:");
  for(inti = 0; i < length; i++) {
    Serial.print((char)payload[i]);
  }
  Serial.println();
}

*) To be precise: the subscription is to a topic, the client then receives the topic + the payload coming with that topic. The payload is sometimes called ‘the message’ but confusingly the topic+payload is sometimes also called “the message”

In the above example we have only subscribed to one specific topic:
test/esp8266.

The topic “test/esp8266/node1” for instance, will not arrive. For that we would need to specify a wildcard “#”

The main loop

In the main loop function, we use the loop method of the PubSubClient. This function is called to allow the client to process incoming messages and maintain its connection to the server.

voidloop()
{
  client.loop();
}

The full code

The full code now is as follows:

#include <WiFiClient.h>
#include <ESP8266WiFi.h>
#include <PubSubClient.h>
constchar* ssid = "YourWiFiSSID ";
constchar* password =  "YourWiFiPassword";
constchar* mqttServer = "test.mosquitto.com";
constintmqttPort =1883;
constchar* mqttUser = "YourMQTTUsername";
constchar* mqttPassword = "YourMQTTPassword";
WiFiClient espClient;
PubSubClient client(espClient);
voidsetup()
{
  Serial.begin(115200);
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.println("Connecting to WiFi..");
  }
  Serial.println("Connected to the WiFi network");
  client.setServer(mqttServer, mqttPort);
  client.setCallback(callback);
  while (!client.connected()) {
    Serial.println("Connecting to MQTT...");
    if(client.connect("ESP8266Client", mqttUser, mqttPassword )) 
{
     Serial.println("connected");
    }else{
      Serial.print("failed with state ");
      Serial.print(client.state());
      delay(2000);
    }
  }
  client.publish("test/esp8266","Hello ESP World");
  client.subscribe("test/esp8266");// here is where you later add a wildcard
}
voidcallback(char* topic, byte* payload, unsigned intlength)
{
  Serial.print("Message arrived in topic: ");
  Serial.println(topic);
  Serial.print("Message:");
  for(int i = 0; i < length; i++) {
    Serial.print((char)payload[i]);
  }
  Serial.println();
}
voidloop()
{
  client.loop();
}


Testing

To test the code, upload and run it on your ESP8266. Open the Arduino IDE serial monitor to print the output.

The ESP8266 will send the “Hello ESP World” message, however, that will not be printed on the serial monitor, but as the program also subscribed to that same topic, it will receive that same messages, coming back from the broker and then print it on the serial monitor.

If any other client would send message to the “test/esp8266” topic, then that also will be printed in the serial monitor

Now if it does not work and you are sure the code is OK, is very wise to have MQTT-Spy installed on your computer. MQTT-spy monitors the MQTT traffic on your network and it can publish and subscribe to any desired MQTT topic.

OK, so now what? How do I send a variable?

After your initial joy that everything is working you immediately would want to send something useful for instance that reading of a DHT22 or a DS18B20 sensor and suddenly you need to make the step of the string “Hello World” to a temperature variable. Ofcourse you will try and just insert the variable in place of the “Hello World” string and hope for the best, but that will fail, as the PubSub library (and the MQTT protocol) will only accept strings.
You will see it will not even accept a string variable, so what to do? Well there are several ways to put a variable in the MQTT payload:

1- take another library.
There is a fork of the PubSubClient library by Ian Tester. It will accept variables if defined like this:
client.publish("topic", String (variable));
However, it is often not a drop in replacement, you will need to make minor adjustments in setting up the various calls.

2-use some C++language
Suppose you want to publish some interger variable  called ‘temperature’

sprintf(buff_msg, "%d", temperature);
client.publish("topic", buff_msg);

ofcourse you will need to set up the buff_msg buffer in your global variable declarations like this:

buff_msg[32]; // mqtt message

A very frequently used  payload to send will be OFF or ON. Although often it  might be just as easy to define an IF statement for your payload like this:

if (status==1){client.publish("topic","ON");}
if (status==0){client.publish("topic","OFF");}

it is also possible to do that with  the sprintf command:

sprintf(buff_msg, "%d",  status);
client.publish("topic",buff_msg);

So, can I send a JSON with MQTT?

Sure you can. I am pretty sure it works with the above method, after all a JSON is just some type of string. However, I prefer another method for a JSON.

Suppose I want to send the following JSON:
"{\"garden\":temperature\",\"data\":["+String(temperature_outside)+","+String(temperature_greenhouse)+","+String(temperature_compost)+","+String(temperature_coldframe)+","+String(temperature_soil)+"]}"
do that as follows:

String payload="{\"garden\":temperature\",\"data\":["+String(temperature_outside)+","+String(temperature_greenhouse)+","+String(temperature_compost)+","+String(temperature_coldframe)+","+String(temperature_soil)+"]}";

payload.toCharArray(data, (payload.length() + 1));
client.publish("topic",data);

I like to send my IP number as MQTT, is that possible?

Ofcourse. There are several methods and in the end all roads lead to Rome. Using c_str is sometimes advised but I prefer the following.

in your general variable declaration declare:
String IP;

Then after  you connect to WiFi  state:
IP = WiFi.localIP().toString();

Then where you handle your publishing, do the following:

for (i = 0; i < 16; i++) {
buff_msg[i] = IP[i];
}
buff_msg[i] = '\0';
client.publish("IPnumber", buff_msg);
}

Can I send my MAC

Yes you can. Again there are several methods. I prefer  this one:
In your general variable declarations, declare:

String MAC;
Then after you connect to WiFi state:
MAC = WiFi.macAddress();  (unlike the IP nr, this does not need the 'toString()' function.

Then where you handle your publishing, do the following:

for (i = 0; i < 16; i++) {
buff_msg[i] = IP[i];
}
buff_msg[i] = '\0';
client.publish("MAC", buff_msg);
}

An easier way though is to use a PubSubClient that in fact an send variables, like the PubSubClient from Ian Tester that I mentioned earlier.
This allows you to do:

IP = WiFi.localIP().toString(); MAC = WiFi.macAddress(); client.publish("home/ip", String(IP)); client.publish("home/mac", String(MAC));

Can I send variables in the topic?

Yes you can. This is in fact not so different from sending variables in the payload:
In your variable declaraion  declare a buffer:
buff_topic[30]; // mqtt topic

Now suppose you want to send the ID number or the name of your  ESP8266 as part of the topic and an error as the payload:

in your  global declarations you have defined the number or name of your ESP:
#define nodeId 13 // node ID

and then where you send your  messages do:

sprintf(buff_topic, "home/nb/node%02d/error", nodeId);
sprintf(buff_msg, "syntax error %d", error);
client.publish(buff_topic, buff_msg);

What about receiving MQTT messages?

Well, I spoke about the callback function that handles the incoming messages, but I admitt that just waiting for one standard solitary “Hello World” message isn’t very useful.
To subscribe to more than one message we need to use a wildcard, so
client.subscribe(“test/esp8266”);
becomes
client.subscribe(“test/esp8266/#”);
this will catch  messages such as:
“test/esp8266/Light1”
“test/esp8266/Light2”
The most obvious way to use these would be to check the incoming topics is it Light1 or Light2, then check the incoming payload (is it “ON” or “OFF”) and then act upon the content of the payload.

Especially if you have MQTT  topics with varying lengths this requires quite some checking of possibly long topics.
Say you have:
test/esp8266/kitchenlight
test/esp8266/Livingroomlight
test/esp8266/washingmachine
test/esp/8266/firstfloor/airco

Although it is very well possible to check for all these messages and then determine what the payload is, it is memory consuming.
A more elegant way is to keep the MQTT messages all the same length and only say vary the last 2 characters of the topic to indicate what it is about. That gives room for 100 different topics, while uou only have to check the last 2 characters, rather than a long string.
Lets give an example:
Suppose your MQTT messages all have the format:  “home/nodeID/devID” or -in practice:
“home/node13/dev00”
…….
“home/node13/dev99”

Your subscription then would be to: “home/node13/#”
and you would only have to check on the last two digits.

An example of  your callback then would be

void callBack(char* topic, byte* payload, unsigned int length) { // receive and handle MQTT messages
int i;
for (int i = 0; i < length; i++) {
Serial.print((char)payload[i]);
}
if (strlen(topic) == 17) { // correct topic length ?
DID = (topic[15] - '0') * 10 + topic[16] - '0'; // extract device ID from MQTT topic
payload[length] = '\0'; // terminate string with '0'
String strPayload = String((char*)payload); // convert to string
if (length == 0) {
error = 2; // no payload sent
}else {
if (DID == 0) {
device0 = true;
}
if (DID == 1) {
device1 = true;
}
if (DID == 2) {
device2 = true;
}
if (DID == 99) {
device99 = true;
}
}

What the code does here is to just look at the last 2 digits (DID = Device ID) and then set a flag (device0 -device 99) that can be checked in the main loop upon which action can be taken, depending on the content of  the ‘strPayload’  string.

For an extensive example of this check the gateway projects of Computourist on github.

Advertisements

OpenHab GPIO Example

I know this might not directly be Arduino or ESP8266 related, but I thought I’d publish it anyway as it may come in handy with openhab

First of all, install the GPIO binding in the PaperUI. After one has done that, there isnt really much else to do on that binding. There is no services/gpio.cfg although some of the documentation mentions that file.

Before I go further it is good to note that I used the openhabian distro and I presume a lot of things are already OK in that, but if you did a manual install, it might be good to check if the user ‘openhab’ is member of ‘gpio’ and if not, set that with

sudo adduser openhab gpio

That’s basically all one needs to do.
Let’s setup an ‘input’ which in openhab is a ‘Contact’ and an ‘output’, which in openhab is a Switch. In fact, those are the only states the binding allows.

There is no pressing need to define anything in your sitemap file if e.g. you add the items to a group in your itemsfile, but if you prefer a sitemap then add this to your sitemap file:

Switch item=GPIO_LAMP
Text item=GPIO_BUTTON

Let’s pick some pins to use. It is easiest to use pins on the Raspberry header that are close to a ground pin.

Looking at the pin out of the raspi, we see that for instance GPIO pins 21,13,5, 7, 10, 27,22,24,25,12,16,26, 3, 4, 8 and 17 are next to a ground pin, but some of those have reserved functions so i stayed away from those.
I picked GPIO23 and GPIO24 as that one is also close to a 3V3 pin and when using that as input one can always decide later to make that have a HIGH or a LOW input.The configuration string for the binding is defined as follows:

gpio="pin:PIN_NUMBER [debounce:DEBOUNCE_INTERVAL] [activelow:yes|no] [force:yes|no] [initialValue:high|low]"

for those who consider this as double dutch, it will become clear in the items definition:

Add this to your items file:

Switch GPIO_LAMP "Lamp" (LivingRoom) { gpio="pin:23 force:yes" }
Contact GPIO_BUTTON "Button [%s]" (LivingRoom) { gpio="pin:24 activelow:yes" }

The group ‘LivingRoom’ is not mandatory if you have defined your sitemap and of course you may use another group that suits you more.

The Switch that I set up is quite simple. In fact it already would be enough if it would have been:

Switch GPIO_LAMP "Lamp" {gpio="pin:23"}.

But with ‘force:yes’ we force the use of the pin even if it would have been in use by another process. Obviously that might not be the most elegant way to do it, but as i am pretty sure I am not using it for another process, I just wanted to make sure Openhab would have the pin available.

The current of a GPIO pin is far to small to drive any significant load as it can deliver some 60mA I think.
But the output can drive a small LED to test it… but use a 330 ohm series resistor, no matter what other websites tell you: you dont want to fry your GPIO pin. Eventually it would be more useful to drive a small relay module as long as it has a separate transistor to drive the relay (I know, I am cautious)

The configuration allows for some other parameters.
i already mentioned ‘force:yes|no

activelow yes|no  this means that when the switch is off, there is no voltage on the pin
initialValue:high:low determines the state the pin will take during initialisation.
debounce:interval is used to set the debounce value for the switch in milisecs

If we look at the output, I have not added an activelow because I thought that was the default value (but correct me if I am wrong)
If we look at our input, I set that to activelow, because the way the button works is it applies voltage to trigger the pin and I added it because I was not sure about the default
I did not set a debounce for the contact, but you could state:

Contact GPIO_BUTTON "Button [%s]" (LivingRoom) { gpio="pin:24 debounce:10 activelow:yes" }