Although there are ample ready made programs for the ESP8266 to connect to MQTT, I still get questions of people on how to do it. So I will explain it step by step here. If you already know exactly on how to do this, you may just as well skip this post, although there might be some things of a more general interest towards the end.
For those who really have no idea what MQTT actually is, it is a lightweight protocol for machine to machine communication. It consists of a topic and a message (also called payload) that is used often in iot and homeautomation projects. In short, and pulling it a bit in absurdum, with MQTT your refrigerator can talk to your oven and your coffee-maker, provided these are all ‘smart devices. However, they do not talk directly to eachother, they do that via a ‘broker’. A broker is a machine that keeps track of all the MQTT traffic and knows what messages need to go to what device. That means that when your fridge wants to talk to the coffee maker, there is no need for the oven to listen to that message.
So first of all you need an MQTT broker. That could be a public broker or your own broker that you run locally (for instance on a Raspberry Pi). Mosquitto is a popular broker to run yourself (check here how). If you do not want to run your own broker you have options like test.mosquitto.org, CloudMQTT, HiveMQ, MQTT Dashboard (though that uses Hive), RabbitMQ and io.adafruit . To get up and running fast, test.mosquitto.org is the easiest. However, it is a completely open test broker. There is no security on it whatsoever so it is adviseable to change to a private or password protected broker as soon as you have your mqtt up and running.
For the ESP8266 program that we will be making, there will be 4 important credentials that we need. Those are the server, the user, the password and the port. Mind you that “test.mosquitto.org” does not require a user log-in and also when you are running your own broker you do not need (but can set) a log-in, but I will introduce these two variables in the program for when you decide you need/want a log in.
The ESP8266 using the MQTT protocol is considered a ‘client’ of the ‘broker’. To let the ESP8266 talk to the broker there is very popular library called PubSubClient. It is called ‘PubSub’ because it allows the ESP to ‘publish’ MQTT messages to the broker and to ‘Subscribe’ to MQTT messages of the broker. Do you remember I said the oven des not need to listen to the fridge talking to the coffeemaker? Well that is what the ‘subscription’ is for: The specific ESP8266 is only getting messages that it ‘subscribed’ to. The library can be installed via Arduino IDE library manager.
The code
Like in all programs, we begin by including the libraries that are needed for all the functionality we want/need. We need the ESP8266WiFi library, for the ESP8266 to connect to a WiFi network, and the PubSubClient library, for the ESP8266 to connect to a MQTT broker and publish/subscribe messages in topics.
#include <PubSubClient.h>
#include <ESP8266WiFi.h>
#include <WiFiClient.h
Next, we will declare the global variables for our connections. We need the WiFi credentials, to connect to the WiFi network.
constchar* ssid = "YourWiFiSSID"; constchar* password = "YourWiFiPassword";
Next we need the information of the MQTT server. As explained earlier, you will need the server address, the port, the username and the password, but for our test we can leave the user and password empty. In most MQTT brokers, the port is 1883
const char* mqttServer = "test.mosquitto.com";
const int mqttPort = 1883;
const char* mqttUser = "YourMQTTUsername";
const char* mqttPassword = "YourMQTTPassword";
Next we will declare two ‘objects’ (like with all libraries): the WiFiClient object establishes a connection to a specific IP and port. The PubSubClient receives input of the WiFiClient that we just created
We will keep the PubSubclient object simple with just one argument, but it can receive more arguments. For that you may want to check the the API documentation.
WiFiClient espClient;
PubSubClient client(espClient);
In the setup function we connect to the WiFi network and for convenience we print the progress to the serial monitor
Serial.begin(115200);
WiFi.begin(ssid, password);
while(WiFi.status()
!= WL_CONNECTED) {
delay(500);
Serial.print("Connecting to WiFi..");
}
Serial.println("Connected to the WiFi network");
Then we set the address and the port of the MQTT server. We do this with the setServer method of the PubSubClient object.
client.setServer(mqttServer, mqttPort);
We are almost there, but we still need to tell the program what to do with incoming MQTT messages. For that we use the setCallback method of the PubSub object to set a function that is executed when a MQTT message is received. Traditionally this function is called ‘callback’, but you can give it any other name
client.setCallback(callback);
Now, we need the Setup function to connect to the MQTT server. Preferably we do this in a loop till we get succes.
while
(!client.connected()) {
Serial.println("Connecting to MQTT...");
if (client.connect("ESP8266Client", mqttUser, mqttPassword )) {
Serial.println("connected");
}else{
Serial.print("failed with state ");
Serial.print(client.state());
delay(2000);
}
}
The setup function is a good place to publish our first MQTT message to tell the world we are there. Kind of an “Hello World” to let the broker know we are on line. We do this with the publish method, that takes a topic and a message as arguments. As a topic we choose “test/esp8266”, and for a message “Hello ESP world”.
client.publish("test/esp8266","Hello ESP world");
Leaves us to subscribe to a topic to let the broker know what messages from other publishers we like top receive. We do that with the subscribe method, with the topic name as argument. We can take any topic we desire, but for now lets subscribe to the same topic as we publish.
client.subscribe("test/esp8266");
The full setup function now looks like this:
voidsetup()
{
Serial.begin(115200);
WiFi.begin(ssid,password);
while(WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.println("Connecting to WiFi..");
}
Serial.println("Connected to the WiFi network");
client.setServer(mqttServer, mqttPort);
client.setCallback(callback);
while(!client.connected()) {
Serial.println("Connecting to MQTT...");
if(client.connect("ESP8266Client", mqttUser, mqttPassword )) {
Serial.println("connected");
}else{
Serial.print("failed with state ");
Serial.print(client.state());
delay(2000);
}
}
client.publish("test/esp8266", "Hello ESP world");
client.subscribe("test/esp8266");
}
The Topic
This would be a good moment to say some words about the format of the topic. In its smallest form, the topic only needs to be one character long.
Traditionally though, topics have a hierarchy that is defined by backslashes. A topic for instance could be “home\garden\lawn\sprinkler”. This hierarchie allows
subscription on several levels. You could for instance subscribe to the topic “home\#” to get all the messages that deal with ‘home’. You could subscribe to the topic “home\garden\#” to get all the messages that have to do with your garden etc. etc. Often you will see a topic starting with a backslash, like
“\home\garden\lawn\sprinkler” There is absolutely no gain in starting a topic with a backslash.
The callback function
The callback function receives the MQTT messages (topic + payload) that our ESP PubSub client has subscribed*) to. The arguments of the callback function are the name of the topic, the payload and the length of that payload.
As illustration, we will setup the function to print
the topic to the serial port, and then print the received payload.
Since we also have the length of the payload as argument of the
function, this can be easily done in a loop.
voidcallback(char* topic, byte* payload, unsigned intlength)
{
Serial.print("Message arrived in topic: ");
Serial.println(topic);
Serial.print("Message:");
for(inti = 0; i < length; i++) {
Serial.print((char)payload[i]);
}
Serial.println();
}
*) To be precise: the subscription is to a topic, the client then receives the topic + the payload coming with that topic. The payload is sometimes called ‘the message’ but confusingly the topic+payload is sometimes also called “the message”
In the above example we have only subscribed to one specific topic:
test/esp8266.
The topic “test/esp8266/node1” for instance, will not arrive. For that we would need to specify a wildcard “#”
The main loop
In the main loop function, we use the loop method of the PubSubClient. This function is called to allow the client to process incoming messages and maintain its connection to the server.
voidloop()
{
client.loop();
}
The full code
The full code now is as follows:
#include <WiFiClient.h>
#include <ESP8266WiFi.h>
#include <PubSubClient.h>
constchar* ssid = "YourWiFiSSID ";
constchar* password = "YourWiFiPassword";
constchar* mqttServer = "test.mosquitto.com";
constintmqttPort =1883;
constchar* mqttUser = "YourMQTTUsername";
constchar* mqttPassword = "YourMQTTPassword";
WiFiClient espClient;
PubSubClient client(espClient);
voidsetup()
{
Serial.begin(115200);
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.println("Connecting to WiFi..");
}
Serial.println("Connected to the WiFi network");
client.setServer(mqttServer, mqttPort);
client.setCallback(callback);
while (!client.connected()) {
Serial.println("Connecting to MQTT...");
if(client.connect("ESP8266Client", mqttUser, mqttPassword ))
{
Serial.println("connected");
}else{
Serial.print("failed with state ");
Serial.print(client.state());
delay(2000);
}
}
client.publish("test/esp8266","Hello ESP World");
client.subscribe("test/esp8266");// here is where you later add a wildcard
}
voidcallback(char* topic, byte* payload, unsigned intlength)
{
Serial.print("Message arrived in topic: ");
Serial.println(topic);
Serial.print("Message:");
for(int i = 0; i < length; i++) {
Serial.print((char)payload[i]);
}
Serial.println();
}
voidloop()
{
client.loop();
}
Testing
To test the code, upload and run it on your ESP8266. Open the Arduino IDE serial monitor to print the output.
The ESP8266 will send the “Hello ESP World” message, however, that will not be printed on the serial monitor, but as the program also subscribed to that same topic, it will receive that same messages, coming back from the broker and then print it on the serial monitor.
If any other client would send message to the “test/esp8266” topic, then that also will be printed in the serial monitor
Now if it does not work and you are sure the code is OK, is very wise to have MQTT-Spy installed on your computer. MQTT-spy monitors the MQTT traffic on your network and it can publish and subscribe to any desired MQTT topic.
OK, so now what? How do I send a variable?
After your initial joy that everything is working you immediately would want to send something useful for instance that reading of a DHT22 or a DS18B20 sensor and suddenly you need to make the step of the string “Hello World” to a temperature variable. Ofcourse you will try and just insert the variable in place of the “Hello World” string and hope for the best, but that will fail, as the PubSub library (and the MQTT protocol) will only accept strings.
You will see it will not even accept a string variable, so what to do? Well there are several ways to put a variable in the MQTT payload:
1- take another library.
There is a fork of the PubSubClient library by Ian Tester. It will accept variables if defined like this:
client.publish("topic", String (variable));
However, it is often not a drop in replacement, you will need to make minor adjustments in setting up the various calls.
2-use some C++language
Suppose you want to publish some interger variable called ‘temperature’
sprintf(buff_msg, "%d", temperature);
client.publish("topic", buff_msg);
ofcourse you will need to set up the buff_msg buffer in your global variable declarations like this:
buff_msg[32]; // mqtt message
A simpler method is with the use of ‘c_str’:
Suppose I want to send a float variable called “room_temperature” that i got by reading a ds18b20.I can then send that as follows:
client.publish("home/living/temp",String(room_temperature).c_str());
A very frequently used payload to send will be OFF or ON. Although often it might be just as easy to define an IF statement for your payload like this:
if (status==1){client.publish("topic","ON");}
if (status==0){client.publish("topic","OFF");}
it is also possible to do that with the sprintf command:
sprintf(buff_msg, "%d", status);
client.publish("topic",buff_msg);
So, can I send a JSON with MQTT?
Sure you can. I am pretty sure it works with the above method, after all a JSON is just some type of string. However, I prefer another method for a JSON.
Suppose I want to send the following JSON:
"{\"garden\":temperature\",\"data\":["+String(temperature_outside)+","+String(temperature_greenhouse)+","+String(temperature_compost)+","+String(temperature_coldframe)+","+String(temperature_soil)+"]}"
do that as follows:
String payload="{\"garden\":temperature\",\"data\":["+String(temperature_outside)+","+String(temperature_greenhouse)+","+String(temperature_compost)+","+String(temperature_coldframe)+","+String(temperature_soil)+"]}";
payload.toCharArray(data, (payload.length() + 1));
client.publish("topic",data);
Other example, suppose I want to send the temperature and humidity from a DHT12 as a JSON
DHTtemp = am2320.readTemperature();
DHThumid = am2320.readHumidity();
String payload = "{\"dht12temp\":" + String(DHTtemp) + ",\"dht12humid\":" + String(DHThumid) + "}";
payload.toCharArray(data, (payload.length() + 1));
client.publish("home/bak/dht12", data);
I like to send my IP number as MQTT, is that possible?
Ofcourse. There are several methods and in the end all roads lead to Rome. Using c_str is sometimes advised but I prefer the following.
in your general variable declaration declare:
String IP;
Then after you connect to WiFi state:
IP = WiFi.localIP().toString();
Then where you handle your publishing, do the following:
for (i = 0; i < 16; i++) {
buff_msg[i] = IP[i];
}
buff_msg[i] = '\0';
client.publish("IPnumber", buff_msg);
}
Can I send my MAC
Yes you can. Again there are several methods. I prefer this one:
In your general variable declarations, declare:
String MAC;
Then after you connect to WiFi state:
MAC = WiFi.macAddress(); (unlike the IP nr, this does not need the 'toString()' function.
Then where you handle your publishing, do the following:
for (i = 0; i < 16; i++) {
buff_msg[i] = IP[i];
}
buff_msg[i] = '\0';
client.publish("MAC", buff_msg);
}
An easier way though is to use a PubSubClient that in fact an send variables, like the PubSubClient from Ian Tester that I mentioned earlier.
This allows you to do:
IP = WiFi.localIP().toString(); MAC = WiFi.macAddress(); client.publish("home/ip", String(IP)); client.publish("home/mac", String(MAC));
Can I send the name of my sketch with MQTT?
Yes you can. In tour variable declaration state:
String filename = (__FILE__); // filename
This will give you the full path with filename and that might not be interesting/too much. So if you just want the filename, you need to parse that string. Do that in the setup section of your sketch, like this:
//strip path from filename
byte p1 = filename.lastIndexOf('\\');
byte p2 = filename.lastIndexOf('.');
filenaam = filename.substring(p1 + 1, p2);
Then when you want to send it do:
client.publish("home/sketch/name", filename.c_str());
That should do it.
Can I send variables in the topic?
Yes you can. This is in fact not so different from sending variables in the payload:
In your variable declaraion declare a buffer:
buff_topic[30]; // mqtt topic
Now suppose you want to send the ID number or the name of your ESP8266 as part of the topic and an error as the payload:
in your global declarations you have defined the number or name of your ESP:
#define nodeId 13 // node ID
and then where you send your messages do:
sprintf(buff_topic, "home/nb/node%02d/error", nodeId);
sprintf(buff_msg, "syntax error %d", error);
client.publish(buff_topic, buff_msg);
What about receiving MQTT messages?
Well, I spoke about the callback function that handles the incoming messages, but I admit that just waiting for one standard solitary “Hello World” message isn’t very useful.
To subscribe to more than one message we need to use a wildcard, so
client.subscribe(“test/esp8266”);
becomes
client.subscribe(“test/esp8266/#”);
this will catch messages such as:
“test/esp8266/Light1”
“test/esp8266/Light2”
The most obvious way to use these would be to check the incoming topics is it Light1 or Light2, then check the incoming payload (is it “ON” or “OFF”) and then act upon the content of the payload.
Especially if you have MQTT topics with varying lengths this requires quite some checking of possibly long topics.
Say you have:
test/esp8266/kitchenlight
test/esp8266/Livingroomlight
test/esp8266/washingmachine
test/esp/8266/firstfloor/airco
Although it is very well possible to check for all these messages and then determine what the payload is, it is memory consuming.
A more elegant way is to keep the MQTT messages all the same length and only say vary the last 2 characters of the topic to indicate what it is about. That gives room for 100 different topics, while you only have to check the last 2 characters, rather than a long string.
Lets give an example:
Suppose your MQTT messages all have the format: “home/nodeID/devID” or -in practice:
“home/node13/dev00”
…….
“home/node13/dev99”
Your subscription then would be to: “home/node13/#”
and you would only have to check on the last two digits.
An example of your callback then would be
void callBack(char* topic, byte* payload, unsigned int length) { // receive and handle MQTT messages
int i;
for (int i = 0; i < length; i++) {
Serial.print((char)payload[i]);
}
if (strlen(topic) == 17) { // correct topic length ?
DID = (topic[15] - '0') * 10 + topic[16] - '0'; // extract device ID from MQTT topic
payload[length] = '\0'; // terminate string with '0'
String strPayload = String((char*)payload); // convert to string
if (length == 0) {
error = 2; // no payload sent
}else {
if (DID == 0) {
device0 = true;
}
if (DID == 1) {
device1 = true;
}
if (DID == 2) {
device2 = true;
}
if (DID == 99) {
device99 = true;
}
}
What the code does here is to just look at the last 2 digits (DID = Device ID) and then set a flag (device0 -device 99) that can be checked in the main loop upon which action can be taken, depending on the content of the ‘strPayload’ string.
For an extensive example of this check the gateway projects of Computourist on github.
(Note: if you consider building this… I am working on a similar station with a bare ESP12 and a BME280 for even better energy efficiency)
Arduino, ESP8266, ESP32 & Raspberry Pi stuff 