Connecting an ESP8266 to the outside World

I admit that this is a subject that is really really basic and no doubt explained at many places. Yet, I still see the question posed time after time at various fora and sadly the replies are often not on the ball. Things like “You need REST API”, “Just use Blynk”, “Just use IFTTT” or “Why would you want to do that”, surely will not help the struggling novice on their way.
So, rather than having to repeat my explanation again and again, it is better, I just write an article, explaining it…..for the novices amongst us.

Finding 2 IP addresses
OK, so you have an ESP8266 that reads a sensor and you want to be able to get the reading when you are away from home, using internet. It is quite simple, but you need to know 2 distinct IP addresses.
1- you need to know the IP address that connects your computer to the internet. You can easily find that on websites like “whatismyip.com“. It is also called ‘Public IP’ or ‘WAN IP’
2- you need to know the IP address that your ESP8266 has in your local network. You need to check the DHCP table in your router, but many ESP8266 programs actually print that number as well in the serial port. These numbers almost always looks like “192.168.x.yyy”. You can also use a phone app like ‘port authority’ to find the number for you.

Port forwarding
So what happens when you are on your holiday address in Reykjavik or Kabul and you want to know what your sensor back home says? You open a browser, type in the IP address you found under “1”. well that connects you via the internet to your router back home. However, your router when it gets that call, doesn’t have a clue what to do with it, let alone that it knows it should forward that call to the ESP8266 that is connected to it, because you didn’t tell it what to do with it.

So when you get back home you have to instruct your router to forward the calls to your ESP8266 and that is done via “IP forwarding” or “Port forwarding”.
What makes that a bit complicated is that it is done differently in various routers, so you would have to check your specific router for it. Look for “IP forwarding” or “Port forwarding” and in some routers it is called ‘virtual server’.
When you found that you usually have to fill out 2 things. The port that is being forwarded and the IP address it has to be forwarded to. For the port you choose “80” and for the IP address to forward to, you use the local ip address, you found under point “2”.It is the address that started with “192.168.”
If it doesnt allow you to forward a specific port, but it does allow you to forward a service, choose ‘http’ for that service.
If you really cant find the port forwarding, but You happen to find something that is called “DMZ” choose that to forward to the ip address of your ESP8266 (the address starting with “192.168”) that is less elegant, but it will work.

Setting up a Webserver
So you are back on holiday and you want to check again. You fill out the internet IP number of your home system (the number you found under “1” and yes, this time the call comes in to your router and that one knows exactly what to do,so it forwards the call to your ESP8266. The ESP duly receives the call and……….has no idea what to do with it because chances are you didnt tell it what to so with it.
Your ESP8266 needs a program that presents the data in “Webpage”. That is not hard, there are plenty of programs who do that. These programs usually are called “Webservers”. Plenty of those available. (Look in your Arduino IDE examples or check e.g. randomnerdstutorial.)

It works!!!! for now: MAC and Address reservation
So, you now have programmed your ESP8266 with a webserver program that presents the sensor data on its own little webpage.
In order to test it when you are at home you open your browser and fill out the Local IP number (that you found under “2” starting with 192.168) and yes, you see the webpage. To make sure you also fill out the address you found under “1” (thus the address that your system has on the worldwide interweb) and great it all works. You go back on holiday and check from the other side of the world and great still works. Mission accomplished!!!

…..or is it. Suddenly it doesn’t work anymore. What on earth is wrong????? Well chances are that you have restarted your ESP8266 a few times and suddenly it has received a different IP address in your Local network. Say it once had 192.168.1.103, but now you find out it has 192.168.1.110. Your router however still sends the incoming call to the first IP number because that is what you told it to do. Now it is hardly feasible that you constantly check if the local IP is still the same and if it isnt that you keep changing your forward instructions.
Fortunately, most routers allow you to reserve a specific IP number in your network, for a specific device, and this is how you do it:
When you earlier found the IP number of your ESP8266 (the one starting with 192.168), you probably also found its MAC number. A MAC number (or addres rather) looks something like: “5D:CF:7F:AC:61:66”
Now you need to look in your router that is called “MAC binding” or “Address reservation”. The object here is that you give the MAC number, as well as the IP number (192.168.xx.yyy) that should always be reserved just for the device with that specific MAC stands for “media access control”.

Yes, works again
OK you got that done,so now whenever you go to the internet address of your system back home, the call comes into your router, the router knows what internal address to send it to, that address is always the same, and the device on that address knows what to do: it presents (“serves”) a webpage to the computer half across the world behind which you are sitting. That webpage can present data, but it may just as well receive commands from you, e.g. to turn a lamp on or off. Realuse though that anybody who knows your public IP address can also check that webpage. That might not be so bad if it just presents temperature, but it might give a problem if strangers can switch a lamp off and on in your home. So once you set it all up, you may consider making a password  protected webpage.

Let me compare the situation with making a phonecall to a hotel: you need the hotel’s phonenumber (compare to the internet IP) and you need to know what room someone is in (the Local IP). The person in the room needs to know what to do with your questions (serving a webpage). That all works well as long as the person is in the same room (the address reservation: when Mr MAC comes to the hotel, he always gets room 192.168.1.103).

A final touch: DNS providers
So, everything works now………..but what if there is a rezoning and the hotel’s phonenumber changes???? Yes, you could look up the new number, but it is hardly practical.
But what does that mean in internet terms? Well, most of us get a so called ‘Dynamic IP address’ from our provider. Usually those addresses stay the same for a long time, but when you happen to restart your router you could get a different internet IP address. Sure you could check your internet address regularly, but that is only possible when you are at home. What to do when you are away from home????
The solution is a (free) DNS provider.
DNS providers provide you with an internet name, that you can use rather than an IP number. The essence though is that you get to put a tiny bit of software on your computer, that regularly checks your internet address and if that changes, it tells your DNS provider what the new number is. So you only have to provide a domain name to your browser and the DNS provider then knows the most up to date IP number to connect to.
A popular freeDNS provider for instance was “noip.com”, but there are more.

Advertisements

Using the ESP8285 M2, with a warning

Next to the previously discussed M3, there is also an ESP8285 M2 (in fact, there is also an M1). Like the M3 it is based on an ESP8285 that has 1Mb on board.
It has 11 I/O pins (including Rx and Tx) as well as an ADC.

An advantage though is that it is available on a development board that has a Vcc and Grnd connection per I/O pin (but read the caveat below), whereas e.g. the Wemos D1 mini only has 1 ground and 3V3 pin.
The board can be programmed directly from the Arduino IDE, but it does not have a circuit present that puts it automatically in flash mode. Therefore it is necessary to push the Flash button, hold it and then press the Reset button and only then release the Flash button.

There is a big caveat with this board though: The row of positive pins opposite the I/O pins, is connected to the INPUT voltage, so even if you feed this board via USB, you get 5Volt on the “V” pins. If you feed your board 3.3 Volt via the 3V3 connector……there is no voltage at all on the ‘V’ pins. Whether this is a design flaw or intentional I do not know.

Showing ESP8266 battery voltage in a graph

In a previous post, I dabbled around with presenting sensor readings with the Highcharts graphic library, based on a post on the randomnerdtutorials website.

It might be educational to add another channel to it that reads battery voltage and presents that in a graph as well.
In the picture above there are two colours: when the voltage is above 3.3Volt, the colour is green, when it goes below 3.3Volt the colour turns red.

This is how to do it:
First, grab the Arduino Sketch file from the randomnerdsturorial website. Open it up and add two procedures:

1 locate the procedure called readBME280Humidity() and right above that, add the following lines:

String readBatt(){
float v=analogRead(A0);
v=4.2*(v/1023);
return String(v);
}

2 locate the section that says // Start server.
right above that add:

server.on("/voltage", HTTP_GET, [](AsyncWebServerRequest *request){
    request->send_P(200, "text/plain", readBatt().c_str());
  });

That’s all we need to do in the Arduino Sketch. Now get the index.html file from the randomnerdstutorial website.
Open up the file in an editor like Notepad++ and find the </script> tag.
Right above that tag, add the following code.

var chartV = new Highcharts.Chart({
chart:{ renderTo:'chart-voltage' },
title: { text: 'LipoVoltage' },
series: [{
showInLegend: false,
name: 'Voltage',
data: [],

zones: [{
value: 0,
color: '#0000ff'
}, {
value: 3.3,
color: '#ff0000'
}, {
color: '#00ff00'
}]

}],
plotOptions: {
line: { animation: false,
dataLabels: { enabled: true }
}
},
xAxis: {
type: 'datetime',
dateTimeLabelFormats: { second: '%H:%M:%S' }
},
yAxis: {

title: { text: 'Voltage (V)' },
min: 0,
max:5
},
credits: { enabled: false }
});

setInterval(function ( ) {
var xhttp = new XMLHttpRequest();
xhttp.onreadystatechange = function() {
if (this.readyState == 4 && this.status == 200) {
var x = (new Date()).getTime(),
y = parseFloat(this.responseText);
//console.log(this.responseText);
if(chartV.series[0].data.length > 40) {
chartV.series[0].addPoint([x, y], true, true, true);
} else {
chartV.series[0].addPoint([x, y], true, false, true);
}
}
};
xhttp.open("GET", "/voltage", true);
xhttp.send();
}, 30000 ) ;

Next, locate the following section in the file, and add the indicated line

Save the html file in the data folder of your Arduino sketch.

Now we have to take care of the hardware section.
The ADC of the ESP8266 chip can only read a maximum of 1Volt, whereas a lipo battery will have a max voltage of 4.2 Volt. So we need a voltage divider that makes 1 Volt uit of 4.2 Volt. If you are using the Wemos D1 mini, that already has a voltage divider in place. It connects the A0 of the Wemos with the ADC of the ESP8266 chip.

A 220k over 100k however will not be giving the required ratio. We will need to add an extra 100k resistor between the battery and the A0 pin. That gives us a ratio of 100/(100+220+100)=100/420=1/4.2.
If you happen to use the new, version 1.3 battery shield, I think that already has an extra 130k in place, which gives a slightly different division factor of 100/(130+220+100)=1/4.5.

Upload the Arduinosketch to an ESP8266 and upload the HTML file to the SPIFFS. Open up your IDE terminal window to check the ip nr and open a browser to that ip nr.

ESP8266 Plot Sensor Readings in Real Time Charts on a Web Server

Until recent, I had not bothered much about adding graphics to my ESP8266 programs, as I was using Openhab and Influx/Grafana for graphical presentation of data. In fact I never fully realized what was possible on an ESP8266. Then I saw some great graphics for a BME280 on the randomnerdtutorials site from Rui and Sara Santos.
Their approach is to use the ‘Highcharts’ graphic javascript library that is available on line on cloudflare, something that in all honesty I had never thought of.
Anyway, they create 3 separate graphs, for each of the modalities (temperature, humidity and pressure) of the BME280. That works wonder well. I however was more interested in having various (temperature) results in one graph. Before I go further, the fact that Sara and Rui make 3 requests, makes the program easy to follow. They could ofcourse have made one request, receive the temperature, humidity and air pressure in one json message and parse that. Anyway, back to my adaptation of the code.
I found some examples that were using a different library: Chart.js, but I found the program that Sara and Rui presented cleaner and clearer. Also, they fully separate the HTML file from the ino sketch (storing the HTML in SPIFFS) and that was something I wanted too, so I set about to adapt their program to my needs. I must confess that I am not an expert on javascript, asynchronous websites, websockets and AJAX programming, but this seemed like a great way to dive deeper.
I started with identifying where in the program the data was sent and where it was further processed
In the BME280 program 3 separate calls are made in which subsequently temperature, humidity and pressure are sent to the server.
For the temperature this goes as follows:

server.on("/temperature", HTTP_GET, [](AsyncWebServerRequest *request){
    request->send_P(200, "text/plain", readBME280Temperature().c_str());

The request->send_P is a way to send an entire webpage from PROGMEM, but in fact, only the result of the temperature reading is being sent. It seemed to me the best way to add temperature readings (from several DS18B20’s) they needed to be added here somehow. It seemed most sensible to do that with a JSON format.

I found the data was received/processed in the HTML file by the setInterval(function ( ) in the line y = parseFloat(this.responseText);. Obviously I needed to add one or more variables here and instead of using parsefloat, I would need to parse the JSON like this:

var myObj = JSON.parse(this.responseText);
var y=myObj.Temp[0];
var z=myObj.Temp[1];

OK, so that seemed covered, now the Highcharts library needed some further digging in to for me in order to actually add the values to a second and third line in the chart. There are some examples on the cloudflare and jsfiddle website, but those seemed to have little in common with the structure I already had, so obviously this needed some trial and error.

To keep things manageble as an example, I will add 1 channel to the temperature chart that Sara and Rui Santos made and will leave the Humidity and AirPressure graphs untouched.

So first lets adapt the ino file.
Go to the server.on("/temperature" procedure and replace that one by:

server.on("/temperature", HTTP_GET, [](AsyncWebServerRequest *request){
     String payload = "{\"Temp\":[" + String(readBME280Temperature()) + "," + String(readDS18B20()) + "]}";
    Serial.println(payload);
    request->send(200, "text/plain", payload.c_str());

The readDS18B20() should be your procedure to read the result of a DS18B20 (or any other sensor). and yes, I am using ‘send‘, not ‘send_P.’
The result -i.e. the JSON, can be checked on /temparature20190807_200243.jpg

The HTML file needs various changes,not only in the AJAX-serverrequest section, but also in the Highcharts definitions, so I best just post that here in totality.
You could play around with the various Highchart settings. You may have noticed that the colors of both lines are the same. Might be a good excercise for you to try and define a different color . A hint: in the index.html file, as far as I understand Highcharts, the color statement that Rui&Sara use (series: { color: '#059e8a' }) under ‘plot options’ is a general one for the entire graph (1 of the 3). That is a possibility with more than one line too, but you can also define colors per line. Try it, it is not so hard. (I believe that if you do not define any color at all, Highcharts will use default colours, but I did not try that.)

BME3

It is also easy to change the line markers,BME4

or make a dotted line, give it a try.

BME5

or display negative results in a different color:

bme6

(Yes, I know I am not giving you the exact code for that, trying to stimulate, and really, only takes minutes. You may find this link of interest)

Turning LED OFF and ON on the ESP32 camera module, using bluetooth.

The popular ESP32 camera module that I discussed in my previous post has a large build in LED.
In the standard build in camera program, that LED is always on. In the ESP32 camera example that comes with the ESP32 core in the Arduino IDE, that LED is OFF, both programs have no possibility to toggle the state of the LED, which is a pity.
I had a glance at adding a button, that would be visible in the UI, but that seemed quite an undertaking, and I thought it would be easier to use bluetooth for that.
In order to do that, we need to do the following:
Open the Camera example in the IDE and add the below statements to your declaration section (say right above the “// WARNING!!! Make sure that you have either selected ESP32 Wrover Module”)

#include "BluetoothSerial.h"
#if !defined(CONFIG_BT_ENABLED) || !defined(CONFIG_BLUEDROID_ENABLED)
#error Bluetooth is not enabled! Please run `make menuconfig` to and enable it
#endif
BluetoothSerial SerialBT;
int incoming;
#define LED_BUILTIN 4

Once you have done that, go to the setup() section and add:

Serial.begin(115200);
  SerialBT.begin("ESP32Camera"); //Bluetooth device name
  Serial.println("The device started, now you can pair it");
  Serial.setDebugOutput(true);
  Serial.println();
  pinMode (LED_BUILTIN, OUTPUT);//Specify that LED pin is output

Subsequently go to the “loop” section. That should only have a delay(10000); statement. Replace the entire ‘void loop() by:

void loop() {
  // put your main code here, to run repeatedly:
   if (SerialBT.available()) //Check if we receive anything from Bluetooth
  {
    incoming = SerialBT.read(); //Read what we recevive 
    Serial.print("Received:"); 
    Serial.println(incoming);

    if (incoming == 49)
        {
        digitalWrite(LED_BUILTIN, HIGH);
        SerialBT.println("LED turned ON");
        }
        
    if (incoming == 48)
        {
        digitalWrite(LED_BUILTIN, LOW);
        SerialBT.println("LED turned OFF");
        }     
  }
  delay(20);
}

Upload that program and check if the camera functions as normal. Then you need some kind of Bluetooth application on your phone. There are various options, but I like “Serial Bluetooth Terminal“.
To pair the Camera, open the Serial Bluetooth Terminal and tap the 3 horizontal bars in the top left.
Then choose ‘Devices’, make sure you are in the Bluetooth Classic tab, and tap the cogwheel in the top right. There you can scan for available devices and once found pair with ESP32Camera.

Then go back to the ‘Devices’ menu you were before and choose ESP32Camera (it is likely the only one there). Tap the back arrow, so you are in the main terminal menu and tap the ‘connect’  icon in the top right (it is just left of the ‘delete bin’.


The terminal window will show you the connection status.
Then tap in the command window (the one at the bottom), your keyboard comes up and press 1, to turn on the LED and 0 to turn off the LED, press ‘send’ on yr keyboard, or the right arrow in the terminal to send yr command.


Full software found here.

ESP32 Camera module

20190803_150613.jpgVarious Chinese webstores sell an ESP32 based Camera module. A known one is from LilyGo, but there are  very cheap, simpler modules available too. The cheapest I found is the module as seen in the picture at 4.44 Euro, shipping included.

Using the built in camera program

Although it might not be the case for similar modules from various suppliers, mine actually already came with a camera application present in the ESP32.
To use that built-in program, one needs to connect to the ESP32-CAM acces-point in your WiFi list:

and then point your browser to ‘192.168.4.1/jpg_stream’ for  streaming video, or ‘192.168.4.1/jpg’ for stills.
The documentation on this webserver is very scarce, only a bit of information is given here. But that does not really give much extra info. The Firmware in the camera could easily be something like this.

Uploading your own camera program

So far so good if you only want a simple IP cam and you do not mind it has its own AP. It is fairly easy though to upload a camera program, that for instance does not need connection to a separate AccessPoint, but what can be found within ones own regular network.

Such a program can be found among the ESP32 examples in the IDE:

We need to make a few changes in the program:
For my specific  module, I needed to select (lines 9-14 in the program)
“#define CAMERA_MODEL_AI_THINKER”
and comment out the other defines.
For you that might be another model.
If you happen to have a TTGO model, then you find the proper pin definitions here.
Now right under that, we define a string with the name we like the module to have in the DHCP list
Then find the “WiFi.begin()” command and add  2 lines in front, like so:

You will find it here.  Mind you though that this program needs ESP32 core 1.02 (or higher) to compile !!!! The 1.01 core comes with is own version of this program.
Now we are (almost) ready for upload. As this ESP32-CAM module has no USB port for uploading, we need a 3V3 FTDI adapter to upload programs. Make sure it is a 3v3 FTDI adaptor (NOT a 5 Volt)

The connections are as follows

FTDI <-> ESP32
Ground <-> Ground
Rx <-> UOT
Tx<-> UOR
3V3<->3V3

Additionally on the ESP32 module Pin IO0 needs to be connected with GND to put the module in Programming mode.
For my specific  module, I needed to select (lines 9-14 in the program)
“#define CAMERA_MODEL_AI_THINKER”
and comment out the other defines.
Then in the IDE, under Tools, I needed to select
Board: “ESP32 Wrover module”
Partition Scheme: Huge APP (3MB No OTA)
Once the program is uploaded, Remove the  connection between IO0 and GND and press the reset button.
Now you need to find out the IP number. You can find this by opening the serial monitor, where the IP number is printed.
IF in the serial monitor you may see a ‘Brownout detector’  warning and repeated failed attempts to start the software, do not despair, it just means that your USB port does not provide enough power.
If that happens, connect the module to a proper powersource and check in the DHCP client list of your router, what the IP number is. The module should identify itself with the name we previously defined in the program. If you didn’t do that, it will identify itself as ‘expressif’.

Then open a browser and direct it to the IP number of the module.
In the screen that subsequently opens, scroll down and click “Start Stream”.
This should start the  camera stream.

There is also a button that says “Still”. That one should just make a still picture, but in the example program that we used, it will not store that picture. But that should not be too hard to add. I may do that in the future.Should you want to dabble into that, the following pins are connected to the SDCard:

  • GPIO 2: Data 0
  • GPIO 4: Data 1 (also connected to the on-board whiteLED)
  • GPIO 12: Data 2
  • GPIO 13: Data 3 (also connected to a small red led)
  • GPIO 14: CLK
  • GPIO 15: CMD

Mind you that the program will only allow one client to view the stream

Face Detection
The webserver has buttons for face detection. Sadly there is a bug. Face detection is working in the 1.01 core but not in the 1.02&1.03 core. So if you have core 1.02, you have to roll back AND use the ESP32CameraWebServer example program coming with THAT core. If you cant find that, it is (still) on the randomnerd github from Rui & Sara Santos

The site of Rui & Sara Santos is definitely good source on the ESP32Camera module is.

LED
The module has a nice fat LED on board. If you want to control that for your camera, check here.

Solved: Starting programs at boot didn’t work on raspberry

Today I made a new fresh install of Raspbian-Stretch on my old RaspberryPi 1B. That board mainly has some python programs on it that MQTT info onto my network to OpenHab. The previous install had become a bit broken I think, it would crash every two days and rather than sorting that out, I decided a fresh install would be the quickest.

After installing a fresh Stretch image, I put all the programs and required libraries back on, a quick test and that all worked.
OK, now it is time to have the programs start automatically at boot.
That is also simple, I usually add the programs I want to start at boot in my /etc/rc.local file.

Always worked for me.
However, this time it didn’t and for the life of me I couldn’t figure it out. Why is a setup that used to work suddenly not working anymore, starting the programs at boot, while the programs itself work when started manually.
Trying trying trying. Writing the output from each command to a log file didn’t help me, logfiles stayed empty.
Then I decided to write the output of the entire rc.local file to a logfile with:

exec > /tmp/rc-local.out 2>&1;set -x

put that in the rc.local file right before the line that says “# Print the IP address”:

I rebooted and checked the file: /tmp/rc-local.out.

Now it became clear…. the paho (mqtt) module that these python programs were using was not found.
Odd.. I am sure I installed it……and when I manually start the programs they work 100%

I was thinking….. in my previous setup, I had installed the paho.mqtt library manually. It resided in a folder in my /home/pi directory. This time I had installed it with pip. Now it wouldn’t be that…… no, it couldn’t, could it?
The commands in the rc.local file, as far as I know are given by root and my /home/pi directory was owned by ‘pi’…….but that hadn’t changed between two installs and root has privilege anywhere right?

After some more searching I learned that the library “paho” had been installed by pip in the folder “/home/pi/.local/lib/python2.7/site-packages” but “sudo python” searches this library in the folder “/usr/local/lib/python2.7/dist-packages”.

I was not using sudo… but that is in fact what the rc.local does, so I was guessing that that was where my problem lied.

So, I needed to link the one directory to the other somehow.

This is what did that:

cd /usr/lib/python2.7/dist-packages
sudo ln -s /home/pi/.local/lib/python2.7/site-packages/paho

Then it all worked again.

Interesting info on how to start your programs at boot can be found here.