Very DeepSleep and energy saving on ESP8266 – Part 2, Sending data with HTTP

In an earlier post,  I showed how to  try and do some major energy saving in case you are using a battery. I gave a working framework sketch, that did all but actually read data or send data as it just focussed on the energy saving.

Though it isn’t hard to add that to the program, as I indicated where you could add a ‘read sensor’ and ‘senddata’ procedure, I realize that that may still cause some obstacles for a beginner, so I will show how to add some code to send the data somewhere.

Though MQTT is a popular way to send data, in this example I have chosen to send data to a MySQL database (in fact I am using MariaDB, but that works the same).

As there is no reason to reinvent the wheel, for setting up the database and the needed  php files, I refer to the randomnerdstutorial website that has 2 excellent tutorials about this. W will use their server-side code (database and php files) to send the data to.

We will use the BME280 to gather the data.

Since the BME280 is used as I2C device, we need to inlude the Wire library. As we will use the Adafruit library to read the data, we will actually need two  more libraries, that we add in the global section of the program. We will also declare the object:

#include <Wire.h>
#include <Adafruit_Sensor.h>
#include <Adafruit_BME280.h> //defaults to 0x77, change that if yr BME is 0x76
Adafruit_BME280 bme;

In the Setup() section, we have to initialize the library.  The library defaults to the BME having an I2C address of 0x76, but if you happen to have a 0x77 module, here is where you can change the default.
We then do am optional check to see if the BME280 is detected and in principle you are good to go. However, in my previous article I mentioned you could put the BME280 to sleep in between readings and that is what we will do. Therefore for now I only added a jump to a procedure that i call ‘BMEsetup’. We will fill it in later

Wire.begin();
//bme.begin() will return True if the sensor was found, and False if not. If you get a False value back, check your wiring!
status = bme.begin(); //bme.begin(address) 0x76 or 0x77
if (!status) {
Serial.println("no BME detected");
delay(1);
}
BMEsetup(); // set weathermonitormode

So now we have to do 2 things: we have to read the sensor  and send those in an HTTP request.

As we are using Forced Mode we have to wake the sensor and tell it it has to take a new measurement:

bme.takeForcedMeasurement();

We can read the sensor and  build  our HTTP request string in one go like so:

// Prepare your HTTP POST request data

httpRequestData = "api_key=" + apiKeyValue + "&sensor=" + sensorName
+ "&location=" + sensorLocation + "&value1=" + String(bme.readTemperature())
+ "&value2=" + String(bme.readPressure() / 100.0F) + "&value3=" + String(bme.readHumidity())+ "";

Sadly, the Adafruit library is extremely inefficient in reading the data. The datasheet advises a ‘burst read’ of the data, both for speed and to prevent mix-up of data. The adafruit library has chosen to do separate register readings, but to make matters worse, it precedes the pressure and humidity readings each with a temperature reading first, which is then discarded. So 3 measurements become 5 which adds at least 4 ms to the total. As we are reading the temperature first in our httprequest string, we could opt to remove the extra reads from the Adafruit library. The Sparkfun library does not have these extra reads.

The total conversion time of the 3 measurements comes to 13msec and it is better to do that while the radio is still off, especially when the library adds anothe 4 msecs

As I plan to send the http request in a seperate procedure (to keep it all clear and transparent), we have to define the String variable httpRequestData as a global variable (by adding it in the global section of the program), like so:

String httpRequestData;

Then it is time to Switch the WiFi on like described before and make our http request.

void sql() {
//Check WiFi connection status
if (WiFi.status() == WL_CONNECTED) {
HTTPClient http;

// Your Domain name with URL path or IP address with path
http.begin(serverName);// deprecated.

// Specify content-type header
http.addHeader("Content-Type", "application/x-www-form-urlencoded");

// Send the request and collect the response
int httpResponseCode = http.POST(httpRequestData);

if (httpResponseCode > 0) {
Serial.print("HTTP Response code: ");
Serial.println(httpResponseCode);
}
else {
Serial.print("Error code: ");
Serial.println(httpResponseCode);
}
// Free resources
http.end();
}
else {
Serial.println("WiFi Disconnected");
}
}

The http.begin(serverName); command, is or will be deprecated in near future. I understand the proper command then will be: http.begin(client,serverName);. Current code is working.

Now we only have to define the BMEsetup routine that we called earlier. Thats an easy one:

// weather monitoring
void BMEsetup() {
Serial.println("-- Weather Station Scenario --");
Serial.println("forced mode, 1x temperature / 1x humidity / 1x pressure oversampling,");
Serial.println("filter off");
bme.setSampling(Adafruit_BME280::MODE_FORCED,
Adafruit_BME280::SAMPLING_X1, // temperature
Adafruit_BME280::SAMPLING_X1, // pressure
Adafruit_BME280::SAMPLING_X1, // humidity
Adafruit_BME280::FILTER_OFF );
}

Download the entire file here.  After you got it working feel free to delete the print statements.

A word on voltage monitoring
In my previous article I mentioned that voltage monitoring -a popular thing to do when using a battery- needs a voltage divider that can be a considerable, constant drain on on the battery.
If you are using a LiPobattery that can be as high as 4.2Volts, a 320k over 1ook voltage divider is a decent choice, but still adds to a discharge of the batteries. From that aspect, measuring the internal Vcc is an interesting alternative.
Sure, if you use a stabiliser to get the battery voltage down to 3.3 Volt, initially you have no info on the battery voltage, but you will know when it gets critical.
Presume you have an HT78333 that has a voltage drop og 360 mV. Suppose the output is a rock solid 3.3Volt, then the only thing you know is that your battery is still OK, i.e >3.63 Volt. The moment your Vcc drops below 3.3Volt you know yr battery is dropping below its nominal 3.7 Volt, while there still is plenty of juice to feed your ESP.
If you want to monitor the internal Vcc, then just add.

ADC_MODE(ADC_VCC);

to the declaration section and read the voltage with

ESP.getVcc()/1023.0F

Adding fields
Should you want to add a field, e.g. for the battery voltage, then there are several things to do:

    1. add  String(ESP.getVcc()/1023.0F) to the httpRequest string
    2. add a field to the database. you can do that with
      ALTER TABLE yourtable ADD newfieldname VARCHAR() after fieldname

      but it is better to use your own specific database management system. (I use webmin)

    3. Adapt the post-esp-data.php file to collect and store data, to cater for an extra field. Go ahead, try, it is simple
    4. Do the same for the esp_data.php file

Board
The best board to use would be an ESP12F:

In a follow up article I will show how to implement MQTT with this deep sleep example

Very DeepSleep and energy saving on ESP8266 – Part 1

Intro: For the seasoned ESP user I might be over explaining things here, but apparently some people struggle with basic things around deepsleep, hence my perhaps somewhat overly long explanation.

For battery fed projects, low energy consumption is a must.
De ESP8266 can help with that, using DeepSleep.

Earlier I published a rather simple DeepSleep program, combined with MQTT, but from an energy conserving point of view  there are more things in setting up DeepSleep that can make a difference. There are also couple of other things to take into consideration.

The ideas here are not new. I have gotten some inspiration from Erik Bakke’s OppoverBakke site, even though there was a mistake in his code snippets.

I will just describe what I did to bring my ESP8266 to minimal energy consumption.

1. Pick the proper ESP8266 board.
You want your ESP board as lean as possible. No other power hungry other components such as FTDI chip or LEDs. This basically excludes most modules such as Wemos D1 mino or NodeMCU and only leaves the bare chips such as ESP-03, ESP-04 and ESP12 on a carrier board. The ESP12 is the easiest though as the ESP-03 and ESP-04 for instance do not have the GPIO16 or RST pin broken out to a side connector, but to a pad on the PCB.
The ESP8266-01 would be perfect after you remove the LED, but it does not have the DO/GPIO16 pin broken out. If you have young eyes and a steady hand you can try to solder a wire on the GPIO16 pin.

2. Pick the right voltage regulator.
If you are using 2×1.5V AA cells (unlikely) you may get by without a regulator, but if for instance you are using 3×1.2V NiCd-s or a 3.7 Lipo (that can go up as high as 4.2Volt), you are going to need a regulator. You dont want a regulator that needs a big input voltage in order to maintain 3.3Volt, so you will need a Low drop regulator, a so called ‘LDO’
Possible candidates are:

Type Quiescent current Drop Voltage Max Input Voltage Max Output Package
Torex XC6206 1uA ‎ 160mV@100mA 6Volt 200mA SOT23-3, SOT89,USP-6B
Analog Devices LTC3525-3.3 ‎ 7uA ‎ Step up converter ‎ 6Volt ‎ 60mA at 3.3V from a 1V Input, or 140mA at 3.3V from a 1.8V Input‎
Holtek HT7333 4uA 90mV@40mA 12V 250mA SOT89,TO92
Holtek HT7833 4uA 360mV@500mA 8.5V 500mA SOT89,TO92
ME6211 40uA 100mV@100mA 6.5V 300-500mA SOT23, SOT89
SPX3819M5 90uA 340mV@full load 16V 800mA SOT23-5,SOIC8,DFN8
Microchip MCP1700 1.6uA 178 mV@ 250 mA 6V 250mA SOT23, SOT89, TO92
Microchip MCP1825 120uA 210mV@ 500 mA 6V 500mA SOT23, SOT89, TO92
LT-1763 30uA 300 20V 500mA S8
TPS783xx 500nA 130-175 6V 150mA SOT5

The problem here is that many of the LDO’s that have a very low quiescent current, also have an output current that is on the edge of what the ESP8266 needs at startup.

The MCP1700 as well as the HT7333 are often advised,but they are really at the verge of what an ESP8266 needs in current using WiFi.

In all honesty I would advise AGAINST using any regulator giving less than 300mA.

Apparently it is possible to use two HT7333 chips in parallel. The HT7833 seems a pretty good choice. It typically needs a 1uF elco on the input and a 2.2uF elco on the output to be stable. A higher value is ofcourse a good idea.

The ME6211 is used on the Wemos D1 mini board (which does not mean it is the best choice for low-drop/low Iq). Same goes for the SPX3819M5 on the Lolin NodeMCU.

3. Pick the right internet connection
Often we pick a DHCP connection by default. A DHCP connection usually  takes longer to establish than a static IP connection. It is therefore best to chose for the latter. With the ESP it is simply a matter of providing the following details (mind you, the IPnrs are just an example)

//We will use static ip
IPAddress ip(192, 168, 1, 35);// 
IPAddress dns(192.168.1.1);
IPAddress gateway(192, 168, 1, 1); //
IPAddress subnet(255, 255, 255, 0); // this one is pretty standard

and then make the connection with

WiFi.config( ip, dns, gateway, subnet );
WiFi.begin( WLAN_SSID, WLAN_PASSWD );

That is not the only thing we can do making the connection. We can try circumvent some oddities that the ESP8266 shows in connecting to the WiFi network:

the ESP8266 persists the network connection information to flash, and then reads it back when it next starts the WiFi function. It does this every time, and it takes at least 1.2 seconds (source Erik Bakke).

The chip does this even when you pass connection information to WiFi.begin(), like:

WiFi.begin( WLAN_SSID, WLAN_PASSWD );

This will actually load the connection information from flash, ignore it and use the values you specify instead, connect to the WiFi and then finally write your values back to flash.

We can disable that with the command WiFi.persistent( false )

There is more we can do to save time though. The Wifi.begin(SSID,PASSWORD) command does a scan for the proper network amongst the ones advertised. It is possible to supply a WiFi channel number and a BSSID to the WiFi.begin() command that takes away the need for a scan.
The ESP does not know the channel number or BSSID, so we need to read that info from the first connection, store it, and submit it at the next connection.
For that we need to replace the Wifi.begin(); statement with a piece of code that retrieves and manages the necessary information.
Fortunately there is such code available, and as I saw it in various places, I would not know  who to credit for it.
We begin with defining a structure:

A structure aka. struct is a way to group variables together, possibly of different types. One can have several instances of a declared structure. The variables within a structure are called members.

We define the following structure:

struct {
  uint32_t crc32;   // 4 bytes
  uint8_t channel;  // 1 byte,   5 in total
  uint8_t ap_mac[6];// 6 bytes, 11 in total
  uint8_t padding;  // 1 byte,  12 in total
} rtcData;

We use a CRC checksum to see if the data we use perhaps got corrupted. We use 1 bte for the channel and 6 bytes for the Accespoint information.
We add one superfluous byte to come to a multiple of 4. (here is more about storing data in rtc memory).

Then on connecting we make a few checks:

        1. is the CRC data correct? If not, make a normal connection.
        2. if after 100 tries it is not working, we reset the WiFi and go make a normal connection
        3. If  after 600 tries still no connection, we put the ESP8266 and will try later again.

We do need a routine to do the CRC-check, but that’s easily done.
Once we made the WiFi connection, we can connect to where we need to go: an MQTT broker or make an HTTP request for instance.

4. Make sure you do not connect to WiFi before you have to.
Making the connection at the right moment is important though. You do not want to waste energy on your 2.4 GHz signal when you are still reading sensors. As it is, the ESP8266   already has its radio on by default.
So the first thing we need to do upon wake up, is to switch off the radio.
We do that like this:

void setup() {   // disable WiFi, coming from DeepSleep, as we do not need it right away
  WiFi.mode( WIFI_OFF );
  WiFi.forceSleepBegin();
  delay( 1 );

Only once we have done that, we let the program do what it must, like reading sensors, and only then do we start thinking about making the  WiFi connection. Ofcourse we first need to switch the modem/radio on and we do that with:

//Switch Radio back On
WiFi.forceSleepWake();
delay( 1 );

That brings us almost to the end of what we need to do, software wise:
We Switch off the radio, read the sensors, switch on the radio, try make a quick connection and if necessary a regular connection, we then connected to where we want to go (Thingspeak,  MQTTbroker, whatever)and then it is finally time to put the ESP8266 into DeepSleep.

5.Putting the ESP8266 into deepsleep
The simplest form of a DeepSleep program is to do the stuff you need to do  (read yr sensors,make connections and then call:
ESP.deepSleep(SLEEPTIME)
There is however more that we can do. remember that I said that on wake up the radio is on by default? Well we can give the ESP8266 a wake up instruction so it knows what to do when waking up. We can already tell the ESP8266 that it should NOT switch on the radio by default. We do this with:

ESP.deepSleep(SLEEPTIME, WAKE_RF_DISABLED);

I know that in an earlier step we started with switching the radio off and strictly speaking that step would no longer be necessary once we use the wake up instruction, but I left it in anyway, just to make sure/for illustration. You could opt to leave it out though.
Truthfully there is a bit more superfluous code. e.g. it is not really necessary to switch off the WiFi immediately before DeepSleep.

Only one thing remains now: How long will we put the ESP8266 to sleep?

That ofcourse is a very personal decision, but lets see how long we CAN put the ESP to sleep.

The max deepsleep USED to be max 71 minutes. That was simply because the system_deep_sleep parameter was defined as an unsigned 32 bit integer. The max value of a 32bit unsigned integer is 0xFFFFFFFF, that is 71 min in usecs.
Since the 2.4.1 core, it is a 64 byte unsigned integer. That comes to 18446744073709551615 us or 5,124,095,576 hrs or about half a million year. Nobody wants to wait that long, so expressif seem to have capped that to about 3.5 hrs

This capped number is defined in the ESP.deepSleep(Max) parameter. So when I defined my deepsleep call as ESP.deepSleep(ESP.deepSleepMax(),WAKE_RF_DISABLED );, my ESP went to sleep for 3hrs and 9 min.

Testing on a Wemos D1 mini. Connect RST pin with a jumper tp pin D0, as shown here

You will find a DeepSleep program as described above, here. The only thing you need to add other than your web credentials is some code to read your sensors and some code to send your data to where you want to store it (e.g. Thingspeak or yr own webserver, or a broker.
Upload the code and AFTER your upload, connect D0/gpio16 to the RST pin.

6. Think about your sensors
Your sensors use power as well, even when your ESP8266 is sleeping. Perhaps there is a sensor that does the same as another sensor, but uses less power. Consider feeding yr sensors from a gpio pin that you switch LOW when the sensor is not used. Keep in mind that some sensors need a warm up or settling time.  If you do that, ALWAYS make sure your sensor does not draw more than an ESP8266 pin can deliver (12mA), but chances are that if it does, it isn’t one that you should feed from a battery anyway. If you really have to or want to, you can opt for a FET to switch on the sensor. A very popular ‘sensor’ in battery fed projects is measuring the battery voltage. As the ADC of the ESP8266 only goes up to 1 Volt, some form of resistor voltage divider is necessary. This means a constant drain on the battery, so it is best to chose high values for the divider. If you use a 420k divider, you would be constantly draining 10uA on a full battery. That may not seem much, but it is also what your entire ESP8266 uses in deepsleep. If you are not using a power regulator, for instance when you are using 2×1.5Volt batteries, then it is better to use the internal measurement of the Vcc. Do that with:

ADC_MODE(ADC_VCC);
float vccVolt = ESP.getVcc()/1024.0F;

Some sensors –like the BME280– can be programmed to go to sleep in between measurements. For the BME280 this is the ‘Forced Mode’ if you then have a 1/min reading (Bosch calls this the ‘weather monitoring state’) the power consumption is 0.16uA. Using forced mode is also a good way to prevent/limit internal warming up of the chip, which would give skewed readings.

In a follow up article, I will publish some code that uses this deepsleep framework to read a sensor and send it to a database.

 

Using Telegram with ESP8266 or ESP32

Telegram is a ‘Whatsapp’ like app for mobile phones to exchange messages.
An advantage over Whatsapp or Viber is that it also allows you to communicate with your (micro)computer.
That means your phone can receive messages sent from your ESP8266 or ESP32, but also that your phone can be used to control your ESP8266 or ESP32.

In order to do so you need to have the telegram app installed, create your own bot to handle the traffic, get an API_key and the chat_id for your bot

Install Telegram
Visit Telegram official website and download the official application for your smartphone, or get it from the playstore

Create your Bot
Create your bot by using a bot called “BotFather”. To do this, add BotFather to your contacts or visit telegram.me/botfather and follow the instructions to create your own bot.

Get your API
When your bot is created, BotFather gives you the HTTP API token which follows a pattern like 123456789:AABBCCDD_abcdefghijklmnopqrstuvwxyz. You need this token to send messages.

Get your chat_id
You will need this ID to send messages, you do not need it to receive messages. Getting this ID is not straightforward and most of the methods i have seen on the internet do not work (anymore).
The only ones I found working is to the IDBot (and follow the instructions). and the get_id_bot. The latter can be a bit slow though. (I will mention a 3rd option further down)

Getting your chat ID with IDBot
Getting your chat ID with get_id bot

The id is an integer and it follows a pattern like 123456789

Now it is time to test your bot.
You can do that with cURL

curl --data chat_id="892915001" --data "text=Hello World" "https://api.telegram.org/bot977142506:AAGYWypUbMN8BCthML-tFM25M403r2yCsFj/sendMessage

from your commandline prompt

or with a GET URL in a browser like this:

https://api.telegram.org/bot977142506:AAGYWypUbMN8BCthML-tFM25M403r2yCsFj/sendMessage?chat_id=892915001&text=Hello-World

There is an other element that can be added at the end and that is &parse_mode=, but for normal operations we can leave that out or make it “&parse_mode=HTML”.  A bit more on this further down

Using GET
So now we have our communication with Telegram working, we can go do something useful with it.
The fact that the API can be reached with a GET message opens up possibilities for an ESP8266, ESP32, Raspberry Pi or other controller that is connected to internet.

You would do that with

botclient.print(String("GET ") + url + " HTTP/1.1\r\n" +
"Host: " + host + "\r\n" +
"User-Agent: BuildFailureDetectorESP8266\r\n" +
"Connection: close\r\n\r\n");

in which

const char* host = "api.telegram.org"

and the
String url is the sum of the strings and variables you want to send

In your declaration section you would be stating:

WiFiClientSecure botclient;

The reason I am not putting a complete and ready “Telegram via GET” program here is because there is actually a pretty decent library program available, so if you insist on using your own GET statements rather than a library, I presume you do that to make very lean code in which case I presume you are versed enough in coding to build your own routine with the help of the above info.
If you use a GET URL to connect your ESP to Telegram and you are sure your code is correct, you may have a secure connection problem. Read On

Using a library
There are several libraries for Telegram available and you probably already find them in your library manager. The “UniversalTelegramBot” is a pretty decent one and the examples should get you up and running fast, even if from first glance it is not always clear what they do 🙂 .
So, install the library via library manager and lets have a look at the “PhotoFromURL”example.

When you fill in your WiFi credentials and Telegram API, you may wonder “where the heck goes my chat_id”.
Well, remember earlier I said you only needed the chat_id when you wanted to send something from yr micro to your phone? Well, that’s the case right here.
“Yes, but this program IS supposed to send me something and I clearly see a variable called chat_id, so where do I put mine” I hear you say.
Well this program will send you a picture ONLY when you tell it to do that and then the program will just read your chat_id from the incoming message and use that. This is also the 3rd method of finding your client ID

So here is  what you do to use that example: After you upload it to your ESP8266, go to your Telegram app on your phone. select the bot you just created to send a message to and message /start.

After a few moments the bot responds with an overview of the commands you can give (only one in this case). Tap on that command to execute it.

It is not working
If nothing happens, it is time to open the serial monitor.
You should see the IP number being printed. Make sure you type the correct command in your Telegram app and make sure you are sending it to the proper bot.

If you don’t see the ‘got response’ message appearing and you are sure you have inserted the right API, then apparently something else is keeping the program from opening a socket to the bot.

Could it be that you are still using the 2.5.0 ESP8266 core? There are plenty good reasons to use that core, but apparently it has a problem with secure connections. But not all is lost. Here is what you do:

Go to setup() and add

client.setInsecure();

and then compile and upload the file again.

Beware! In this example the WiFiSecure() client IS called client. That maybe different in other examples (in the Bulkmessages example it is called ‘secured_client’. Change your setInsecure statement accordingly.

Controlling your ESP8266/ESP32 via Telegram
It is possible to control your ESP via your Telegram App. In fact, that is what the previous example was already doing: we  instructed it to send a photo.

The FlashLed example shows very clear how to control your ESP. If for testing you want to use the inbuild led from the Wemos, the ledPin should be ‘2’, the ledon routine should make the pin LOW and the ledoff routine should make the pin HIGH.

You could combine functions like so:

/*******************************************************************
 *  How to use basic functiond of a Telegrambot
 *  A completely changed example based on a sketch from  
 *      Vadim Sinitski
 *******************************************************************/
#include <ESP8266WiFi.h>
#include <WiFiClientSecure.h>
#include <UniversalTelegramBot.h>

// Initialize Wifi connection to the router
char ssid[] = "NETGEAR1";     // your network SSID (name)
char password[] = "SECRET"; // your network key

// Initialize Telegram BOT
#define BOTtoken "XXXXXXXXX:YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY"  // your Bot Token (Get from Botfather)

WiFiClientSecure client;
UniversalTelegramBot bot(BOTtoken, client);

int Bot_mtbs = 1000; //mean time between scan messages
long Bot_lasttime;   //last time messages' scan has been done
bool Start = false;

const int ledPin = 2;
int ledStatus = 0;

String test_photo_url = "https://img.srgcdn.com/e/w:750/cjh2NWp4S0FiNWxJQVNsQ3JUU28uanBn.jpg"; // can insert any picture URL

void handleNewMessages(int numNewMessages) {
  Serial.println("handleNewMessages");
  Serial.println(String(numNewMessages));


  for (int i = 0; i < numNewMessages; i++) {
    String chat_id = String(bot.messages[i].chat_id);
    String text = bot.messages[i].text;

    String from_name = bot.messages[i].from_name;
    if (from_name == "") from_name = "Guest";

    if (text == "/get_photo") {
      bot.sendPhoto(chat_id, test_photo_url, "Who");
    }

    //-------------
    //add more commands here
    //-------------
    
    if (text == "/ledon") {
      digitalWrite(ledPin, LOW);   // turn the LED on (HIGH is the voltage level)
      ledStatus = 1;
      bot.sendMessage(chat_id, "Led is ON", "");
    }

    if (text == "/ledoff") {
      ledStatus = 0;
      digitalWrite(ledPin, HIGH);    // turn the LED off (LOW is the voltage level)
      bot.sendMessage(chat_id, "Led is OFF", "");
    }

    if (text == "/status") {
      if (ledStatus) {
        bot.sendMessage(chat_id, "Led is ON", "");
      } else {
        bot.sendMessage(chat_id, "Led is OFF", "");
      }
    }
    

    if (text == "/start") {
      String welcome = "Welcome to Telegram Bot, " + from_name + ".\n";
      welcome += "This will show several properies.\n\n";
      welcome += "/get_photo : getting photo\n";
      welcome += "/ledon : to switch the Led ON\n";
      welcome += "/ledoff : to switch the Led OFF\n";
      welcome += "/status : Returns current status of LED\n";


      bot.sendMessage(chat_id, welcome, "");
    }
  }
}

void setup() {
  Serial.begin(115200);

  // Set WiFi to station mode and disconnect from an AP if it was Previously connected
  WiFi.mode(WIFI_STA);
  WiFi.disconnect();
  delay(100);

  // attempt to connect to Wifi network:
  Serial.print("Connecting Wifi: ");
  Serial.println(ssid);
  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    Serial.print(".");
    delay(500);
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());
  client.setInsecure(); // if yr core cannot handle secure connections

  pinMode(ledPin, OUTPUT); // initialize digital ledPin as an output.
  delay(10);
  digitalWrite(ledPin, HIGH); // initialize pin as off

}

void loop() {
  if (millis() > Bot_lasttime + Bot_mtbs)  {
    int numNewMessages = bot.getUpdates(bot.last_message_received + 1);

    while (numNewMessages) {
      Serial.println("got response");
      handleNewMessages(numNewMessages);
      numNewMessages = bot.getUpdates(bot.last_message_received + 1);
    }

    Bot_lasttime = millis();
  }
}

Or download here.

I just want to have my ESP send notifications
Well in fact the previous examples kinda did that, but if you really just want to have notifications sent to your phone from your ESP, then the only command you will need from the library is

bot.sendMessage(chat_id, "My message", "");

Provided of course you called your library object ‘bot’. Those empty quotes at the end are there for the parse_mode (read on).
Instead of “my message” you can also insert a String variable in which you add text and variables and you can even use simple HTML coding in that string.  In that case the command needs to be:

bot.sendMessage(chat_id, messagestring, "HTML");

your messagestring could for instance be:

String messagestring="The temperature is  <b>"+String(tempreading+ "</b>Celsius";

more about the API here!

Storing sensor data on a (local) server using POST or GET: client side

In a previous article I described PHP scripts that would store sensor data in a CSV file and display  the data in a graph.

Now I wanted to give two Arduino/ENC28J60 programs that would send that data. One using POST, the other one using GET, using the EtherCard library. That proved a bit harder than I thought.

Using POST was easy, the program below does that. It reads the value of two analog value and sends those to a server side PHP script that stores the data in a CSV file

#include <EtherCard.h>
#define PATH "POST-data.php"

// ethernet interface mac address, must be unique on the LAN
byte mymac[] = { 0x74, 0x69, 0x69, 0x2D, 0x30, 0x31 };

const char website[] PROGMEM = "yoursite.com";// in case you use that

byte Ethernet::buffer[700];
uint32_t timer;
Stash stash;

void setup () {
Serial.begin(57600);
Serial.println("POST Data");

if (ether.begin(sizeof Ethernet::buffer, mymac, 10) == 0)
Serial.println( "Failed to access Ethernet controller");
if (!ether.dhcpSetup())
Serial.println("DHCP failed");

ether.printIp("IP: ", ether.myip);
ether.printIp("GW: ", ether.gwip);
ether.printIp("DNS: ", ether.dnsip);

if (!ether.dnsLookup(website))
Serial.println("DNS failed");
ether.parseIp (ether.hisip, "192.168.2.10");// in case of local server

ether.printIp("SRV: ", ether.hisip); //in case of local server
}

void loop () {
ether.packetLoop(ether.packetReceive());

if (millis() > timer) {
timer = millis() + 60000;
byte val = random(255);
String val1 = String(val);
val = random(100) * analogRead(A2) / 4;
String val2 = String(val);
byte sd = stash.create();
stash.print("temp=");
stash.print(val1);
stash.print("&");
stash.print("temp2=");
stash.print(val2);
stash.save();


Stash::prepare(PSTR("POST /$F HTTP/1.1" "\r\n"
"Host: $F" "\r\n"
"Content-Length: $D" "\r\n"
"Content-Type: application/x-www-form-urlencoded" "\r\n"
"\r\n"
"$H"),
PSTR(PATH), website, stash.size(), sd);

// send the packet - this also releases all stash buffers once done
ether.tcpSend();

}
}

Trying this with GET, proved a lot harder, and in fact I have to admitt defeat here. I just could not get it done, even when I tested code that was said to work, it didnt. It seems the problem has mainly to do with writing to a local server, not a distant server, though that ‘bug’  was supposedly fixed. Another issue is the sparse documentation on the ‘Stash’ routine.

Sure, doing a GET request with a W5100 based  board is not that hard and I presume using the UIPEthernetlibrary also makes it easier, but I wanted to try with Ethercard. I guess I failed. It is quite possible to do a GET call to a remote website, but to a local server remains a problem
So I fell back to the UIPEthernet library:

#include <SPI.h>
#include <UIPEthernet.h> //https://github.com/ntruchsess/arduino_uip/releases/tag/UIPEthernet_v1.59
int dd = 0;// mock variable 1
String ddd = String(dd);
int ww = 255;// mock variable2
String www = String(ww);

byte mac[] = { 0x74, 0x69, 0x69, 0x2D, 0x30, 0x31 };
// if you don't want to use DNS (and reduce your sketch size)
// use the numeric IP instead of the name for the server:
IPAddress server(192, 168, 2, 10); // numeric IP for website (no DNS)
//char server[] = "yourwebsite.com"; // name address for website (using DNS)

// Set the static IP address to use if the DHCP fails to assign
IPAddress ip(192, 168, 2, 40);

// Initialize the Ethernet client library
// with the IP address and port of the server
// that you want to connect to (port 80 is default for HTTP):
EthernetClient client;

void setup() {
// Open serial communications and wait for port to open:
Serial.begin(9600);
Serial.println("start");

// start the Ethernet connection:
if (Ethernet.begin(mac) == 0) {
Serial.println("Failed to configure Ethernet using DHCP");
// try to congifure using IP address instead of DHCP:
Ethernet.begin(mac, ip);
}
// give the Ethernet shield a second to initialize:
Serial.println("wait a moment");
delay(1000);
}

void sendData()
{
Serial.println("connecting...");
if (client.connect(server, 80)) {
Serial.println("connected");
// Make your GET request:
client.println("GET /saveTemp.php?pwd=raspiTemp&temp=" + ddd + "&temp2=" + www + " HTTP/1.1");
client.println("Host: 192.168.2.10");
client.println("Connection: close");
client.println();
}
else {
// if you didn't get a connection to the server:
Serial.println("connection failed");
}
// if there are incoming bytes available
// from the server, read them and print them:
if (client.available()) {
char c = client.read();
Serial.print(c);
}
}

void loop()
{
sendData();

// if the server's disconnected, stop the client:
if (!client.connected()) {
Serial.println();
Serial.println("disconnecting.");
client.stop();
}
delay(10000);
dd = dd + 1;
ww = ww - 1;
ddd = String(dd);
www = String(ww);
}

Storing Arduino sensor data on a (local) server using POST or GET and PHP and presenting it in a graph

In an earlier article, I described how to store your sensor data in GoogleSheet I also described how to make graphical presentations of data with the Highcharts library (much based on an article of Sarah&Rui Santos of Randomnerdstutorials, but extended by me).

That method was especially suited for an ESP8266 or ESP32 as they can hold and process a lot of data in memory.

So what if we use a processor with less memory, say an Arduino with an ethernet connector? Well, there are some possibilities for that too. It involves storing data on a (local) server. Yes that could be in say MySQL/MariaDB, but for much purposes a simple CSV file is enough. We will do everything in simple PHP with the help of the jpGraph library

So for now I will focus on the serverside: how to  store the data and how to show it nicely in a graph. In a subsequent article I will talk about the client side: the Arduino with an (old) ENC28J60 module.

For now let’s assume we have a local server, most likely a Raspberry with PHP and Apache installed.

The directory where we will store the PHP files as well as the data, is the ‘root’ website directory which is /var/www/html

POST-ing data
<?php 
define("LOG_FILE", "./data.csv"); 
$temp=$_POST['temp'];
$temp2=$_POST['temp2']; 
$file_handler = fopen(LOG_FILE, "a+") or die("Unable to open file");
 #fwrite($file_handler, time() . "," . $temp . "\n");
 fwrite($file_handler, time() . "," . $temp . "," . $temp2 . "\n");
 fflush($file_handler);
 echo "OK";
 ?>

Let’s call the script “POST-data.php”

GET-ing data
<?php

define("PASSWORD", "raspianTemp");
define("LOG_FILE", "./data.csv");

if(!isset($_GET["temp"])) die("NO - no value present");
if(!isset($_GET["pwd"])) die("NO - no password present");

$pwd = $_GET["pwd"];
if($pwd != PASSWORD) die("NO - wrong password");

$temp = $_GET["temp"];
if(!is_numeric($temp)) die("NO - not a number");
echo $temp;

$temp2= $_GET["temp2"];
#if(!is_numeric($temp)) die("NO - not a number");

$file_handler = fopen(LOG_FILE, "a+") or die("Unable to open file");
#fwrite($file_handler, time() . "," . $temp . "\n");
fwrite($file_handler, time() . "," . $temp . "," . $temp2 . "\n");
fflush($file_handler);
echo "OK";

?>

The GET script looks a bit more complicated than the POST script, but that’s simply because it has more checks: on a password and the entered values. That is because the GET method can also be used through an URLsuch as:

http://192.168.xx.yyy/saveTemp.php?pwd=raspianTemp&temp=12.5&temp2=9
But you can opt to leave them out or use them in the POST method as well.

Let’s call this script “GET-data.php”

Oh, darn. ‘privileges’
set the /var/www/html directory to ‘775’ (with chmod) and make sure the apache user is owner of the data.csv file. If  you let the php scripts create the file, then the apache user is the owner. If you happen to have created the file yrself, need to change ownership with:
sudo chown www-data data.csv

jpGraph

jpGraph is a graphic library meant for php. It can be downloaded here. for now we will install it in the /var/www/html folder.
Download the tar file and put it in the /var/www/html folder.
Currently jpgraph-4.2.10.tar.gz is the latest version.
‘Install’ it by unpacking it and then rename the created directory as follows:

tar xzf jpgraph-4.2.10.tar.gz
sudo mv jpgraph-4.2.10 jpgraph

 

Once that is all done we can show the data

Showing the data
<?php

define("LOG_FILE", "./data.csv");
require_once('./jpgraph/src/jpgraph.php');
require_once('./jpgraph/src/jpgraph_line.php');

$times = array();
$values1 = array();
$values2=array();
$file_lines = file(LOG_FILE, FILE_IGNORE_NEW_LINES|FILE_SKIP_EMPTY_LINES);

//Get the data
foreach($file_lines as $line_num => $line_value) {
$line_elements = explode(",", $line_value);
#$times[] = date("H:i:s", $line_elements[0]);
$times[] = date("d-m-y H:i", $line_elements[0]);
$values1[] = $line_elements[1];
$values2[] = $line_elements[2];
}
// setup the graph
$graph = new Graph(1100, 550);
$graph->SetFrame(false);
#$graph->SetScale('intint');
$graph->SetScale('textlin');
$graph->SetShadow($aShowShadow=true,$aShadowWidth=5,$aShadowColor=array(102,102,102));
$graph->ygrid->SetFill(true,'#EFEFEF@0.5','#BBCCFF@0.5');
$graph->ygrid->Show();
$graph->xgrid->Show();

#$graph->img->SetAntiAliasing(false);
//li,re,bo/on
$graph->SetMargin(50,30,36,90);
$graph->SetMarginColor('white');

//Setup Title and axis
$graph->title->Set("Temperatures");
$graph->xaxis->SetLabelAngle('50');
$graph->xaxis->SetTickLabels($times);
$graph->xaxis->SetLabelSide(SIDE_BOTTOM);

$graph->yaxis->scale->SetAutoMin(0);
$graph->yaxis->title->Set("&deg;C");

$graph->ygrid->SetFill($aFlg=true, $aColor1='white', $aColor2='gray9');
// Create the first line
$lineplot = new LinePlot($values1);
$lineplot->SetLegend('Living Room');
#https://jpgraph.net/features/src/show-example.php?target=new_line1.php
$graph->Add($lineplot);
$lineplot->SetColor("green");

// Create the second line
$p2 = new LinePlot($values2);
$graph->Add($p2);
$p2->SetColor("#FF1493");
$p2->SetLegend('Bedroom');

$graph->legend->SetFrameWeight(1);
// Finally send the graph to the browser
$graph->Stroke();
?>

The above methods can come in handy for every kind of data generated by a data monitoring system, but as said, I wanted to make some arduino/ENC28J60/EtherCard lib examples. The POST method was fairly simple (done) but the GET method caused some headaches.

Currently, if you have stored a lot of data, the graph becomes too condensed. That can be remedied by for example only showing the last 10 or 20 lines. I will see if i can add that.

(The idea to use jGraph came from Luca Dentella, who has done much work with the ENC26J60)

Using a MAX6675 temperature sensor without a library

When you are using a sensor, or other type of peripheral, it is always handy to use one of the available libraries. Sometimes though there is no library, or the instructions for the peripheral are simple enough to not use a library, or the library gives a lot of overhead.
In an earlier article, I showed how the (excellent) datasheet of the good old BMP180 offers a blueprint on using the sensor without library on an attiny85.

Purely as a learning moment, I will now do the same for an SPI sensor, the MAX6675 temperature sensor on an ESP8266. Yes I know, there is a library available, but consider this just as an exercise in dealing with SPI peripherals.
The MAX6675 datasheet states the following:

Force CS low and apply a clock signal at SCK to read the results at SO. Forcing CS low immediately stops any conversion process. Initiate a new conversion process by forcing CS high. Force CS low to output the first bit on the SO pin. A complete serial interface read requires 16 clock cycles.Read the 16 output bits on the falling edge of the clock.

The first bit, D15, is a dummy sign bit and is always zero. Bits D14–D3 contain the converted temperature in the order of MSB to LSB. Bit D2 is normally low and goes high when the thermocouple input is open. D1 is low to provide a device ID for the MAX6675 and bit D0 is three-state.

There is another piece of information hidden in the Datasheet, saying that the 12 Bit output is in steps of 0.25 degrees.
Another piece of info that would come in handy, but that is void in the datasheet is the signal timings. Yes, there are some timing diagrams in the datasheet, but they mention no numbers, so we have to just do some trial and error. The speed of the SPI-bus itself is half of the CPU processor speed so in the ESP8266 it runs at 40Mhz. We willl just leave it at that.

In the table of the SO output register we see that apart from MSB (bit15) we also do not really need the 3 LSB’s, but we will read them anyway and deal with them. Bit 2 does have some function as it indicates whether the Sensor is connected or not.
Finally, we have to multiply the 12 bit value we found with 0.25 because  the raw value found gives the number of quarter degrees.

So in order to get the temperature we need to do the following things:

  • Begin with CS low to stop any conversion. To start fresh so to say
  • Then make CS high, to start the conversion
  • Pull CS low to output the bits to MISO on falling edge of clock.
  • Cycle the clock to read all the 16 incoming bits on the MISO pin
  • Ignore the MSB (bit 15, remember, we count from 0)
  • Discard the 3 LSB’s by a right shift 3
  • Multiply the found value with 0.25 (or divide by four).

In Software that goes like this:
Define the pins and variables we will use

#define cs D8 //cs
#define clk D5 //sck
#define miso D6 // so
//define variables
int v = 0;
float tempC;//for temperature in degrees Celsius
float tempF;//for temperature in degrees Fahrenheit

In the setup, we define the pinModes and the begin states of the pins

void setup() {
Serial.begin(115200);
pinMode(cs, OUTPUT);
pinMode(clk, OUTPUT);
pinMode(miso, INPUT);
digitalWrite(cs, HIGH);
digitalWrite(clk, LOW);
}

In the loop, we  call a procedure that reads and returns the MISO bits

void loop() {
v = spiRead();
if (v == -1) {
  Serial.print("Sensor not found");
}
else {
  tempC = v * 0.25; //Celsius (read on,the 0.25 will become clear)
  tempF= tempC * 9.0/5.0 + 32;//Fahrenheit
  Serial.print(tempC);
}
delay(500);
}

Then in the spiRead procedure, we read the  the first MSB (bit15) and discard it.  Then in a loop we read the remaining 15 bits (bit 0-bit 14) and add them to a variable, we then discard bit 0,1 and 2, so we have a 12 bit number left. This number than still needs to be multiplied by 0.25 to give the result in degrees celsius

int spiRead() 
{
int value = 0;
digitalWrite(cs,LOW);
delay(2);
digitalWrite(cs,HIGH);
delay(200);

//Deal with bit 15 first
digitalWrite(cs,LOW);//Pull CS low to start conversion
digitalWrite(clk,HIGH);
delay(1);
digitalWrite(clk,LOW);// bit is read on downflank

//Read bits 14-0 from MAX6675 and store in 'rawTmp'
for (int i=14; i>=0; i--) 
{
  digitalWrite(clk,HIGH);
  rawTmp += digitalRead(miso) << i;   digitalWrite(clk,LOW); } // if bit D2 is HIGH no sensor connected   if ((rawTmp & 0x04) == 0x04) return -1; //remaining 14 bits read, now discard bits 0,1 and 2 //we do that with a shift right three places return rawTmp >> 3;

The section

// if bit D2 is HIGH no sensor connected
  if ((rawTmp & 0x04) == 0x04) return -1;

needs a bit of explaining. Once we read all 15 bytes, we can check bit 2 to see if that is set or not, to see if the sensor is detached or connected. the hex value 0x04 is equal to 0b00000100, so no matter what the value of our 15 bit raw temperature is, if we ‘AND’ it with 0b0b00000100, only bit number 2 will matter. If that bit is set, the result is 0b00000100 again, which is 0x04. So if we AND the raw temperature with 0x04 and the result is 0x04, we know that bit was HIGH, meaning the sensor was not connected.

Once that is done, we remove the  3 unnecessary bits (2,1 and 0) by the rightshift with 3.

It will probably be clear that if  we would have defined the command:

for (int i=14; i>=0; i--)

as

for (int i=14; i>=2; i--)

we would only need to do a rightshift 1 like so

return rawTmp >> 1;

As we have to multiply the rawTmp value with 0.25, or divide it by 4. You might be enticed to do that with  a bitshift because after all, division by 4 is a rightshift with 2.
So, in theory it is possible to combine:
v=rawTmp >>3
tempC=v*0.25

as :
v=rawTmp >>5
Yet that is not a good idea.
Suppose we find the value “101001000” (=328)
with a rightshift 3 that becomes “101001” (=41)
divided by 4 that is 10.25 degrees.
However, if we would have combined it in a rightshift 5, we would have gotten “1010” which is 10.

Once it is all done we can convert the degrees Celsius to degrees Fahrenheit by the known formula: *9/5 +32
Download the code here.
In a next article, I’ll hook up the MAX6675 to an ESP8266-01 and will make a monitoring system for a fireplace or cooktop

A cheap 1 channel relaymodule for ESP8266-01. Modifications revisited

In an earlier post, I discussed 2 popular 1 channel ESP8266-01 relay boards that are widely available on  ebay and chinese webstores. One of these boards -the simplest- seems to cause a lot of problems with the unsuspecting buyer, as it doesn’t work.
So, it might be good to address the problems around this board again, in a more practical session.

Be aware that this circuit is intended for the ESP8266-01S, but it can be made to work with the regular ESP8266-01.

The main reason why this board  does not work, is because the designer of  the PCB was drunk, or high on drugs when he designed it, or maybe he just didn’t understand the workings of the ESP8266-01.
His failure becomes immediately clear when we look at the circuit:
The designer used GPIO0 to trigger the relay pin.
There is nothing wrong with that, but as GPIO0 needs to be pulled high to let the ESP8266 start in UART mode (i.e. a normal start). The consequence is that the relay is activated on start-up. “No problem” must the designer have thought, “I just add a 2k resistor to pull it down”. Sure, that works, but the ESP8266 then just will not boot anymore.
So, we need to remove that 2k resistor that is indicated in the circuit above. On your PCB it is indicated as R2 and labelled with ‘202’.

A simple nudge with a hot soldering iron suffices, but make sure you remove the right one.

The next issue is that there is no connection between the Chipselect (EN or CH_PD) and the Vcc. So that needs to be fixed. Ideally that can be done with a 10k resistor, but it is most practical to just make the wire bridge as indicated in the circuit above, like this:

If you have an ESP8266-01S, then you are set, the board should work.

If you have a classic 512kRAM ESP8266-01, then you need to do one more thing: you need to add 10k pullup resistors between GPIO0 and 3V3 and between GPIO2 and 3V3.
When you have done this, the module will also work with an ESP8266-01.

Now  you have your 01 or 01S working  but, the relay will be activated on startup. That can be handled in several ways:
Add a “digitalWrite(0,HIGH);” instruction to the beginning of your program….or….
Use the NC contact to make your connection….or….
reroute the track to pin 0, to another pin (GPIO 1 or GPIO3) and give that pin a pulldown resistor.

If you want to control the board via MQTT, this program might be of interest to you. If you want to use a webpage on your phone, a simple Webserver will do.

Just one more thing about removing the 2k resistor. You may have come across people that do not mention removing the 2k resistor. They solder a 3k3 resistor between 3V3 and GPIO0. Well, yes that works. Why? Well effectively they make the pull up resistor on GPIO0  about 2k5 (the 10k in combo with the 3k3). So that gives a voltage divider of 2k5 over 2k, which makes 1.46 Volt out of  3.3V, which is high enough to be seen as HIGH by gpio0. That is indeed a less elegant method, but it works.

Now if you happen to have the  other relay board, the one with the STC15F104-SOP8 micro controller in addition to the ESP8266, and you dont get that working, begin with checking if the CH_PD/ EN pin is indeed pulled up to 3V3.