Storing sensor data on a (local) server using POST or GET: client side

In a previous post I described PHP scripts that would store sensor data in a CSV file and display the data in a graph.

Now I wanted to give two Arduino/ENC28J60 programs that would send that data. One using POST, the other one using GET, using the EtherCard library. That proved a bit harder than I thought.

Using POST was easy, the program below does that. It reads the value of two analog value and sends those to a server side PHP script that stores the data in a CSV file

#include <EtherCard.h>
#define PATH "POST-data.php"

// ethernet interface mac address, must be unique on the LAN
byte mymac[] = { 0x74, 0x69, 0x69, 0x2D, 0x30, 0x31 };

const char website[] PROGMEM = "yoursite.com";// in case you use that

byte Ethernet::buffer[700];
uint32_t timer;
Stash stash;

void setup () {
Serial.begin(57600);
Serial.println("POST Data");

if (ether.begin(sizeof Ethernet::buffer, mymac, 10) == 0)
Serial.println( "Failed to access Ethernet controller");
if (!ether.dhcpSetup())
Serial.println("DHCP failed");

ether.printIp("IP: ", ether.myip);
ether.printIp("GW: ", ether.gwip);
ether.printIp("DNS: ", ether.dnsip);

if (!ether.dnsLookup(website))
Serial.println("DNS failed");
ether.parseIp (ether.hisip, "192.168.2.10");// in case of local server

ether.printIp("SRV: ", ether.hisip); //in case of local server
}

void loop () {
ether.packetLoop(ether.packetReceive());

if (millis() > timer) {
timer = millis() + 60000;
byte val = random(255);
String val1 = String(val);
val = random(100) * analogRead(A2) / 4;
String val2 = String(val);
byte sd = stash.create();
stash.print("temp=");
stash.print(val1);
stash.print("&");
stash.print("temp2=");
stash.print(val2);
stash.save();


Stash::prepare(PSTR("POST /$F HTTP/1.1" "\r\n"
"Host: $F" "\r\n"
"Content-Length: $D" "\r\n"
"Content-Type: application/x-www-form-urlencoded" "\r\n"
"\r\n"
"$H"),
PSTR(PATH), website, stash.size(), sd);

// send the packet - this also releases all stash buffers once done
ether.tcpSend();

}
}

Trying this with GET, proved a lot harder, and in fact I have to admitt defeat here. I just could not get it done, even when I tested code that was said to work, it didnt. It seems the problem has mainly to do with writing to a local server, not a distant server, though that ‘bug’ was supposedly fixed. Another issue is the sparse documentation on the ‘Stash’ routine.

Sure, doing a GET request with a W5100 based board is not that hard and I presume using the UIPEthernetlibrary also makes it easier, but I wanted to try with Ethercard. I guess I failed. It is quite possible to do a GET call to a remote website, but to a local server remains a problem
So I fell back to the UIPEthernet library:

#include <SPI.h>
#include <UIPEthernet.h> //https://github.com/ntruchsess/arduino_uip/releases/tag/UIPEthernet_v1.59
int dd = 0;// mock variable 1
String ddd = String(dd);
int ww = 255;// mock variable2
String www = String(ww);

byte mac[] = { 0x74, 0x69, 0x69, 0x2D, 0x30, 0x31 };
// if you don't want to use DNS (and reduce your sketch size)
// use the numeric IP instead of the name for the server:
IPAddress server(192, 168, 2, 10); // numeric IP for website (no DNS)
//char server[] = "yourwebsite.com"; // name address for website (using DNS)

// Set the static IP address to use if the DHCP fails to assign
IPAddress ip(192, 168, 2, 40);

// Initialize the Ethernet client library
// with the IP address and port of the server
// that you want to connect to (port 80 is default for HTTP):
EthernetClient client;

void setup() {
// Open serial communications and wait for port to open:
Serial.begin(9600);
Serial.println("start");

// start the Ethernet connection:
if (Ethernet.begin(mac) == 0) {
Serial.println("Failed to configure Ethernet using DHCP");
// try to congifure using IP address instead of DHCP:
Ethernet.begin(mac, ip);
}
// give the Ethernet shield a second to initialize:
Serial.println("wait a moment");
delay(1000);
}

void sendData()
{
Serial.println("connecting...");
if (client.connect(server, 80)) {
Serial.println("connected");
// Make your GET request:
client.println("GET /saveTemp.php?pwd=raspiTemp&temp=" + ddd + "&temp2=" + www + " HTTP/1.1");
client.println("Host: 192.168.2.10");
client.println("Connection: close");
client.println();
}
else {
// if you didn't get a connection to the server:
Serial.println("connection failed");
}
// if there are incoming bytes available
// from the server, read them and print them:
if (client.available()) {
char c = client.read();
Serial.print(c);
}
}

void loop()
{
sendData();

// if the server's disconnected, stop the client:
if (!client.connected()) {
Serial.println();
Serial.println("disconnecting.");
client.stop();
}
delay(10000);
dd = dd + 1;
ww = ww - 1;
ddd = String(dd);
www = String(ww);
}

Storing Arduino sensor data on a (local) server using POST or GET and PHP and presenting it in a graph

In an earlier article, I described how to store your sensor data in GoogleSheet I also described how to make graphical presentations of data with the Highcharts library (much based on an article of Sarah&Rui Santos of Randomnerdstutorials, but extended by me).

That method was especially suited for an ESP8266 or ESP32 as they can hold and process a lot of data in memory.

So what if we use a processor with less memory, say an Arduino with an ethernet connector? Well, there are some possibilities for that too. It involves storing data on a (local) server. Yes that could be in say MySQL/MariaDB, but for much purposes a simple CSV file is enough. We will do everything in simple PHP with the help of the jpGraph library

So for now I will focus on the serverside: how to  store the data and how to show it nicely in a graph. In a subsequent article I will talk about the client side: the Arduino with an (old) ENC28J60 module.

For now let’s assume we have a local server, most likely a Raspberry with PHP and Apache installed.

The directory where we will store the PHP files as well as the data, is the ‘root’ website directory which is /var/www/html

POST-ing data
<?php 
define("LOG_FILE", "./data.csv"); 
$temp=$_POST['temp'];
$temp2=$_POST['temp2']; 
$file_handler = fopen(LOG_FILE, "a+") or die("Unable to open file");
 #fwrite($file_handler, time() . "," . $temp . "\n");
 fwrite($file_handler, time() . "," . $temp . "," . $temp2 . "\n");
 fflush($file_handler);
 echo "OK";
 ?>

Let’s call the script “POST-data.php”

GET-ing data
<?php

define("PASSWORD", "raspianTemp");
define("LOG_FILE", "./data.csv");

if(!isset($_GET["temp"])) die("NO - no value present");
if(!isset($_GET["pwd"])) die("NO - no password present");

$pwd = $_GET["pwd"];
if($pwd != PASSWORD) die("NO - wrong password");

$temp = $_GET["temp"];
if(!is_numeric($temp)) die("NO - not a number");
echo $temp;

$temp2= $_GET["temp2"];
#if(!is_numeric($temp)) die("NO - not a number");

$file_handler = fopen(LOG_FILE, "a+") or die("Unable to open file");
#fwrite($file_handler, time() . "," . $temp . "\n");
fwrite($file_handler, time() . "," . $temp . "," . $temp2 . "\n");
fflush($file_handler);
echo "OK";

?>

The GET script looks a bit more complicated than the POST script, but that’s simply because it has more checks: on a password and the entered values. That is because the GET method can also be used through an URLsuch as:

http://192.168.xx.yyy/saveTemp.php?pwd=raspianTemp&temp=12.5&temp2=9
But you can opt to leave them out or use them in the POST method as well.

Let’s call this script “GET-data.php”

Oh, darn. ‘privileges’
set the /var/www/html directory to ‘775’ (with chmod) and make sure the apache user is owner of the data.csv file. If  you let the php scripts create the file, then the apache user is the owner. If you happen to have created the file yrself, need to change ownership with:
sudo chown www-data data.csv

jpGraph

jpGraph is a graphic library meant for php. It can be downloaded here. for now we will install it in the /var/www/html folder.
Download the tar file and put it in the /var/www/html folder.
Currently jpgraph-4.2.10.tar.gz is the latest version.
‘Install’ it by unpacking it and then rename the created directory as follows:

tar xzf jpgraph-4.2.10.tar.gz
sudo mv jpgraph-4.2.10 jpgraph

 

Once that is all done we can show the data

Showing the data
<?php

define("LOG_FILE", "./data.csv");
require_once('./jpgraph/src/jpgraph.php');
require_once('./jpgraph/src/jpgraph_line.php');

$times = array();
$values1 = array();
$values2=array();
$file_lines = file(LOG_FILE, FILE_IGNORE_NEW_LINES|FILE_SKIP_EMPTY_LINES);

//Get the data
foreach($file_lines as $line_num => $line_value) {
$line_elements = explode(",", $line_value);
#$times[] = date("H:i:s", $line_elements[0]);
$times[] = date("d-m-y H:i", $line_elements[0]);
$values1[] = $line_elements[1];
$values2[] = $line_elements[2];
}
// setup the graph
$graph = new Graph(1100, 550);
$graph->SetFrame(false);
#$graph->SetScale('intint');
$graph->SetScale('textlin');
$graph->SetShadow($aShowShadow=true,$aShadowWidth=5,$aShadowColor=array(102,102,102));
$graph->ygrid->SetFill(true,'#EFEFEF@0.5','#BBCCFF@0.5');
$graph->ygrid->Show();
$graph->xgrid->Show();

#$graph->img->SetAntiAliasing(false);
//li,re,bo/on
$graph->SetMargin(50,30,36,90);
$graph->SetMarginColor('white');

//Setup Title and axis
$graph->title->Set("Temperatures");
$graph->xaxis->SetLabelAngle('50');
$graph->xaxis->SetTickLabels($times);
$graph->xaxis->SetLabelSide(SIDE_BOTTOM);

$graph->yaxis->scale->SetAutoMin(0);
$graph->yaxis->title->Set("&deg;C");

$graph->ygrid->SetFill($aFlg=true, $aColor1='white', $aColor2='gray9');
// Create the first line
$lineplot = new LinePlot($values1);
$lineplot->SetLegend('Living Room');
#https://jpgraph.net/features/src/show-example.php?target=new_line1.php
$graph->Add($lineplot);
$lineplot->SetColor("green");

// Create the second line
$p2 = new LinePlot($values2);
$graph->Add($p2);
$p2->SetColor("#FF1493");
$p2->SetLegend('Bedroom');

$graph->legend->SetFrameWeight(1);
// Finally send the graph to the browser
$graph->Stroke();
?>

The above methods can come in handy for every kind of data generated by a data monitoring system, but as said, I wanted to make some arduino/ENC28J60/EtherCard lib examples. The POST method was fairly simple (done) but the GET method caused some headaches.

Currently, if you have stored a lot of data, the graph becomes too condensed. That can be remedied by for example only showing the last 10 or 20 lines. I will see if i can add that.

(The idea to use jGraph came from Luca Dentella, who has done much work with the ENC26J60)

Using a MAX6675 temperature sensor without a library

When you are using a sensor, or other type of peripheral, it is always handy to use one of the available libraries. Sometimes though there is no library, or the instructions for the peripheral are simple enough to not use a library, or the library gives a lot of overhead.
In an earlier article, I showed how the (excellent) datasheet of the good old BMP180 offers a blueprint on using the sensor without library on an attiny85.

Purely as a learning moment, I will now do the same for an SPI sensor, the MAX6675 temperature sensor on an ESP8266. Yes I know, there is a library available, but consider this just as an exercise in dealing with SPI peripherals.
The MAX6675 datasheet states the following:

Force CS low and apply a clock signal at SCK to read the results at SO. Forcing CS low immediately stops any conversion process. Initiate a new conversion process by forcing CS high. Force CS low to output the first bit on the SO pin. A complete serial interface read requires 16 clock cycles.Read the 16 output bits on the falling edge of the clock.

The first bit, D15, is a dummy sign bit and is always zero. Bits D14–D3 contain the converted temperature in the order of MSB to LSB. Bit D2 is normally low and goes high when the thermocouple input is open. D1 is low to provide a device ID for the MAX6675 and bit D0 is three-state.

There is another piece of information hidden in the Datasheet, saying that the 12 Bit output is in steps of 0.25 degrees.
Another piece of info that would come in handy, but that is void in the datasheet is the signal timings. Yes, there are some timing diagrams in the datasheet, but they mention no numbers, so we have to just do some trial and error. The speed of the SPI-bus itself is half of the CPU processor speed so in the ESP8266 it runs at 40Mhz. We willl just leave it at that.

In the table of the SO output register we see that apart from MSB (bit15) we also do not really need the 3 LSB’s, but we will read them anyway and deal with them. Bit 2 does have some function as it indicates whether the Sensor is connected or not.
Finally, we have to multiply the 12 bit value we found with 0.25 because  the raw value found gives the number of quarter degrees.

So in order to get the temperature we need to do the following things:

  • Begin with CS low to stop any conversion. To start fresh so to say
  • Then make CS high, to start the conversion
  • Pull CS low to output the bits to MISO on falling edge of clock.
  • Cycle the clock to read all the 16 incoming bits on the MISO pin
  • Ignore the MSB (bit 15, remember, we count from 0)
  • Discard the 3 LSB’s by a right shift 3
  • Multiply the found value with 0.25 (or divide by four).

In Software that goes like this:
Define the pins and variables we will use

#define cs D8 //cs
#define clk D5 //sck
#define miso D6 // so
//define variables
int v = 0;
float tempC;//for temperature in degrees Celsius
float tempF;//for temperature in degrees Fahrenheit

In the setup, we define the pinModes and the begin states of the pins

void setup() {
Serial.begin(115200);
pinMode(cs, OUTPUT);
pinMode(clk, OUTPUT);
pinMode(miso, INPUT);
digitalWrite(cs, HIGH);
digitalWrite(clk, LOW);
}

In the loop, we  call a procedure that reads and returns the MISO bits

void loop() {
v = spiRead();
if (v == -1) {
  Serial.print("Sensor not found");
}
else {
  tempC = v * 0.25; //Celsius (read on,the 0.25 will become clear)
  tempF= tempC * 9.0/5.0 + 32;//Fahrenheit
  Serial.print(tempC);
}
delay(500);
}

Then in the spiRead procedure, we read the  the first MSB (bit15) and discard it.  Then in a loop we read the remaining 15 bits (bit 0-bit 14) and add them to a variable, we then discard bit 0,1 and 2, so we have a 12 bit number left. This number than still needs to be multiplied by 0.25 to give the result in degrees celsius

int spiRead() 
{
int value = 0;
digitalWrite(cs,LOW);
delay(2);
digitalWrite(cs,HIGH);
delay(200);

//Deal with bit 15 first
digitalWrite(cs,LOW);//Pull CS low to start conversion
digitalWrite(clk,HIGH);
delay(1);
digitalWrite(clk,LOW);// bit is read on downflank

//Read bits 14-0 from MAX6675 and store in 'rawTmp'
for (int i=14; i>=0; i--) 
{
  digitalWrite(clk,HIGH);
  rawTmp += digitalRead(miso) << i;   digitalWrite(clk,LOW); } // if bit D2 is HIGH no sensor connected   if ((rawTmp & 0x04) == 0x04) return -1; //remaining 14 bits read, now discard bits 0,1 and 2 //we do that with a shift right three places return rawTmp >> 3;

The section

// if bit D2 is HIGH no sensor connected
  if ((rawTmp & 0x04) == 0x04) return -1;

needs a bit of explaining. Once we read all 15 bytes, we can check bit 2 to see if that is set or not, to see if the sensor is detached or connected. the hex value 0x04 is equal to 0b00000100, so no matter what the value of our 15 bit raw temperature is, if we ‘AND’ it with 0b0b00000100, only bit number 2 will matter. If that bit is set, the result is 0b00000100 again, which is 0x04. So if we AND the raw temperature with 0x04 and the result is 0x04, we know that bit was HIGH, meaning the sensor was not connected.

Once that is done, we remove the  3 unnecessary bits (2,1 and 0) by the rightshift with 3.

It will probably be clear that if  we would have defined the command:

for (int i=14; i>=0; i--)

as

for (int i=14; i>=2; i--)

we would only need to do a rightshift 1 like so

return rawTmp >> 1;

As we have to multiply the rawTmp value with 0.25, or divide it by 4. You might be enticed to do that with  a bitshift because after all, division by 4 is a rightshift with 2.
So, in theory it is possible to combine:
v=rawTmp >>3
tempC=v*0.25

as :
v=rawTmp >>5
Yet that is not a good idea.
Suppose we find the value “101001000” (=328)
with a rightshift 3 that becomes “101001” (=41)
divided by 4 that is 10.25 degrees.
However, if we would have combined it in a rightshift 5, we would have gotten “1010” which is 10.

Once it is all done we can convert the degrees Celsius to degrees Fahrenheit by the known formula: *9/5 +32
Download the code here.
In a next article, I’ll hook up the MAX6675 to an ESP8266-01 and will make a monitoring system for a fireplace or cooktop