Creating a JSON structure for PySense

[smbunn]

I am trying to create a JSON structure for all (well almost all) of the values present in the pysense. This will then be sent over LoRa to an MQTT gateway on my Node-red computer.

    mp = MPL3115A2(py,mode=ALTITUDE) # Returns height in meters. Mode may also be set to PRESSURE, returning a value in Pascals
    si = SI7006A20(py)
    lt = LTR329ALS01(py)
    li = LIS2HH12(py)
    print("Count=", count)
    print("MPL3115A2 temperature: " + str(mp.temperature()))
    print("Altitude: " + str(mp.altitude()))
    mpp = MPL3115A2(py,mode=PRESSURE) # Returns pressure in Pa. Mode may also be set to ALTITUDE, returning a value in meters
    print("Pressure: " + str(mpp.pressure())) #Pressure does not work too well
    print("Temperature: " + str(si.temperature())+ " deg C and Relative Humidity: " + str(si.humidity()) + " %RH")
    print("Dew point: "+ str(si.dew_point()) + " deg C")
    t_ambient = 24.4
    print("Humidity Ambient for " + str(t_ambient) + " deg C is " + str(si.humid_ambient(t_ambient)) + "%RH")
    print("Light (channel Blue lux, channel Red lux): " + str(lt.light()))
    print("Acceleration: " + str(li.acceleration()))
    print("Roll: " + str(li.roll()))
    print("Pitch: " + str(li.pitch()))
    vt = py.read_battery_voltage()
    dew = si.dew_point()
    print("Battery voltage: " +str(vt))
    pycom.rgbled(blue)
    time.sleep(0.5)
    pycom.rgbled(off)
    time.sleep(0.5)
    pycom.rgbled(red)
    time.sleep(0.5)
    pycom.rgbled(off)
    s.setblocking(True)
    mqData = {}
    mqData["data"] = {}
    mqData["data"]["mov"] = {}
    mqData["data"]["mov"]["acc"] = {}
    accTuple = li.acceleration()
    mqData["data"]["mov"]["acc"]["x"] = accTuple[0]
    mqData["data"]["mov"]["acc"]["y"] = accTuple[1]
    mqData["data"]["mov"]["acc"]["z"] = accTuple[2]
    mqData["data"]["mov"]["rol"] = li.roll()
    mqData["data"]["mov"]["pit"] = li.pitch()
    mqData["data"]["loc"] = {}
    data = ujson.dumps(mqData)
    print ('Data = ',data)
    s.send(data)  # send buffer to TTN

But this gives me an error message "OSError: [Errno 90] EMSGSIZE"

Anyone got a working structure that loads into JSON?

[jcaron]

You definitely DO NOT want to use JSON for data sent over LoRaWAN. It’s much too verbose and not adapted for the very very low bandwidth available over LoRa, especially for the slower (longer range) data rates.

You should instead pack your data as bytes, then decode the data network side to a more readable format like JSON.

[smbunn]

@jcaron I started doing that but had problems with node-red which does not appear to have any string functions at all. I cannot do the equivalent of leftstr(), rightstr() or midstr().
So my data payload arrives as 16 hex characters and I cannot strip it out into 4 hex chunks. It seems weird that there are no string functions in node-red as it handles mainly strings!

[jcaron]

@smbunn I’m not familiar with node-Red but from what I understand it’s based on JavaScript and you should be able to use any JavaScript function. I would be surprised if you actually received a string though, more probably a Buffer or some kind of array...

[smbunn]

The data is received as one very large string. You can use Node Red to parse it into JSON but this is sometimes more trouble than it is worth. I found a Node-Red add on that permits string functions. (https://flows.nodered.org/node/node-red-contrib-string)

The payload sent by my Python code above is a string. I have changed the code to :

    s.setblocking(True)
    data = bytearray(16)
    data[0:4] = bytearray(struct.pack(">i", count))
    data[4:8] = bytearray(struct.pack(">f", vt))
    data[8:12] = bytearray(struct.pack(">f", dew))
    data[12:16] = bytearray(struct.pack(">f", mp.temperature()))
    print ('Data = ',count,vt,dew, mp.temperature() )
    s.send(data)  # send buffer to TTN

Count is an integer that will keep track of how many packets I have sent. My Node-Red is working and it definitely gets the payload as a string:

3/26/2018, 8:11:03 AMnode: 2nd
$aws/things/BE7A0200000001A0/shadow/update/accepted : msg.payload : string[8]
"0000033040945e6642060db8425a0000"

[jcaron]

@smbunn I don’t quite get how a 16-byte array can turn into an 8-digit hex string? It should be quite the opposite...

[smbunn]

@jcaron Sorry, I showed the truncated part of the string that has the PySense battery voltage. I have modified my response to show the full string

[jcaron]

@smbunn It would probably make more sense if the message was delivered directly as a Buffer rather than a string, but you can convert it yourself: new Buffer(msg.payload,'hex') will return a Buffer with the relevant data. You can then get any byte of the buffer using the usual subscript syntax:

var buf = new Buffer(msg.payload);
var byte0 = buf[0]

You could also use librairies such as https://github.com/ryanrolds/bufferpack to unpack the values you packed on the LoPy.

[smbunn]

Have it all working nicely using buffers in Node-Red. Here is the code in Python (still have to sort out the tuple used for acceleration, any hints welcome!

    vt = py.read_battery_voltage()
    print("Battery voltage: " +str(vt))
    dew = si.dew_point()
    print("Dew point: "+ str(dew) + " deg C")
    mp = MPL3115A2(py,mode=ALTITUDE) # Returns height in meters. Mode may also be set to PRESSURE, returning a value in Pascals
    print("MPL3115A2 temperature: " + str(mp.temperature()))
    temp1 = mp.temperature()
    alt1 = mp.altitude()+100.0
    print("Altitude: " + str(alt1))
    mpp = MPL3115A2(py,mode=PRESSURE) # Returns pressure in Pa. Mode may also be set to ALTITUDE, returning a value in meters
    press1 = mpp.pressure()
    print("Pressure: " + str(press1)) #Pressure does not work too well
    temp2 = si.temperature()
    hum1 = si.humidity()
    print("Temperature: " + str(temp2)+ " deg C and Relative Humidity: " + str(hum1) + " %RH")
    t_ambient = 24.4
    relhum = si.humid_ambient(t_ambient)
    print("Humidity Ambient for " + str(t_ambient) + " deg C is " + str(relhum) + "%RH")
    print("Light (channel Blue lux, channel Red lux): " + str(lt.light()))
    acc1 = li.acceleration()
    roll1 = li.roll()
    pitch1 = li.pitch()
    print("Acceleration: " + str(acc1))
    print("Roll: " + str(roll1))
    print("Pitch: " + str(pitch1))
    # Flash the light every time a payload is sent
    pycom.rgbled(blue)
    time.sleep(0.5)
    pycom.rgbled(off)
    time.sleep(0.5)
    pycom.rgbled(red)
    time.sleep(0.5)
    pycom.rgbled(off)
    data = bytearray(48)
    data[0:4] = bytearray(struct.pack(">i", count))
    data[4:8] = bytearray(struct.pack(">f", vt))
    data[8:12] = bytearray(struct.pack(">f", dew))
    data[12:16] = bytearray(struct.pack(">f", temp1))
    data[16:20] = bytearray(struct.pack(">f", alt1))
    data[20:24] = bytearray(struct.pack(">f", press1))
    data[24:28] = bytearray(struct.pack(">f", temp2))
    data[28:32] = bytearray(struct.pack(">f", hum1))
    data[32:36] = bytearray(struct.pack(">f", relhum))
    data[36:40] = bytearray(struct.pack(">f", roll1))
    data[40:44] = bytearray(struct.pack(">f", roll1))
    data[44:48] = bytearray(struct.pack(">f", pitch1))
    print ('Data = ',count,vt,dew, temp1, alt1, press1, temp2, hum1, relhum, acc1, roll1, pitch1 )
    s.setblocking(True)
    s.send(data)  # send buffer to AWS

and in Node-Red, as an example to read dew-point I use:

var buf = new Buffer(msg.payload,'hex');
value1 = parseFloat(buf.readFloatBE(8).toFixed(2));
msg.payload = value1;
msg.topic = "Dew Point";
return msg;

so really the only thing I need to change is the value inside readFloatBE(x) where x is a multiple of 4, first item is 0, Voltage is 4, dew is 8, etc

[jcaron]

@smbunn It's probably much easier if you use a single pack with a single format string and all the values you want to pack in there. Otherwise you'll end up with issues of incorrect offsets or lengths at some point when you'll edit formats.