Anyone with a good compressed payload for PySense

[smbunn]

The PySense presents a lot of data, much of it in floating point. Has anyone created a nice compression code in Python to pack everything into the fewest hex bytes for the payload to send via LoRa? Also do they have the matching JavaScript to unpack it when it is received? Seems a worthwhile project if someone has not done this already.

I can see some interesting tradeoffs between payload size and data resolution.

I would like to see everything the PySense can send so I can see this in my database once received:

MPL3115A2 temperature: 46.0
Altitude: -5.75
Pressure: 101403.0
Temperature: 48.06712 deg C and Relative Humidity: 22.30887 %RH
Dew point: 21.12161 deg C
Humidity Ambient for 24.4 deg C is 82.00996%RH
Light (channel Blue lux, channel Red lux): (22, 37)
Acceleration: (-0.05383301, -0.01623535, 0.9763184)
Roll: 3.061144
Pitch: 0.7916171
Battery voltage: 4.395663

Simon

[jcaron]

@smbunn Not sure how you want to send the data, but if you're Wi-Fi for instance you usually don't care much about saving every bit you can, convenience is key, and JSON is probably your best friend.

If you intend to use LoRaWAN on the other hand, you really want to save every bit you can, but using deltas is probably not a good idea as packets are usually unconfirmed and can thus be lost, which would break all subsequent readings.

[MarcoBenini]

Some of the values of PySense sensors are calculated in the libraries or on the sensor itself, so one way to reduce LoRa payload is to send only the measured data.

For example:
"... Dew point: 21.12161 deg C
Humidity Ambient for 24.4 deg C is 82.00996%RH ..."

Dew point and humidity ambient for ... are calculated in the library from temperature and humidity values.

Altitude and pressure: in MPL3115A2 sensor the measured value is the pressure, the altitude is derived, using a formula that can be found in the datasheet. So only one of the two values could be transported.

Than one could consider to transport as float values or the original bytes and than calculate the floats on arrive point.

[smbunn]

All good points. I used to program PLCs in the 80s and we were forever packing data into as few bytes as possible by following the techniques you mentioned.

You can also use prior knowledge. For example on send barometric pressure as the delta from 1000. Plus or minus 128 fits nicely into 8 bits.

Agree this is not classical compression more data packing.

[bmarkus]

First question what is the resolution you need? For example even if you see 5 digits after the decimal point in temperature figure accuracy of the sensor is much less. Send only as much digits as it has real value, do not send noise. A single fractional digit must be enough for temp, like 48.1 Also consider the measurement range. Convert value to a signed or unsigned integer and transmit an 8-bit or 16-bit integer (24-bit). For example humidity measurement integer part is enough, use an unsigned integer with 0---255 representation range. For battery voltage, multiply it by 10 and send singly byte only. You can put multiple sensors data into a single integer (bit field) also, but in most cases it doesn't give too much saving. Do not call it compression, better to say coding.