Float to bytearray conversion for sigfox payload

[tttadam]

@robert-hh Thanks for the tip. Both of them worked. The first is when I send it as a string (it's temperature data), the second when I send it as byte. But I am not sure how to decode the second one. If you have some tips, or link to other discussion that would be great.

2018-11-03 21:42:35 32352e3539383034 ASCII: 25.59804
2018-11-03 21:37:55 a404d241 ASCII: ...A

My main goal is that send some cordinates also alongside sensor data, but I got the message size.
Is there a way to compress my sentence?
My sentence is like this:([47.45194, 'N'] [19.02091, 'E']);temp;25.68384

[jcaron]

@tttadam I strongly recommend you convert the data to some integer representation (e.g. multiplying by 100 and truncating to an integer), depending on expected range and desired accuracy. It will make things much simpler for you, and if you pick the right integer size, will probably save you a few bytes, which for media like LoRa or Sigfox, can be essential.

[tttadam]

@jcaron Converting to int saved a few bites for me for sure! Thanks!
But I am still struggling to push GPS data into 12 bytes
([47.45194, 'N'] [19.02091, 'E']);temp;25.68384

2018-11-04 06:41:20 32313235 ASCII: 2125
2018-11-03 21:42:35 32352e3539383034 ASCII: 25.59804

[robert-hh]

@tttadam What's the problem. You have 3 numbers for latitude, longitude and temperature. Each of them can be converted to a 4 byte signed int, resulting in 12 bytes. So you have +/- latitude, +/- longitude and +/- temp. If you set latitude and longitude to 1 millionth degree, there is still sufficient resolution in a long int. You do not need to send the ASCII decoration in the message. The receiver can restore that, when unpacking the data.
What kind of coding is available at the receiver side?

[rcolistete]

The Si7021 humidity/temperature sensor :
https://www.silabs.com/products/sensors/humidity/si7006-13-20-21-34
https://www.silabs.com/documents/public/data-sheets/Si7021-A20.pdf
has 0.3-0.4 C accuracy and repeatability/noise of 0.01-0.08 C. So any reading after centesimals (hundredths) has almost no meaning.
Other possibility is the node to make many (10-100) measurements and calculate the mean to have a more stable value, but usually temperature sensor gives a slow changing value.

I suggest to use :

temp_pack = struct.pack(">h", round(si7021.readtemp()*100))

where '>h' is signed int with 2 bytes :
https://docs.python.org/3/library/struct.html#format-characters
https://docs.micropython.org/en/latest/library/ustruct.html
so accepting temperatures between [-327.68, +327.67] C.

Depending on the use (IoT Cloud service, etc), the temperature should have up to decimals (like Cayenne), i. e. :

temp_pack = struct.pack(">i", round(si7021.readtemp()*10))

P.S. : fixed '>i' (4 bytes)' to '>h' (2 bytes).

[rcolistete]

Like @roberto-th said, which is the code at the receiver side ?

One option is to convert the GPS numbers to integer, like Cayenne LPP :
https://mydevices.com/cayenne/docs/lora/#lora-cayenne-low-power-payload
https://community.mydevices.com/t/cayenne-lpp-2-0/7510
for GPS :
Latitude : 0.0001 ° Signed MSB
Longitude : 0.0001 ° Signed MSB
Altitude : 0.01 meter Signed MSB

[tttadam]

@rcolistete @rcolistete Regading to the receiver side:
There isn't any so far, so it could be anything. :)
But I am thinking about google cloud platform (maps api)

Thanks for tips, I will try to imlement them, and I will be back with the result.

[tttadam]

@tttadam I am not sure if anyone still follows this topic, but this is the current status.
I think I can't compress the gps coordinates anymore, but there is still space some other telemetry data.

sfMessage=(str(my_gps.latitude[0]*100000)[1:-2]+str(my_gps.longitude[0]*100000)[1:-2]).encode()
sf.send(bytes(sfMessage))

[tttadam]

@tttadam Nope, I think there is still place for improving.
This send the caracters as string not as byte array.

[robert-hh]

@tttadam But using a string representation requires always more space. As binary representation, you encode 8 bit per byte, as a (decimal) string, you encode about 3 bit per byte.
Example: A 32 bit integer (or float) always requires 4 byte space, wheras as string it may take need up to 11 bytes for integers and up to 14 bytes for floats.
P.S.: I recall that you had problems to decode binary numbers at the receiving side.

[tttadam]

@robert-hh I don't really have a problem with decoding on the receiving side, because there is no receing side. only chack the backend.sigfox with my eyes only. My issue right now that I can't convert this string to bytes. I am not sure what method should I use. Probably something trivial I think.

This is the useful information:

lat =47.497132 
lon =19.063296 #(famous ruinpub in Budapest)
str(lat*100000)[1:-2]+str(lon*100000)[1:-2]

[robert-hh]

@tttadam You use struct.pack() to put these two floats into an 8 byte array without converting them to a string. That results in the shortest possible message, not readable for human eyes withou doing an unpack.

import ustruct
lat =47.497132 
lon =19.063296
msg = ustruct.pack("ff", lat, lon)
# ..
# and unpack
lat, lon = ustruct.unpack("ff", msg)

[tttadam]

@robert-hh said in Float to bytearray conversion for sigfox payload:

ustruct.unpack("ff", msg)

Nice, thank you.
I think is still posible save some more space if cut the first caracter and the dot. (And then possible to include more information)
Which I try and fail to doing here. I think the unpacked value is rounded somehow.

>>> lat
47.49713
>>> lon
19.0633
>>> msg2 =int((str(lat*100000)[1:-2]+str(lon*100000)[1:-2]))
>>> msg = ustruct.pack("ff", msg2)
>>> ret = ustruct.unpack("ff", msg)
>>> print(ret)
(7.497139e+11, 0.0)

[jcaron]

@tttadam What you are doing is covering two floats to strings, concatenating them, converting the total to an int, packing that in as a float, and unpacking.

Forget about str. Just use pack directly with the original values: ustruct.pack("ff",lat,lon)

Or if you want to avoid sending floats over the air, you may convert them to ints and pack than as such instead.

[robert-hh]

@tttadam msg2 contains an single int. So you must not use "ff" for pack, since that tells, that two float numbers are supplied. You have to use "i", which will pack (and unpack) it into a 4 byte format.
Please have a look at the documentation of struct: https://docs.python.org/3/library/struct.html

[tttadam]

@robert-hh said in Float to bytearray conversion for sigfox payload:

I see now. Okay. But I think I am still missing some information. Because I can't get back the same number.

>>> lat
47.49713
>>> lon
19.0633
>>> msg2 =int(str(lat*100000)[1:-2]+str(lon*100000)[1:-2])
>>> msg2
749713906330
>>> msg = ustruct.pack("i", msg2)
>>> msg
b'\x9aZn\x8e'
>>> ret = ustruct.unpack("i", msg)
>>> ret
(-1905370470,)

[robert-hh]

@tttadam The numer is too long for the i format, then you have to use the "q" format (long long). But that results in 8 bytes, and then you can also directly pack the two floats and have no further need for reformatting or scaling (see my post below).
Using the "q" format, you could eventually drop the last three bytes, they are zero, and add them before unpacking. But that does not seem to be worth the effort.

[robert-hh]

@tttadam Or you could split the number:

>>> msg2
749713906330
>>> msg = ustruct.pack("IB", msg2 // 256, msg2 % 256)
>>> msg
b'Zn\x8e\xae\x9a'
>>> msb,lsb = struct.unpack("IB", msg)
>>> ret=msb * 256 + lsb
>>> ret
749713906330

But still that is overkill compared to the float packing, unless you fight for every single byte.

[tttadam]

@robert-hh So basicly your implamentation is the best. :)
Thank you.

[jcaron]

@tttadam why do you keep doing the same stuff you have already been told twice is not correct?

You have been told twice that the solution is very simply ustruct.pack(“ff”,lat,long). What’s wrong with that? Stop with the str and the concatenation and all that. Just pack the two values, and you’re done.