cat m1 overheads

[kjm]

I decided to have a look at lte cat m1 data overheads. First thing I did was to change from https to http & the mean packet size dropped from 7.3 to 6.1 kb, so dump all that encryption & you use less data, makes sense.

Then I had a peek at some individual server uploads;
to send {'12': [0, 'TAC8313 twr528221 CID135224587 earfcn9410 ss-72 cinr10.7', 1600473753], '4': [0, 'c1 New Version: 131, release 1.20.1.r1', 1600473746], '6': [102, '2.28', 1600473747], '10': [100, 10.7, 1600473752], '8': [103, '0.0', 1600473747], '0': [100, 11.3, 1600472619], '2': [0, ' ta da! FZtank site is up, ver 131', 1600473746]} used 6.08kb
to send {'0': [102, '2.27', 1600472609], '4': [100, 9.8, 1600472620], '2': [103, '0.0', 1600472609]} used 6.31kb
Such poor correlation between characters sent & data used suggests most of it must be overheads?

[Gijs]

Hi,

For my personal tests using LTE (If I remember correctly, it was Cat M1), we were able to make http requests to the mozilla firefox uptime server for only 600 bytes a piece. Im not sure what happens on your downlink, maybe that is where the data is spent (the firefox uptime server only prints "OK" or the like back). Now I remember the sim-provider we used had a smallest delimiter of 1kb per connection.
I wanted to comment this on your other thread, but thought the difference would be found in https.
It is quite a lot of over head (6kB) tha cannot be attributed to the message directly.. I dont have the example we used here right now, but I could probably post it coming Monday!
Gijs

[SciWax]

@kjm I think I can see that too actually. At the moment, I'm sending around 135 bytes of data per minute and I didnt even do it this whole month (Maybe I've experimented for 10 - 15 days in September with data transmissions every minute).... I already used 117 Megabytes of data according to 1nce for this month. It doesn't make sense to me, because I only send such a small amount of data. Granted, I've kept the Cat m1 connection between transmission, because I'm sending every minute anyways. I will investigate this issue more, sending 3 minute averages of measurements every 3 minutes and closing the cat m1 connection between my transmit intervals. I'm also using urequests for POST-requests and check, if I get a 200 back to see if the data transmission were successfull.

[SciWax]

@kjm Now I'm pretty sure, that I have the same problem. I'm sending around 100 bytes every minute, but around 6.1 KB get spent during a single transmission, which doesn't make alot of sense to me. I'm using an 1nce SIM card and the cat m1 network of the telekom in Germany. Which provider for cat m1 do you use?

Hello @Gijs ,
could you find the example already?

kind regards,

Sci

[kjm]

@SciWax I'm using Telstra in Australia. I've experimented with hologram sims but the http/https overheads burn through a 1Mb/month hologram sim in a single day (10 min uploads).

[SciWax]

@kjm Thank you for the information. I really want to know, what the overhead is related too..... I would love to use the example of Gijs to see, if this huge overhead isn't the standard.

[Gijs]

Hi,

Thanks for the reminder
Here, we use this to test the LTE connection main.py: (The LTE callback does not work in 100% of the cases)

from network import LTE
import time
from http_get import http_get
import sys
import machine
def cb_handler(arg):
    print("CB: LTE Coverage lost")
    s = 120 # wait a while just to confirm that this wasn't a false positive and we truely lost connection
    print("CB: sleep", s)
    time.sleep(s)
    print("CB: deinit")
    lte.deinit()
    print("CB: reset")
    machine.reset()
    # print("arg", type(arg))
    # print("arg", arg)
    # sys.exit()

print("init")
lte = LTE()
lte.lte_callback(LTE.EVENT_COVERAGE_LOSS, cb_handler)
print("attaching..",end='')
lte.attach(band=20, apn='simpoint.m2m')
for t in range(30):
    if lte.isattached():
        break
    time.sleep(1)
    print(".",end='')
if lte.isattached():
    print("attached")
else:
    print("Could not attach ... resetting")
    machine.reset()
lte.connect()
print("connecting [##", end='')
while not lte.isconnected():
    time.sleep(1)
    print("#",end='')
print("] connected")


for i in range(20):
    print("[ {} ]".format(time.ticks_ms()),end=' ')
    print(http_get())
    s = 2 ** i
    print(i, s, s/60, s/3600)
    time.sleep(s)

print("[ {} ]".format(time.ticks_ms()), " done")

With http_get() (and note that this is http only):

import socket
import time
def_url = 'http://detectportal.firefox.com/'
def http_get(url = def_url):
    time_start = time.time()
    dur_send = 0
    dur_recv = 0
    dur_total = 0
    bps_send = 0
    bps_recv = 0
    len_recv = 0
    cnt_recv = 0
    success = False
    try:
        print("http_get(", url, ")")
        _, _, host, path = url.split('/', 3)
        print('getaddrinfo')
        ip_port = socket.getaddrinfo(host, 80)[0][-1]
        print(ip_port)
        print('socket')
        s = socket.socket()
        print('connect')
        s.connect(ip_port)
        request = bytes('GET /%s HTTP/1.0\r\nHost: %s\r\n\r\n' % (path, host), 'utf8')
        time_send = time.time()
        print('send request')
        s.send(request)
        dur_send = time.time() - time_send
        if dur_send:
            bps_send = len(request) / dur_send
        print('recv response')
        time_recv = time.time()
        s.settimeout(60)
        while True:
            data = s.recv(100)
            if data:
                cnt_recv += 1
                len_recv += len(data)
                dur_recv = time.time() - time_recv
                if dur_recv:
                    bps_recv = len_recv / dur_recv
                if cnt_recv % 1000 == 0:
                    print("received[", time.time(), "]:", len_recv, bps_recv)
                success = True
                # print(str(data, 'utf8'), end='')
                pass
            else:
                break
        print("recv: done")
        print("close")
        s.close()
    except Exception as e:
        print("http_get failed:", e)
    dur_total = time.time() - time_start
    return (success, dur_send, dur_recv, dur_total, len_recv, bps_send, bps_recv)
# def http_gets(url = def_url, count=1):
#     for c in range(0, count):
#         try:
#             http_get(url)
#             print("success[", c, "]")
#             # break
#         except Exception as e:
#             print("failure[", c, "]:", e)
def http_gets(url = def_url, count=1):
    for c in range(0, count):
        print(c, http_get(url))
if __name__ == "__main__":
    # http_get()
    # http_get('http://micropython.org/ks/test.html')
    http_gets('http://detectportal.firefox.com/')
    # https_gets('http://pycom.io')
    # http_get('http://ftp.snt.utwente.nl/pub/docs/rfc/rfc1498.json') # 1000 B
    # http_get.http_gets('http://ftp.snt.utwente.nl/pub/docs/rfc/rfc1535.html') #   10 KiB
    # http_get('http://ftp.snt.utwente.nl/pub/docs/rfc/rfc753.html')  #  100 KiB
    # http_get('http://ftp.snt.utwente.nl/pub/test/1M')               #  977 KiB
    # http_get('http://ftp.snt.utwente.nl/pub/test/100M')
    # http_get.http_gets('http://ftp.snt.utwente.nl/pub/test/10M')

Atm I have no simcard on hand to properly test it, but I made no changes since last time I used it

Gijs