FiPy: MQTT publishing over CAT-M1

securigy

So I gave up on Hologram. I do use their SIM card and I was able to send some data to it, but that the only positive I can say... and since they do not have any mean to visualize the data I am trying to use MQTT to send my data elsewhere.
To get started I decided on free CloudMQTT (also by Amazon but it is not AWS IoT). It also runs Mosquitto. In fact using Python on Raspberry Pi 3B I was able to connect and send data to it both with and without TLS (TLS must be 1.2).

So now I wrote MycroPython code (that BTW works over LoRa) that reads the data from the sensor and tries to send it to CloudMQTT over CAT-M1. Since I am here it is obvious that I have some problems/questions about it... because I cannot see any data on CloudMQTT as if it never gets it... I already posted a question to its tech support but just want to make sure that I do not do anything stupid with my code:

HOST = "postman.cloudmqtt.com"
PORT = 13240
TOPIC = "dwm/position"

# Connect to CloudMQTT Broker
mqttClient = MQTTClient(b'FiPy_Node_Client', HOST, port=PORT, user="xxxxxxxx", password="xxxxxxxxxxxxxxx")

#sslParams = {'cert_reqs':ssl.CERT_REQUIRED, 'ca_certs':'/flash/cert/addtrustexternalcaroot.crt'}
#client = MQTTClient("secureTry1", "postman.cloudmqtt.com", port=23240, ssl=True, user="fxzlcorl", password="Rblo3fKx7G6l", ssl_params=sslParams)
print("Created mqttClient")

#mqttClient.settimeout(5)
resp = mqttClient.connect(clean_session=True)
print("mqttClient connect call returned: {}".format(resp))
if (resp == 0):
    print("Successfully connected to MQTT Broker")

try:
    #####################################################################################
    # The constructor creates a class with a thread that reads the data from the sensor
    ECG = DFRobotECG('P18', 460)
    #####################################################################################

    idx = 1

    # Loop for delivery sensor information to CloudMQTT
    while(idx < 5):

        # give the sensor a bit time to generate data
        utime.sleep(0.1)

        try:
            
            # Get data from the sensor thread
            message = ECG.GetData()
            print("message: {} json string: {}".format(idx, message))
            gc.collect()
 
            # sleep for 1 sec
            utime.sleep(1) 

            try:
                mqttClient.publish(topic=TOPIC, msg=message)
                print("message {} published".format(idx))

            except Exception as ex:
                print("Exception publishing message {}: {}".format(idx, os.strerror(ex.args[0])))

            idx = idx + 1

        except Exception as e:
            print("Exception from while loop: {}".format(os.strerror(e.args[0]), e.message))

    print("Data-sending loop completed OK. Calling disconnect on MQTT Client...") 
    mqttClient.disconnect()
    gc.collect()

except Exception as e2:
    print("Exception from main code: %s" % os.strerror(e2.args[0]))
    
finally:
    endLTE()

...and here is the output:

Disable MicroPython control and heartbeat of LED
Instantiated LTE connection.
Resetting LTE modem ... OK
Configuring LTE .. OK
Attaching to LTE network ..... OK
Connecting on LTE network  OK
getLTE returned lte: <LTE>
Created mqttClient
mqttClient connect call returned: 0
Successfully connected to MQTT Broker

message: 1 json string: [[0, 0], [1, 1860], [8, 1861], [15, 1862], [22, 1853], [29, 1854], [36, 1852], [43, 1854], [50, 1855], [57, 1862], [64, 1861], [71, 1878], [78, 1848], [85, 1863], [92, 1858]]
message 1 published

Main.py: Started loop iteration at local time: (1970, 1, 1, 0, 0, 24, 3, 1)
message: 2 json string: [[145, 1860], [152, 1864], [159, 1852], [166, 1853], [173, 1866], [180, 1852], [187, 1865], [194, 1867], [201, 1854], [208, 1866], [215, 1851], [222, 1852], [229, 1863], [236, 1861], [243, 1854], [250, 1847], [257, 1852], [264, 1863], [271, 1852], [278, 1854], [285, 1853], [292, 1860], [299, 1866], [306, 1846], [313, 1850], [320, 1854], [327, 1859], [334, 1867], [341, 1847], [348, 1851], [355, 1852], [362, 1854], [369, 1859], [376, 1852], [383, 1866], [390, 1862], [397, 1853], [404, 1852], [411, 1853], [418, 1847], [425, 1854], [432, 1853], [439, 1847], [446, 1861], [453, 1853], [460, 1862], [467, 1862], [474, 1851], [481, 1860], [488, 1864], [495, 1854], [502, 1861], [509, 1859], [516, 1851], [523, 1858], [530, 1854], [537, 1860], [544, 1860], [551, 1861], [558, 1851], [565, 1864], [572, 1859], [579, 1852], [586, 1853], [593, 1852], [600, 1854], [607, 1849], [614, 1859], [621, 1852], [628, 1852], [635, 1853], 
[642, 1849], [649, 1851], [656, 1849], [663, 1854], [670, 1854], [677, 1847], [684, 1852], [691, 1860], [698, 1851], [705, 1862], [712, 1850], [719, 1847], [726, 1853], [733, 1854], [740, 1852], [747, 1861], [754, 1852], [761, 1854], [768, 1854], [775, 1854], [782, 1847], [789, 1854], [796, 1861], [803, 1854], [810, 1860], [817, 1850], [824, 1847], [831, 1850], [838, 1854], [845, 1851], [852, 1847], [859, 1849], [866, 1847], [873, 1854], [880, 1854], [887, 1860], [894, 
1854], [901, 1849], [908, 1853], [915, 1851], [922, 1852], [929, 1854], [936, 1853], [943, 1845], [950, 1852], [957, 1847], [964, 1852], [971, 1849], [978, 1849], [985, 1847], [992, 1854], [999, 1850], [1006, 1853], [1013, 1854], [1020, 
1847], [1027, 1847], [1034, 1851], [1041, 1861], [1048, 1847], [1055, 1862], [1062, 1850], [1069, 1850], [1076, 1854], [1083, 1852], [1090, 1852], [1097, 1847], [1104, 1854], [1111, 1861], [1118, 1854], [1125, 1847], [1132, 1860], [1139, 1860], [1146, 1851], [1154, 1853], [1161, 1854], [1168, 1850], [1175, 1851], [1182, 1847], [1189, 1858], [1196, 1852], [1203, 1854], [1210, 1846], [1217, 1847], [1224, 1860], [1231, 1862], [1238, 1854], [1245, 1853], [1252, 1850]]      
message 2 published

Main.py: Started loop iteration at local time: (1970, 1, 1, 0, 0, 25, 3, 1)
message: 3 json string: [[1473, 1853], [1480, 1854], [1487, 1849], [1494, 1847], [1501, 1853], [1508, 1854], [1515, 1849], [1522, 1851], [1529, 1851], [1536, 1846], [1543, 1846], [1550, 1850], [1557, 1850], [1564, 1861], [1571, 1842], [1578, 1849], [1585, 1851], [1592, 1847], [1599, 1859], [1606, 1850], [1613, 1859], [1620, 1860], [1627, 1847], [1634, 1847], [1641, 1852], [1648, 1854], [1655, 1853], [1662, 1860], [1669, 1862], [1676, 1853], [1683, 1858], [1690, 1849], [1697, 1852], [1704, 1851], [1711, 1850], [1718, 1854], [1725, 1851], [1732, 1852], [1739, 1854], [1746, 1851], [1753, 1852], [1760, 1861], [1767, 1851], [1774, 1847], [1781, 1852], [1788, 1859], [1795, 1844], [1802, 1854], [1809, 1842], 
[1816, 1860], [1823, 1851], [1830, 1857], [1837, 1849], [1844, 1849], [1851, 1854], [1858, 1851], [1865, 1853], [1872, 1850], [1879, 1858], [1886, 1861], [1893, 1851], [1900, 1857], [1907, 1852], [1914, 1850], [1921, 1853], [1928, 1850], [1935, 1849], [1942, 1854], [1949, 1852], [1956, 1852], [1963, 1854], [1970, 1850], [1977, 1854], [1984, 1849], [1991, 1853], [1998, 1857], [2005, 1850], [2012, 1854], [2019, 1860], [2026, 1849], [2033, 1853], [2040, 1864], [2047, 1852], [2054, 1850], [2061, 1846], [2068, 1850], [2075, 1853], [2082, 1851], [2089, 1843], [2096, 1859], [2103, 1845], [2110, 1861], [2117, 1851], [2124, 1853], [2131, 1862], [2138, 1852], [2145, 1859], [2152, 1854], [2159, 1851], [2166, 1850], [2173, 1850], [2180, 1854], [2187, 1852], [2194, 1854], [2201, 1852], [2208, 1851], [2215, 1854], [2222, 1847], [2229, 1850], [2236, 1847], [2243, 1859], [2250, 1854], [2257, 1847], [2264, 1853], [2271, 1860], [2278, 1854], [2285, 1860], [2292, 1859], [2299, 1864], [2306, 1854], [2313, 1847], [2320, 1847], [2327, 1852], [2334, 1852], [2341, 1850], [2348, 1847], [2355, 1854], [2362, 1853], [2369, 1854], [2376, 1852], [2383, 1851], [2390, 1860], [2397, 1861], [2404, 1854], [2411, 1861], [2418, 1859], [2425, 1851], [2432, 1849], [2439, 1852], [2446, 1849], [2453, 1860], [2460, 1853], [2467, 1852], [2474, 1852], [2483, 1851], [2490, 1845], [2497, 1851], [2504, 1851], [2511, 1846], [2518, 1847], [2525, 1853], [2532, 1851], [2539, 1854], [2546, 1857], [2553, 1849], [2560, 1852], [2567, 1853], [2574, 1854], [2581, 1849]]
message 3 published

Main.py: Started loop iteration at local time: (1970, 1, 1, 0, 0, 27, 3, 1)
message: 4 json string: [[2813, 1845], [2821, 1853], [2828, 1847], [2835, 1847], [2842, 1853], [2849, 1844], [2856, 1847], [2863, 1854], [2870, 1854], [2877, 1849], [2884, 1847], [2891, 1862], [2898, 1845], [2905, 1844], [2912, 1852], [2919, 1854], [2926, 1852], [2933, 1857], [2940, 1853], [2947, 1844], [2954, 1852], [2961, 1862], [2968, 1849], [2975, 1852], [2982, 1850], [2989, 1852], [2996, 1866], [3003, 1853], [3010, 1851], [3017, 1854], [3024, 1852], [3031, 1857], [3038, 1849], [3045, 1851], [3052, 1847], [3059, 1849], [3066, 1847], [3073, 1857], [3080, 1854], [3087, 1847], [3094, 1854], [3101, 1850], [3108, 1851], [3115, 1851], [3122, 1854], [3129, 1850], [3136, 1843], [3143, 1858], [3150, 1859], 
[3157, 1859], [3164, 1852], [3171, 1859], [3178, 1849], [3185, 1858], [3192, 1843], [3199, 1854], [3206, 1852], [3213, 1847], [3220, 1854], [3227, 1852], [3234, 1858], [3241, 1861], [3248, 1859], [3255, 1847], [3262, 1846], [3269, 1858], [3276, 1847], [3283, 1865], [3290, 1853], [3297, 1859], [3304, 1854], [3311, 1854], [3318, 1844], [3325, 1847], [3332, 1857], [3339, 1847], [3346, 1852], [3353, 1862], [3360, 1852], [3367, 1853], [3374, 1850], [3381, 1850], [3388, 1853], [3395, 1846], [3402, 1854], [3409, 1854], [3416, 1849], [3423, 1852], [3430, 1854], [3437, 1847], [3444, 1854], [3451, 1850], [3458, 1854], [3465, 1846], [3472, 1850], [3479, 1847], [3486, 1849], [3493, 1854], [3500, 1850], [3507, 1852], [3514, 1852], [3521, 1858], [3528, 1849], [3535, 1852], [3542, 1849], [3549, 1852], [3556, 1852], [3563, 1860], [3570, 1854], [3577, 1850], [3584, 1854], [3591, 1850], [3598, 1852], [3605, 1859], [3612, 1861], [3619, 1854], [3626, 1853], [3633, 1852], [3640, 1850], [3647, 1853], [3654, 1854], [3661, 1852], [3668, 1854], [3675, 1859], [3682, 1849], [3689, 1851], [3696, 1850], [3703, 1846], [3710, 1854], [3717, 1854], [3724, 1861], [3731, 1849], [3738, 1861], [3745, 1852], [3752, 1847], [3759, 1846], [3766, 1847], [3773, 1860], [3780, 1854], [3787, 1852], [3794, 1851], [3801, 1851], [3808, 1852], [3815, 1860], [3823, 1852], [3830, 1863], [3837, 1844], [3844, 1853], [3851, 1859], [3858, 1847], [3865, 1859], [3872, 1847], [3879, 1847], [3886, 1851], [3893, 1854], [3900, 1858], [3907, 1854], [3914, 1859], [3921, 1852]]

Disonnecting LTE ... Traceback (most recent call last):
  File "main.py", line 300, in <module>
  File "main.py", line 166, in endLTE
OSError: the requested operation failed
Pycom MicroPython 1.20.1.r1 [15b6d69] on 2019-11-02; FiPy with ESP32
Type "help()" for more information.
>>> Unhandled exception in thread started by <bound_method>
IndexError:

I send 4 messages and it is obvious that I do get the 4th print "message 4 published" despite I have exception handling around...and somehow mysteriously the code jumps to endLTE() function which for another mysterious reason throws exception executing lte.disconnect(). Any ideas? Is a payload too big? This is my main question... the next one I may be able to handle by just killing the thread before exiting...

Then it gets to REPL and sits there for awhile. Then suddenly it throws exception IndexError. So, does it mean that the thread that I sarted with ECG constructor was still looping despite that my main exited??

EDIT: Dec 16, 2019
I resolved the problem... It has to do how much data being sent in one chunk and how frequently...The bottom line: send small chunks of data and frequently (for example: you can send 120 bytes every 7 milliseconds and it all goes through without any problems). If you want more details on how to push vast amount of data quickly - ping me...
And yet, the problem I was having calling lete.disconnect() continues - I constantly get exception on this...

EDIT: Dec 17, 2019
Apparently, the data delivery saga continues... My previous joy of resolving it was premature...
Now that I sent 117 bytes string every 50 msec over mqttClient.publish the call does not return from it when trying to send 43rd message. I tried different combinations of the payload and the transmission frequency and I always get that at some point after 2-2.5 sec of transmitting data (that is 4KB-4.5KB) the call to mqttClient.publish never returns (that is returns after it timed out). And the displays Exception code 118. Initially, I tought that it is multithreading screws things up but now there are no threads left...

Despite the enormous time and effort I invested in this, the above behavior makes this project (or combination of FiPy + MicroPython not viable). So I definitely will appreciate and open to try any new ideas how to resolve this problem...
I also wonder if the root of the problem is in the following commands despite that the connection goes through fine and always connects to AT&T.

print("Configuring LTE ", end='')
    lte.send_at_cmd('AT+CGDCONT=1,"IP","hologram"')  # Changed this from origninal
    print(".", end='')
    lte.send_at_cmd('AT!="RRC::addscanfreq band=4 dl-earfcn=9410"') # no idea what earfcn=9410 is... is it correct??
    print(".", end='')
    lte.send_at_cmd('AT+CFUN=1')
    print(" OK")

Here is the latest code:

    idx = 1
    attempt = 0


    # Loop for delivery sensor information over CAT-M1
    while(idx < 1000):

        print("Main.py: Started loop iteration at local time: {}".format(utime.localtime()))
    
        try:

            diff = 0
            message = ""

            start = time.ticks_ms()

            #print("Start Ticks: {}".format(start))
            while (diff < 50):
                    
                # get the voltage value
                v = adc_channel.voltage()
                
                diff = utime.ticks_diff(time.ticks_ms(), start)

                message += "{},{} ".format(diff, v)

                gc.collect()

            try:
                mqttClient.publish(topic=TOPIC, msg=message)
                print("Num: {}  Len: {}  Msg: {}".format(idx, len(message), message))
                gc.collect()

            except Exception as ex:
                print("Exception publishing {}".format(ex.args[0]))
                attempt += 1
                if (attempt > 3):
                    break
                gc.collect()

           idx = idx + 1

        except Exception as e:
            print("Exception from while loop: {}".format(e.args[0]))

    print("Data-sending loop exited. Calling disconnect on MQTT Client...") 
    mqttClient.disconnect()
    gc.collect()

 
except Exception as e2:
    print("Exception from main code: %s" % e2.args[0])
    
finally:
    print("Enable MicroPython control and heartbeat of LED")
    pycom.heartbeat(True)

    endLTE()

Explore Pybytes | Official Documentation | Report a Firmware Bug/Issue | GitHub

FiPy: MQTT publishing over CAT-M1

Pycom on Twitter