GPy rebooting because of Pybytes watchdog?



  • I have a GPy that is currently connected via wifi to my home network. The wifi network is stable as far as I can tell, but it seems like the GPy is failing regularly to connect to the Pybytes MQTT broker and reboots the device.
    Is there a way to disable this (rebooting when Pybytes fails to connect via MQTT)?

    Below is the log output with pycom.nvs_set('pybytes_debug', 99). It shows that first the Protocol.__keep_connection(alarm=<Alarm>) calls succeed, but then fail with Socket send error [Errno 104] ECONNRESET. Once this happens the GPy reboots, and it happens regularly (sometimes every 10 minutes, sometimes every 20 or 30 minutes, but usually multiple times per hour.

    Checking the content of machine.reset_cause() returns 2, which according to the documentation means machine.WDT_RESET: Reset by watchdog timer. I'm not using a custom watchdog machine.WDT, so I assume it is an internal Pybytes one.

    All lines starting with [INFO] are my own log output, and I set a simple Timer.Alarm to output the current time once per minute to keep track of when this issue happens.

    AutoConnect enabled, ignoring 'address' setting (see Global Settings)
    Searching for PyCom boards on serial...
    Connecting to /dev/tty.usbmodemPy85be0a1...
    Wifi connection attempt: 2
    WLAN connected? False
    (...list of wifi networks...)
    Connecting with <SSID> and <PASSWORD>
    WiFi connection established
    MQTT Protocol
    Packet sent. (Length: 114)
    This is PybytesProtocol.start_MQTT
    Packet sent. (Length: 44)
    Connected to MQTT mqtt.pybytes.pycom.io
    Pybytes connected successfully (using the built-in pybytes library)
    This is pack_info_message()
    __pack_message: b'010504a00584000000'
    MQTT Protocol
    Packet sent. (Length: 50)
    Pybytes configuration read from /flash/pybytes_config.json
    [INFO] Starting firmware
    [INFO] Network connection: WLAN
    [INFO] Waiting for time syncronization...
    [INFO] 2021-05-21T11:27:00Z Time synchronized
    Pycom MicroPython 1.20.2.r4 [v1.11-ffb0e1c] on 2021-01-12; GPy with ESP32
    Pybytes Version: 1.6.1
    Type "help()" for more information.
    >>> [INFO] 2021-05-21T11:27:00Z
    [INFO] 2021-05-21T11:28:00Z
    [INFO] 2021-05-21T11:29:00Z
    [INFO] 2021-05-21T11:30:00Z
    [INFO] 2021-05-21T11:31:00Z
    [INFO] 2021-05-21T11:32:00Z
    [INFO] 2021-05-21T11:33:00Z
    [INFO] 2021-05-21T11:34:00Z
    [INFO] 2021-05-21T11:35:00Z
    [INFO] 2021-05-21T11:36:00Z
    This is PybytesProtocol.__keep_connection(alarm=<Alarm>)
    MQTT Protocol
    Packet sent. (Length: 42)
    This is PybytesProtocol.__process_recv_message()
    This is PybytesLibrary.unpack_message(message=b'\x07')
    header: 7 body: b''
    network_type: 0
    message_type: 7
    [INFO] 2021-05-21T11:37:00Z
    [INFO] 2021-05-21T11:38:00Z
    [INFO] 2021-05-21T11:39:00Z
    [INFO] 2021-05-21T11:40:00Z
    [INFO] 2021-05-21T11:41:00Z
    [INFO] 2021-05-21T11:42:00Z
    [INFO] 2021-05-21T11:43:00Z
    [INFO] 2021-05-21T11:44:00Z
    [INFO] 2021-05-21T11:45:00Z
    [INFO] 2021-05-21T11:46:00Z
    This is PybytesProtocol.__keep_connection(alarm=<Alarm>)
    MQTT Protocol
    Packet sent. (Length: 42)
    [INFO] 2021-05-21T11:47:00Z
    [INFO] 2021-05-21T11:48:00Z
    [INFO] 2021-05-21T11:49:00Z
    [INFO] 2021-05-21T11:50:00Z
    [INFO] 2021-05-21T11:51:00Z
    [INFO] 2021-05-21T11:52:00Z
    [INFO] 2021-05-21T11:53:00Z
    [INFO] 2021-05-21T11:54:00Z
    [INFO] 2021-05-21T11:55:00Z
    [INFO] 2021-05-21T11:56:00Z
    This is PybytesProtocol.__keep_connection(alarm=<Alarm>)
    MQTT Protocol
    Socket send error [Errno 104] ECONNRESET
    [INFO] 2021-05-21T11:57:00Z
    ets Jun  8 2016 00:22:57
    
    rst:0x7 (TG0WDT_SYS_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
    configsip: 0, SPIWP:0xee
    clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
    mode:DIO, clock div:1
    load:0x3fff8020,len:8
    load:0x3fff8028,len:2128
    load:0x4009fa00,len:19760
    entry 0x400a05bc
    {'wifi': {'ssid': 'SSID', 'password': 'PASSWORD'}, 'cfg_msg': 'Pybytes configuration read from /flash/pybytes_config.json', 'dump_ca': False, 'username': 'mail@example.org', 'server': 'mqtt.pybytes.pycom.io', 'ssl': False, 'lte': {'apn': 'em', 'cid': 1, 'reset': False, 'carrier': 'standard', 'band': 20, 'type': 'IP'}, 'device_id': '00000000-0000-0000-0000-000000000000', 'network_preferences': ['wifi', 'lte'], 'ota_server': {'port': 443, 'domain': 'software.pycom.io'}, 'wlan_antenna': 0, 'pybytes_autostart': True}
     ssl_params={'ca_certs': '/flash/cert/pycom-ca.pem'} 
    LIS2HH12 not imported
     ssl_params={'ca_certs': '/flash/cert/pycom-ca.pem'} 
    WMAC: 840D8E122018
    Firmware: 1.20.2.r4
    Pybytes: 1.6.1
    {'wifi': {'ssid': 'SSID', 'password': 'PASSWORD'}, 'cfg_msg': 'Pybytes configuration read from /flash/pybytes_config.json', 'dump_ca': False, 'username': 'mail@example.org', 'server': 'mqtt.pybytes.pycom.io', 'ssl': False, 'lte': {'apn': 'em', 'cid': 1, 'reset': False, 'carrier': 'standard', 'band': 20, 'type': 'IP'}, 'device_id': '00000000-0000-0000-0000-000000000000', 'network_preferences': ['wifi', 'lte'], 'ota_server': {'port': 443, 'domain': 'software.pycom.io'}, 'wlan_antenna': 0, 'pybytes_autostart': True}
    Attempting to connect with network wifi
    Initialized watchdog for WiFi and LTE connection with timeout 1260000 ms
    WLAN connected? False
    Wifi connection attempt: 1
    WLAN connected? False
    (...list of wifi networks...)
    Connecting with <SSID> and <PASSWORD>
    Wifi connection attempt: 2
    WLAN connected? False
    (...list of wifi networks...)
    Connecting with <SSID> and <PASSWORD>
    WiFi connection established
    MQTT Protocol
    Packet sent. (Length: 114)
    This is PybytesProtocol.start_MQTT
    Packet sent. (Length: 44)
    Connected to MQTT mqtt.pybytes.pycom.io
    Pybytes connected successfully (using the built-in pybytes library)
    This is pack_info_message()
    __pack_message: b'010504a00584000000'
    MQTT Protocol
    Packet sent. (Length: 50)
    Pybytes configuration read from /flash/pybytes_config.json
    [INFO] Starting firmware
    [INFO] Network connection: WLAN
    [INFO] Waiting for time syncronization...
    [INFO] 2021-05-21T11:58:07Z Time synchronized
    Pycom MicroPython 1.20.2.r4 [v1.11-ffb0e1c] on 2021-01-12; GPy with ESP32
    Pybytes Version: 1.6.1
    Type "help()" for more information.
    >>> [INFO] 2021-05-21T11:59:07Z 
    [INFO] 2021-05-21T12:00:07Z 
    [INFO] 2021-05-21T12:01:07Z 
    >>> machine.reset_cause()
    2
    >>> [INFO] 2021-05-21T12:02:07Z 
    [INFO] 2021-05-21T12:03:07Z 
    [INFO] 2021-05-21T12:04:07Z 
    [INFO] 2021-05-21T12:05:07Z 
    [INFO] 2021-05-21T12:06:07Z 
    [INFO] 2021-05-21T12:07:07Z 
    [INFO] 2021-05-21T12:08:07Z 
    This is PybytesProtocol.__keep_connection(alarm=<Alarm>)
    MQTT Protocol
    Packet sent. (Length: 42)
    [INFO] 2021-05-21T12:09:07Z 
    [INFO] 2021-05-21T12:10:07Z 
    [INFO] 2021-05-21T12:11:07Z 
    [INFO] 2021-05-21T12:12:07Z 
    [INFO] 2021-05-21T12:13:07Z 
    [INFO] 2021-05-21T12:14:07Z 
    [INFO] 2021-05-21T12:15:07Z 
    [INFO] 2021-05-21T12:16:07Z 
    [INFO] 2021-05-21T12:17:07Z 
    [INFO] 2021-05-21T12:18:07Z 
    This is PybytesProtocol.__keep_connection(alarm=<Alarm>)
    MQTT Protocol
    Socket send error [Errno 104] ECONNRESET
    ets Jun  8 2016 00:22:57
    
    rst:0x7 (TG0WDT_SYS_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
    configsip: 0, SPIWP:0xee
    clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
    mode:DIO, clock div:1
    load:0x3fff8020,len:8
    load:0x3fff8028,len:2128
    load:0x4009fa00,len:19760
    entry 0x400a05bc
    {'wifi': {'ssid': 'SSID', 'password': 'PASSWORD'}, 'cfg_msg': 'Pybytes configuration read from /flash/pybytes_config.json', 'dump_ca': False, 'username': 'mail@example.org', 'server': 'mqtt.pybytes.pycom.io', 'ssl': False, 'lte': {'apn': 'em', 'cid': 1, 'reset': False, 'carrier': 'standard', 'band': 20, 'type': 'IP'}, 'device_id': '00000000-0000-0000-0000-000000000000', 'network_preferences': ['wifi', 'lte'], 'ota_server': {'port': 443, 'domain': 'software.pycom.io'}, 'wlan_antenna': 0, 'pybytes_autostart': True}
     ssl_params={'ca_certs': '/flash/cert/pycom-ca.pem'} 
    LIS2HH12 not imported
     ssl_params={'ca_certs': '/flash/cert/pycom-ca.pem'} 
    WMAC: 840D8E122018
    Firmware: 1.20.2.r4
    Pybytes: 1.6.1
    {'wifi': {'ssid': 'SSID', 'password': 'PASSWORD'}, 'cfg_msg': 'Pybytes configuration read from /flash/pybytes_config.json', 'dump_ca': False, 'username': 'mail@example.org', 'server': 'mqtt.pybytes.pycom.io', 'ssl': False, 'lte': {'apn': 'em', 'cid': 1, 'reset': False, 'carrier': 'standard', 'band': 20, 'type': 'IP'}, 'device_id': '00000000-0000-0000-0000-000000000000', 'network_preferences': ['wifi', 'lte'], 'ota_server': {'port': 443, 'domain': 'software.pycom.io'}, 'wlan_antenna': 0, 'pybytes_autostart': True}
    Attempting to connect with network wifi
    Initialized watchdog for WiFi and LTE connection with timeout 1260000 ms
    WLAN connected? False
    Wifi connection attempt: 1
    WLAN connected? False
    (...list of wifi networks...)
    Connecting with <SSID> and <PASSWORD>
    Wifi connection attempt: 2
    WLAN connected? False
    (...list of wifi networks...)
    Connecting with <SSID> and <PASSWORD>
    WiFi connection established
    MQTT Protocol
    Packet sent. (Length: 114)
    This is PybytesProtocol.start_MQTT
    Packet sent. (Length: 44)
    Connected to MQTT mqtt.pybytes.pycom.io
    Pybytes connected successfully (using the built-in pybytes library)
    This is pack_info_message()
    __pack_message: b'010504a00584000000'
    MQTT Protocol
    Packet sent. (Length: 50)
    Pybytes configuration read from /flash/pybytes_config.json
    [INFO] Starting firmware
    [INFO] Network connection: WLAN
    [INFO] Waiting for time syncronization...
    [INFO] 2021-05-21T12:19:34Z Time synchronized
    Pycom MicroPython 1.20.2.r4 [v1.11-ffb0e1c] on 2021-01-12; GPy with ESP32
    Pybytes Version: 1.6.1
    Type "help()" for more information.
    >>> [INFO] 2021-05-21T12:20:34Z 
    [INFO] 2021-05-21T12:21:34Z 
    [INFO] 2021-05-21T12:22:34Z 
    [INFO] 2021-05-21T12:23:34Z 
    [INFO] 2021-05-21T12:24:34Z 
    


  • Ah sorry, my mistake! Good to hear it works :)



  • Thanks for the quick reply! This sounds like what I am looking for, but when I try to set the configuration key I get this error:

    >>> pybytes.update_config("connection_watchdog", value=False)
    Error updating configuration!
    Traceback (most recent call last):
      File "_pybytes.py", line 339, in update_config
    KeyError: connection_watchdog
    

    Edit: This seems to fail because the pybytes_config.json does not have this key present. It works when using pybytes.set_config(...) though :)



  • I've suspected this issue for my devices for the past few days as well (but they did not have the pybytes debugging turned on, so Im not 100% sure). We did have some issues with the Pybytes service, but all should be operational now.

    Disabling the watchdog can be done in the pybytes_config.json file by adding the key 'connection_watchdog' with value 'False'. You can use pybytes.update_config(...) as described here: https://docs.pycom.io/pybytes/api/#pybytesupdate_configkey-valuenone-permanenttrue-silentfalse-reconnectfalse


Log in to reply
 

Pycom on Twitter