Potential device lockdown - Network providers

tealbrains

Hi,

I have come to the conclusion that the Cat-M1 capable Pycom devices (GPy and FiPy) can't connect to the Telstra network in Australia anymore. We had 9!!! yes 9! devices working, and not all with the same firmware. Some using HTTP, some using MQTT... They won't connect to the network anymore! Not a server issue, it's not even getting there, they are stuck on

lte.attach()

You could say it's a network outage, but we had three different sets in 3 major Australian cities, more than 2500km apart. I just tested the network with a Quectel BG96, and was able to make a data connection.

Many boards were ordered, and they are just going to become expensive bricks. The project is coming to an abrupt end if this is not fixed soon...

I have posted in the right thread but thought this was worth a new one.

@tealbrains said in General LTE-M (Cat M1) connection problems thread:

@Prawnhead @timh

Are you still able to connect to the Telstra network? The gpy boards I have no longer work. I thought it was a network outage issue but it appears to be something else.

I have been running the exact same program so it can't be firmware as it was working, posting data, etc.

I have tested it with :

Aldi APN: mdata.com.au

Telstra Prepaid : APN: telstra.internet

M2Mone APN: tesltra.m2m

Band 28, Earfcn 9410 for all three.

I thought maybe the modem on the Gpy may have been damaged (no reason why though) so I tested another 2 Gpys I had (which also used to work). Same problem. It just gets stuck at lte.attach(), where it was working before (most of the time anyway...).

I am able to send SMS and have data connectivity from a smartphone -> SIM cards OK.

I also have an IoT device that uses a Quectel BG96 and uses an M2Mone sim card. It was able to send data -> Network is OK.

No changes made to firmware or hardware

I have exhausted all my options. Any ideas?

Any help would be appreciated.
Thank you.

IOAg

@daniel I understand Telstra connectivity issue should be resolved. I don't see how to test it myself, Sequans firmware being unavailable at the moment.

Did you get positive feedback in this forum?

micksta

@timh

Thanks t for the follow up/post, appreciated.

timh

@tealbrains Just received this from Telstra

Thanks for your email and sorry for the delay. I was basically consulting number of different teams to have clear answer for you.

I just had confirmation from our engineering team that they are working with “Sequans” on certifying the Cat M1 module. (same chipset as your devices). This will be mostly completed by end of August and we will have Telstra module FW to support these devices.

I will keep following it up and let you know once this activity has been completed.

Given various other people found their pycoms started working late last week, it looks like we have a few moving targets with SEQUANS firmware and Telstra network.

timh

@timh To fix this I found I had to dupterm to a null device. This is odd and make development hard. This appears to have been introduced with the latest firmware. I don't know how my code could introduce such behaviour.

timh

@tealbrains Hmm not so sure.
I am finding that if I don't have a terminal connected when using LTE, after a while I am getting socket errors.
I don't get this when terminal is connected.

Disconnecting for a short time, means mqtt packets don't get sent, reconnecting the terminal in time means they are sent back to back quickly (seems they are queued). Reconnecting after a longer period (minutes) I see socket errors.

Where as the same code with a WIFI connection doesn't have the problem.

I am struggling to understand why having a terminal connection makes any difference, but it is repeatable, across a GPY and FiPY.

tealbrains

@timh @micksta @Prawnhead @reclusio Yep, so I just tested it and it's working now.

The firmware update was not the solution.
Telstra has been tinkering with the network as we all know. This last change, which has fixed the problem, seems to have happened at the same time of the week, exactly one month later. It is worth noting that they have just done the following : https://www.telstra.com.au/aboutus/media/media-releases/Telstra-deploys-Australias-first-LTE-Broadcast-technology-to-support-sports-streaming . I wouldn't be surprised if it is connected to the issue.
As for me, I will not be developing any products around the Gpy/Fipy until their modem is certified. I am supporting a couple of projects that are already using them. I am developing a comms board with a certified modem. Such a shame...wasted so much time and effort. I might still use these boards for quick tests but nothing more.

I guess we can consider the case closed (fingers crossed) for now.

timh

@micksta really. Thats interesting. Have Telstra rolled back some changes.
I will ask the support engineer on Monday of they know of any changes

blindman2k

Same here. Everything was working one day, nothing working the next day, everything working again today.

micksta

@timh

Hey lads, changed nothing here. Just thought I'd try and connect - Success here as well.

(sysname='GPy', nodename='GPy', release='1.17.5.b6', version='v1.8.6-849-56d00234 on 2018-05-18', machine='GPy with ESP32')

send_at_cmd_pretty('AT+CGDCONT=1,"IP","mdata.net.au"') # Aldi sim / "mdata.net.au" Aldi APN (aka Telstra network)
time.sleep(1)
send_at_cmd_pretty("AT!=\"RRC::addscanfreq band=28 dl-earfcn=9410\"")
time.sleep(1)
send_at_cmd_pretty('AT+CFUN=1') # enable modem & set to full functionality mode
time.sleep(1)

timh

@tealbrains So I finally found time to test the SEQUANS firmware.

I performed the firmware upgrade as per the instructions.

The upgrade performed as per instructions on a FiPy

And the result, working fine. - Success for me at least.

(sysname='FiPy', nodename='FiPy', release='1.17.3.b1', version='v1.8.6-849-83e2f7f on 2018-03-19', machine='FiPy with ESP32', lorawan='1.0.2', sigfox='1.0.1')

    lte.send_at_cmd('AT+CGDCONT=2,"IP","telstra.internet"')
    lte.send_at_cmd("AT!=\"RRC::addscanfreq band=28 dl-earfcn=9410\"")
    lte.send_at_cmd('AT!="RRC::addScanBand band=28"')
    lte.send_at_cmd('AT+CFUN=1')

    for i in range(retries):
        if wdt:
            wdt.feed()

        if (lte.isattached()):
              print("attached")
              lte.connect()
              break
        time.sleep(1)

I will keep this running over the weekend, but looks good for me on at least one device.

The other devices are on the other side of the country in mechanical pumps so need to organise people to get me access to perform the upgrade ;-(

timh

@prawnhead I haven't been able to spend time on this (heavy workload on non rpycom related projects).

It does seem worrisome that no one seems to have it working.

Prawnhead

Anyone who's got working LTE with Optus or Vodafone wanna share your setup? Just looking for APN/channel configuration that would make testing other carriers quicker. Thanks, Chris.

Dylan

Hey everybody,

I have been using a FiPy and having the same trouble that appears to be happening to most people (maybe everybody) in Australia.

I was in the middle of writing something up for a new topic on FiPy but since this is basically it I'll put it here.

Currently the following occurs:

LTE connects successfully many times in a row, but can then go for hours without a connection (With 60 second sleep cycles). Just like discussed here:
https://forum.pycom.io/topic/2878/fipy-lte-connection-working-with-telstra/18
and
https://forum.pycom.io/topic/2769/telstra-lte-cat-m-working-on-fipy-just/24
When doing longer sleeps between messages (15minutes) I very rarely get a connection
Today I have simply not been able to get a connection at all.
Sometimes when I do connect, it will say message sent however it is never received by adafruit
Last successful connection was last night around 9pm.

I just installed the new sequans firmware and having the same result.

I am using SIM cards that are given out by telstra for m2m.

My code:

import machine
from network import LTE
from pytrack import Pytrack
import os
import time
import socket
import pycom
from MQTT import MQTTClient
from L76GNSS import L76GNSS


def sub_cb(topic, msg):
    print(msg)

lte = LTE()
# lte.reset()
print("Line 15") #lots of prints for debugging
time.sleep(1)
lte.send_at_cmd('AT^RESET') #trying this to see if it helps on restart
print(lte.send_at_cmd('AT^RESET'))
print("Line 16")
time.sleep(1)

lte.send_at_cmd('AT+CFUN=0')
print("Line 17")
time.sleep(1)
lte.send_at_cmd('AT!="clearscanconfig"')
print(lte.send_at_cmd('AT!="clearscanconfig"'))
print("Line 18")
time.sleep(1)

lte.send_at_cmd('AT+CGDCONT=1,"IP","telstra.m2m"')
print(lte.send_at_cmd('AT+CGDCONT=1,"IP","telstra.m2m"'))
print("Line 19")
time.sleep(1)

lte.send_at_cmd("AT!=\"RRC::addscanfreq band=28 dl-earfcn=9410\"")
print(lte.send_at_cmd("AT!=\"RRC::addscanfreq band=28 dl-earfcn=9410\""))
print("Line 20")
time.sleep(1)

lte.send_at_cmd('AT+CFUN=1')
print(lte.send_at_cmd('AT+CFUN=1'))
print("Line 21")
time.sleep(1)

print("Line 22")
time.sleep(1)

while not lte.isattached():
    print("Attaching...")
    time.sleep(0.1)
print("Attached")

lte.connect()

while not lte.isconnected():
    print("Connecting...")
    time.sleep(0.1)
print("Connected!")
time.sleep(2)
def settimeout(duration): 
   pass
print("line41")
def sub_cb(topic, msg): 
   print(msg)
print("line44")

client = MQTTClient("Demo","io.adafruit.com",user="Dylan", password="xxxx", port=1883)
print("Line 38")
client.set_callback(sub_cb)
print("Line 41")
client.connect()
print("Line 43")
time.sleep(1)
client.subscribe(topic="Dylan/feeds/test")
print("Line 46")
py = Pytrack()
battery = py.read_battery_voltage()
l77 = L76GNSS(py, timeout=30)
coord = l77.coordinates()
coord = str(coord) + str(',') + str(battery)

print("Sending ON")
client.publish(topic="Dylan/feeds/test", msg=coord)
print("Line 51")
time.sleep(1)
print("Line 55")
client.check_msg()

py.setup_sleep(60)
py.go_to_sleep()

Prawnhead

@tealbrains said in Potential device lockdown - Network providers:

Not only it is not working for me but it is even worse now (I did not think it could get any worse but it did.)

I am not able to get a CCID from the (M2Mone) SIM card when issuing:
send_at_cmd_pretty('AT+SQNCCID')
I thought it would maybe be an issue with the SIM card itself so used a GPY that still had the previous modem firmware (FIPY_GPY_CATM1_33988.dup) with the same SIM card and it worked.

So, summarising:

GPy on stable 1.18.0 with FIPY_GPY_CATM1_33988 -> Lost Telstra connectivity a few weeks ago but it can talk to the SIM card.

GPy on stable 1.18.0 with FIPY_GPY_CATM1_36417 -> Can't talk to SIM card, which I believe is what is producing an ERROR response when issuing
send_at_cmd_pretty('AT!="RRC::addScanBand band=28"')
It would seem we are taking one step forward and two steps backward...

If @micksta, @timh and @Prawnhead can confirm this behaviour the problem is still there.

I'm using this as minimum viable code:

import os
import time
import machine
from network import LTE

def at_cmd(command):
    print("Command:  {}".format(command))
    result = lte.send_at_cmd(command)
    print("Response: {}".format(result))

print(os.uname())

lte = LTE()

at_cmd('AT+CGDCONT=1,"IP","telstra.internet"')
at_cmd('AT!="addscanfreq band=28 dl-earfcn=9410"')
at_cmd('AT+CFUN=1')

for x in range(20):
    print("Attached? {}".format(lte.isattached()))
    time.sleep(1)

machine.reset()

Result is always:

>>> Running Z:\Content\Content2\Chris\Pycom\Pycom GPy\Pycom Forum 2878\connect0.py

>>>
>>>
(sysname='GPy', nodename='GPy', release='1.18.0', version='v1.8.6-849-046b350 on 2018-06-01', machine='GPy with ESP32')
Command:  AT+CGDCONT=1,"IP","telstra.internet"
Response:
OK

Command:  AT!="RRC::addScanBand band=28"
Response:

OK

Command:  AT+CFUN=1
Response:
OK

Attached? False
Attached? False
Attached? False
Attached? False
Attached? False
Attached? False
Attached? False
Attached? False
Attached? False
Attached? False
Attached? False
Attached? False
Attached? False
Attached? False
Attached? False
Attached? False
Attached? False
Attached? False
Attached? False
Attached? False
ets Jun  8 2016 00:22:57

rst:0x7 (TG0WDT_SYS_RESET),boot:0x17 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff8028,len:8
load:0x3fff8030,len:1728
load:0x4009fa00,len:0
load:0x4009fa00,len:14584
entry 0x400a059c
Pycom MicroPython 1.18.0 [v1.8.6-849-046b350] on 2018-06-01; GPy with ESP32
Type "help()" for more information.
>>>

I have tried 1 x FiPy and 1 x GPy on firmware FIPY_GPY_CATM1_33988.dup and FIPY_GPY_CATM1_36417.dup and have not had a successful lte.isattached() in 20+ attempts. However, I'm not completely confident I've got the minimum viable code right. If someone using Optus or Vodafone could post their minimum code or confirm mine (with whatever settings changes are required) I'd appreciate it!

Chris

micksta

@tealbrains
Sure CCID no prob's --> same no Connection. Had a look at some alt. LTE M1 products (googled "Certified Modules and Devices - Telstra") & ordered several SIM7000E. Will be interesting to compare.. Hopefully the Pyboards make the list soon.

tealbrains

@micksta I am guessing the AT+SQNCCID is what works without problems, right? It's a different story for the Cat-M1 connection. If you can't connect either I guess the issue is not resolved then...

micksta

@tealbrains

Here works no probs, have 2 GPY's - different revs of modem f/w.

Using (Aldi Sim aka Telstra & below GPy f/w)
(sysname='GPy', nodename='GPy', release='1.17.5.b6', version='v1.8.6-849-56d00234 on 2018-05-18', machine='GPy with ESP32')

send_at_cmd_pretty('AT+SQNCCID?') # reads the ICCID ID on the SIM card.

I'm also using (as @Daniel suggested below AT!)

send_at_cmd_pretty("AT!=\"RRC::addScanBand band=28\"")
 - System FSM & Synchro Stats still loops with no connection (although it does seem to see the tower) . CREG still stuck  --> +CREG: 2,0

tealbrains

@tealbrains said in Potential device lockdown - Network providers:

Not only it is not working for me but it is even worse now (I did not think it could get any worse but it did.)

I am not able to get a CCID from the (M2Mone) SIM card when issuing:
send_at_cmd_pretty('AT+SQNCCID')
I thought it would maybe be an issue with the SIM card itself so used a GPY that still had the previous modem firmware (FIPY_GPY_CATM1_33988.dup) with the same SIM card and it worked.

So, summarising:

GPy on stable 1.18.0 with FIPY_GPY_CATM1_33988 -> Lost Telstra connectivity a few weeks ago but it can talk to the SIM card.

GPy on stable 1.18.0 with FIPY_GPY_CATM1_36417 -> Can't talk to SIM card, which I believe is what is producing an ERROR response when issuing
send_at_cmd_pretty('AT!="RRC::addScanBand band=28"')
It would seem we are taking one step forward and two steps backward...

If @micksta and @Prawnhead can confirm this behaviour the problem is still there.

EDIT:
The SIM command does work, eventually. I think the new pycom firmware blocks some commands sometimes if the modem is busy.

@daniel
A bit of extra info would be nice. The way the first command you mentioned needs to be sent is slightly different.

 send_at_cmd_pretty("AT!=\"RRC::addScanBand band=28\"")

It must include the escape characters, otherwise it seems to produce an ERROR. The modem won't attach though.

tealbrains

Not only it is not working for me but it is even worse now (I did not think it could get any worse but it did.)

I am not able to get a CCID from the (M2Mone) SIM card when issuing:

send_at_cmd_pretty('AT+SQNCCID')

I thought it would maybe be an issue with the SIM card itself so used a GPY that still had the previous modem firmware (FIPY_GPY_CATM1_33988.dup) with the same SIM card and it worked.

So, summarising:

GPy on stable 1.18.0 with FIPY_GPY_CATM1_33988 -> Lost Telstra connectivity a few weeks ago but it can talk to the SIM card.
GPy on stable 1.18.0 with FIPY_GPY_CATM1_36417 -> Can't talk to SIM card, which I believe is what is producing an ERROR response when issuing

send_at_cmd_pretty('AT!="RRC::addScanBand band=28"')

It would seem we are taking one step forward and two steps backward...

If @micksta, @timh and @Prawnhead can confirm this behaviour the problem is still there.

Explore Pybytes | Official Documentation | Report a Firmware Bug/Issue | GitHub

Potential device lockdown - Network providers

Pycom on Twitter