LoPy stops detecting BLE advertisements

[papasmurph]

@jmarcelino Sadly, scanning (or detection) still stops though. After an hour or so get_adv stopped returning data, despite there being beacons close by that I hadn't moved. A hard reset got it started again.

[papasmurph]

@papasmurph When this happens, neither is it possible to attach to the LoPy's WiFi access point. My code still runs though.

[Duane Kaufman]

@papasmurph I think I might have a bad board, as I have not been able to get BLE to see devices I can detect with my phone around me. My code:

`from network import Bluetooth
import time
import binascii

bluetooth = Bluetooth()
bluetooth.init()

bluetooth.start_scan(20) # start scanning with 10s timeout
while bluetooth.isscanning():
time.sleep(1)
adv_list = []
adv_list = bluetooth.get_adv()
if adv_list != None:
for adv in adv_list:
print("Time: {0} adv.mac: {1}".format(time.time(), binascii.hexlify(adv.mac)))
else:
print("Time: {0} adv.mac: {1}".format(time.time(), "None"))
`
and I get:
MicroPython v1.8.6-607-g9c8a0e9e on 2017-05-01; LoPy with ESP32
Login as: micro
Password:
Login succeeded!
Type "help()" for more information.

import BLE_Raw
Time: 229 adv.mac: None
Time: 230 adv.mac: None
Time: 231 adv.mac: None
Time: 232 adv.mac: None
Time: 233 adv.mac: None
Time: 234 adv.mac: None
Time: 235 adv.mac: None
Time: 236 adv.mac: None
Time: 237 adv.mac: None
Time: 238 adv.mac: None
Time: 239 adv.mac: None
Time: 240 adv.mac: None
Time: 241 adv.mac: None
Time: 242 adv.mac: None
Time: 243 adv.mac: None
Time: 244 adv.mac: None
Time: 245 adv.mac: None
Time: 246 adv.mac: None
Time: 247 adv.mac: None
Time: 248 adv.mac: None`

Thanks for your review,
Duane
`

[papasmurph]

@Duane-Kaufman get_adv returns a single tuple, not an array, so "for" doesn't make sense here. See the code I posted initially.

Example: (mac=b'\xb4\x99Lm\x80\xcd', addr_type=0, adv_type=0, rssi=-87, data=b'\x02\x01\x06\x03\x02\xf0\xff\x15\xffBH\xcd\x80mL\x99\xb4\xa2@\x10\x01\x00\x00\xff\xff\x0f\xff\xff\xff\x00\x00')

[Duane Kaufman]

@papasmurph Thanks for looking this over. Changed code to this, but still no response:

import time
import binascii

bluetooth = Bluetooth()
bluetooth.init()

bluetooth.start_scan(20)    # start scanning with 10s timeout
while bluetooth.isscanning():
	time.sleep(1)
	adv_list = bluetooth.get_adv()
	if adv_list != None:
		print("Time: {0} adv.rssi: {1}".format(time.time(), adv.rssi))
	else:
		print("Time: {0} adv.mac: {1}".format(time.time(), "None"))```

and I still get nothing :( as before

Duane

[papasmurph]

@Duane-Kaufman It works fine if you fix a few bugs. Well, at least it did for me.

from network import Bluetooth
import time

bluetooth = Bluetooth()
bluetooth.init()
bluetooth.start_scan(20)

while bluetooth.isscanning():
    adv_list = bluetooth.get_adv()
    if adv_list != None:
        print("Time: {0} adv.rssi: {1}".format(time.time(), adv_list.rssi))

    time.sleep(0.05)

print("I give up!")

[Duane Kaufman]

@papasmurph Using your code (modified a little :):

import time

bluetooth = Bluetooth()
bluetooth.init()
bluetooth.start_scan(20)

while bluetooth.isscanning():
    adv_list = bluetooth.get_adv()
    if adv_list != None:
        print("Time: {0} adv.rssi: {1}".format(time.time(), adv_list))

    time.sleep(1.0)

print("I give up!")```

I do not get anything (generally). Once in a great while, I will pick up one of the three BLE devices nearby, but probably only 1 in ten times I run the script.

Duane

[papasmurph]

@Duane-Kaufman I don't strictly see all advertisements either and after some time (randomly, but usually after an hour or so) it stops entirely, so something is wrong. In my own code I now use the event callback, which enables it to do other things in parallel, but it still happens though.

[RichardTXD]

@papasmurph
Hi.
I've been working on this, following your progress and the great community support - thanks. This is a show stopper for me if it can't be made to scan reliably 24/7.

Have you made progress?

I found that it just stops scanning early without an error message. I told it to scan for "-1" seconds = forever. I thought it was broken packets but I (currently) think that is wrong.

I've been tinkering with your code. I found it sensitive to putting in a gc.collect() just before fetching a packet (device = bluetooth.get etc) - that made the difference between scanning for approx 1 minute to many minutes (still going after 10 and I must leave the office).
Much more testing required.

[papasmurph]

@RichardTXD With callback to an event handler it can work for an hour or so, but still eventually stops. It's time Pycom takes a look at it.

[stef]

@papasmurph ,
I confirm that , still stoping after time . nothing to do but reset.
New firmware doesn't matter .

[RichardTXD]

@papasmurph
I am Running WiPy 2.0 with latest firmware flashed yesterday.
Windows 7 and Atom. I'm using an expansion board.
I set my PC to not shutdown/go to sleep etc.

Hi. There is also a concern with callback. I tried it also. Then I went back to a non-call back program. The callback was still triggering.
It would appear that unless you press reset of the board, the callback lingers in memory.
I would not be surprised at a difference. I'm guessing memory is deallocated when the callback is terminated and that may be at the heart of it all.

@stef
Hi. You are quite correct. It is unstable.
I've seen different crashes depending on if garbage collection is done. If I don't garbage collect it seems to crash quicker - I will go back to testing that later.

I'm still trying to get a feel for how and where it crashes but so far it seems random - a few hours to tens of minutes.

I modded the program to tell me how many beacons were received. I have a lot of beacons in the office. I empty the queue on each iteration.

I did 3 tests late yesterday.
I set it to scan for -1 seconds = infinitely
There is a garbage collection before each read of bluetooth into its variable.
I press reset between each program run
Test-1. It received 103,139 beacons and failed after about 2.5 hours
Test-2. It received 17,094 beacons and failed - not all that long.
Test-3. It received 31,945 beacons and failed.
Each fail was the code hanging up. Stopped.
I just now put in an elapsed seconds on the end of each result.
The outer while loop repeats till it is not scanning. An inner loop empties the queue. I assume it was still scanning, yet nothing was received.

Until we get past this issue, I can't ship product of course, same as others I suppose. I'm trying to think of a workaround.

[papasmurph]

@RichardTXD As this is discussed in Issues&Bugs, let's hope Pycom acts on it.

[RichardTXD]

@papasmurph
Yes. I truly hope so and soon. It is unstable. Same for WiPy 2.0

Recap : My code is basically yours with extra diagnositcs for number of beacons read, heap space, number of items in the queue.

When it fails, I press control-C and each time it says the last line was "while device != None" (I pull all packets out of the queue when it wakes up).

Is this a hint to anyone?

I've spent the day runing tests on it. Best is about 3.6 hours. Worst about 10 minutes. Putting in garbage collection, taking it out. Changing "sleep" for "idle()", adding diagnostic messages. No closer to nailing down what it is.

Tempted to run the program without any bluetooth to see if it is the underlying system that is flaky. Maybe tomorrow.

Rather frustrating.

[stef]

@RichardTXD
yes , the only workarround i found for this problem is to use deepsleep ( as it act like a reset).
Another problem i try to solve is to avoid all the 'miss' on a short scan time.
I use Kontack Beacons , they push a message every 250ms.
With a 2 secondes scan , on 3 Beacons surrounding , one tag is missing on 1 / 4 loop.

any hint ?

[papasmurph]

@stef You can reconfigure any beacon (including Kontakt's) to ping less often. 0.25 is usually unnecessarily short and drains the battery fast, unless they are USB-attached of course. Try 1 second. I recommend that you scan for 10 seconds and collect all detected beacons in a list, send the list (in some clever format) and then activate sleep. I don't understand why you miss any pings with that high frequency. The Pycom should detect all and queue them (up to 8) that should all be visible via get_adv if you just call it often enough (at least every 50 milliseconds or so), or better, use a callback with an internal loop for "strays", the way I do it now. I've noticed not all pings show up, but I have assumed that's a bug.

[RichardTXD]

@stef
Hi Stef. We use Kontakt beacons also. I have something like 90 to 100 beacons in the office with me, set on Level-2 power and 500mS interval.
I notice a speed difference in scanning depending on code. I can get close to 1,000 packets a minute or down below 600 depending on the code. One run I got 103,000 packets in about 2 hours. A different set of code with different garbage collection I got about 90,000 packets in 3.5 hours approx. Quite a difference.
I'm not that worried about lost packets; the possibility of collisions and broken packets is very high - and then there are my 2 or 3 WiFi devices in the office also pushing out RF. Broken and lost packets are a given imho. There is no way they won't collide.
With 3 beacons and 250mS gap there would be less collisions but I would not rule it out. Maybe set a longer interval in the beacons and increase the scanning window.
Latest test-tweak of the code has run for 1hr 20m approx. Nothing special. I would have thought 24 to 48 hours 'good' but not great if it falls over then. The problem is that it falls over and seems to die.No error message. No recovery.

[RichardTXD]

@papasmurph
I mostly agree - and strongly agree on battery.
To a small degree I'd be surprised with 3 beacons if you collected all advertizing packets every time for a period or more than a few seconds.
Problem/advantage is that shorter intervals give better geolocation. iDevices seem to do an averaging and lots of packets make it more stable iirc. They work via RSSI and it is a real struggle. Android seems to report everything (could be wrong on that).
We use 500mS as we need better battery life - and accept the lesser accuracy.
The marketing material I see that claims 1 metre and better accuracy makes me smile. In the office standing still it is very variable with 500mS.
Problem with 1 second and low power, Level-0, is that it is very possible to move past it and not get a reading in an iDevice as it doesn't get enough hits. Battery life is great tho.
I suppose it depends on the application & tradeoffs.
Packet collisions... that is my theory as to what is happening. I may be wrong - happy to be adviced by those more knowledgable but it would seem logical.
Multipath is also an issue we see in the field. Reflections of the same packet arriving time shifted as a result of a reflecting surface. Seen this in the field with just one beacon and small dead spots in a zone where it is working all around. Same effect as 'ghosting' all those years ago on television but leads to broken packets.
Agree, a longer sample time would be good.

[papasmurph]

@RichardTXD I have an iOS/Android app (CliqTags Spotter) for detecting beacons and geofences, and it has a hidden page/pane that shows all beacon activity in a stable list (once a beacon is found it's stays in its place, unless it goes away for a longer time, so it's easy to see what happens with any specific beacon), and it shows many more pings than my Python app, so there's a difference. I also tried with BLE Scanner and got similar results. In a setting with only one or two beacons near by the app detects almost all pings. I was not aware of the use case. I use beacons mostly as pure proximity markers, not for absolute location, even though that's no doubt an interesting field that I'd like to get into, and I understand the relevance of a higher frequency for that use. Regarding accuracy: Try holding a beacon between you and the detecting device. Water and metal are beacon killers. But the main issue here is of course that it stops.

[stef]

@papasmurph
I start a simple test on Lopy and an generic board : Scan for 60 secondes a Kontact Beacon ( only one - 350ms message freq ) , the beacon is put on the Lopy .
Then i just count how many time this beacon appear for 60s

TOTAL CAST : 60000/350 =170
TOTAL for python test on Lopy ( 4x - reset after each ) :100-103-130-128
TOTAL for C test (4x) on ESP32_core_Board_V2 ( esp32-idf ) :99-110-120-115

so , in the two cases , we have min. 30% packet lost.

Someone can test with other beacons or hardware ?

Stéphane A.