I2C unstable

[Steven Boonstoppel]

Hi guys,
I am working with 60 LoPy4 modules (+ Expansion Board v3.1).
53 of them work flawlessly, but 7 modules have problems with I2C.
(They are all on identical hardware and software (1.20.2.r6))

Connected are an SSD1306 (0x3C == 60), BME680 (0x77 == 119), VEML6070 (0x38 and 0x39 == 56 and 57, and TSL2591 (0x29 == 41) which are initialized in that exact order.

Initiation of I2C which always works:

>>> i2c_bus = machine.I2C(0)
>>> print(i2c_bus.scan())
[41, 56, 57, 60, 119]

However, when running the software, the I2C bus fails after a very short while. Usually on the SSD1306 immediately, but when adding the i2c_bus.scan() as above, that one succeeds and it fails on the BME680. That same party trick does not work once again, however. I confirmed through swapping that the BME680 works on all nodes (and it also returns its address nicely anyway).

Furthermore I tried to no avail:

• Swapping I2C cables from LoPy4 to breadboard
• Different frequencies (10k, 20k, 50k, 200k)
• Different I2C pins (e.g. P10/P11 instead of the usual P9/P10)

Any suggestions?

[robert-hh]

@Steven-Boonstoppel Did you check with an oscilloscope that the levels & timing ate fine, or at least with a logic analyzer for differences in the timing?

[Steven Boonstoppel]

@robert-hh no, unfortunately I do not have access to either of those. I would have to order them and need to learn to use it.. which would take days if not weeks. Any other ideas?

[robert-hh]

@Steven-Boonstoppel A simple logic analyzer like this one: https://www.amazon.nl/AZDelivery-Logische-Analyser-compatibel-inclusief/dp/B01MUFRHQ2/ref=sr_1_5?__mk_nl_NL=ÅMÅŽÕÑ&crid=2HFH23S0ONIP7&keywords=logic+analyzer&qid=1661762680&sprefix=logic+analyzer%2Caps%2C85&sr=8-5
is affordable and easy to use. I consider it as extremely helpful when dealing with external devices.
I asked because it could be some kind of "analog" interface problem. Can you test at a failing LoPy to connect the external devices one-by-one? Maybe it is a kind of interference or to heavy bus load, causing the signal levels be at the margin.

[Steven Boonstoppel]

@robert-hh I will order that (from https://www.tinytronics.nl/shop/nl/gereedschap-en-montage/meten/oscilloscopen-en-logic-analyzers/logic-analyzer-8-kanaals-usb).

I do suspect that it is an analog issue, really. I tried on two of the defective modules to disconnect sensors and alter the software to only speak to a selective number of sensors, and I cannot find patterns but sometimes it works, sometimes it doesn't. Sometimes it even works fully for a longer time but it might break next time.

[robert-hh]

@Steven-Boonstoppel A analog issue would not be directly visible with a logic analyzer. With that, you can change with the logic levels, and compare the timing of 'good' vs 'bad' devices, especially the pulse widths.

[Brian Wyld]

Do you have the pullup resistors (often 10k) somewhere on the I2C SDL/SCK lines? Or maybe too many pullups if you have multiple modules and they all decided to integrate the pullups!!

I found this discussion about the right pullup values and where to place them useful: https://electronics.stackexchange.com/questions/65538/i2c-pull-up-resistors-calculation-where-to-place-them

[Steven Boonstoppel]

@Brian-Wyld thanks for your response!
I did not have them installed, but I just tried both 4.7k and 10k pull-up on SCL/SDA but it did not solve my problems. I2C bus still fails after successful SSD1306 initialization.. :( (and i2c.scan() still lists all addresses present)

[Brian Wyld]

@Steven-Boonstoppel
Did you try swapping the SSD1306 between a 'good' and 'bad' unit? it only takes 1 misbehving I2C module to ruin the party for everyone...

[Steven Boonstoppel]

@Brian-Wyld all modules have been swapped. Disconnecting and removing the SSD1306 from software makes the LoPy stumble on the BME680 still. Disconnecting and commenting out the BME680 makes the LoPy trip on the VEML6070.. all the while happily showing their addresses each time, proving that the cables and connections are (somewhat) working and I2C is (somewhat) correctly initialized.

[robert-hh]

@Steven-Boonstoppel scan() is not overly demanding for the communication, since only the address byte is sent and the ACK/NAK state for this byte is used to tell, whether a device is present. I have seen it that scan() worked, but data transfer failed.
But in your test: did you only remove the devices by software? That's not sufficient, if it is a hardware problem. You have also to disconnect the devices from the I2C bus.

[Steven Boonstoppel]

@robert-hh I have done a lot of trial and error the past two hours. There's now a very interesting situation on one of the nodes:
a) All works fine when scanning for the addresses first.
b) All fails when performing no address scan.

That's literally it... how...

[robert-hh]

@Steven-Boonstoppel That's strange. scan() is stateless. The only difference I see in the code is by the I2C ID, being <2 or >= 2. Which I2C device do you use?

[Steven Boonstoppel]

@robert-hh I am sorry - I don't understand what you mean by I2C ID being <2 or >=2.
I am using LoPy4's default I2C(0) as master at 100 kHz, using pins 9 and 10.

[jcaron]

@Steven-Boonstoppel Is there any other code running on the LoPy which could use I2C and change the I2C device configuration (pins, etc.)?

For instance if you have code to handle stuff for the PySense/PyTrack/PyScan (to detect their presence for instance) it would use I2C device 0 with pins P22/P21 rather than the default 9 and 10.

[robert-hh]

@Steven-Boonstoppel So the ID is 0.

[Steven Boonstoppel]

@jcaron nope, nothing. Full code is available at https://github.com/StevenCellist/MJLO/tree/main/software.

[jcaron]

@Steven-Boonstoppel Is Pybytes disabled? How are you powering the board? Maybe you have a current spike related to all the sensors starting up at the same time? Monitoring input current and voltage during the awake cycle may be interesting.

[Steven Boonstoppel]

@jcaron Pybytes is disabled, powering over USB (tried battery and both as well). Current spike happens when the voltage regulator to SDS011 and MQ135 is enabled which happens after all I2C sensors have been used. Until then the regulator is in shutdown mode.

[jcaron]

@Steven-Boonstoppel what’s the max total current used? What’s the USB power source? Is the USB cable adequate? What kind of battery?

Just trying to eliminate a possibility…