Repeatable hardware I2C EIO errors with multiple boards

[snizbatch]

Hello all, this is my first post here, but I am not new to the PICO or Python. I am a freelance/hobby Python developer, a commercial pilot and an electronics hobbyist. I've been attempting to tie all this together by designing a model aircraft autopilot with a Raspberry Pi PICO and MicroPython (I know I have unrealistic expectations but it's about pushing boundaries). I've been very successful with individual sensors, servos, GPS, LORA, and TOF sensors but every time I attempt to tie the I2C boards together I run into unresolvable but erratic EIO errors. After the initial EIO error, re-plugging in an I2C device always results in EIO errors on every device connected. Devices are ALWAYS detectable on the bus, but when using writeto_mem they throw an EIO error. It seems as though letting everything sit, then connecting a single device the next day and running, for example, the BerryIMU library, there are no EIO errors. But then, if I change the pin configuration at all, EIO errors start and don't resolve even after power cycles. Here are my questions:

1. Is it possible to configure machine.Pin incorrectly prior to instantiating an I2C class or does I2C reconfigure the Pins as required for operation?

2. At one point during testing I reversed the SCL and SDA lines on the 5v side of a Pololu logic level shifter. This understandably caused EIO errors but I don't imagine that this may have caused any hardware damage. I'm not, however, ruling it out. On my custom circuit board (see the circuit below) I have 2 Pololu voltage regulators. One for 3.3v and one for 5v. I tied the 3v3 rail on the PICO to the 3v3 voltage regulator output. I'm wondering if this too may have damaged any components. HOWEVER, I purchased a new PICO and individually connected an I2C device that was never in the original circuit (A simple OLED screen) and it too experiences occasional EIO errors. Hooking it up to a Raspberry Pi Zero results in no errors. I'm convinced I'm doing something wrong with my MicroPython code, but I can't be 100% certain. Here is code that ALWAYS results in EIO errors...

from machine import Pin, I2C
import sys
from time import sleep

class I2C_Scanner:
    def __init__(self):
        print(sys.implementation[2])
        if "Raspberry Pi Pico" in sys.implementation[2]:
            self.data_pins = {0:[0,4,8,12,16,20], 1:[2,6,10,14,18,26]}
            self.clock_pins = {0:[1,5,9,13,17,21], 1:[3,7,11,15,19,27]}
        elif "Adafruit QT Py RP2040 with RP2040" in sys.implementation[2]:
            self.data_pins = {0:[28,24,20,12,4], 1:[26,6,22]}
            self.clock_pins = {0:[29,25,5,21], 1:[11,27,3,23]}
        else:
            raise Exception("Device is not a PICO")
        self.bad_pins = []
        self.devices = []
        self.__test_pins()
        self.length = 0
        self.__reprs = []
        self.__i2cbusses = []
        self.scan()
        
        
    def __repr__(self):
        return "\n".join(self.__reprs)

    def bus_id(self, index=None):
        if index:
            return self.__i2cbusses[index][0]
        elif len(self.__i2cbusses) > 0:
            return self.__i2cbusses[0][0]
        else:
            return None

    def sda(self, index=None):
        if index:
            return self.__i2cbusses[index][1]
        elif len(self.__i2cbusses) > 0:
            return self.__i2cbusses[0][1]
        else:
            return None

    def scl(self, index=None):
        if index:
            return self.__i2cbusses[index][2]
        elif len(self.__i2cbusses) > 0:
            return self.__i2cbusses[0][2]
        else:
            return None

    def __test_pins(self):
        self.bad_pins = []
        anybad = False        
        if sys.implementation[2] == "Adafruit QT Py RP2040 with RP2040":
            import neopixel
            Pin(11, Pin.OUT).on()
            np = neopixel.NeoPixel(Pin(12), 1)
            for i in range(3):
                pixelist = [0,0,0]
                pixelist[i] = 255
                np[0] = pixelist
                np.write()
                sleep(0.5)
            Pin(11, Pin.OUT).off()
            for pnum in [3,4,5,6,11,12,20,22,23,24,25,26,27,28,29]:
                Pin(pnum, Pin.OUT).on()
                Pin(pnum, Pin.OUT).off()
                if not Pin(pnum, Pin.IN, Pin.PULL_UP).value() or Pin(pnum, Pin.IN, Pin.PULL_DOWN).value():
                    self.bad_pins.append(pnum)
                    anybad = True
                    return False
            
        elif "Raspberry Pi Pico with RP2040" in sys.implementation[2]:
            for pnum in range(30):
                if pnum!= 24:  # This is the power pin (USB/VBUS = 1  VSYS = 0)
                    Pin(pnum, Pin.OUT).on()
                    Pin(pnum, Pin.OUT).off()
                    if not Pin(pnum, Pin.IN, Pin.PULL_UP).value() or Pin(pnum, Pin.IN, Pin.PULL_DOWN).value():
                        self.bad_pins.append(pnum)
                        anybad = True
                        return False
                
        if not anybad:
            return True
    
    def scan(self):
        scan = []
        self.__i2cbusses = []
        
        print("Scanning all i2c pin combinations...")
        for busnum in [0,1]:
            for try_sda in self.data_pins[busnum]:
                for try_scl in self.clock_pins[busnum]:
                    try:
                        i2c = I2C(busnum, sda=try_sda, scl=try_scl)
                        #sleep(0.1)
                        scan = i2c.scan()
                        if scan:
                            self.devices += scan
                            hexlist = ",".join([hex(s) for s in scan])
                            self.__reprs.append(f"I2C{busnum} - {try_sda}(SDA) {try_scl}(SCL): Devices-> {hexlist}")
                            self.__i2cbusses.append((busnum,try_sda,try_scl))
                            for bpin in [try_sda, try_scl]:
                                if bpin in self.bad_pins:
                                    self.bad_pins.remove(bpin)
                    except Exception as e:
                        pass
        self.length = len(self.__i2cbusses)
        return self.length
        
    def writeReg(self, device_addr, reg_address, data):
        res = bytearray(2)
        i2c.writeto_mem(device_addr, reg_address, bytes([data]))

    def readReg(self, device_addr, reg_address):
        try:
            res = i2c.readfrom_mem(device_addr, reg_address, 1)
        except:
            res = None
        return res

if __name__ == "__main__":
    i2cscan = I2C_Scanner()
    print(i2cscan)
    print(f"Found: {i2cscan.length} connection{"" if i2cscan.length == 1 else "s"}.")
    print(f"{"Pin " if i2cscan.bad_pins else "No pins"}{", ".join([str(bp) for bp in i2cscan.bad_pins])} tested bad.")
    
    if i2cscan.length == 1:
        i2c = I2C(id=i2cscan.bus_id(), sda=i2cscan.sda(), scl=i2cscan.scl())
        if 0x6A in i2cscan.devices:      # BerryIMU
            i2cscan.writeReg(0x6A, 0x10, 0b10011111)
            writeReg(0x6A, 0x17, 0b11001000)
        elif 0x29 in i2cscan.devices:    # VL53L1X
            print(i2c.readfrom_mem(0x29, 0x010F, 2, addrsize=16))

Here's my circuit:

Any help steering me in the right direction would be awesome!
-Jim Hunt

[ricksorensen]

Pull-up on SDA/SCL lines?

[snizbatch]

Rick,
When I first created my circuit board and connected everything I had 2 5K's on the 5v side of the level shifter SCL and SDA lines. I needed 5v for a serial GPS and the 5v I2C from an ATTiny85 that calculates RPM and sends it to the PICO over I2C. When things went sideways, I removed everything from the board. The 3v3 SDA and SCL lines on the BerryIMU have built-in 10K pull ups on them. I didn't want to go overboard with pull-ups when they are already present on both the BerryIMU and my Pololu VL53L1X laser ranging TOF carrier board. Placing too many would effectively reduce the resistance slowing down or completely disabling the SDA and SCL lines from what I understand.

Jim

[GitHubsSilverBullet]

A couple of observations.

machine.I2C() is initialised with 400kbitb/s (50kbyte/s). Maybe try a slower 100k speed and check whether that's sufficient. (slower is more forgiving, especially in a dirty/noisy environment)

machine.I2C() requires (according to docs) a Pin object, not a GPIO number.
Try to setup Pin() before handing them over to machine.I2C().
In addition, using mem32[] and PADS_BASE, one can vary individual slewrate and drive as well.

[robert-hh]

@snizbatch Lower pull-up resistors allow higher baud rates. The lowest acceptable value depends on the current that can be sunk into the driver, which is typically at least 10mA. This means, that at 3V3 the lowest value for the Pull-up resistor is about 330 Ohm So it's not critical. But having values in the range of 2k2 to 10k protects the Pico in case that you accidentally connected the pull-ups to 5V.
@GitHubsSilverBullet For machine.I2C (and machine.SPI) one can use both the GPIO number or a Pin name as string from pins.csv of the board or a Pin object.

[snizbatch]

@GitHubsSilverBullet thanks for pointing that out... My "I2C_Scanner" class was working for detecting devices present when iterating integer pin number pairs but it didn't register until you said something that I was actually sending try_sda and try_scl as integers. I suppose it's possible that the I2C.scan() works with integers while the writeto_mem function needs Pin objects. I'm not sure. I do know that when I run my custom pin scan, I always encounter EIO errors afterwards. @robert-hh very much appreciate the info on the pullups. I've always wondered the effect of combining multiple breakout boards with built in pull-ups and why things always start to fail when I load up on them. I'm not sure exactly how I'm going to tackle managing the conglomeration of boards without desoldering resistors though. Even my logic level shifter has 10k's on both the high side and low side. https://www.pololu.com/product/2595. It seems I'm trippling up on 10kΩ resistors resulting in 3k3... On the low side of your recommendation but still within the range. Theoretically, as long as I can supply enough current, these values should allow even higher speeds. I'm going to do some more tinkering over the next week and see what I can come up with. Thank you all so much.

[robert-hh]

I suppose it's possible that the I2C.scan() works with integers while the writeto_mem function needs Pin objects.

The pins for sda and scl are specified in the constructor. And with the actual firmware you can use integers, names as strings and Pin objects. So sda=20, sda=Pin(20) and sda="GP20" will all result in the same pin to be used ind all I2C methods. Note that there are cases where i2c.scan() succeeds but longer transfer fail. You could try to use SoftI2C instead of Hard I2C. The RP2 port uses Soft I2C for single byte tran sfers like the one being used of i2c.scan(). There may be a problem with clock stretching.
SoftI2C does not require a device number and can be used with any GPIO Pin pair.
Edit: Are you sure that there are not address collisions with your sensors? Do you have a means to monitor the bus signals, e.g. with a logic analyzer? For slow speeds you can use a RP2040 as logic analyzer with this firmware: https://github.com/dotcypress/ula and Pulseview and user interface.

[snizbatch]

@robert-hh ok, so no issues using integers for pins... As far as the SoftI2C goes, I've been on the fence about trying this since I feel as though it is a workaround rather than a true solution. I feel this way because anything you can do through hardware should consume fewer resources. But if it's the only solution than I suppose I'll give it a shot. The clock stretching issue you speak of I encountered when designing my 5v ATTiny85 RPM calculator (It just uses an interrupt to count pulse duration and makes itself available as a very simple Arduino I2C slave device). Multiple sources reported clock stretching issues when interfacing much of the same with a Raspberry Pi Zero. I was eventually able to get mine working but I believe I simply watched for errors and threw away the garbage results. Not an ideal solution. It has been a few years since I did that project and never truly understood what clock stretching was. I'll go do my homework on it before I pick your brain for more info. Thank you.

[snizbatch]

Well, this isn't really a solution to my EIO errors... They are gone due to not messing with the GPIO pins prior to initializing I2C, separating devices and only using 3v3 devices. The 5v ATTiny85 I2C device I had on the level shifter seemed to be causing a lot of issues. I found the following tutorial that removed the need for a separate microcontroller to count RPM pulses: Accurate frequency & timing measurement on the Pi Pico RP2040 using Python It only takes 1 PWM pin and doesn't disrupt the main loop excessively to count RPM pulses. The Hall effect sensor I'm using DOES use 5v and a level shifter to convert the pulses to 3v3 for GPIO but it works flawlessly.

In addition, I was level shifting a 5v GPS which should have had no effect on the completely separate I2C lines but who knows. I discovered that the GPS is able to be powered via 3v3, and I obtained an airspeed pressure sensor that also runs on 3v3. This removes the level shifter from the I2C line completely. I now only have 1 unsolved issue. GPS UART on the PICO is coming through incredibly noisy. Below is an example (see line 6 where the $GNRMC interrupts the prior line without a newline or carriage return. I hooked it up in parallel to a FTDI serial device at 3v3 and an oscilloscope simultaneously and the FTDI device receives clean uninterrupted lines as the PICO prints out mostly good lines, but every 6 or 8 lines it is missing a chunk of data, to include the CR and LF. I tried bumping the BAUD rate up to 57600 and messing with the character timeout and the UART timeout values to test, but still no luck. It also often receives a "More than 100 frame errors, UART RX was disabled" message from the GPS but only on the PICO. I checked the settings on parity, stop bits and flow control on the PC and matched it on the PICO but received the same results. I also verified that the RX line is indeed high (3v3) on the oscilloscope between reads.

$GPGSV,1,1,00*79
$GLGSV,1,1,00*65
$GNGLL,,,,,172251.00,V,N*$GNRMC,172251.50,V,,,,,,,310724,,,N*67
$GNVTG,,,,,,,,,N*2E
$GNGGA,172251.50,,,,,0,00,99.99,,,,,,*7F
$GNGSA,A,1,,,,,,,,,,,,,99.99,99.99$GNRMC,172252.00,V,,,,,,,310724,,,N*61
$GNVTG,,,,,,,,,N*2E
$GNGGA,172252.00,,,,,0,00,99.99,,,,,,*79
$GNGSA,A,1,,,,,,,,,,,,,99.99,99.99,99.99*2E
$GNGSA,A,1,,,,,,,,,,,,,99.99,99.99,99.99*2E
$GPGSV,1,1,00*79
$GLGSV,1,1,00*65
$GNGLL,,,,,172252.00,V,N*$GNRMC,172252.50,V,,,,,,,310724,,,N*64
$GNVTG,,,,,,,,,N*2E
$GNGGA,172252.50,,,,,0,00,99.99,,,,,,*7C
$GNGSA,A,1,,,,,,,,,,,,,99.99,99.99$GNRMC,172253.00,V,,,,,,,310724,,,N*60
$GNVTG,,,,,,,,,N*2E
$GNGGA,172253.00,,,,,0,00,99.99,,,,,,*78
GLGSV,3,2,12,73,11,034,26,74,34,073,32,75,29,138,19,81,02,338,*69
GLGSV,3,3,12,86,07,180,,87,41,219,23,88,38,299,,,,,43*59
GNGLL,30xx.03895,N,081xx.63516,W,205518.00,A,A*69
GNZDA,205518.00,30,07,2024,00,00*73
GNTXT,01,01,01,More than 100 frame errors, UART RX was disabled*70
GNRMC,205519.00,A,30xx.03875,N,081xx.63521,W,0.070,,300724,,,A*7E

Here's the waveform:

from machine import UART, Pin
from time import ticks_ms, ticks_diff

class Easy_comms:
 
    uart_id = 0
    baud_rate = 9600
    
    def __init__(self, uart_id:int, baud_rate:int=None, txpin=0, rxpin=1, line_end=None):
        if line_end == None:
            self.line_end = chr(13)
        else:
            self.line_end = line_end
            
        self.uart_id = uart_id
        if baud_rate: self.baud_rate = baud_rate

        self.uart = UART(self.uart_id,self.baud_rate, tx=Pin(txpin), rx=Pin(rxpin))
        # invert = UART.INV_TX | UART.INV_RX

        # Initialise the UART serial port
        self.uart.init(timeout=500)
        self.uart.flush()
        self.message = ""
    
    def send(self, message:str):
        print(f"Sending: {message} to UART{str(self.uart_id)}...", end="")
        if type(message) is list:
            for item in message:
                self.uart.write(item + self.line_end + "\n")
        elif type(message) is str:
            self.uart.write(message + self.line_end + "\n")
        
    def wait_for_response(self, timeout=3000):
        timeout_tick = ticks_ms()
        try_read = self.read()
        while (not try_read) and (ticks_diff(ticks_ms(), timeout_tick) < timeout):
            try_read = self.read()
        return try_read
    
    def write(self, text):
        self.uart.write(text)
    
    def read(self)->str:
        if self.uart.any():
            readtext = self.uart.readline()
            if readtext:
                try:
                    return readtext.decode().strip()  # 'utf-8'
                except:
                    pass
        return None
        
if __name__ == "__main__":
    from time import sleep
    # Easy_comms(uart_id=1, baud_rate=9600, txpin=8, rxpin=9)
    e = Easy_comms(0, 57600, 12, 13)
    sleep(0.1)
    
    while True:        
        resp = e.read()
        if resp:
            print(resp)
        sleep(0.1)

[snizbatch]

Norbert,    Thank you very much for your input! When I get in from my trip I'll give your suggestions a try and see if things improve! JimOn Jul 13, 2024 23:15, Norbert ***@***.***> wrote: A couple of observations. machine.I2C() is initialised with 400kbitb/s (50kbyte/s). Maybe try a slower 100k speed and check whether that's sufficient. (slower is more forgiving, especially in a dirty/noisy environment) machine.I2C() requires (according to docs) a Pin object, not a GPIO number. Try to setup Pin() before handing them over to machine.I2C(). In addition, using mem32[] and PADS_BASE, one can vary individual slewrate and drive as well. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>