RTC - an occasional problem

Hello,

we've encountered a problem with the A9X module and its RTC that occurs on some modules.

It's possible that it happens on all modules, but due to the random nature of the problem, I cannot say it for sure.

Randomly (at least we couldn't specify under which condition it happens) RTC PFC8563 is not found after linux boot is finished.

Sometimes it occures after 200-500 ms reset, but other times it happens even after the power is off for like 10 seconds or more.

When linux starts, the driver fails with a reading error from the RTC - is is the first time driver reads from this circuit.

Here is a selection from the system event log:

[ 0.655499] rtc-pcf8563 2-0051: chip found, driver version 0.4.3

[ 0.763043] rtc-pcf8563 2-0051: pcf8563_read_block_data: read error

[ 0.763133] rtc-pcf8563 2-0051: pcf8563_probe: read error

[ 0.763195] rtc-pcf8563: probe of 2-0051 failed with error -5

[ 1.946588] hctosys: unable to open rtc device (rtc0)

The observed supply voltage of the module and the PONRES signal correspond to the desired behavior, we did not find any incorrect values on them.

While searching for the cause, we "sacrificed" one module and soldered the probes directly to the RTC circuit.

When this problem occurs, then communication via I2C looks like this:

The address byte 0x51 is fine, but the register address byte (0x01) has its 8th bit "truncated".

It looks as if the I2C slave (RTC) pulled the data to 0 in an attempt to ACK one clock pulse earlier.

Thanks for some the advice.

Kamil

Hello,

here is CPU/Board info of one of the modules (MAC 00:05:51:1A:A4:BC):

CPU: Freescale i.MX6SX rev1.4, 996 MHz (running at 792 MHz)

CPU: Extended Commercial temperature grade (-20C to 105C)

Reset: POR

Board: PicoCOMA9X Rev 1.20 (LAN, 1x DRAM)

I2C: ready

DRAM: 512 MiB

NAND: 256 MiB

U-Boot and linux are generated from buildroot B2019.11, but there are many custom changes in U-boot, kernel and device tree. However, none of these changes were in RTC and I2C driver.

MAC of one of modules in which the problem occured is 00:05:51:1A:A4:BC, but I saw serveral other modules with the same behavior.

I don't have them here now, but I have MAC addresses of two of them so that you can find their board revisions:

00:05:51:1A:A4:C1

00:05:51:23:4F:55

U-boot and linux were the same in all modules.

Thank you,

Kamil

Hello again.

In last months I received two more PicoCOMA9X modules with this problematic behaviour and here are my observations and findings.

It seems to be the same problem as before, but it manifests itself a little differently - immediately after startup (already in U-Boot) the signal of the internal I2C3 bus SDA is held in 0.

The result is that neither the RTC nor the audio in linux work (audio - via the sgtl5000 driver - is connected to the same I2C bus).

The situation appears after resets typically performed by mechanical disconnecting and connecting the power supply - hence there appears some kind of trembling or oscillations on the supply voltage. I tested whether the behavior changes if the battery for the RTC is connected or not - but there was not difference.

I also checked the /RESET input signal on the oscilloscope and it seems to be correct.

I managed to find a workaround for this situation - in U-Boot it is necessary to generate artificial pulses on I2C3 SCL (in U-Boot this is just GPIO) and by this the bus can be "unblocked" and both RTC and audio are then working in Linux.

Apparently when disconnecting and connecting the power supply, some participant on the I2C3 bus does not perform a correct reset and remains somewhere "in the middle" of the transmission. It gets out of this with help of artificially generated clock pulses.

If this only happenes when the RTC battery is connected, it could be the caused by RTC circuit, which could interpret some disturbing pulses (generated during the power off/on) as an I2C communication. However, the problematic behavior could be seen even without the RTC battery connected.

Currently - with the workaround in U-Boot - the I2C3 bus does not stop working in Linux anymore (at least on the two PicoCOMA9 modules mentioned above).

However, the workaround is done more or less blindly - it tries to generate enough SCL pulses to bring SDA back to 1.

It is questionable whether this will work under all circumstances. Therefore I would like you to check the PicoCOMA9X module behaviour - wheather reset of I2C3 participants is performed correctly in such situations.

Thank you,

Kamil