piControl: AIO addr 31: communication failed (req:20,ret:-5)

[in06khattab]

Hi,

I noticed the following errors in dmesg on a Revpi Connect 4.
Sometimes the red power light turns on the front of the device.
I noticed someone else has the same error: viewtopic.php?p=15160

Code: Select all

[  354.692296] piControl: AIO addr 30: communication failed (req:20,ret:-5)
[  364.253774] pi-bridge serial1-0: receive head error in io-req
[  364.253781] piControl: AIO addr 31: communication failed (req:20,ret:-5)
[  383.291964] pi-bridge serial1-0: receive head error in io-req
[  383.291972] piControl: AIO addr 31: communication failed (req:20,ret:-5)
[  383.294647] pi-bridge serial1-0: invalid checksum (expected: 0x01, got 0xc1
[  383.294653] piControl: AIO addr 30: communication failed (req:20,ret:-74)
[  383.305962] pi-bridge serial1-0: receive data error in io-req(len: 20)
[  383.305969] piControl: AIO addr 31: communication failed (req:20,ret:-5)
[  385.492314] pi-bridge serial1-0: receive data error in io-req(len: 20)
[  385.492323] piControl: AIO addr 31: communication failed (req:20,ret:-5)
[  386.128728] pi-bridge serial1-0: receive head error in io-req
[  386.128736] piControl: AIO addr 30: communication failed (req:20,ret:-5)
[  386.141751] pi-bridge serial1-0: failed to discard 9 bytes within timeout
[  386.141760] pi-bridge serial1-0: received packet truncated (9 bytes missing)
[  386.141765] piControl: AIO addr 31: communication failed (req:20,ret:-74)
[  418.492061] pi-bridge serial1-0: receive head error in io-req
[  418.492069] piControl: AIO addr 30: communication failed (req:20,ret:-5)
[  431.692200] pi-bridge serial1-0: receive head error in io-req
[  431.692207] piControl: AIO addr 30: communication failed (req:20,ret:-5)
[  469.182110] pi-bridge serial1-0: receive head error in io-req
[  469.182118] piControl: AIO addr 31: communication failed (req:20,ret:-5)
[  469.182831] pi-bridge serial1-0: unexpected response addr 0x1f
[  469.182835] piControl: AIO addr 30: communication failed (req:20,ret:-74)
[  469.194167] pi-bridge serial1-0: receive data error in io-req(len: 20)
[  469.194176] piControl: AIO addr 31: communication failed (req:20,ret:-5)

Code: Select all

pi@Pi1:~ $ piTest -d
Found 4 devices:

Address: 0 module type: 136 (0x88) RevPi Connect 4 V1.0
Module is present
     input offset: 178 length: 6
    output offset: 184 length: 7

Address: 31 module type: 103 (0x67) RevPi AIO V1.5
Module is present
     input offset: 89 length: 20
    output offset: 109 length: 4

Address: 30 module type: 103 (0x67) RevPi AIO V1.5
Module is present
     input offset: 0 length: 20
    output offset: 20 length: 4

Address: 64 module type: 24580 (0x6004) ModbusRTU Master Adapter V0.0
Module is present
     input offset: 191 length: 101
    output offset: 292 length: 73

Code: Select all

pi@Pi1:~ $ piTest -f
Virtual modules don't have firmware to update

Code: Select all

pi@Pi1:~ $ uname -a
Linux Pi1 6.1.46-rt13-v7l #1 SMP PREEMPT_RT Mon, 5 Feb 2024 15:02:39 +0100 armv7l GNU/Linux

Code: Select all

 cat /boot/config.txt
# For more options and information see
# http://rpf.io/configtxt
# Some settings may impact device functionality. See link above for details

# uncomment if you get no picture on HDMI for a default "safe" mode
#hdmi_safe=1

# uncomment this if your display has a black border of unused pixels visible
# and your display can output without overscan
disable_overscan=1

# uncomment the following to adjust overscan. Use positive numbers if console
# goes off screen, and negative if there is too much border
#overscan_left=16
#overscan_right=16
#overscan_top=16
#overscan_bottom=16

# uncomment to force a console size. By default it will be display's size minus
# overscan.
#framebuffer_width=1280
#framebuffer_height=720

# uncomment if hdmi display is not detected and composite is being output
hdmi_force_hotplug=1

# uncomment to force a specific HDMI mode (this will force VGA)
hdmi_group=1
hdmi_mode=31

# uncomment to force a HDMI mode rather than DVI. This can make audio work in
# DMT (computer monitor) modes
hdmi_drive=2

# uncomment to increase signal to HDMI, if you have interference, blanking, or
# no display
#config_hdmi_boost=4

# uncomment for composite PAL
#sdtv_mode=2

#uncomment to overclock the arm. 700 MHz is the default.
#arm_freq=800

# Uncomment some or all of these to enable the optional hardware interfaces
dtparam=i2c_arm=on
#dtparam=i2s=on
#dtparam=spi=on

# Uncomment this to enable infrared communication.
#dtoverlay=gpio-ir,gpio_pin=17
#dtoverlay=gpio-ir-tx,gpio_pin=18

# Additional overlays and parameters are documented /boot/overlays/README

# Enable audio (loads snd_bcm2835)
dtparam=audio=on

# Enable DRM VC4 V3D driver
#dtoverlay=vc4-kms-v3d
#max_framebuffers=2

[pi4]
[cm4s]
dtoverlay=dwc2,dr_mode=host

[all]
enable_uart=1

I was not able to perform an SOS report as the tool is not installed.

Code: Select all

pi@Pi1:~ $ sudo apt update
Get:1 http://raspbian.raspberrypi.org/raspbian bullseye InRelease [15.0 kB]
Get:2 http://packages.revolutionpi.de bullseye InRelease [14.2 kB]
Hit:3 http://packages.revolutionpi.de bullseye-backports InRelease
Hit:4 http://repository.teamviewer.com/deb stable InRelease
Get:5 http://raspbian.raspberrypi.org/raspbian bullseye/main armhf Packages [13.2 MB]
Get:6 https://archive.raspberrypi.org/debian bullseye InRelease [23.6 kB]
Get:7 http://packages.revolutionpi.de bullseye/main armhf Packages [33.9 kB]
Get:8 https://archive.raspberrypi.org/debian bullseye/main armhf Packages [313 kB]
Fetched 13.6 MB in 11s (1,223 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
22 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: Target Packages (tvrevpi/binary-armhf/Packages) is configured multiple times in /etc/apt/sources.list.d/teamviewer-iot-agent.list:3 and /etc/apt/sources.list.d/teamviewer-revpi.list:2
W: Target Packages (tvrevpi/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/teamviewer-iot-agent.list:3 and /etc/apt/sources.list.d/teamviewer-revpi.list:2
W: Target Translations (tvrevpi/i18n/Translation-en_GB) is configured multiple times in /etc/apt/sources.list.d/teamviewer-iot-agent.list:3 and /etc/apt/sources.list.d/teamviewer-revpi.list:2
W: Target Translations (tvrevpi/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/teamviewer-iot-agent.list:3 and /etc/apt/sources.list.d/teamviewer-revpi.list:2
W: Target Packages (tvrevpi/binary-armhf/Packages) is configured multiple times in /etc/apt/sources.list.d/teamviewer-iot-agent.list:3 and /etc/apt/sources.list.d/teamviewer-revpi.list:2
W: Target Packages (tvrevpi/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/teamviewer-iot-agent.list:3 and /etc/apt/sources.list.d/teamviewer-revpi.list:2
W: Target Translations (tvrevpi/i18n/Translation-en_GB) is configured multiple times in /etc/apt/sources.list.d/teamviewer-iot-agent.list:3 and /etc/apt/sources.list.d/teamviewer-revpi.list:2
W: Target Translations (tvrevpi/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/teamviewer-iot-agent.list:3 and /etc/apt/sources.list.d/teamviewer-revpi.list:2
pi@Pi1:~ $ sudo apt install revpi-tools
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
revpi-tools is already the newest version (3.4.0-1+revpi11+1).
0 upgraded, 0 newly installed, 0 to remove and 22 not upgraded.
pi@Pi1:~ $ revpi-sos
-bash: revpi-sos: command not found

Photo of setup:

I have attached the Dmesg log.

dmesg.zip

(17.37 KiB) Downloaded 9 times

[in06khattab]

After a few hours, the problem has gotten worse, I get the following errors in the log.
The red power light is on.
The data still appears to be correct when reading the inputs.

Code: Select all

Apr 18 16:14:35 Pi1 kernel: piControl: too many communication errors -> set inputs to default 0 255 0 0   0 0 0 0
Apr 18 16:14:35 Pi1 kernel: piControl: too many communication errors -> set inputs to default 0 255 0 0   0 0 0 0
Apr 18 16:14:35 Pi1 kernel: piControl: too many communication errors -> set inputs to default 0 255 0 0   0 0 0 0
Apr 18 16:14:35 Pi1 kernel: piControl: too many communication errors -> set inputs to default 0 255 0 0   0 0 0 0
Apr 18 16:14:35 Pi1 kernel: piControl: too many communication errors -> set inputs to default 0 255 0 0   0 0 0 0
Apr 18 16:14:35 Pi1 kernel: piControl: too many communication errors -> set inputs to default 0 255 0 0   0 0 0 0
Apr 18 16:14:35 Pi1 kernel: piControl: too many communication errors -> set inputs to default 0 255 0 0   0 0 0 0
Apr 18 16:14:35 Pi1 kernel: piControl: too many communication errors -> set inputs to default 0 255 0 0   0 0 0 0
Apr 18 16:14:35 Pi1 kernel: piControl: too many communication errors -> set inputs to default 0 255 0 0   0 0 0 0
Apr 18 16:14:35 Pi1 kernel: piControl: too many communication errors -> set inputs to default 0 255 0 0   0 0 0 0
Apr 18 16:14:35 Pi1 kernel: piControl: too many communication errors -> set inputs to default 0 255 0 0   0 0 0 0
Apr 18 16:14:35 Pi1 kernel: piControl: too many communication errors -> set inputs to default 0 255 0 0   0 0 0 0
Apr 18 16:14:35 Pi1 kernel: piControl: too many communication errors -> set inputs to default 0 255 0 0   0 0 0 0
Apr 18 16:14:35 Pi1 kernel: piControl: too many communication errors -> set inputs to default 0 255 0 0   0 0 0 0
Apr 18 16:14:35 Pi1 kernel: piControl: too many communication errors -> set inputs to default 0 255 0 0   0 0 0 0
Apr 18 16:14:35 Pi1 kernel: piControl: too many communication errors -> set inputs to default 0 255 0 0   0 0 0 0
Apr 18 16:14:35 Pi1 kernel: piControl: too many communication errors -> set inputs to default 0 255 0 0   0 0 0 0
Apr 18 16:14:35 Pi1 kernel: piControl: too many communication errors -> set inputs to default 0 255 0 0   0 0 0 0
Apr 18 16:14:35 Pi1 kernel: piControl: too many communication errors -> set inputs to default 0 255 0 0   0 0 0 0
Apr 18 16:14:35 Pi1 kernel: piControl: too many communication errors -> set inputs to default 0 255 0 0   0 0 0 0

Updated dmesg file:

dmesg (2).zip

(8.36 KiB) Downloaded 10 times

[in06khattab]

I wonder if firmware 1.4 is more reliable, can I try to revert the AIO firmware from 1.5 to 1.4?

[nicolaiB]

Hi,

thanks for the detailed report. The revpi-sos tool is part of the revpi-sos-report package (sudo apt install -y revpi-sos-report). Which image are you using (cat /etc/revpi/image-release)? Is there a special reason why you run a 32-bit system? The connect 4 is usually shipped with 64-bit image by default. Does the error persists if you move one AIO to the right side?

Regarding 1.4 more stable than 1.5: Shouldn't make any difference, as 1.5 only fixed some overvoltage indication. See https://gitlab.com/revolutionpi/revpi-f ... 9cd022cec5

Nicolai

[in06khattab]

Hi Nicolai,

Attached is the SOS report.

Unfortunately, it is an upgraded image which was upgraded to Buster and then to Bullsesye which is required for the Connect 4. The same symptoms are with this forum poster who has also upgraded. viewtopic.php?p=15160

Code: Select all

cat /etc/revpi/image-release
2019-03-14-revpi-stretch.img

I received the same errors after moving an AIO to the right side.

The person in that thread had success when reverting to the 5.10 kernel. But that would be quite limiting and would prevent any future upgrades.

Let me know what more information I can provide to help troubleshoot this issue.

[in06khattab]

SOS report attached.

sosreport-Pi1-2024-04-23-kwzbbkh.tar

(810 KiB) Downloaded 7 times

[nicolaiB]

Thanks for the sos report. After a quick look I can see that your system runs on very old bootloader firmware version:

Dec 1 2021 15:01:54
Copyright (c) 2012 Broadcom
version 71bd3109023a0c8575585ba87cbb374d2eeb038f (clean) (release) (start)

on a similar Connect 4 with Bullseye Image this should be

Mar 17 2023 10:50:39
Copyright (c) 2012 Broadcom
version 82f3750a65fadae9a38077e3c2e217ad158c8d54 (clean) (release) (start)

In order to be sure that this is not something which relates to the upgrade vom Stretch to Buster to Bulllseye (which is officially not supported), it would be great if you can make a backup of the current system and try a fresh Bullseye Image (prefereably arm64 like we deliver the Connect 4 with).

Nicolai

[in06khattab]

I know it is due to the upgrade as I was not getting the error message before the upgrade to Bullseye.

Do you want me to update the bootloader?

Is there anything I can do to debug the piControl driver?
I am also reading the RS485 channels and the AIO and I saw the update to kernel 6.1 relates to the RS485.

Should this post be moved into software?

[nicolaiB]

Is hard to debug a system which has been upgraded with several major os version. The bootloader firmware is part of the package raspberrypi-bootloader (file /boot/start4.elf) and thus _should_ have been updated during your updated. My guess is that this (and maybe other?) packages are not updated.

You can check which packages from Buster are still installed:

dpkg -l | grep -E '^ii.*deb10'

From Stretch

dpkg -l | grep -E '^ii.*deb9'

Also make sure that all repos have been updated in /etc/apt/sources.list and /etc/apt/sources.list.d. Also double check if there are pending upgrades with dist-upgrade. As I mentioned before: The best way to find out if this relates to the current bullseye packages or the upgrade is to flash the last bullseye release (arm64) and test again. If you make a backup of your image it can be flashed again after testing. I assume you're familar with this procedure, as you already flashed and older image to the Connect 4.

Nicolai

[in06khattab]

No packages awaiting to be upgraded.

I was thinking about how to upgrade the raspberrypi-bootloader and I tried the following in two separate attempts:
sudo rpi-update
sudo rpi-update 22d9210842402780c9e6c74668b2d3eae94c5192 didn’t work

However, I would get errors such as:
This appears on startup

Code: Select all

failed to load kernel modules

Code: Select all

piTest -d Cannot retrieve device list: No such device , 

This appears constantly in journatctl

Code: Select all

piModbusMaster[1239]: write to process image failed: -19. 

So I reverted the image from backup

Code: Select all

pi@Pi1:~ $ dpkg -l | grep -E '^ii.*deb10'
ii  e2fslibs:armhf                    1.44.5-1+deb10u3                  armhf        transitional package
ii  libhogweed4:armhf                 3.4.1-1+deb10u1                   armhf        low level cryptographic library (public-key cryptos)
ii  libnettle6:armhf                  3.4.1-1+deb10u1                   armhf        low level cryptographic library (symmetric and one-way cryptos)
ii  multiarch-support                 2.28-10+rpt2+rpi1+deb10u2         armhf        Transitional package to ensure multiarch compatibility
pi@Pi1:~ $ dpkg -l | grep -E '^ii.*deb9'
ii  libdns-export162                  1:9.10.3.dfsg.P4-12.3+deb9u12     armhf        Exported DNS Shared Library
ii  libisc-export160                  1:9.10.3.dfsg.P4-12.3+deb9u12     armhf        Exported ISC Shared Library
ii  libprocps6:armhf                  2:3.3.12-3+deb9u1                 armhf        library for accessing process information from /proc
ii  libssl1.0.2:armhf                 1.0.2u-1~deb9u7                   armhf        Secure Sockets Layer toolkit - shared libraries

I did try a fresh bullseye image and I get no errors. I would like to get the upgraded image working.