EPICS ioc with RTEMS on Raspberry Pi
Miroslaw Dach <[email protected]> Wed, 22 Apr 2026 11:25:08 -0700
| Newsgroups | gmane.os.rtems.user |
|---|---|
| Message-ID | <CAPAFRJdGBARVvdrUDW25SjdYcih8cFH_S0qwWN+Nzd8yAwrpmw@mail.gmail.com> |
--===============4263124216103028876==
Content-Type: multipart/alternative; boundary="0000000000007a0706065010a754"
--0000000000007a0706065010a754
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Hi All,
I'm running an EPICS ioc server (EPICS 7.0.10) with RTEMS 6.2 on a
Raspberry Pi 3B+ with rtems-libbsd (6-freebsd-14) and
encountering a reproducible system hang after several minutes of operation.
Through systematic elimination testing I've narrowed the root cause to the
DWC OTG USB controller driver. I'd appreciate any recommendations on how
to address this. I can of course use the EPICS ioc server under linux on
RPi but just tried to have the RTEMS - Hard real time system which is much
more deterministic.
The boot time for the RPi with EPICS/RTEMS is around 7 sec which is
extremely fast!
Environment
-----------
- Board: Raspberry Pi 3B+ (BCM2837, boardrev a020d3)
- RTEMS: rtems-6.2 (ARM/ARMv4/raspberrypi2)
- Network stack: rtems-libbsd (RTEMS_BSD_CONFIG_BSP_CONFIG +
RTEMS_BSD_CONFIG_INIT)
- Ethernet: LAN7515 USB Ethernet (muge driver, via DWC OTG)
- Application: EPICS IOC (but hang occurs with minimal/empty IOC as well)
- Console: UART serial (/dev/ttyS0)
Symptom
-------
The system boots and runs normally, then the entire system freezes =E2=80=
=94
including the UART serial console (which is not USB-dependent). No crash
message, no stack dump =E2=80=94 a hard hang requiring power cycle.
The time-to-hang depends on the system tick rate:
- CONFIGURE_MICROSECONDS_PER_TICK=3D500 (2 kHz): hangs after ~2 minutes
- CONFIGURE_MICROSECONDS_PER_TICK=3D10000 (100 Hz): hangs after ~5-6 minute=
s
During the hang, the UART becomes completely unresponsive, suggesting a
kernel-level deadlock or interrupt handler issue rather than an
application-level problem.
Elimination testing performed
-----------------------------
I systematically disabled components to isolate the cause. All tests below
used CONFIGURE_MICROSECONDS_PER_TICK=3D10000 (100 Hz):
1. Disabled all EPICS database records (no record processing) -> still hang=
s
2. Disabled periodic NTP sync (no socket operations) -> still hangs
3. Disabled IP configuration (no ifconfig, no route, no network traffic,
but USB/DWC OTG still initialised via RTEMS_BSD_CONFIG_BSP_CONFIG)
-> still hangs (~5-6 min), USB hub enumeration continues in background:
ugen1.2: <vendor 0x0424 product 0x2514> at usbus1
uhub1 on uhub0
...
Console output is garbled by concurrent USB enumeration messages,
suggesting interrupt contention.
4. Commented out RTEMS_BSD_CONFIG_BSP_CONFIG and the
#include <bsp/nexus-devices.h> to prevent DWC OTG initialisation
-> STABLE, ran for 14+ minutes with no hang (test stopped manually)
The libbsd software stack (loopback, sockets, telnetd) continues to
function in test 4 =E2=80=94 only the hardware BSP devices (DWC OTG, muge,
uhub, ukphy) are excluded.
Minimal reproduction
--------------------
Build an RTEMS 6.2 application for raspberrypi2 BSP with:
#define RTEMS_BSD_CONFIG_BSP_CONFIG
#define RTEMS_BSD_CONFIG_INIT
#include <machine/rtems-bsd-config.h>
#include <bsp/nexus-devices.h> /* in one translation unit */
#define CONFIGURE_MICROSECONDS_PER_TICK 10000
The application does not need to configure any network interface or
perform any USB transfers =E2=80=94 the DWC OTG hub polling alone triggers
the hang after ~5-6 minutes.
Commenting out RTEMS_BSD_CONFIG_BSP_CONFIG and the nexus-devices.h
include eliminates the hang.
Boot log (abbreviated, from hanging configuration)
--------------------------------------------------
RTEMS RPi 3B+ 1.3 (1GB) [00a020d3]
nexus0: <RTEMS Nexus device>
dwcotg0: <DWC OTG 2.0 integrated USB controller> on nexus0
usbus1 on dwcotg0
usbus1: 480Mbps High Speed USB v2.0
ugen1.1: <DWCOTG OTG Root HUB> at usbus1
uhub0 on usbus1
uhub0: <DWCOTG OTG Root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
uhub0: 1 port with 1 removable, self powered
ugen1.2: <vendor 0x0424 product 0x2514> at usbus1
uhub1 on uhub0
uhub1: <vendor 0x0424 product 0x2514, class 9/0, rev 2.00/b.b3, addr 2> on
usbus1
uhub1: 4 ports with 3 removable, self powered
ugen1.3: <vendor 0x0424 product 0x2514> at usbus1
uhub2 on uhub1
uhub2: <vendor 0x0424 product 0x2514, class 9/0, rev 2.00/b.b3, addr 3> on
usbus1
uhub2: 3 ports with 2 removable, self powered
ugen1.4: <vendor 0x0424 product 0x7800> at usbus1
muge0: <vendor 0x0424 product 0x7800, rev 2.10/3.00, addr 4> on usbus1
muge0: Chip ID 0x7800 rev 0002
miibus0: <MII bus> on muge0
ukphy0: <Generic IEEE 802.3u media interface> PHY 1 on miibus0
info: ue0: <USB Ethernet> on muge0
[system hangs after ~5-6 minutes, UART unresponsive]
Questions
---------
1. Is this a known issue with the DWC OTG driver on RPi 3B+?
2. Are there any configuration options (hub polling interval, interrupt
coalescing, DMA settings) that might work around the problem?
3. Would a newer version of rtems-libbsd contain fixes for this?
4. Is there an alternative Ethernet driver approach for RPi 3B+ that
avoids the DWC OTG USB path?
5. Is there any known project which uses RPi with RTEMS?
(it looks like that the option to run RTEMS on RPi4 or RPi 5 can not be
considered since the BSP in RTEMS kernel is not yet finalised.
The RPi4 or RPi 5 would be much better candidates vs RPi 3 B+ since they
use direct connection to Ethernet instead of the USB-Ethernet)
Thank you for any guidance.
Mirek
--0000000000007a0706065010a754
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Hi All,<div><br></div><div>I'm running an EPICS ioc se=
rver (EPICS 7.0.10)=C2=A0 with RTEMS 6.2 on a Raspberry Pi 3B+ with rtems-l=
ibbsd=C2=A0 (6-freebsd-14) and<br>encountering a reproducible system hang a=
fter several minutes of operation.<br>Through systematic elimination testin=
g I've narrowed the root cause to the<br>DWC OTG USB controller driver.=
I'd appreciate any recommendations on how<br>to address this. I can of=
course use the EPICS ioc server under linux on RPi but just tried to have =
the RTEMS - Hard real time system which is much more deterministic.</div><d=
iv>The boot time for the RPi with EPICS/RTEMS is around 7 sec which is extr=
emely=C2=A0fast!</div><div><br>Environment<br>-----------<br>- Board: Raspb=
erry Pi 3B+ (BCM2837, boardrev a020d3)<br>- RTEMS: rtems-6.2 (ARM/ARMv4/ras=
pberrypi2)<br>- Network stack: rtems-libbsd (RTEMS_BSD_CONFIG_BSP_CONFIG + =
RTEMS_BSD_CONFIG_INIT)<br>- Ethernet: LAN7515 USB Ethernet (muge driver, vi=
a DWC OTG)<br>- Application: EPICS IOC (but hang occurs with minimal/empty =
IOC as well)<br>- Console: UART serial (/dev/ttyS0)<br><br>Symptom<br>-----=
--<br>The system boots and runs normally, then the entire system freezes =
=E2=80=94<br>including the UART serial console (which is not USB-dependent)=
. No crash<br>message, no stack dump =E2=80=94 a hard hang requiring power =
cycle.<br><br>The time-to-hang depends on the system tick rate:<br>- CONFIG=
URE_MICROSECONDS_PER_TICK=3D500 =C2=A0(2 kHz): hangs after ~2 minutes<br>- =
CONFIGURE_MICROSECONDS_PER_TICK=3D10000 (100 Hz): hangs after ~5-6 minutes<=
br><br>During the hang, the UART becomes completely unresponsive, suggestin=
g a<br>kernel-level deadlock or interrupt handler issue rather than an<br>a=
pplication-level problem.<br><br>Elimination testing performed<br>---------=
--------------------<br>I systematically disabled components to isolate the=
cause. All tests below<br>used CONFIGURE_MICROSECONDS_PER_TICK=3D10000 (10=
0 Hz):<br><br>1. Disabled all EPICS database records (no record processing)=
-> still hangs<br>2. Disabled periodic NTP sync (no socket operations) =
-> still hangs<br>3. Disabled IP configuration (no ifconfig, no route, n=
o network traffic,<br>=C2=A0 =C2=A0but USB/DWC OTG still initialised via RT=
EMS_BSD_CONFIG_BSP_CONFIG)<br>=C2=A0 =C2=A0-> still hangs (~5-6 min), US=
B hub enumeration continues in background:<br>=C2=A0 =C2=A0 =C2=A0 ugen1.2:=
<vendor 0x0424 product 0x2514> at usbus1<br>=C2=A0 =C2=A0 =C2=A0 uhu=
b1 on uhub0<br>=C2=A0 =C2=A0 =C2=A0 ...<br>=C2=A0 =C2=A0Console output is g=
arbled by concurrent USB enumeration messages,<br>=C2=A0 =C2=A0suggesting i=
nterrupt contention.<br>4. Commented out RTEMS_BSD_CONFIG_BSP_CONFIG and th=
e<br>=C2=A0 =C2=A0#include <bsp/nexus-devices.h> to prevent DWC OTG i=
nitialisation<br>=C2=A0 =C2=A0-> STABLE, ran for 14+ minutes with no han=
g (test stopped manually)<br><br>The libbsd software stack (loopback, socke=
ts, telnetd) continues to<br>function in test 4 =E2=80=94 only the hardware=
BSP devices (DWC OTG, muge,<br>uhub, ukphy) are excluded.<br><br>Minimal r=
eproduction<br>--------------------<br>Build an RTEMS 6.2 application for r=
aspberrypi2 BSP with:<br><br>=C2=A0 #define RTEMS_BSD_CONFIG_BSP_CONFIG<br>=
=C2=A0 #define RTEMS_BSD_CONFIG_INIT<br>=C2=A0 #include <machine/rtems-b=
sd-config.h><br>=C2=A0 #include <bsp/nexus-devices.h> =C2=A0 =C2=
=A0 /* in one translation unit */<br><br>=C2=A0 #define CONFIGURE_MICROSECO=
NDS_PER_TICK 10000<br><br>The application does not need to configure any ne=
twork interface or<br>perform any USB transfers =E2=80=94 the DWC OTG hub p=
olling alone triggers<br>the hang after ~5-6 minutes.<br><br>Commenting out=
RTEMS_BSD_CONFIG_BSP_CONFIG and the nexus-devices.h<br>include eliminates =
the hang.<br><br>Boot log (abbreviated, from hanging configuration)<br>----=
----------------------------------------------<br>RTEMS RPi 3B+ 1.3 (1GB) [=
00a020d3]<br>nexus0: <RTEMS Nexus device><br>dwcotg0: <DWC OTG 2.0=
integrated USB controller> on nexus0<br>usbus1 on dwcotg0<br>usbus1: 48=
0Mbps High Speed USB v2.0<br>ugen1.1: <DWCOTG OTG Root HUB> at usbus1=
<br>uhub0 on usbus1<br>uhub0: <DWCOTG OTG Root HUB, class 9/0, rev 2.00/=
1.00, addr 1> on usbus1<br>uhub0: 1 port with 1 removable, self powered<=
br>ugen1.2: <vendor 0x0424 product 0x2514> at usbus1<br>uhub1 on uhub=
0<br>uhub1: <vendor 0x0424 product 0x2514, class 9/0, rev 2.00/b.b3, add=
r 2> on usbus1<br>uhub1: 4 ports with 3 removable, self powered<br>ugen1=
.3: <vendor 0x0424 product 0x2514> at usbus1<br>uhub2 on uhub1<br>uhu=
b2: <vendor 0x0424 product 0x2514, class 9/0, rev 2.00/b.b3, addr 3> =
on usbus1<br>uhub2: 3 ports with 2 removable, self powered<br>ugen1.4: <=
vendor 0x0424 product 0x7800> at usbus1<br>muge0: <vendor 0x0424 prod=
uct 0x7800, rev 2.10/3.00, addr 4> on usbus1<br>muge0: Chip ID 0x7800 re=
v 0002<br>miibus0: <MII bus> on muge0<br>ukphy0: <Generic IEEE 802=
.3u media interface> PHY 1 on miibus0<br>info: ue0: <USB Ethernet>=
on muge0<br>[system hangs after ~5-6 minutes, UART unresponsive]<br><br>Qu=
estions<br>---------<br>1. Is this a known issue with the DWC OTG driver on=
RPi 3B+?<br>2. Are there any configuration options (hub polling interval, =
interrupt<br>=C2=A0 =C2=A0coalescing, DMA settings) that might work around =
the problem?<br>3. Would a newer version of rtems-libbsd contain fixes for =
this?<br>4. Is there an alternative Ethernet driver approach for RPi 3B+ th=
at<br>=C2=A0 =C2=A0avoids the DWC OTG USB path?</div><div>5. Is there any k=
nown project which uses RPi with RTEMS?</div><div>(it looks like that the o=
ption to run RTEMS on RPi4 or RPi 5 can not be considered=C2=A0since the BS=
P in RTEMS kernel=C2=A0 is not yet finalised.<br>The=C2=A0RPi4 or RPi 5 wou=
ld be much better candidates=C2=A0vs RPi 3 B+ since they use direct connect=
ion to Ethernet instead of the USB-Ethernet)=C2=A0<br><br>Thank you for any=
guidance.<br><br>Mirek<br><br><br></div></div>
--0000000000007a0706065010a754--
--===============4263124216103028876==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
users mailing list
[email protected]
http://lists.rtems.org/mailman/listinfo/users
--===============4263124216103028876==--