Re: EPICS ioc with RTEMS on Raspberry Pi
John Howard <[email protected]> Sat, 25 Apr 2026 12:36:59 -0500
| Newsgroups | gmane.os.rtems.user |
|---|---|
| Message-ID | <[email protected]> |
--===============1162736012330876168== Content-Type: multipart/alternative; boundary=Apple-Mail-E4F878F6-AE27-4839-9816-E4A93F70B467 Content-Transfer-Encoding: 7bit --Apple-Mail-E4F878F6-AE27-4839-9816-E4A93F70B467 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html class=3D"apple-mail-supports-explicit-dark-mode"><head><meta http-equi= v=3D"content-type" content=3D"text/html; charset=3Dutf-8"></head><body dir=3D= "auto"><div dir=3D"ltr"></div><div dir=3D"ltr">Hi Mirek,</div><div dir=3D"lt= r"><br></div><div dir=3D"ltr">I don't know what to tell you about this issue= . You know more about the details than I do.</div><div dir=3D"ltr"><br></div= ><div dir=3D"ltr">(I don't know what EPICS IOC does.)</div><div dir=3D"ltr">= <br></div><div dir=3D"ltr">I knew RTEMS didn't use FIQ, but I assumed a 32/6= 4-bit FreeBSD for Raspberry Pi didn't either. Why not a continuing FIQ issue= on their mailing list after all this time? I haven't used FreeBSD for RPi (= too many security holes to me). I use Debian Bullseye for my 3B+ and I am no= t happy with Linux.</div><div dir=3D"ltr"><br></div><div dir=3D"ltr">I don't= expect this to occur on RPi Zero 2 W because it lacks the Ethernet subsyste= m and has a newer revision of the 3B+ subsystem. (I will deal with it later d= uring my app testing phase.)</div><div dir=3D"ltr"><br></div><div dir=3D"ltr= ">I am not the CODEOWNER at RTEMS.org for that Dynaware/Synopsis driver.</di= v><div dir=3D"ltr"><br></div><div dir=3D"ltr">I do know that Synopsis releas= ed it unlicensed AS IS but is willing to license it under obligation for the= ir customers.</div><div dir=3D"ltr"><br></div><div dir=3D"ltr">I recently lo= oked at an alternative kernel memory map document where FIQ stacks are prese= nt for RPi 3B+ when Aarch32, but I can't find them when Aarch64 which standa= rdized calling conventions.</div><div dir=3D"ltr"><br></div><div dir=3D"ltr"= >Sorry I couldn't be of much help to you on this issue. Dealing with FIQ han= dling is as idiosyncratic as it gets on Raspberry Pi and is a deal-breaker t= o me.</div><div dir=3D"ltr"><br></div><div dir=3D"ltr">-- John</div><div dir= =3D"ltr"><br><blockquote type=3D"cite">On Apr 22, 2026, at 10:28=E2=80=AFPM,= Miroslaw Dach <[email protected]> wrote:<br><br></blockquote></= div><blockquote type=3D"cite"><div dir=3D"ltr">=EF=BB=BF<div dir=3D"ltr">Hi J= ohn,<br><br>Thanks for your e-mail and your suggestion.<div><br>Follow-up to= my earlier report about the DWC OTG USB hang on RPi 3B+<br>with RTEMS 6. I'= ve done extensive instrumentation and applied BCM2835-<br>specific errata wo= rkarounds from the Linux dwc2 driver. Here are the<br>complete findings:<br>= <br>I added diagnostic counters (#ifdef __rtems__) to the FreeBSD dwc_otg<br= >driver, printed every 1 second from the 10ms timer callback:<br><br>- yld: &= nbsp; times dwc_otg_interrupt_poll_locked() hit its 16-iteration cap<br>- af= ail: times dwc_otg_host_channel_alloc() failed (no free channels)<br>- hok: &= nbsp; successful halt completions processed<br>- rxd: RX data discard= ed (no active endpoint listening)<br>- stuck: channels with wait_halted=3D1 b= ut allocated=3D0 (leaked channels)<br>- filt: filter ISR entry/exit co= unt (to detect stuck ISR)<br>- thr: thread handler entry/exit count<b= r>- gi: last GINTSTS register value<br><br>Results: channel exh= austion theory DISPROVED<br>Across all tests, every counter remained perfect= ly healthy right up to<br>the instant of the freeze:<br><br> DWC_OTG t= =3D17500: yld=3D0 afail=3D0 hok=3D180038 rxd=3D0 ch=3D8/16<br> &= nbsp; alloc=3D0 wh=3D0 stuck=3D0 filt=3D738178/738178 thr=3D7280/7280<br>&nb= sp; gi=3D0x04000031<br><br>- yld=3D0: the 16-ite= ration poll loop cap was NEVER hit<br>- afail=3D0: channel allocation= NEVER failed<br>- stuck=3D0: NO channels ever leaked (wait_halted wa= s always cleared)<br>- filt balanced: filter ISR entered and exited the same= number of times<br>- thr balanced: thread handler entered and exited t= he same number of times<br>- gi=3D0x04000031: benign (host mode + TX FIFOs e= mpty + RX FIFO non-empty)<br><br>The ~1054 halt completions/second were stea= dy with no degradation.<br><br>Results: total CPU freeze, not software deadl= ock<br><br>I also tried an independent RTEMS timer (rtems_timer_fire_after) a= s a<br>heartbeat, firing outside the USB bus lock. Both the USB timer and th= e<br>independent heartbeat stopped simultaneously, confirming the entire ARM= <br>core freezes =E2=80=94 not just the USB subsystem.<br><br>No RTEMS fatal= error handler was triggered (I installed one via<br>CONFIGURE_INITIAL_EXTEN= SIONS). This means it is NOT a standard ARM<br>data abort =E2=80=94 the CPU s= imply stops executing, likely due to an AHB bus<br>lockup caused by the DWC O= TG controller hardware.<br><br>BCM2835 errata workarounds applied<br><br>Bas= ed on the Linux dwc2 driver (params.c, core.c) and the Ultibo<br>project doc= umentation, I applied three workarounds:<br><br>1. FIFO size cap: BCM2835 ha= rdware reports 4096 words but only 4080<br> exist. Cap sc_fifo_s= ize to 4080*4 bytes to prevent FIFO overrun.<br> (In practice, t= he hardware on my board reported <=3D 4080, so this<br> cap w= as NOT triggered.)<br><br>2. GAHBCFG AHB burst configuration: Broadcom redef= ined bits [4:1] of<br> GAHBCFG for AXI burst control. Linux dwc2= sets ahbcfg=3D0x10 for<br> BCM2835. Changed from GAHBCFG_GLBLIN= TRMSK (0x01) to 0x11.<br><br>3. AHB idle wait after core reset: Added a loop= waiting for<br> GRSTCTL_AHBIDLE after GRSTCTL_CSFTRST, plus 250= ms settling delay<br> (Linux dwc2 uses 100+ ms).<br><br>Results w= ith workarounds:<br>- Before: hangs after ~86 seconds (2 kHz tick)<br>- Afte= r: hangs after ~175 seconds (2 kHz tick)<br><br>The workarounds approx= imately doubled the time-to-hang but did NOT<br>eliminate it. All software c= ounters remained healthy throughout.<br><br>The hang is caused most probably= by a combination of factors:<br><br>1. The DWC OTG controller on BCM2835/BC= M2837 requires sub-125=C2=B5s<br> interrupt response times for U= SB split transaction phases (Start<br> Split =E2=86=92 Complete S= plit through the USB hub). The RPi 3B+ Ethernet<br> goes through= a USB hub (LAN7515), making every packet a split<br> transactio= n.<br><br>2. The Linux kernel addresses this with a dedicated FIQ (Fast Inte= rrupt<br> Request) handler (dwc_otg_fiq_fsm.c) that executes com= plete split<br> transactions in FIQ context, bypassing the norma= l interrupt stack.<br> Without FIQ, "certain USB devices become c= ompletely unusable."<br><br>3 The FreeBSD dwc_otg driver used by RTEMS handl= es all split<br> transactions in normal interrupt context. On RT= EMS, the interrupt<br> filter and thread handler run back-to-bac= k in the interrupt server<br> task (nexus_intr_with_filter in rt= ems-kernel-nexus.c), with no<br> preemption point between them.<= br><br>4 When the controller's split transaction timing is violated, it<br>&= nbsp; enters an unrecoverable state that locks the AHB bus, freezing t= he<br> entire ARM core including UART and system timers.<br><br>= The timing correlation with tick rate confirms this: higher tick rate<br>=3D= more frequent scheduling =3D more interrupt latency jitter =3D faster<br>ti= ming violation.</div><div><br>A proper fix requires most probably implementi= ng FIQ-based split transaction handling<br>in the RTEMS BSP for BCM2835/BCM2= 837, similar to what Linux does in<br>dwc_otg_fiq_fsm.c. This is a significa= nt undertaking but is essential<br>for reliable USB operation on RPi 3B+ (an= d RPi Zero 2 W, which uses<br>the same SoC).<br><br>The GAHBCFG and reset se= quence workarounds should also be applied as<br>they improve stability.<br><= br>Please give me your thoughts on that. Maybe it is easier to finalise the B= SP for RPi 4 and 5?<br><br>Best Regards<br>Mirek<br></div></div><br><div cla= ss=3D"gmail_quote gmail_quote_container"><div dir=3D"ltr" class=3D"gmail_att= r">=C5=9Br., 22 kwi 2026 o 13:17 John Howard <<a href=3D"mailto:echo= [email protected]">[email protected]</a>> napisa=C5=82(a):<br></div= ><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left:1px solid rgb(204,204,204);padding-left:1ex">Fascinating. Thanks for th= at detailed report.<br> <br> You indicated USB enumerating continues running in the background.<br> <br> I am educated-guessing that a counter maximum is reached, and then mistakenl= y breached. I would look for a test of that counter and correct it from allo= wing greater-than comparing.<br> <br> Let us know how it turns out.<br> <br> I am developing an app for Raspberry Pi Zero 2 W (stripped-down 3B+). I wasn= 't expecting any potential problem like this.<br> <br> -- John<br> <br> > On Apr 22, 2026, at 1:25=E2=80=AFPM, Miroslaw Dach <<a href=3D"mailt= o:[email protected]" target=3D"_blank">[email protected]</a>>= wrote:<br> > <br> > =EF=BB=BF<br> > Hi All,<br> > <br> > I'm running an EPICS ioc server (EPICS 7.0.10) with RTEMS 6.2 on a= Raspberry Pi 3B+ with rtems-libbsd (6-freebsd-14) and<br> > encountering a reproducible system hang after several minutes of operat= ion.<br> > Through systematic elimination testing I've narrowed the root cause to t= he<br> > DWC OTG USB controller driver. I'd appreciate any recommendations on ho= w<br> > to address this. I can of course use the EPICS ioc server under linux o= n RPi but just tried to have the RTEMS - Hard real time system which is much= more deterministic.<br> > The boot time for the RPi with EPICS/RTEMS is around 7 sec which is ext= remely fast!<br> > <br> > Environment<br> > -----------<br> > - Board: Raspberry Pi 3B+ (BCM2837, boardrev a020d3)<br> > - RTEMS: rtems-6.2 (ARM/ARMv4/raspberrypi2)<br> > - Network stack: rtems-libbsd (RTEMS_BSD_CONFIG_BSP_CONFIG + RTEMS_BSD_= CONFIG_INIT)<br> > - Ethernet: LAN7515 USB Ethernet (muge driver, via DWC OTG)<br> > - Application: EPICS IOC (but hang occurs with minimal/empty IOC as wel= l)<br> > - Console: UART serial (/dev/ttyS0)<br> > <br> > Symptom<br> > -------<br> > The system boots and runs normally, then the entire system freezes =E2=80= =94<br> > including the UART serial console (which is not USB-dependent). No cras= h<br> > message, no stack dump =E2=80=94 a hard hang requiring power cycle.<br>= > <br> > The time-to-hang depends on the system tick rate:<br> > - CONFIGURE_MICROSECONDS_PER_TICK=3D500 (2 kHz): hangs after ~2 m= inutes<br> > - CONFIGURE_MICROSECONDS_PER_TICK=3D10000 (100 Hz): hangs after ~5-6 mi= nutes<br> > <br> > During the hang, the UART becomes completely unresponsive, suggesting a= <br> > kernel-level deadlock or interrupt handler issue rather than an<br> > application-level problem.<br> > <br> > Elimination testing performed<br> > -----------------------------<br> > I systematically disabled components to isolate the cause. All tests be= low<br> > used CONFIGURE_MICROSECONDS_PER_TICK=3D10000 (100 Hz):<br> > <br> > 1. Disabled all EPICS database records (no record processing) -> sti= ll hangs<br> > 2. Disabled periodic NTP sync (no socket operations) -> still hangs<= br> > 3. Disabled IP configuration (no ifconfig, no route, no network traffic= ,<br> > but USB/DWC OTG still initialised via RTEMS_BSD_CONFIG_BSP= _CONFIG)<br> > -> still hangs (~5-6 min), USB hub enumeration continue= s in background:<br> > ugen1.2: <vendor 0x0424 product 0x2514>= at usbus1<br> > uhub1 on uhub0<br> > ...<br> > Console output is garbled by concurrent USB enumeration me= ssages,<br> > suggesting interrupt contention.<br> > 4. Commented out RTEMS_BSD_CONFIG_BSP_CONFIG and the<br> > #include <bsp/nexus-devices.h> to prevent DWC OTG in= itialisation<br> > -> STABLE, ran for 14+ minutes with no hang (test stopp= ed manually)<br> > <br> > The libbsd software stack (loopback, sockets, telnetd) continues to<br>= > function in test 4 =E2=80=94 only the hardware BSP devices (DWC OTG, mu= ge,<br> > uhub, ukphy) are excluded.<br> > <br> > Minimal reproduction<br> > --------------------<br> > Build an RTEMS 6.2 application for raspberrypi2 BSP with:<br> > <br> > #define RTEMS_BSD_CONFIG_BSP_CONFIG<br> > #define RTEMS_BSD_CONFIG_INIT<br> > #include <machine/rtems-bsd-config.h><br> > #include <bsp/nexus-devices.h> /* i= n one translation unit */<br> > <br> > #define CONFIGURE_MICROSECONDS_PER_TICK 10000<br> > <br> > The application does not need to configure any network interface or<br>= > perform any USB transfers =E2=80=94 the DWC OTG hub polling alone trigg= ers<br> > the hang after ~5-6 minutes.<br> > <br> > Commenting out RTEMS_BSD_CONFIG_BSP_CONFIG and the nexus-devices.h<br> > include eliminates the hang.<br> > <br> > Boot log (abbreviated, from hanging configuration)<br> > --------------------------------------------------<br> > RTEMS RPi 3B+ 1.3 (1GB) [00a020d3]<br> > nexus0: <RTEMS Nexus device><br> > dwcotg0: <DWC OTG 2.0 integrated USB controller> on nexus0<br> > usbus1 on dwcotg0<br> > usbus1: 480Mbps High Speed USB v2.0<br> > ugen1.1: <DWCOTG OTG Root HUB> at usbus1<br> > uhub0 on usbus1<br> > uhub0: <DWCOTG OTG Root HUB, class 9/0, rev 2.00/1.00, addr 1> on= usbus1<br> > uhub0: 1 port with 1 removable, self powered<br> > ugen1.2: <vendor 0x0424 product 0x2514> at usbus1<br> > uhub1 on uhub0<br> > uhub1: <vendor 0x0424 product 0x2514, class 9/0, rev 2.00/b.b3, addr= 2> on usbus1<br> > uhub1: 4 ports with 3 removable, self powered<br> > ugen1.3: <vendor 0x0424 product 0x2514> at usbus1<br> > uhub2 on uhub1<br> > uhub2: <vendor 0x0424 product 0x2514, class 9/0, rev 2.00/b.b3, addr= 3> on usbus1<br> > uhub2: 3 ports with 2 removable, self powered<br> > ugen1.4: <vendor 0x0424 product 0x7800> at usbus1<br> > muge0: <vendor 0x0424 product 0x7800, rev 2.10/3.00, addr 4> on u= sbus1<br> > muge0: Chip ID 0x7800 rev 0002<br> > miibus0: <MII bus> on muge0<br> > ukphy0: <Generic IEEE 802.3u media interface> PHY 1 on miibus0<br= > > info: ue0: <USB Ethernet> on muge0<br> > [system hangs after ~5-6 minutes, UART unresponsive]<br> > <br> > Questions<br> > ---------<br> > 1. Is this a known issue with the DWC OTG driver on RPi 3B+?<br> > 2. Are there any configuration options (hub polling interval, interrupt= <br> > coalescing, DMA settings) that might work around the probl= em?<br> > 3. Would a newer version of rtems-libbsd contain fixes for this?<br> > 4. Is there an alternative Ethernet driver approach for RPi 3B+ that<br= > > avoids the DWC OTG USB path?<br> > 5. Is there any known project which uses RPi with RTEMS?<br> > (it looks like that the option to run RTEMS on RPi4 or RPi 5 can not be= considered since the BSP in RTEMS kernel is not yet finalised.<br> > The RPi4 or RPi 5 would be much better candidates vs RPi 3 B+ since the= y use direct connection to Ethernet instead of the USB-Ethernet) <br> > <br> > Thank you for any guidance.<br> > <br> > Mirek<br> > <br> > <br> > _______________________________________________<br> > users mailing list<br> > <a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a= ><br> > <a href=3D"http://lists.rtems.org/mailman/listinfo/users" rel=3D"norefe= rrer" target=3D"_blank">http://lists.rtems.org/mailman/listinfo/users</a><br= > </blockquote></div> </div></blockquote></body></html>= --Apple-Mail-E4F878F6-AE27-4839-9816-E4A93F70B467-- --===============1162736012330876168== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ users mailing list [email protected] http://lists.rtems.org/mailman/listinfo/users --===============1162736012330876168==--