Re: EPICS ioc with RTEMS on Raspberry Pi
Miroslaw Dach <[email protected]> Thu, 21 May 2026 14:41:50 -0700
| Newsgroups | gmane.os.rtems.user |
|---|---|
| Message-ID | <CAPAFRJfZXkLLhFi3NdPLMWmrbPKALuACFdUcPmfedaesHwk9kg@mail.gmail.com> |
--===============9109511382912199927== Content-Type: multipart/alternative; boundary="00000000000054019f06525ac885" --00000000000054019f06525ac885 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Gedare, Thanks for the good news about the RPi4. Please let me know when you have some results and working RPi4 with RTEMS. Did anybody try the lwIP network stack for RTEMS? Best Regards Mirek wt., 19 maj 2026 o 15:18 Gedare Bloom <[email protected]> napisa=C5=82(a): > Just a quick note that the preferred venue for community engagement is > shifting toward https://users.rtems.org > > I think the RPi4 is maturing quite rapidly. We have a GSoC project > focused on it as well. > > lwIP might also be an option to consider. > > On Fri, May 8, 2026 at 4:00=E2=80=AFPM Miroslaw Dach <miroslaw.dach@gmail= .com> > wrote: > > > > Hi John, > > > > Many Thanks for your reply and all of the hints. It looks like that at > present there is no good solution to use RTEMS on RPi 3 b+ due to the iss= ue > with the > > usb-Network bridge. The most promising approach would be to use the PRi > v4 or v5 since the network chip is directly connected to the system but > the BSP is not yet complete for RTEMS so maybe in the future. > > > > Best Regards > > Mirek > > > > sob., 25 kwi 2026 o 10:37 John Howard <[email protected]> > napisa=C5=82(a): > >> > >> Hi Mirek, > >> > >> I don't know what to tell you about this issue. You know more about th= e > details than I do. > >> > >> (I don't know what EPICS IOC does.) > >> > >> I knew RTEMS didn't use FIQ, but I assumed a 32/64-bit FreeBSD for > Raspberry Pi didn't either. Why not a continuing FIQ issue on their maili= ng > list after all this time? I haven't used FreeBSD for RPi (too many securi= ty > holes to me). I use Debian Bullseye for my 3B+ and I am not happy with > Linux. > >> > >> I don't expect this to occur on RPi Zero 2 W because it lacks the > Ethernet subsystem and has a newer revision of the 3B+ subsystem. (I will > deal with it later during my app testing phase.) > >> > >> I am not the CODEOWNER at RTEMS.org for that Dynaware/Synopsis driver. > >> > >> I do know that Synopsis released it unlicensed AS IS but is willing to > license it under obligation for their customers. > >> > >> I recently looked at an alternative kernel memory map document where > FIQ stacks are present for RPi 3B+ when Aarch32, but I can't find them wh= en > Aarch64 which standardized calling conventions. > >> > >> Sorry I couldn't be of much help to you on this issue. Dealing with FI= Q > handling is as idiosyncratic as it gets on Raspberry Pi and is a > deal-breaker to me. > >> > >> -- John > >> > >> On Apr 22, 2026, at 10:28=E2=80=AFPM, Miroslaw Dach <miroslaw.dach@gma= il.com> > wrote: > >> > >> =EF=BB=BF > >> Hi John, > >> > >> Thanks for your e-mail and your suggestion. > >> > >> Follow-up to my earlier report about the DWC OTG USB hang on RPi 3B+ > >> with RTEMS 6. I've done extensive instrumentation and applied BCM2835- > >> specific errata workarounds from the Linux dwc2 driver. Here are the > >> complete findings: > >> > >> I added diagnostic counters (#ifdef __rtems__) to the FreeBSD dwc_otg > >> driver, printed every 1 second from the 10ms timer callback: > >> > >> - yld: times dwc_otg_interrupt_poll_locked() hit its 16-iteration ca= p > >> - afail: times dwc_otg_host_channel_alloc() failed (no free channels) > >> - hok: successful halt completions processed > >> - rxd: RX data discarded (no active endpoint listening) > >> - stuck: channels with wait_halted=3D1 but allocated=3D0 (leaked chann= els) > >> - filt: filter ISR entry/exit count (to detect stuck ISR) > >> - thr: thread handler entry/exit count > >> - gi: last GINTSTS register value > >> > >> Results: channel exhaustion theory DISPROVED > >> Across all tests, every counter remained perfectly healthy right up to > >> the instant of the freeze: > >> > >> DWC_OTG t=3D17500: yld=3D0 afail=3D0 hok=3D180038 rxd=3D0 ch=3D8/16 > >> alloc=3D0 wh=3D0 stuck=3D0 filt=3D738178/738178 thr=3D7280/7280 > >> gi=3D0x04000031 > >> > >> - yld=3D0: the 16-iteration poll loop cap was NEVER hit > >> - afail=3D0: channel allocation NEVER failed > >> - stuck=3D0: NO channels ever leaked (wait_halted was always cleared= ) > >> - filt balanced: filter ISR entered and exited the same number of time= s > >> - thr balanced: thread handler entered and exited the same number of > times > >> - gi=3D0x04000031: benign (host mode + TX FIFOs empty + RX FIFO non-em= pty) > >> > >> The ~1054 halt completions/second were steady with no degradation. > >> > >> Results: total CPU freeze, not software deadlock > >> > >> I also tried an independent RTEMS timer (rtems_timer_fire_after) as a > >> heartbeat, firing outside the USB bus lock. Both the USB timer and the > >> independent heartbeat stopped simultaneously, confirming the entire AR= M > >> core freezes =E2=80=94 not just the USB subsystem. > >> > >> No RTEMS fatal error handler was triggered (I installed one via > >> CONFIGURE_INITIAL_EXTENSIONS). This means it is NOT a standard ARM > >> data abort =E2=80=94 the CPU simply stops executing, likely due to an = AHB bus > >> lockup caused by the DWC OTG controller hardware. > >> > >> BCM2835 errata workarounds applied > >> > >> Based on the Linux dwc2 driver (params.c, core.c) and the Ultibo > >> project documentation, I applied three workarounds: > >> > >> 1. FIFO size cap: BCM2835 hardware reports 4096 words but only 4080 > >> exist. Cap sc_fifo_size to 4080*4 bytes to prevent FIFO overrun. > >> (In practice, the hardware on my board reported <=3D 4080, so this > >> cap was NOT triggered.) > >> > >> 2. GAHBCFG AHB burst configuration: Broadcom redefined bits [4:1] of > >> GAHBCFG for AXI burst control. Linux dwc2 sets ahbcfg=3D0x10 for > >> BCM2835. Changed from GAHBCFG_GLBLINTRMSK (0x01) to 0x11. > >> > >> 3. AHB idle wait after core reset: Added a loop waiting for > >> GRSTCTL_AHBIDLE after GRSTCTL_CSFTRST, plus 250ms settling delay > >> (Linux dwc2 uses 100+ ms). > >> > >> Results with workarounds: > >> - Before: hangs after ~86 seconds (2 kHz tick) > >> - After: hangs after ~175 seconds (2 kHz tick) > >> > >> The workarounds approximately doubled the time-to-hang but did NOT > >> eliminate it. All software counters remained healthy throughout. > >> > >> The hang is caused most probably by a combination of factors: > >> > >> 1. The DWC OTG controller on BCM2835/BCM2837 requires sub-125=C2=B5s > >> interrupt response times for USB split transaction phases (Start > >> Split =E2=86=92 Complete Split through the USB hub). The RPi 3B+ Et= hernet > >> goes through a USB hub (LAN7515), making every packet a split > >> transaction. > >> > >> 2. The Linux kernel addresses this with a dedicated FIQ (Fast Interrup= t > >> Request) handler (dwc_otg_fiq_fsm.c) that executes complete split > >> transactions in FIQ context, bypassing the normal interrupt stack. > >> Without FIQ, "certain USB devices become completely unusable." > >> > >> 3 The FreeBSD dwc_otg driver used by RTEMS handles all split > >> transactions in normal interrupt context. On RTEMS, the interrupt > >> filter and thread handler run back-to-back in the interrupt server > >> task (nexus_intr_with_filter in rtems-kernel-nexus.c), with no > >> preemption point between them. > >> > >> 4 When the controller's split transaction timing is violated, it > >> enters an unrecoverable state that locks the AHB bus, freezing the > >> entire ARM core including UART and system timers. > >> > >> The timing correlation with tick rate confirms this: higher tick rate > >> =3D more frequent scheduling =3D more interrupt latency jitter =3D fas= ter > >> timing violation. > >> > >> A proper fix requires most probably implementing FIQ-based split > transaction handling > >> in the RTEMS BSP for BCM2835/BCM2837, similar to what Linux does in > >> dwc_otg_fiq_fsm.c. This is a significant undertaking but is essential > >> for reliable USB operation on RPi 3B+ (and RPi Zero 2 W, which uses > >> the same SoC). > >> > >> The GAHBCFG and reset sequence workarounds should also be applied as > >> they improve stability. > >> > >> Please give me your thoughts on that. Maybe it is easier to finalise > the BSP for RPi 4 and 5? > >> > >> Best Regards > >> Mirek > >> > >> =C5=9Br., 22 kwi 2026 o 13:17 John Howard <[email protected]> > napisa=C5=82(a): > >>> > >>> Fascinating. Thanks for that detailed report. > >>> > >>> You indicated USB enumerating continues running in the background. > >>> > >>> I am educated-guessing that a counter maximum is reached, and then > mistakenly breached. I would look for a test of that counter and correct = it > from allowing greater-than comparing. > >>> > >>> Let us know how it turns out. > >>> > >>> I am developing an app for Raspberry Pi Zero 2 W (stripped-down 3B+). > I wasn't expecting any potential problem like this. > >>> > >>> -- John > >>> > >>> > On Apr 22, 2026, at 1:25=E2=80=AFPM, Miroslaw Dach <miroslaw.dach@g= mail.com> > wrote: > >>> > > >>> > =EF=BB=BF > >>> > Hi All, > >>> > > >>> > I'm running an EPICS ioc server (EPICS 7.0.10) with RTEMS 6.2 on a > Raspberry Pi 3B+ with rtems-libbsd (6-freebsd-14) and > >>> > encountering a reproducible system hang after several minutes of > operation. > >>> > Through systematic elimination testing I've narrowed the root cause > to the > >>> > DWC OTG USB controller driver. I'd appreciate any recommendations o= n > how > >>> > to address this. I can of course use the EPICS ioc server under > linux on RPi but just tried to have the RTEMS - Hard real time system whi= ch > is much more deterministic. > >>> > The boot time for the RPi with EPICS/RTEMS is around 7 sec which is > extremely fast! > >>> > > >>> > Environment > >>> > ----------- > >>> > - Board: Raspberry Pi 3B+ (BCM2837, boardrev a020d3) > >>> > - RTEMS: rtems-6.2 (ARM/ARMv4/raspberrypi2) > >>> > - Network stack: rtems-libbsd (RTEMS_BSD_CONFIG_BSP_CONFIG + > RTEMS_BSD_CONFIG_INIT) > >>> > - Ethernet: LAN7515 USB Ethernet (muge driver, via DWC OTG) > >>> > - Application: EPICS IOC (but hang occurs with minimal/empty IOC as > well) > >>> > - Console: UART serial (/dev/ttyS0) > >>> > > >>> > Symptom > >>> > ------- > >>> > The system boots and runs normally, then the entire system freezes = =E2=80=94 > >>> > including the UART serial console (which is not USB-dependent). No > crash > >>> > message, no stack dump =E2=80=94 a hard hang requiring power cycle. > >>> > > >>> > The time-to-hang depends on the system tick rate: > >>> > - CONFIGURE_MICROSECONDS_PER_TICK=3D500 (2 kHz): hangs after ~2 > minutes > >>> > - CONFIGURE_MICROSECONDS_PER_TICK=3D10000 (100 Hz): hangs after ~5-= 6 > minutes > >>> > > >>> > During the hang, the UART becomes completely unresponsive, > suggesting a > >>> > kernel-level deadlock or interrupt handler issue rather than an > >>> > application-level problem. > >>> > > >>> > Elimination testing performed > >>> > ----------------------------- > >>> > I systematically disabled components to isolate the cause. All test= s > below > >>> > used CONFIGURE_MICROSECONDS_PER_TICK=3D10000 (100 Hz): > >>> > > >>> > 1. Disabled all EPICS database records (no record processing) -> > still hangs > >>> > 2. Disabled periodic NTP sync (no socket operations) -> still hangs > >>> > 3. Disabled IP configuration (no ifconfig, no route, no network > traffic, > >>> > but USB/DWC OTG still initialised via RTEMS_BSD_CONFIG_BSP_CONFI= G) > >>> > -> still hangs (~5-6 min), USB hub enumeration continues in > background: > >>> > ugen1.2: <vendor 0x0424 product 0x2514> at usbus1 > >>> > uhub1 on uhub0 > >>> > ... > >>> > Console output is garbled by concurrent USB enumeration messages= , > >>> > suggesting interrupt contention. > >>> > 4. Commented out RTEMS_BSD_CONFIG_BSP_CONFIG and the > >>> > #include <bsp/nexus-devices.h> to prevent DWC OTG initialisation > >>> > -> STABLE, ran for 14+ minutes with no hang (test stopped > manually) > >>> > > >>> > The libbsd software stack (loopback, sockets, telnetd) continues to > >>> > function in test 4 =E2=80=94 only the hardware BSP devices (DWC OTG= , muge, > >>> > uhub, ukphy) are excluded. > >>> > > >>> > Minimal reproduction > >>> > -------------------- > >>> > Build an RTEMS 6.2 application for raspberrypi2 BSP with: > >>> > > >>> > #define RTEMS_BSD_CONFIG_BSP_CONFIG > >>> > #define RTEMS_BSD_CONFIG_INIT > >>> > #include <machine/rtems-bsd-config.h> > >>> > #include <bsp/nexus-devices.h> /* in one translation unit */ > >>> > > >>> > #define CONFIGURE_MICROSECONDS_PER_TICK 10000 > >>> > > >>> > The application does not need to configure any network interface or > >>> > perform any USB transfers =E2=80=94 the DWC OTG hub polling alone t= riggers > >>> > the hang after ~5-6 minutes. > >>> > > >>> > Commenting out RTEMS_BSD_CONFIG_BSP_CONFIG and the nexus-devices.h > >>> > include eliminates the hang. > >>> > > >>> > Boot log (abbreviated, from hanging configuration) > >>> > -------------------------------------------------- > >>> > RTEMS RPi 3B+ 1.3 (1GB) [00a020d3] > >>> > nexus0: <RTEMS Nexus device> > >>> > dwcotg0: <DWC OTG 2.0 integrated USB controller> on nexus0 > >>> > usbus1 on dwcotg0 > >>> > usbus1: 480Mbps High Speed USB v2.0 > >>> > ugen1.1: <DWCOTG OTG Root HUB> at usbus1 > >>> > uhub0 on usbus1 > >>> > uhub0: <DWCOTG OTG Root HUB, class 9/0, rev 2.00/1.00, addr 1> on > usbus1 > >>> > uhub0: 1 port with 1 removable, self powered > >>> > ugen1.2: <vendor 0x0424 product 0x2514> at usbus1 > >>> > uhub1 on uhub0 > >>> > uhub1: <vendor 0x0424 product 0x2514, class 9/0, rev 2.00/b.b3, add= r > 2> on usbus1 > >>> > uhub1: 4 ports with 3 removable, self powered > >>> > ugen1.3: <vendor 0x0424 product 0x2514> at usbus1 > >>> > uhub2 on uhub1 > >>> > uhub2: <vendor 0x0424 product 0x2514, class 9/0, rev 2.00/b.b3, add= r > 3> on usbus1 > >>> > uhub2: 3 ports with 2 removable, self powered > >>> > ugen1.4: <vendor 0x0424 product 0x7800> at usbus1 > >>> > muge0: <vendor 0x0424 product 0x7800, rev 2.10/3.00, addr 4> on > usbus1 > >>> > muge0: Chip ID 0x7800 rev 0002 > >>> > miibus0: <MII bus> on muge0 > >>> > ukphy0: <Generic IEEE 802.3u media interface> PHY 1 on miibus0 > >>> > info: ue0: <USB Ethernet> on muge0 > >>> > [system hangs after ~5-6 minutes, UART unresponsive] > >>> > > >>> > Questions > >>> > --------- > >>> > 1. Is this a known issue with the DWC OTG driver on RPi 3B+? > >>> > 2. Are there any configuration options (hub polling interval, > interrupt > >>> > coalescing, DMA settings) that might work around the problem? > >>> > 3. Would a newer version of rtems-libbsd contain fixes for this? > >>> > 4. Is there an alternative Ethernet driver approach for RPi 3B+ tha= t > >>> > avoids the DWC OTG USB path? > >>> > 5. Is there any known project which uses RPi with RTEMS? > >>> > (it looks like that the option to run RTEMS on RPi4 or RPi 5 can no= t > be considered since the BSP in RTEMS kernel is not yet finalised. > >>> > The RPi4 or RPi 5 would be much better candidates vs RPi 3 B+ since > they use direct connection to Ethernet instead of the USB-Ethernet) > >>> > > >>> > Thank you for any guidance. > >>> > > >>> > Mirek > >>> > > >>> > > >>> > _______________________________________________ > >>> > users mailing list > >>> > [email protected] > >>> > http://lists.rtems.org/mailman/listinfo/users > > > > _______________________________________________ > > users mailing list > > [email protected] > > http://lists.rtems.org/mailman/listinfo/users > --00000000000054019f06525ac885 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Hi Gedare,<div><br></div><div>Thanks for the good news abo= ut the=C2=A0RPi4. Please let me know when you have some results and working= =C2=A0RPi4 with RTEMS.<br>Did anybody try the=C2=A0lwIP network stack for R= TEMS?<br><br>Best Regards</div><div>Mirek</div></div><br><div class=3D"gmai= l_quote gmail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">wt., 1= 9 maj 2026 o 15:18=C2=A0Gedare Bloom <<a href=3D"mailto:[email protected]= ">[email protected]</a>> napisa=C5=82(a):<br></div><blockquote class=3D"g= mail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204= ,204,204);padding-left:1ex">Just a quick note that the preferred venue for = community engagement is<br> shifting toward <a href=3D"https://users.rtems.org" rel=3D"noreferrer" targ= et=3D"_blank">https://users.rtems.org</a><br> <br> I think the RPi4 is maturing quite rapidly. We have a GSoC project<br> focused on it as well.<br> <br> lwIP might also be an option to consider.<br> <br> On Fri, May 8, 2026 at 4:00=E2=80=AFPM Miroslaw Dach <<a href=3D"mailto:= [email protected]" target=3D"_blank">[email protected]</a>> = wrote:<br> ><br> > Hi John,<br> ><br> > Many Thanks for your reply and all of the hints. It looks like that at= present there is no good solution to use RTEMS on RPi 3 b+ due to the issu= e with the<br> > usb-Network bridge. The most promising approach would be to use the PR= i v4 or v5=C2=A0 since the network chip is directly connected to the system= but the BSP is not yet complete for RTEMS so maybe in the future.<br> ><br> > Best Regards<br> > Mirek<br> ><br> > sob., 25 kwi 2026 o 10:37 John Howard <<a href=3D"mailto:echosoft.l= [email protected]" target=3D"_blank">[email protected]</a>> napisa=C5=82= (a):<br> >><br> >> Hi Mirek,<br> >><br> >> I don't know what to tell you about this issue. You know more = about the details than I do.<br> >><br> >> (I don't know what EPICS IOC does.)<br> >><br> >> I knew RTEMS didn't use FIQ, but I assumed a 32/64-bit FreeBSD= for Raspberry Pi didn't either. Why not a continuing FIQ issue on thei= r mailing list after all this time? I haven't used FreeBSD for RPi (too= many security holes to me). I use Debian Bullseye for my 3B+ and I am not = happy with Linux.<br> >><br> >> I don't expect this to occur on RPi Zero 2 W because it lacks = the Ethernet subsystem and has a newer revision of the 3B+ subsystem. (I wi= ll deal with it later during my app testing phase.)<br> >><br> >> I am not the CODEOWNER at RTEMS.org for that Dynaware/Synopsis dri= ver.<br> >><br> >> I do know that Synopsis released it unlicensed AS IS but is willin= g to license it under obligation for their customers.<br> >><br> >> I recently looked at an alternative kernel memory map document whe= re FIQ stacks are present for RPi 3B+ when Aarch32, but I can't find th= em when Aarch64 which standardized calling conventions.<br> >><br> >> Sorry I couldn't be of much help to you on this issue. Dealing= with FIQ handling is as idiosyncratic as it gets on Raspberry Pi and is a = deal-breaker to me.<br> >><br> >> -- John<br> >><br> >> On Apr 22, 2026, at 10:28=E2=80=AFPM, Miroslaw Dach <<a href=3D= "mailto:[email protected]" target=3D"_blank">[email protected]<= /a>> wrote:<br> >><br> >> =EF=BB=BF<br> >> Hi John,<br> >><br> >> Thanks for your e-mail and your suggestion.<br> >><br> >> Follow-up to my earlier report about the DWC OTG USB hang on RPi 3= B+<br> >> with RTEMS 6. I've done extensive instrumentation and applied = BCM2835-<br> >> specific errata workarounds from the Linux dwc2 driver. Here are t= he<br> >> complete findings:<br> >><br> >> I added diagnostic counters (#ifdef __rtems__) to the FreeBSD dwc_= otg<br> >> driver, printed every 1 second from the 10ms timer callback:<br> >><br> >> - yld:=C2=A0 =C2=A0times dwc_otg_interrupt_poll_locked() hit its 1= 6-iteration cap<br> >> - afail: times dwc_otg_host_channel_alloc() failed (no free channe= ls)<br> >> - hok:=C2=A0 =C2=A0successful halt completions processed<br> >> - rxd:=C2=A0 =C2=A0RX data discarded (no active endpoint listening= )<br> >> - stuck: channels with wait_halted=3D1 but allocated=3D0 (leaked c= hannels)<br> >> - filt:=C2=A0 filter ISR entry/exit count (to detect stuck ISR)<br= > >> - thr:=C2=A0 =C2=A0thread handler entry/exit count<br> >> - gi:=C2=A0 =C2=A0 last GINTSTS register value<br> >><br> >> Results: channel exhaustion theory DISPROVED<br> >> Across all tests, every counter remained perfectly healthy right u= p to<br> >> the instant of the freeze:<br> >><br> >>=C2=A0 =C2=A0DWC_OTG t=3D17500: yld=3D0 afail=3D0 hok=3D180038 rxd= =3D0 ch=3D8/16<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0alloc=3D0 wh=3D0 stuck=3D0 filt=3D738178= /738178 thr=3D7280/7280<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0gi=3D0x04000031<br> >><br> >> - yld=3D0:=C2=A0 =C2=A0 =C2=A0the 16-iteration poll loop cap was N= EVER hit<br> >> - afail=3D0:=C2=A0 =C2=A0channel allocation NEVER failed<br> >> - stuck=3D0:=C2=A0 =C2=A0NO channels ever leaked (wait_halted was = always cleared)<br> >> - filt balanced: filter ISR entered and exited the same number of = times<br> >> - thr balanced:=C2=A0 thread handler entered and exited the same n= umber of times<br> >> - gi=3D0x04000031: benign (host mode + TX FIFOs empty + RX FIFO no= n-empty)<br> >><br> >> The ~1054 halt completions/second were steady with no degradation.= <br> >><br> >> Results: total CPU freeze, not software deadlock<br> >><br> >> I also tried an independent RTEMS timer (rtems_timer_fire_after) a= s a<br> >> heartbeat, firing outside the USB bus lock. Both the USB timer and= the<br> >> independent heartbeat stopped simultaneously, confirming the entir= e ARM<br> >> core freezes =E2=80=94 not just the USB subsystem.<br> >><br> >> No RTEMS fatal error handler was triggered (I installed one via<br= > >> CONFIGURE_INITIAL_EXTENSIONS). This means it is NOT a standard ARM= <br> >> data abort =E2=80=94 the CPU simply stops executing, likely due to= an AHB bus<br> >> lockup caused by the DWC OTG controller hardware.<br> >><br> >> BCM2835 errata workarounds applied<br> >><br> >> Based on the Linux dwc2 driver (params.c, core.c) and the Ultibo<b= r> >> project documentation, I applied three workarounds:<br> >><br> >> 1. FIFO size cap: BCM2835 hardware reports 4096 words but only 408= 0<br> >>=C2=A0 =C2=A0 exist. Cap sc_fifo_size to 4080*4 bytes to prevent FI= FO overrun.<br> >>=C2=A0 =C2=A0 (In practice, the hardware on my board reported <= =3D 4080, so this<br> >>=C2=A0 =C2=A0 cap was NOT triggered.)<br> >><br> >> 2. GAHBCFG AHB burst configuration: Broadcom redefined bits [4:1] = of<br> >>=C2=A0 =C2=A0 GAHBCFG for AXI burst control. Linux dwc2 sets ahbcfg= =3D0x10 for<br> >>=C2=A0 =C2=A0 BCM2835. Changed from GAHBCFG_GLBLINTRMSK (0x01) to 0= x11.<br> >><br> >> 3. AHB idle wait after core reset: Added a loop waiting for<br> >>=C2=A0 =C2=A0 GRSTCTL_AHBIDLE after GRSTCTL_CSFTRST, plus 250ms set= tling delay<br> >>=C2=A0 =C2=A0 (Linux dwc2 uses 100+ ms).<br> >><br> >> Results with workarounds:<br> >> - Before: hangs after ~86 seconds (2 kHz tick)<br> >> - After:=C2=A0 hangs after ~175 seconds (2 kHz tick)<br> >><br> >> The workarounds approximately doubled the time-to-hang but did NOT= <br> >> eliminate it. All software counters remained healthy throughout.<b= r> >><br> >> The hang is caused most probably by a combination of factors:<br> >><br> >> 1. The DWC OTG controller on BCM2835/BCM2837 requires sub-125=C2= =B5s<br> >>=C2=A0 =C2=A0 interrupt response times for USB split transaction ph= ases (Start<br> >>=C2=A0 =C2=A0 Split =E2=86=92 Complete Split through the USB hub). = The RPi 3B+ Ethernet<br> >>=C2=A0 =C2=A0 goes through a USB hub (LAN7515), making every packet= a split<br> >>=C2=A0 =C2=A0 transaction.<br> >><br> >> 2. The Linux kernel addresses this with a dedicated FIQ (Fast Inte= rrupt<br> >>=C2=A0 =C2=A0 Request) handler (dwc_otg_fiq_fsm.c) that executes co= mplete split<br> >>=C2=A0 =C2=A0 transactions in FIQ context, bypassing the normal int= errupt stack.<br> >>=C2=A0 =C2=A0 Without FIQ, "certain USB devices become complet= ely unusable."<br> >><br> >> 3 The FreeBSD dwc_otg driver used by RTEMS handles all split<br> >>=C2=A0 =C2=A0 transactions in normal interrupt context. On RTEMS, t= he interrupt<br> >>=C2=A0 =C2=A0 filter and thread handler run back-to-back in the int= errupt server<br> >>=C2=A0 =C2=A0 task (nexus_intr_with_filter in rtems-kernel-nexus.c)= , with no<br> >>=C2=A0 =C2=A0 preemption point between them.<br> >><br> >> 4 When the controller's split transaction timing is violated, = it<br> >>=C2=A0 =C2=A0 enters an unrecoverable state that locks the AHB bus,= freezing the<br> >>=C2=A0 =C2=A0 entire ARM core including UART and system timers.<br> >><br> >> The timing correlation with tick rate confirms this: higher tick r= ate<br> >> =3D more frequent scheduling =3D more interrupt latency jitter =3D= faster<br> >> timing violation.<br> >><br> >> A proper fix requires most probably implementing FIQ-based split t= ransaction handling<br> >> in the RTEMS BSP for BCM2835/BCM2837, similar to what Linux does i= n<br> >> dwc_otg_fiq_fsm.c. This is a significant undertaking but is essent= ial<br> >> for reliable USB operation on RPi 3B+ (and RPi Zero 2 W, which use= s<br> >> the same SoC).<br> >><br> >> The GAHBCFG and reset sequence workarounds should also be applied = as<br> >> they improve stability.<br> >><br> >> Please give me your thoughts on that. Maybe it is easier to finali= se the BSP for RPi 4 and 5?<br> >><br> >> Best Regards<br> >> Mirek<br> >><br> >> =C5=9Br., 22 kwi 2026 o 13:17 John Howard <<a href=3D"mailto:ec= [email protected]" target=3D"_blank">[email protected]</a>> napi= sa=C5=82(a):<br> >>><br> >>> Fascinating. Thanks for that detailed report.<br> >>><br> >>> You indicated USB enumerating continues running in the backgro= und.<br> >>><br> >>> I am educated-guessing that a counter maximum is reached, and = then mistakenly breached. I would look for a test of that counter and corre= ct it from allowing greater-than comparing.<br> >>><br> >>> Let us know how it turns out.<br> >>><br> >>> I am developing an app for Raspberry Pi Zero 2 W (stripped-dow= n 3B+). I wasn't expecting any potential problem like this.<br> >>><br> >>> -- John<br> >>><br> >>> > On Apr 22, 2026, at 1:25=E2=80=AFPM, Miroslaw Dach <<a= href=3D"mailto:[email protected]" target=3D"_blank">miroslaw.dach@gm= ail.com</a>> wrote:<br> >>> ><br> >>> > =EF=BB=BF<br> >>> > Hi All,<br> >>> ><br> >>> > I'm running an EPICS ioc server (EPICS 7.0.10)=C2=A0 = with RTEMS 6.2 on a Raspberry Pi 3B+ with rtems-libbsd=C2=A0 (6-freebsd-14)= and<br> >>> > encountering a reproducible system hang after several min= utes of operation.<br> >>> > Through systematic elimination testing I've narrowed = the root cause to the<br> >>> > DWC OTG USB controller driver. I'd appreciate any rec= ommendations on how<br> >>> > to address this. I can of course use the EPICS ioc server= under linux on RPi but just tried to have the RTEMS - Hard real time syste= m which is much more deterministic.<br> >>> > The boot time for the RPi with EPICS/RTEMS is around 7 se= c which is extremely fast!<br> >>> ><br> >>> > Environment<br> >>> > -----------<br> >>> > - Board: Raspberry Pi 3B+ (BCM2837, boardrev a020d3)<br> >>> > - RTEMS: rtems-6.2 (ARM/ARMv4/raspberrypi2)<br> >>> > - Network stack: rtems-libbsd (RTEMS_BSD_CONFIG_BSP_CONFI= G + RTEMS_BSD_CONFIG_INIT)<br> >>> > - Ethernet: LAN7515 USB Ethernet (muge driver, via DWC OT= G)<br> >>> > - Application: EPICS IOC (but hang occurs with minimal/em= pty IOC as well)<br> >>> > - Console: UART serial (/dev/ttyS0)<br> >>> ><br> >>> > Symptom<br> >>> > -------<br> >>> > The system boots and runs normally, then the entire syste= m freezes =E2=80=94<br> >>> > including the UART serial console (which is not USB-depen= dent). No crash<br> >>> > message, no stack dump =E2=80=94 a hard hang requiring po= wer cycle.<br> >>> ><br> >>> > The time-to-hang depends on the system tick rate:<br> >>> > - CONFIGURE_MICROSECONDS_PER_TICK=3D500=C2=A0 (2 kHz): ha= ngs after ~2 minutes<br> >>> > - CONFIGURE_MICROSECONDS_PER_TICK=3D10000 (100 Hz): hangs= after ~5-6 minutes<br> >>> ><br> >>> > During the hang, the UART becomes completely unresponsive= , suggesting a<br> >>> > kernel-level deadlock or interrupt handler issue rather t= han an<br> >>> > application-level problem.<br> >>> ><br> >>> > Elimination testing performed<br> >>> > -----------------------------<br> >>> > I systematically disabled components to isolate the cause= . All tests below<br> >>> > used CONFIGURE_MICROSECONDS_PER_TICK=3D10000 (100 Hz):<br= > >>> ><br> >>> > 1. Disabled all EPICS database records (no record process= ing) -> still hangs<br> >>> > 2. Disabled periodic NTP sync (no socket operations) ->= ; still hangs<br> >>> > 3. Disabled IP configuration (no ifconfig, no route, no n= etwork traffic,<br> >>> >=C2=A0 =C2=A0 but USB/DWC OTG still initialised via RTEMS_= BSD_CONFIG_BSP_CONFIG)<br> >>> >=C2=A0 =C2=A0 -> still hangs (~5-6 min), USB hub enumer= ation continues in background:<br> >>> >=C2=A0 =C2=A0 =C2=A0 =C2=A0ugen1.2: <vendor 0x0424 prod= uct 0x2514> at usbus1<br> >>> >=C2=A0 =C2=A0 =C2=A0 =C2=A0uhub1 on uhub0<br> >>> >=C2=A0 =C2=A0 =C2=A0 =C2=A0...<br> >>> >=C2=A0 =C2=A0 Console output is garbled by concurrent USB = enumeration messages,<br> >>> >=C2=A0 =C2=A0 suggesting interrupt contention.<br> >>> > 4. Commented out RTEMS_BSD_CONFIG_BSP_CONFIG and the<br> >>> >=C2=A0 =C2=A0 #include <bsp/nexus-devices.h> to prev= ent DWC OTG initialisation<br> >>> >=C2=A0 =C2=A0 -> STABLE, ran for 14+ minutes with no ha= ng (test stopped manually)<br> >>> ><br> >>> > The libbsd software stack (loopback, sockets, telnetd) co= ntinues to<br> >>> > function in test 4 =E2=80=94 only the hardware BSP device= s (DWC OTG, muge,<br> >>> > uhub, ukphy) are excluded.<br> >>> ><br> >>> > Minimal reproduction<br> >>> > --------------------<br> >>> > Build an RTEMS 6.2 application for raspberrypi2 BSP with:= <br> >>> ><br> >>> >=C2=A0 =C2=A0#define RTEMS_BSD_CONFIG_BSP_CONFIG<br> >>> >=C2=A0 =C2=A0#define RTEMS_BSD_CONFIG_INIT<br> >>> >=C2=A0 =C2=A0#include <machine/rtems-bsd-config.h><b= r> >>> >=C2=A0 =C2=A0#include <bsp/nexus-devices.h>=C2=A0 = =C2=A0 =C2=A0/* in one translation unit */<br> >>> ><br> >>> >=C2=A0 =C2=A0#define CONFIGURE_MICROSECONDS_PER_TICK 10000= <br> >>> ><br> >>> > The application does not need to configure any network in= terface or<br> >>> > perform any USB transfers =E2=80=94 the DWC OTG hub polli= ng alone triggers<br> >>> > the hang after ~5-6 minutes.<br> >>> ><br> >>> > Commenting out RTEMS_BSD_CONFIG_BSP_CONFIG and the nexus-= devices.h<br> >>> > include eliminates the hang.<br> >>> ><br> >>> > Boot log (abbreviated, from hanging configuration)<br> >>> > --------------------------------------------------<br> >>> > RTEMS RPi 3B+ 1.3 (1GB) [00a020d3]<br> >>> > nexus0: <RTEMS Nexus device><br> >>> > dwcotg0: <DWC OTG 2.0 integrated USB controller> on= nexus0<br> >>> > usbus1 on dwcotg0<br> >>> > usbus1: 480Mbps High Speed USB v2.0<br> >>> > ugen1.1: <DWCOTG OTG Root HUB> at usbus1<br> >>> > uhub0 on usbus1<br> >>> > uhub0: <DWCOTG OTG Root HUB, class 9/0, rev 2.00/1.00,= addr 1> on usbus1<br> >>> > uhub0: 1 port with 1 removable, self powered<br> >>> > ugen1.2: <vendor 0x0424 product 0x2514> at usbus1<b= r> >>> > uhub1 on uhub0<br> >>> > uhub1: <vendor 0x0424 product 0x2514, class 9/0, rev 2= .00/b.b3, addr 2> on usbus1<br> >>> > uhub1: 4 ports with 3 removable, self powered<br> >>> > ugen1.3: <vendor 0x0424 product 0x2514> at usbus1<b= r> >>> > uhub2 on uhub1<br> >>> > uhub2: <vendor 0x0424 product 0x2514, class 9/0, rev 2= .00/b.b3, addr 3> on usbus1<br> >>> > uhub2: 3 ports with 2 removable, self powered<br> >>> > ugen1.4: <vendor 0x0424 product 0x7800> at usbus1<b= r> >>> > muge0: <vendor 0x0424 product 0x7800, rev 2.10/3.00, a= ddr 4> on usbus1<br> >>> > muge0: Chip ID 0x7800 rev 0002<br> >>> > miibus0: <MII bus> on muge0<br> >>> > ukphy0: <Generic IEEE 802.3u media interface> PHY 1= on miibus0<br> >>> > info: ue0: <USB Ethernet> on muge0<br> >>> > [system hangs after ~5-6 minutes, UART unresponsive]<br> >>> ><br> >>> > Questions<br> >>> > ---------<br> >>> > 1. Is this a known issue with the DWC OTG driver on RPi 3= B+?<br> >>> > 2. Are there any configuration options (hub polling inter= val, interrupt<br> >>> >=C2=A0 =C2=A0 coalescing, DMA settings) that might work ar= ound the problem?<br> >>> > 3. Would a newer version of rtems-libbsd contain fixes fo= r this?<br> >>> > 4. Is there an alternative Ethernet driver approach for R= Pi 3B+ that<br> >>> >=C2=A0 =C2=A0 avoids the DWC OTG USB path?<br> >>> > 5. Is there any known project which uses RPi with RTEMS?<= br> >>> > (it looks like that the option to run RTEMS on RPi4 or RP= i 5 can not be considered since the BSP in RTEMS kernel=C2=A0 is not yet fi= nalised.<br> >>> > The RPi4 or RPi 5 would be much better candidates vs RPi = 3 B+ since they use direct connection to Ethernet instead of the USB-Ethern= et)<br> >>> ><br> >>> > Thank you for any guidance.<br> >>> ><br> >>> > Mirek<br> >>> ><br> >>> ><br> >>> > _______________________________________________<br> >>> > users mailing list<br> >>> > <a href=3D"mailto:[email protected]" target=3D"_blank">user= [email protected]</a><br> >>> > <a href=3D"http://lists.rtems.org/mailman/listinfo/users"= rel=3D"noreferrer" target=3D"_blank">http://lists.rtems.org/mailman/listin= fo/users</a><br> ><br> > _______________________________________________<br> > users mailing list<br> > <a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</= a><br> > <a href=3D"http://lists.rtems.org/mailman/listinfo/users" rel=3D"noref= errer" target=3D"_blank">http://lists.rtems.org/mailman/listinfo/users</a><= br> </blockquote></div> --00000000000054019f06525ac885-- --===============9109511382912199927== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ users mailing list [email protected] http://lists.rtems.org/mailman/listinfo/users --===============9109511382912199927==--