Re: Anyone working on Octeon SMP?

On Fri, Apr 10, 2026 at 11:45=E2=80=AFPM Nick Hudson <[email protected]=
> wrote:
>
> On 10/04/2026 11:18, Kevin Bowling wrote:
> > On Mon, Apr 6, 2026 at 6:42=E2=80=AFAM Nick Hudson <[email protected].=
uk> wrote:
> >>
> >> On 06/04/2026 10:49, Kevin Bowling wrote:
> >>> On Sat, Mar 28, 2026 at 8:28=E2=80=AFAM Andrew Parker <[email protected]=
t> wrote:
> >>>>
> >>>> On Thursday, 19 March 2026 23:31:36 EDT Kevin Bowling wrote:
> >>>>> On Sun, Feb 2, 2025 at 6:52=E2=80=AFAM Andrew Parker <[email protected]=
et> wrote:
> >>>>>> On 1/29/25 06:23, Nick Hudson wrote:
> >>>>>>> On 20/12/2024 22:01, Andrew Parker wrote:
> >>>>>>>> Hi, I've been working towards getting a MULTIPROCESSOR build mor=
e
> >>>>>>>> stable
> >>>>>>>> on my ER4 and would like to know if anyone else may be working o=
n the
> >>>>>>>> same?
> >>>>>>>>
> >>>>>>>> I've found a few areas that were causing instability on my machi=
ne and
> >>>>>>>> have some patches that help (but mostly just for debugging at th=
is
> >>>>>>>> point). If there's any interest in teaming up and exchanging ide=
as or
> >>>>>>>> patches for SMP on Octeon please let me know.
> >>>>>>>
> >>>>>>> Sure.
> >>>>>>>
> >>>>>>> I've dropped the ball on this and said I had a couple of fixes in=
 mind,
> >>>>>>> but done nothing to share them. Hopefully we can make it stable.
> >>>>>>>
> >>>>>>> Nick
> >>>>>>
> >>>>>> Great!  I'm curious about what you have in mind for improvement.  =
I've
> >>>>>> mostly been looking around TLB invalidation and perhaps a missing =
memory
> >>>>>> barrier.
> >>>>>>
> >>>>>> Anyway, I'll work on getting some stuff cleaned up and contact dir=
ectly
> >>>>>> if that works for you.
> >>>>>
> >>>>> I'm interested in this as well, is there anything to share around
> >>>>> current status or issues as well as if anything is pending out of
> >>>>> tree?
> >>>>
> >>>> I hoped to spend more time on this over the winter but ended up movi=
ng and
> >>>> some of my networking gear is still packed up.
> >>>>
> >>>> It's been a slow process getting my test environment setup again but=
 it would
> >>>> be great to pick this back up...especially if there's continued inte=
rest in
> >>>> it.
> >>>>
> >>>> Give me a week or two to see what I can dig up and I'll be in touch.
> >>> I've made some progress but it turned into a much deeper hole than I
> >>> was anticipating.  I can increase SMP stability with some changes in
> >>> pmap_tlb.c to add some icache syncs
> >>
> >> I have a change I'll commit soon to handle EXECness (more) correctly.
>
> This is committed
>
> https://mail-index.netbsd.org/source-changes/2026/04/10/msg161522.html
>m
>
> >>
> >> Am I right in thinking at least some octeon processors icaches are VIV=
T?
> >> I've forgotten most of the mips stuff I knew... If so there will be mo=
re flushing
> >> required for VIVT.
> > VIPT.  But it has an assortment of fun issues.
>
> fun issues?

It seems to require pretty deliberate management that is different
than other MIPS and archs.

I was able to get SMP stabilized to my own satisfaction last night, in
so far as it can hold up cnmac driver modifications which is what I
was originally started with.

There are a few categories of mandatory fixes:  membar_release is
missing a SYNC_PLUNGER (second syncw), INT_MASKs critically missing in
octeon_intr.c, and a variety of locore fixups.  Beyond that I ended up
redoing octeon_intr.c to distribute interrupts and fix my previous
octeon III patch.  I have changes to the PMAP_TLB_NEED_SHOOTDOWN path
primarily that I am least confident about.  I'll try and organize
everything into a more deliberate patch series, right now it is pretty
messy with attempts and debugging.

>
>
> >>> and guarding around an assert that
> >>> is easy to trigger
> >>> @@ -735,7 +767,9 @@ pmap_tlb_shootdown_bystanders(pmap_t pm)
> >>>                         * And best of all, we avoid an IPI.
> >>>                         */
> >>>                        KASSERT(!kernel_p);
> >>> -                       pmap_tlb_pai_reset(ti, pai, pm);
> >>> +                       if (pai->pai_asid > KERNEL_PID) {
> >>> +                               pmap_tlb_pai_reset(ti, pai, pm);
> >>> +                       }
> >>
> >>
> >> you mean this KASSERT in pmap_tlb_pai_reset?
> > Yeah.  But let me think harder about this.
> >>    252  /*
> >>    253   * We must have an ASID but it must not be onproc (on a proces=
sor).
> >>    254   */
> >>    255  KASSERT(pai->pai_asid > KERNEL_PID);
> >>
> >>> I think there are a variety of pmap changes needed.  But I wonder if
> >>> any MIPS SMP or other PMAP_TLB_NEED_SHOOTDOWN has been heavily
> >>> exercised?
> >>
> >> Almost certainly not.
> > Good to know, I was a bit shy about looking at the MI pmap at first.
> >>> IPI and TLB stuff is probably "simpler" on ARM.  On
> >>> FreeBSD (13), OpenBSD, Linux the MIPS TLB shootdowns are synchronous.
> >>
> >> No other NetBSD architecture (not sure of powerpc booke status actuall=
y) uses
> >> the PMAP_TLB_NEED_SHOOTDOWN stuff.
> >>
> >> I'm about to switch aarch64 to sys/uvm/pmap which uses architecture de=
fined
> >> broadcast TLB operations. RISC-V uses SBI remote fence operations.
> > Arch defined broadcast sounds very suitable for what we'll need to do
> > here.  Is that out of tree?
>
> Arm architecture defined the broadcast TLB operations long ago. Others ha=
ve caught up...
>
> MI PMAP uses tlb_* functions
> https://nxr.netbsd.org/xref/src/sys/arch/aarch64/aarch64/aarch64_tlb.c#84
> https://nxr.netbsd.org/xref/src/sys/arch/aarch64/aarch64/cpufunc_asm_armv=
8.S#221
>
>
>