Re: Anyone working on Octeon SMP?

On 16/04/2026 13:24, Andrew Parker wrote:
> On Monday, 13 April 2026 12:58:34 EDT Kevin Bowling wrote:
>> On Fri, Apr 10, 2026 at 11:45=E2=80=AFPM Nick Hudson =
<[email protected]> wrote:
>>> On 10/04/2026 11:18, Kevin Bowling wrote:
>>>> On Mon, Apr 6, 2026 at 6:42=E2=80=AFAM Nick Hudson =
<[email protected]>
> wrote:
>>>>> On 06/04/2026 10:49, Kevin Bowling wrote:
>>>>>> On Sat, Mar 28, 2026 at 8:28=E2=80=AFAM Andrew Parker =
<[email protected]> wrote:
>>>>>>> On Thursday, 19 March 2026 23:31:36 EDT Kevin Bowling wrote:
>>>>>>>> On Sun, Feb 2, 2025 at 6:52=E2=80=AFAM Andrew Parker =
<[email protected]>
> wrote:
>>>>>>>>> On 1/29/25 06:23, Nick Hudson wrote:
>>>>>>>>>> On 20/12/2024 22:01, Andrew Parker wrote:
>>>>>>>>>>> Hi, I've been working towards getting a MULTIPROCESSOR build =
more
>>>>>>>>>>> stable
>>>>>>>>>>> on my ER4 and would like to know if anyone else may be =
working on
>>>>>>>>>>> the
>>>>>>>>>>> same?
>>>>>>>>>>>=20
>>>>>>>>>>> I've found a few areas that were causing instability on my
>>>>>>>>>>> machine and
>>>>>>>>>>> have some patches that help (but mostly just for debugging =
at
>>>>>>>>>>> this
>>>>>>>>>>> point). If there's any interest in teaming up and exchanging
>>>>>>>>>>> ideas or
>>>>>>>>>>> patches for SMP on Octeon please let me know.
>>>>>>>>>>=20
>>>>>>>>>> Sure.
>>>>>>>>>>=20
>>>>>>>>>> I've dropped the ball on this and said I had a couple of =
fixes in
>>>>>>>>>> mind,
>>>>>>>>>> but done nothing to share them. Hopefully we can make it =
stable.
>>>>>>>>>>=20
>>>>>>>>>> Nick
>>>>>>>>>=20
>>>>>>>>> Great!  I'm curious about what you have in mind for =
improvement.
>>>>>>>>> I've
>>>>>>>>> mostly been looking around TLB invalidation and perhaps a =
missing
>>>>>>>>> memory
>>>>>>>>> barrier.
>>>>>>>>>=20
>>>>>>>>> Anyway, I'll work on getting some stuff cleaned up and contact
>>>>>>>>> directly
>>>>>>>>> if that works for you.
>>>>>>>>=20
>>>>>>>> I'm interested in this as well, is there anything to share =
around
>>>>>>>> current status or issues as well as if anything is pending out =
of
>>>>>>>> tree?
>>>>>>>=20
>>>>>>> I hoped to spend more time on this over the winter but ended up
>>>>>>> moving and
>>>>>>> some of my networking gear is still packed up.
>>>>>>>=20
>>>>>>> It's been a slow process getting my test environment setup again =
but
>>>>>>> it would be great to pick this back up...especially if there's
>>>>>>> continued interest in it.
>>>>>>>=20
>>>>>>> Give me a week or two to see what I can dig up and I'll be in =
touch.
>>>>>>=20
>>>>>> I've made some progress but it turned into a much deeper hole =
than I
>>>>>> was anticipating.  I can increase SMP stability with some changes =
in
>>>>>> pmap_tlb.c to add some icache syncs
>>>>>=20
>>>>> I have a change I'll commit soon to handle EXECness (more) =
correctly.
>>>=20
>>> This is committed
>>>=20
>>> =
https://mail-index.netbsd.org/source-changes/2026/04/10/msg161522.html
>>>=20
>>> m
>>>=20
>>>>> Am I right in thinking at least some octeon processors icaches are
>>>>> VIVT?
>>>>> I've forgotten most of the mips stuff I knew... If so there will =
be
>>>>> more flushing required for VIVT.
>>>>=20
>>>> VIPT.  But it has an assortment of fun issues.
>>>=20
>>> fun issues?
>>=20
>> It seems to require pretty deliberate management that is different
>> than other MIPS and archs.
>>=20
>> I was able to get SMP stabilized to my own satisfaction last night, =
in
>> so far as it can hold up cnmac driver modifications which is what I
>> was originally started with.
>>=20
>> There are a few categories of mandatory fixes:  membar_release is
>> missing a SYNC_PLUNGER (second syncw), INT_MASKs critically missing =
in
>> octeon_intr.c, and a variety of locore fixups.  Beyond that I ended =
up
>> redoing octeon_intr.c to distribute interrupts and fix my previous
>> octeon III patch.  I have changes to the PMAP_TLB_NEED_SHOOTDOWN path
>> primarily that I am least confident about.  I'll try and organize
>> everything into a more deliberate patch series, right now it is =
pretty
>> messy with attempts and debugging.
> I was finally able to boot up my ER-4 last night.  I don't have =
anything to add
> (yet) to the pmap discussion here except I'm guessing I'm in similar =
place you
> are with it. Things are 'stable' mostly through a bunch of extra TLB =
flushes.
> The other change that resulted in a more stable userland for me is =
what
> appears to be a missing memory barrier around cpu_lwp_setprivate().  =
The one I
> added in lwp.c doesn't seem 100% correct but this does resolve =
instability
> that's easily reproduced in unbound (using multiple threads) and =
occasionally
> sshd:
>  diff --git a/sys/arch/mips/mips/cpu_subr.c =
b/sys/arch/mips/mips/cpu_subr.c
> index a80304908774..df854cae4254 100644
> --- a/sys/arch/mips/mips/cpu_subr.c
> +++ b/sys/arch/mips/mips/cpu_subr.c
> @@ -1051,11 +1051,11 @@ cpu_vmspace_exec(lwp_t *l, vaddr_t start, =
vaddr_t end)
>  int
>  cpu_lwp_setprivate(lwp_t *l, void *v)
>  {
> -
>  #if (MIPS32R2 + MIPS64R2) > 0
>         if (l =3D=3D curlwp && MIPS_HAS_USERLOCAL) {
>                 mipsNN_cp0_userlocal_write(v);
>         }
> +       membar_sync();
>  #endif
>         return 0;
>  }
> diff --git a/sys/kern/sys_lwp.c b/sys/kern/sys_lwp.c
> index 7c4e4f27ad23..24cc3315f3e4 100644
> --- a/sys/kern/sys_lwp.c
> +++ b/sys/kern/sys_lwp.c
> @@ -187,7 +187,7 @@ sys__lwp_self(struct lwp *l, const void *v, =
register_t
> *retval)
>  int
>  sys__lwp_getprivate(struct lwp *l, const void *v, register_t *retval)
>  {
> -
> +        membar_sync();
>         *retval =3D (uintptr_t)l->l_private;
>         return 0;
>  }

Thanks for testing.

I can=E2=80=99t say I understand why these memory are needed. Can you =
explain, please?