Re: Some evidence about the PowerMac G5 multiprocessor boot hang ups with the modern VM_MAX_KERNEL_ADDRESS value [found more staging info]

[I needed to allow more time after the 2 resets before
having CPU 0 look at the memory. It was reporting
older values instead of my added writes. The odd
non-zero value was from before the activity of interest.]

I start with the new result found, then give supporting
material.

I've now seen hangs with:

*(unsigned long*)0xc0000000000000f0)=0x10

for CPU 3. So the following completed:

void
cpudep_ap_early_bootstrap(void)
{
#ifndef __powerpc64__
        register_t reg;
#endif

        switch (mfpvr() >> 16) {
        case IBM970:
        case IBM970FX:
        case IBM970MP:
                /* Restore HID4 and HID5, which are necessary for the MMU */

#ifdef __powerpc64__
                mtspr(SPR_HID4, bsp_state[2]); powerpc_sync(); isync();
                mtspr(SPR_HID5, bsp_state[3]); powerpc_sync(); isync();
#else
                __asm __volatile("ld %0, 16(%2); sync; isync;   \
                    mtspr %1, %0; sync; isync;"
                    : "=r"(reg) : "K"(SPR_HID4), "b"(bsp_state));
                __asm __volatile("ld %0, 24(%2); sync; isync;   \
                    mtspr %1, %0; sync; isync;"
                    : "=r"(reg) : "K"(SPR_HID5), "b"(bsp_state));
#endif
                powerpc_sync();
                break;
        case IBMPOWER8:
        case IBMPOWER8E:
        case IBMPOWER9:
#ifdef __powerpc64__
                if (mfmsr() & PSL_HV) {
                        isync();
                        /*
                         * Direct interrupts to SRR instead of HSRR and
                         * reset LPCR otherwise
                         */
                        mtspr(SPR_LPID, 0);
                        isync();
 
                        mtspr(SPR_LPCR, lpcr);
                        isync();
                }
#endif
                break;
        }

        __asm __volatile("mtsprg 0, %0" :: "r"(ap_pcpu));
        powerpc_sync();

        *(unsigned long*)0xc0000000000000f0 = 0x10; // HACK!!!
        powerpc_sync(); // HACK!!!
}

but the following (and later) did not complete:

void
pmap_cpu_bootstrap(int ap)
{
        /*      
         * No KTR here because our console probably doesn't work yet
         */
        
        return (MMU_CPU_BOOTSTRAP(mmu_obj, ap));
 
        *(unsigned long*)0xc0000000000000f0 = 0x20; // HACK!!!
        powerpc_sync(); // HACK!!!
}


Background for reference relative to showing
the intended values. . .

I now have:

        *rstvec = 4;
        powerpc_sync();
        (void)(*rstvec);
        powerpc_sync();
        DELAY(1);
        *rstvec = 0;
        powerpc_sync();
        (void)(*rstvec);
        powerpc_sync();

        if (bootverbose) // HACK!!!
                printf("After reset 4&0 for CPU %d, hwref=%jx, awake=%x, *(unsigned long*)0xc0000000000000e0=0x%jx, *(unsigned long*)0xc0000000000000f0=0x%jx\n",
                    pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
                    pc->pc_awake,(uintmax_t)*(unsigned long*)0xc0000000000000e0,(uintmax_t)*(unsigned long*)0xc0000000000000f0);

        timeout = 10000;
        while (!pc->pc_awake && timeout--)
                DELAY(100);

        if (bootverbose) // HACK!!!
                printf("After attempted wait for awake CPU %d, hwref=%jx, awake=%x, *(unsigned long*)0xc0000000000000e0=0x%jx, *(unsigned long*)0xc0000000000000f0=0x%jx\n",
                    pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
                    pc->pc_awake,(uintmax_t)*(unsigned long*)0xc0000000000000e0,(uintmax_t)*(unsigned long*)0xc0000000000000f0);

The 2nd printf shows the expected values but the first
above shows old memory values.

The memory at 0xc0000000000000e0 hold the get_pcpu() result.
The memory at 0xc0000000000000f0 hold the 0x?? values that I'd reported earlier
although I've added a 0x5E but really use the value at 0xc0000000000000e0 to
check since 0x5e would be replaced by 0x51:

        struct pcpu* thepcpu = get_pcpu(); // HACK!!!
        *(struct pcpu**)0xc0000000000000e0 = thepcpu; // HACK!!!

        if (  thepcpu==&__pcpu[1] // HACK!!!
           || thepcpu==&__pcpu[2]
           || thepcpu==&__pcpu[3]
           )
            *(unsigned long*)0xc0000000000000f0 = 0x5F; // HACK!!!
        else
            *(unsigned long*)0xc0000000000000f0 = 0x5E; // HACK!!!
        powerpc_sync(); // HACK!!!

        PCPU_SET(awake, 1);
        __asm __volatile("msync; isync");

        *(unsigned long*)0xc0000000000000f0 = 0x51; // HACK!!!
        powerpc_sync(); // HACK!!!

        while (ap_letgo == 0)
                __asm __volatile("or 31,31,31");
        __asm __volatile("or 6,6,6");



For reference, a successful boot now looks like:

Adding CPU 0, hwref=cd38, awake=1
Trying to mount root from ufs:/dev/ufs/FBSDG5L2rootfs [rw,noatime]...
Waking up CPU 3 (dev=c480)
powermac_smp_start_cpu 's OF_getprop for CPU 3, hwref=c480, awake=0: res=4, reset=8c
powermac_smp_start_cpu for CPU 3, hwref=c480, awake=0: rstvec_virtbase=0xe000000087fd2000
powermac_smp_start_cpu for CPU 3, hwref=c480, awake=0: rstvec=0xe000000087fd208c
Before reset 4&0 for CPU 3, hwref=c480, awake=0
After reset 4&0 for CPU 3, hwref=c480, awake=0, *(unsigned long*)0xc0000000000000e0=0x0, *(unsigned long*)0xc0000000000000f0=0x0
After attempted wait for awake CPU 3, hwref=c480, awake=1, *(unsigned long*)0xc0000000000000e0=0xc0000000016c6100, *(unsigned long*)0xc0000000000000f0=0x51
cpu_mp_unleash attempting to wait for pc_awake: CPU 3, hwref=c480, awake=1
cpu_mp_unleash after platform_smp_start_cpu and waiting: CPU 3, hwref=c480, awake=1
Adding CPU 3, hwref=c480, awake=1
Waking up CPU 2 (dev=c768)
powermac_smp_start_cpu 's OF_getprop for CPU 2, hwref=c768, awake=0: res=4, reset=8b
powermac_smp_start_cpu for CPU 2, hwref=c768, awake=0: rstvec=0xe000000087fd208b
Before reset 4&0 for CPU 2, hwref=c768, awake=0
After reset 4&0 for CPU 2, hwref=c768, awake=0, *(unsigned long*)0xc0000000000000e0=0xc0000000016c6100, *(unsigned long*)0xc0000000000000f0=0x51
After attempted wait for awake CPU 2, hwref=c768, awake=1, *(unsigned long*)0xc0000000000000e0=0xc0000000016c5100, *(unsigned long*)0xc0000000000000f0=0x51
cpu_mp_unleash attempting to wait for pc_awake: CPU 2, hwref=c768, awake=1
cpu_mp_unleash after platform_smp_start_cpu and waiting: CPU 2, hwref=c768, awake=1
Adding CPU 2, hwref=c768, awake=1
Waking up CPU 1 (dev=ca50)
powermac_smp_start_cpu 's OF_getprop for CPU 1, hwref=ca50, awake=0: res=4, reset=8a
powermac_smp_start_cpu for CPU 1, hwref=ca50, awake=0: rstvec=0xe000000087fd208a
Before reset 4&0 for CPU 1, hwref=ca50, awake=0
After reset 4&0 for CPU 1, hwref=ca50, awake=0, *(unsigned long*)0xc0000000000000e0=0xc0000000016c5100, *(unsigned long*)0xc0000000000000f0=0x51
After attempted wait for awake CPU 1, hwref=ca50, awake=1, *(unsigned long*)0xc0000000000000e0=0xc0000000016c4100, *(unsigned long*)0xc0000000000000f0=0x51
cpu_mp_unleash attempting to wait for pc_awake: CPU 1, hwref=ca50, awake=1
cpu_mp_unleash after platform_smp_start_cpu and waiting: CPU 1, hwref=ca50, awake=1
Adding CPU 1, hwref=ca50, awake=1
machdep_ap_bootstrap before ap_boot_mtx lock: AP CPU #2 launched
machdep_ap_bootstrap before ap_boot_mtx lock: AP CPU #1 launched
machdep_ap_bootstrap before ap_boot_mtx lock: AP CPU #3 launched
SMP: AP CPU #2 launched
SMP: AP CPU #1 launched
SMP: AP CPU #3 launched
machdep_ap_bootstrap after smp_started!=0: AP CPU #2 launched
machdep_ap_bootstrap after smp_started!=0: AP CPU #1 launched
machdep_ap_bootstrap after smp_started!=0: AP CPU #3 launched




===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

_______________________________________________
[email protected] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ppc
To unsubscribe, send any mail to "[email protected]"