Re: Some evidence about the PowerMac G5 multiprocessor boot hang ups with the modern VM_MAX_KERNEL_ADDRESS value [found yet more staging info]

[I referred to 2 resets where I should have referred to
between the last reset and the ps_awake loop. So fixing
that .  . .]

On 2019-Feb-16, at 13:15, Mark Millard <marklmi at yahoo.com> wrote:

> [I got a successful boot and so report its messages
> from starting the CPUs. It has one interesting value
> display.]
> 
> On 2019-Feb-16, at 12:49, Mark Millard <marklmi at yahoo.com> wrote:
> 
>> [I added to moea64_cpu_bootstrap_native to see
>> more staging infomrtion.]
>> 
>> On 2019-Feb-16, at 12:07, Mark Millard <marklmi at yahoo.com> wrote:
>> 
>>> [I needed to allow more time after the 2 resets before
>>> having CPU 0 look at the memory. It was reporting
>>> older values instead of my added writes. The odd
>>> non-zero value was from before the activity of interest.]
>>> 
>>> I start with the new result found, then give supporting
>>> material.
>>> 
>>> I've now seen hangs with:
>>> 
>>> *(unsigned long*)0xc0000000000000f0)=0x10
>>> 
>>> for CPU 3. So the following completed:
>>> 
>>> void
>>> cpudep_ap_early_bootstrap(void)
>>> {
>>> #ifndef __powerpc64__
>>>      register_t reg;
>>> #endif
>>> 
>>>      switch (mfpvr() >> 16) {
>>>      case IBM970:
>>>      case IBM970FX:
>>>      case IBM970MP:
>>>              /* Restore HID4 and HID5, which are necessary for the MMU */
>>> 
>>> #ifdef __powerpc64__
>>>              mtspr(SPR_HID4, bsp_state[2]); powerpc_sync(); isync();
>>>              mtspr(SPR_HID5, bsp_state[3]); powerpc_sync(); isync();
>>> #else
>>>              __asm __volatile("ld %0, 16(%2); sync; isync;   \
>>>                  mtspr %1, %0; sync; isync;"
>>>                  : "=r"(reg) : "K"(SPR_HID4), "b"(bsp_state));
>>>              __asm __volatile("ld %0, 24(%2); sync; isync;   \
>>>                  mtspr %1, %0; sync; isync;"
>>>                  : "=r"(reg) : "K"(SPR_HID5), "b"(bsp_state));
>>> #endif
>>>              powerpc_sync();
>>>              break;
>>>      case IBMPOWER8:
>>>      case IBMPOWER8E:
>>>      case IBMPOWER9:
>>> #ifdef __powerpc64__
>>>              if (mfmsr() & PSL_HV) {
>>>                      isync();
>>>                      /*
>>>                       * Direct interrupts to SRR instead of HSRR and
>>>                       * reset LPCR otherwise
>>>                       */
>>>                      mtspr(SPR_LPID, 0);
>>>                      isync();
>>> 
>>>                      mtspr(SPR_LPCR, lpcr);
>>>                      isync();
>>>              }
>>> #endif
>>>              break;
>>>      }
>>> 
>>>      __asm __volatile("mtsprg 0, %0" :: "r"(ap_pcpu));
>>>      powerpc_sync();
>>> 
>>>      *(unsigned long*)0xc0000000000000f0 = 0x10; // HACK!!!
>>>      powerpc_sync(); // HACK!!!
>>> }
>>> 
>>> but the following (and later) did not complete:
>>> 
>>> void
>>> pmap_cpu_bootstrap(int ap)
>>> {
>>>      /*      
>>>       * No KTR here because our console probably doesn't work yet
>>>       */
>>> 
>>>      return (MMU_CPU_BOOTSTRAP(mmu_obj, ap));
>>> 
>>>      *(unsigned long*)0xc0000000000000f0 = 0x20; // HACK!!!
>>>      powerpc_sync(); // HACK!!!
>>> }
>>> 
>>> 
>>> . . .
>> 
>> The below additions lead to moea64_cpu_bootstrap_native
>> lead to:
>> 
>> *(unsigned long*)0xc0000000000000f0)=0x25
>> 
>> which indicates that moea64_cpu_bootstrap_native
>> got to its end but pmap_cpu_bootstrap (the caller
>> via MMU_CPU_BOOTSTRAP) did not record its:
>> 
>> *(unsigned long*)0xc0000000000000f0 = 0x20;
>> 
>> from after the call. moea64_cpu_bootstrap_native
>> (and MMU_CPU_BOOTRAP) seems to have trouble
>> returning to pmap_cpu_bootstrap.
>> 
>> 
>> The below // HACK!!! lines are what I added:
>> 
>> static void
>> moea64_cpu_bootstrap_native(mmu_t mmup, int ap)
>> {
>>       int i = 0;
>>       #ifdef __powerpc64__
>>       struct slb *slb = PCPU_GET(aim.slb);
>>       register_t seg0;
>>       #endif
>> 
>>       /*
>>        * Initialize segment registers and MMU
>>        */
>> 
>>       mtmsr(mfmsr() & ~PSL_DR & ~PSL_IR);
>> 
>>       *(unsigned long*)0xc0000000000000f0 = 0x21; // HACK!!!
>>       powerpc_sync(); // HACK!!!
>> 
>>       /*
>>        * Install kernel SLB entries
>>        */
>> 
>>       #ifdef __powerpc64__
>>               __asm __volatile ("slbia");
>>               __asm __volatile ("slbmfee %0,%1; slbie %0;" : "=r"(seg0) :
>>                   "r"(0));
>> 
>>               *(unsigned long*)0xc0000000000000f0 = 0x22; // HACK!!!
>>               powerpc_sync(); // HACK!!!
>> 
>>               for (i = 0; i < n_slbs; i++) {
>>                       if (!(slb[i].slbe & SLBE_VALID))
>>                               continue;
>> 
>>                       __asm __volatile ("slbmte %0, %1" ::
>>                           "r"(slb[i].slbv), "r"(slb[i].slbe));
>>               }
>> 
>>               *(unsigned long*)0xc0000000000000f0 = 0x23; // HACK!!!
>>               powerpc_sync(); // HACK!!!
>>       #else
>>               for (i = 0; i < 16; i++)
>>                       mtsrin(i << ADDR_SR_SHFT, kernel_pmap->pm_sr[i]);
>>       #endif
>> 
>>       /*
>>        * Install page table
>>        */
>> 
>>       __asm __volatile ("ptesync; mtsdr1 %0; isync"
>>           :: "r"(((uintptr_t)moea64_pteg_table & ~DMAP_BASE_ADDRESS)
>>                    | (uintptr_t)(flsl(moea64_pteg_mask >> 11))));
>> 
>>       *(unsigned long*)0xc0000000000000f0 = 0x24; // HACK!!!
>>       powerpc_sync(); // HACK!!!
>> 
>>       tlbia();
>> 
>>       *(unsigned long*)0xc0000000000000f0 = 0x25; // HACK!!!
>>       powerpc_sync(); // HACK!!!
>> }
>> 
> 
> From a successful boot, for reference:
> 
> Adding CPU 0, hwref=cd38, awake=1
> Trying to mount root from ufs:/dev/ufs/FBSDG5L2rootfs [rw,noatime]...
> Waking up CPU 3 (dev=c480)
> powermac_smp_start_cpu 's OF_getprop for CPU 3, hwref=c480, awake=0: res=4, reset=8c
> powermac_smp_start_cpu for CPU 3, hwref=c480, awake=0: rstvec_virtbase=0xe000000087fd2000
> powermac_smp_start_cpu for CPU 3, hwref=c480, awake=0: rstvec=0xe000000087fd208c
> Before reset 4&0 for CPU 3, hwref=c480, awake=0
> After reset 4&0 for CPU 3, hwref=c480, awake=0, *(unsigned long*)0xc0000000000000e0=0x0, *(unsigned long*)0xc0000000000000f0=0x25
> After attempted wait for awake CPU 3, hwref=c480, awake=1, *(unsigned long*)0xc0000000000000e0=0xc0000000016c6100, *(unsigned long*)0xc0000000000000f0=0x51
> cpu_mp_unleash attempting to wait for pc_awake: CPU 3, hwref=c480, awake=1
> cpu_mp_unleash after platform_smp_start_cpu and waiting: CPU 3, hwref=c480, awake=1
> Adding CPU 3, hwref=c480, awake=1
> Waking up CPU 2 (dev=c768)
> powermac_smp_start_cpu 's OF_getprop for CPU 2, hwref=c768, awake=0: res=4, reset=8b
> powermac_smp_start_cpu for CPU 2, hwref=c768, awake=0: rstvec=0xe000000087fd208b
> Before reset 4&0 for CPU 2, hwref=c768, awake=0
> After reset 4&0 for CPU 2, hwref=c768, awake=0, *(unsigned long*)0xc0000000000000e0=0xc0000000016c6100, *(unsigned long*)0xc0000000000000f0=0x51
> After attempted wait for awake CPU 2, hwref=c768, awake=1, *(unsigned long*)0xc0000000000000e0=0xc0000000016c5100, *(unsigned long*)0xc0000000000000f0=0x51
> cpu_mp_unleash attempting to wait for pc_awake: CPU 2, hwref=c768, awake=1
> cpu_mp_unleash after platform_smp_start_cpu and waiting: CPU 2, hwref=c768, awake=1
> Adding CPU 2, hwref=c768, awake=1
> Waking up CPU 1 (dev=ca50)
> powermac_smp_start_cpu 's OF_getprop for CPU 1, hwref=ca50, awake=0: res=4, reset=8a
> powermac_smp_start_cpu for CPU 1, hwref=ca50, awake=0: rstvec=0xe000000087fd208a
> Before reset 4&0 for CPU 1, hwref=ca50, awake=0
> After reset 4&0 for CPU 1, hwref=ca50, awake=0, *(unsigned long*)0xc0000000000000e0=0xc0000000016c5100, *(unsigned long*)0xc0000000000000f0=0x51
> After attempted wait for awake CPU 1, hwref=ca50, awake=1, *(unsigned long*)0xc0000000000000e0=0xc0000000016c4100, *(unsigned long*)0xc0000000000000f0=0x51
> cpu_mp_unleash attempting to wait for pc_awake: CPU 1, hwref=ca50, awake=1
> cpu_mp_unleash after platform_smp_start_cpu and waiting: CPU 1, hwref=ca50, awake=1
> Adding CPU 1, hwref=ca50, awake=1
> machdep_ap_bootstrap before ap_boot_mtx lock: AP CPU #3 launched
> machdep_ap_bootstrap before ap_boot_mtx lock: AP CPU #2 launched
> machdep_ap_bootstrap before ap_boot_mtx lock: AP CPU #1 launched
> SMP: AP CPU #3 launched
> SMP: AP CPU #2 launched
> SMP: AP CPU #1 launched
> machdep_ap_bootstrap after smp_started!=0: AP CPU #3 launched
> machdep_ap_bootstrap after smp_started!=0: AP CPU #2 launched
> machdep_ap_bootstrap after smp_started!=0: AP CPU #1 launched

The below has the bad reference to the 2 resets:

> Interstingly the 0x25 shows before the CPU 3 tied resets instead of
> the laster 0x20 (from pmap_cpu_bootstrap), 0x30 (from
> cpudep_ap_bootstrap), or 0x40 (from cpudep_ap_setup). The 0x51
> does show after the pc_awake loop.
> 

The 0x25 was shown after the 2 resets but before the pc_awake
wait loop. The 0x51 was seen after the pc_awake wait loop.

I'm going to delete the code reporting a bunch of information
that has been stable for both successful boots and hang-up
boots (for the modern VM_MAX_KERNEL_ADDRESS value).

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

_______________________________________________
[email protected] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ppc
To unsubscribe, send any mail to "[email protected]"