Re: FYI: powerpc64 headbuilt via devel/powerpc64-xtoolchain-gcc and C++ exceptions for user code built by system-clang or devel/powerpc64-gcc (as of head -r339076 and ports -r480180)
Mark Millard via freebsd-toolchain <[email protected]>
| Newsgroups | gmane.os.freebsd.devel.toolchain,gmane.os.freebsd.devel.ppc,gmane.os.freebsd.devel.hackers |
|---|---|
| Message-ID | <[email protected]> |
[I've found the problem at the low level for
my context of using WITH_LIBCPLUSPLUS= with
the likes of devel/powerpc64-gcc but I do
not have a solution for WITH_LIBCPLUSPLUS=
so far. I give details of what I found.]
On 2018-Oct-14, at 12:40 AM, Mark Millard <marklmi at yahoo.com> wrote:
> On 2018-Oct-12, at 1:59 PM, Mark Millard <marklmi at yahoo.com> wrote:
>
>> I built a powerpc64 head -r339076 via devel/powerpc64-gcc
>> and the like that built clang as cc as well (and
>> WITHOUT_LIB32). This included use of base/binutils to
>> the the powerpc64 set up. The system and kernel are
>> non-debug builds (but with symbols). [system-clang is not
>> used for buildworld buildkernel because of known
>> issues (last I tried).]
>>
>> booting, buildworld, buildkernel, poudriere building
>> what totaled to be somewhat under 400 ports all seem
>> to work. But . . .
>>
>> It been a long time since I've done something analogous
>> and a significant item in the result is different than in
>> the past once I started testing the throwing of C++
>> exceptions in code produced by system-clang or by
>> devel/powerpc64-gcc :
>>
>> Such code ends up stuck using around 100% of a CPU.
>> An example is the program:
>>
>> # more exception_test.cpp
>> #include <exception>
>>
>> int main(void)
>> {
>> try { throw std::exception(); }
>> catch (std::exception& e) {}
>> return 0;
>> }
>>
>> For system-clang it ended up with:
>>
>> # ldd a.out
>> a.out:
>> libc++.so.1 => /usr/lib/libc++.so.1 (0x81006d000)
>> libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x810184000)
>> libm.so.5 => /lib/libm.so.5 (0x8101ab000)
>> libc.so.7 => /lib/libc.so.7 (0x8101eb000)
>> libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x810554000)
>>
>> That program goes into an possibly unbounded execution.
>> (Historically when this program had problems it would
>> stop and produce a core file.)
>>
>> When compiled by devel/powerpc64-gcc the a.out that results
>> does the same thing. ( /usr/local/bin/powerpc64-unknown-freebsd12.0-c++
>> as the compiler path ) So this is not really clang specific
>> in any way. This ended up with:
>>
>> # ldd a.out
>> a.out:
>> libc++.so.1 => /usr/lib/libc++.so.1 (0x81006d000)
>> libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x810184000)
>> libm.so.5 => /lib/libm.so.5 (0x8101ab000)
>> libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x8101eb000)
>> libc.so.7 => /lib/libc.so.7 (0x810211000)
>>
>> (That should not have involved clang or llvm at all.)
>>
>> But compiled by lang/gcc8's g++8 the a.out that results works
>> fine. This ends up with:
>>
>> # ldd a.out
>> a.out:
>> libstdc++.so.6 => /usr/local/lib/gcc8/libstdc++.so.6 (0x81006e000)
>> libm.so.5 => /lib/libm.so.5 (0x8102c7000)
>> libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x810307000)
>> libc.so.7 => /lib/libc.so.7 (0x81032d000)
>>
>> It is not clear if using base/gcc as system cc
>> would do any better than using system-clang does
>> or than devel/powerpc64-gcc does: it is sort of
>> a variant of devel/powerpc64-gcc .
>>
>> It will probably be some time before I figure out
>> much about what is going on.
>>
>> Two things common to the problem cases are:
>>
>> libc++.so.1 => /usr/lib/libc++.so.1 (0x81006d000)
>> libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x810184000)
>>
>> lang/gcc8 avoids those being involved.
>>
>>
>> Notes:
>>
>> . . .
>>
>> WITHOUT_LIB32= is because, for every post-gcc 4.2.1
>> that I've tried, the lib32 produced misuses R30 in
>> crtbeginS code (vs. the ABI for FreeBSD) and 32-bit
>> code just produces core files from the bad so-called
>> address dereference that results.
>>
>> I'd rather have throwing C++ exceptions working and
>> lack of lib32 than have lib32 but not have throwing
>> C++ exceptions working. But at the moment how to have
>> such is not obvious when fairly modern compilers
>> and toolchains are involved.
>
> Here is what I've found so far.
>
> The code is looping in the following routine.
> (I've inserted 2 NOTE: lines for what the
> sustained looping is like.)
>
. . . (the routine was _Unwind_RaiseException) . . .
So far I've found that the following in
_Unwind_RaiseException stays invariant once
initialized --despite the uw_frame_state_for
and uw_update_context calls in
_Unwind_RaiseException 's loop that normally
update fs:
(gdb) print fs
$15 = {regs = {reg = {{loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED} <repeats 31 times>, {loc = {reg = 18446744073709551608, offset = -8,
exp = 0xfffffffffffffff8 <Error reading address 0xfffffffffffffff8: Bad address>}, how = REG_SAVED_OFFSET}, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED} <repeats 33 times>, {
loc = {reg = 16, offset = 16, exp = 0x10 <Error reading address 0x10: Bad address>}, how = REG_SAVED_OFFSET}, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED} <repeats 80 times>},
prev = 0x0}, cfa_offset = 0, cfa_reg = 1, cfa_exp = 0x0, cfa_how = CFA_REG_OFFSET, pc = 0x8101999f8, personality = 0, data_align = -8, code_align = 4, retaddr_column = 65,
fde_encoding = 27 '\033', lsda_encoding = 255 '?', saw_z = 1 '\001', signal_frame = 0 '\0', eh_ptr = 0x0}
It turns out that pc value 0x8101999f8 is a little after the libcxxrt.so
call to _Unwind_RaiseException that is in throw_exception. But
_Unwind_RaiseException returning would be a failure and would end up
in a non-returning, error-reporting code path.
In other words: this is not an appropriate context for following the
return path to unwind out of _Unwind_RaiseException and its internal
caller (throw_exception).
It got to that address from lr containing the address of the
instruction after the one that does bl to the
_Unwind_RaiseException routine.
Overall it needs to unwind past this in the normal case but is stuck
handling the error/no-return case as "the" case.
Supporting details follow.
What lead up to 0x8101999f8 for initialization was
the lr value related to calling _Unwind_RaiseException
(see the address in lr below, also copied to r5):
. . .
(gdb) c
Continuing.
Breakpoint 9, <signal handler called>
1: x/i $pc 0x8101f35d8 <_Unwind_RaiseException+216>: bl 0x8101f2bc0 <uw_init_context_1>
Current language: auto; currently minimal
where the register values being supplied are (see
r5 and lr):
(gdb) info reg
r0 0x8101999f0 34629851632
r1 0x3fffffffffffc320 4611686018427372320
r2 0x810217900 34630367488
r3 0x3fffffffffffd280 4611686018427376256
r4 0x3fffffffffffd930 4611686018427377968
r5 0x8101999f0 34629851632
r6 0xa0 160
r7 0x0 0
r8 0x1 1
r9 0x8101aac10 34629921808
r10 0x1 1
r11 0x28 40
r12 0x28000282 671089282
r13 0x81005d020 34628554784
r14 0x0 0
r15 0x0 0
r16 0x0 0
r17 0x0 0
r18 0x0 0
r19 0x0 0
r20 0x0 0
r21 0x0 0
r22 0x0 0
r23 0x0 0
r24 0x0 0
r25 0x0 0
r26 0x0 0
r27 0x3fffffffffffd280 4611686018427376256
r28 0x810041060 34628440160
r29 0x3fffffffffffc390 4611686018427372432
r30 0x3fffffffffffcd10 4611686018427374864
r31 0x810041008 34628440072
pc 0x8101f35d8 34630219224
ps 0x0 0
cr 0x0 0
lr 0x8101999f0 34629851632
ctr 0x8101f3500 34630219008
xer 0x0 0
fpscr 0xfff80000 -524288
vscr 0x0 0
vrsave 0x0 0
The pc listed in print fs (0x8101999f8) is in the
following from libcxxrt, as is the value in r5 and
lr (0x8101999f0): (some blank lines inserted to
help identify the area and some related material)
(gdb) disass throw_exception
Dump of assembler code for function throw_exception:
0x0000000810199960 <throw_exception+0>: mflr r0
0x0000000810199964 <throw_exception+4>: std r31,-8(r1)
0x0000000810199968 <throw_exception+8>: mr r31,r3
0x000000081019996c <throw_exception+12>: std r0,16(r1)
0x0000000810199970 <throw_exception+16>: stdu r1,-128(r1)
0x0000000810199974 <throw_exception+20>: bl 0x810197ab0 <thread_info>
0x0000000810199978 <throw_exception+24>: ld r10,8(r3)
0x000000081019997c <throw_exception+28>: mr r9,r3
0x0000000810199980 <throw_exception+32>: cmpdi cr7,r10,0
0x0000000810199984 <throw_exception+36>: std r10,24(r31)
0x0000000810199988 <throw_exception+40>: beq- cr7,0x810199a10 <throw_exception+176>
0x000000081019998c <throw_exception+44>: ld r10,0(r9)
0x0000000810199990 <throw_exception+48>: cmpdi cr7,r10,0
0x0000000810199994 <throw_exception+52>: std r10,32(r31)
0x0000000810199998 <throw_exception+56>: beq- cr7,0x8101999d0 <throw_exception+112>
0x000000081019999c <throw_exception+60>: lwz r10,48(r9)
0x00000008101999a0 <throw_exception+64>: addi r3,r31,88
0x00000008101999a4 <throw_exception+68>: addi r10,r10,1
0x00000008101999a8 <throw_exception+72>: stw r10,48(r9)
0x00000008101999ac <throw_exception+76>: bl 0x81018e500 <00000018.plt_call._Unwind_RaiseException@@GCC_3.0>
0x00000008101999b0 <throw_exception+80>: ld r2,40(r1)
0x00000008101999b4 <throw_exception+84>: addi r1,r1,128
0x00000008101999b8 <throw_exception+88>: mr r4,r31
0x00000008101999bc <throw_exception+92>: ld r0,16(r1)
0x00000008101999c0 <throw_exception+96>: ld r31,-8(r1)
0x00000008101999c4 <throw_exception+100>: mtlr r0
0x00000008101999c8 <throw_exception+104>: b 0x8101996b0 <report_failure>
0x00000008101999cc <throw_exception+108>: nop
0x00000008101999d0 <throw_exception+112>: nop
0x00000008101999d4 <throw_exception+116>: addi r3,r31,88
0x00000008101999d8 <throw_exception+120>: ld r10,-30008(r2)
0x00000008101999dc <throw_exception+124>: std r10,32(r31)
0x00000008101999e0 <throw_exception+128>: lwz r10,48(r9)
0x00000008101999e4 <throw_exception+132>: addi r10,r10,1
0x00000008101999e8 <throw_exception+136>: stw r10,48(r9)
0x00000008101999ec <throw_exception+140>: bl 0x81018e500 <00000018.plt_call._Unwind_RaiseException@@GCC_3.0>
0x00000008101999f0 <throw_exception+144>: ld r2,40(r1)
0x00000008101999f4 <throw_exception+148>: addi r1,r1,128
0x00000008101999f8 <throw_exception+152>: mr r4,r31
0x00000008101999fc <throw_exception+156>: ld r0,16(r1)
0x0000000810199a00 <throw_exception+160>: ld r31,-8(r1)
0x0000000810199a04 <throw_exception+164>: mtlr r0
0x0000000810199a08 <throw_exception+168>: b 0x8101996b0 <report_failure>
0x0000000810199a0c <throw_exception+172>: nop
0x0000000810199a10 <throw_exception+176>: nop
0x0000000810199a14 <throw_exception+180>: ld r10,-30000(r2)
0x0000000810199a18 <throw_exception+184>: std r10,24(r31)
0x0000000810199a1c <throw_exception+188>: b 0x81019998c <throw_exception+44>
0x0000000810199a20 <throw_exception+192>: .long 0x0
0x0000000810199a24 <throw_exception+196>: .long 0x90001
0x0000000810199a28 <throw_exception+200>: lwz r0,0(r1)
End of assembler dump.
For: 0x00000008101999f8 (above)
objdump shows: 00000000000159f8 (below):
0000000000015960 <.__cxa_end_catch+0x460> mflr r0
0000000000015964 <.__cxa_end_catch+0x464> std r31,-8(r1)
0000000000015968 <.__cxa_end_catch+0x468> mr r31,r3
000000000001596c <.__cxa_end_catch+0x46c> std r0,16(r1)
0000000000015970 <.__cxa_end_catch+0x470> stdu r1,-128(r1)
0000000000015974 <.__cxa_end_catch+0x474> bl 0000000000013ab0 <._ZdaPv+0x590>
0000000000015978 <.__cxa_end_catch+0x478> ld r10,8(r3)
000000000001597c <.__cxa_end_catch+0x47c> mr r9,r3
0000000000015980 <.__cxa_end_catch+0x480> cmpdi cr7,r10,0
0000000000015984 <.__cxa_end_catch+0x484> std r10,24(r31)
0000000000015988 <.__cxa_end_catch+0x488> beq cr7,0000000000015a10 <.__cxa_end_catch+0x510>
000000000001598c <.__cxa_end_catch+0x48c> ld r10,0(r9)
0000000000015990 <.__cxa_end_catch+0x490> cmpdi cr7,r10,0
0000000000015994 <.__cxa_end_catch+0x494> std r10,32(r31)
0000000000015998 <.__cxa_end_catch+0x498> beq cr7,00000000000159d0 <.__cxa_end_catch+0x4d0>
000000000001599c <.__cxa_end_catch+0x49c> lwz r10,48(r9)
00000000000159a0 <.__cxa_end_catch+0x4a0> addi r3,r31,88
00000000000159a4 <.__cxa_end_catch+0x4a4> addi r10,r10,1
00000000000159a8 <.__cxa_end_catch+0x4a8> stw r10,48(r9)
00000000000159ac <.__cxa_end_catch+0x4ac> bl 000000000000a500 <CXXABI_1.3@@CXXABI_1.3+0xa500>
00000000000159b0 <.__cxa_end_catch+0x4b0> ld r2,40(r1)
00000000000159b4 <.__cxa_end_catch+0x4b4> addi r1,r1,128
00000000000159b8 <.__cxa_end_catch+0x4b8> mr r4,r31
00000000000159bc <.__cxa_end_catch+0x4bc> ld r0,16(r1)
00000000000159c0 <.__cxa_end_catch+0x4c0> ld r31,-8(r1)
00000000000159c4 <.__cxa_end_catch+0x4c4> mtlr r0
00000000000159c8 <.__cxa_end_catch+0x4c8> b 00000000000156b0 <.__cxa_end_catch+0x1b0>
00000000000159cc <.__cxa_end_catch+0x4cc> nop
00000000000159d0 <.__cxa_end_catch+0x4d0> nop
00000000000159d4 <.__cxa_end_catch+0x4d4> addi r3,r31,88
00000000000159d8 <.__cxa_end_catch+0x4d8> ld r10,-30008(r2)
00000000000159dc <.__cxa_end_catch+0x4dc> std r10,32(r31)
00000000000159e0 <.__cxa_end_catch+0x4e0> lwz r10,48(r9)
00000000000159e4 <.__cxa_end_catch+0x4e4> addi r10,r10,1
00000000000159e8 <.__cxa_end_catch+0x4e8> stw r10,48(r9)
00000000000159ec <.__cxa_end_catch+0x4ec> bl 000000000000a500 <CXXABI_1.3@@CXXABI_1.3+0xa500>
00000000000159f0 <.__cxa_end_catch+0x4f0> ld r2,40(r1)
00000000000159f4 <.__cxa_end_catch+0x4f4> addi r1,r1,128
00000000000159f8 <.__cxa_end_catch+0x4f8> mr r4,r31
00000000000159fc <.__cxa_end_catch+0x4fc> ld r0,16(r1)
0000000000015a00 <.__cxa_end_catch+0x500> ld r31,-8(r1)
0000000000015a04 <.__cxa_end_catch+0x504> mtlr r0
0000000000015a08 <.__cxa_end_catch+0x508> b 00000000000156b0 <.__cxa_end_catch+0x1b0>
0000000000015a0c <.__cxa_end_catch+0x50c> nop
0000000000015a10 <.__cxa_end_catch+0x510> nop
0000000000015a14 <.__cxa_end_catch+0x514> ld r10,-30000(r2)
0000000000015a18 <.__cxa_end_catch+0x518> std r10,24(r31)
0000000000015a1c <.__cxa_end_catch+0x51c> b 000000000001598c <.__cxa_end_catch+0x48c>
0000000000015a20 <.__cxa_end_catch+0x520> .long 0x0
0000000000015a24 <.__cxa_end_catch+0x524> .long 0x90001
0000000000015a28 <.__cxa_end_catch+0x528> lwz r0,0(r1)
And dwarfdump shows starting at 0x00015960 as:
< 117><0x00015960:0x00015a2c><><cie offset 0x0000101c::cie index 0><fde offset 0x00001018 length: 0x00000030>
<eh aug data len 0x0>
0x00015960: <off cfa=00(r1) >
0x00015968: <off cfa=00(r1) > <off r31=-8(cfa) > <off r65=r0 >
0x00015974: <off cfa=128(r1) > <off r31=-8(cfa) > <off r65=16(cfa) >
0x000159b8: <off cfa=00(r1) > <off r31=-8(cfa) > <off r65=16(cfa) >
0x000159c8: <off cfa=00(r1) >
0x000159d0: <off cfa=128(r1) > <off r31=-8(cfa) > <off r65=16(cfa) >
0x000159f8: <off cfa=00(r1) > <off r31=-8(cfa) > <off r65=16(cfa) >
0x00015a08: <off cfa=00(r1) >
0x00015a10: <off cfa=128(r1) > <off r31=-8(cfa) > <off r65=16(cfa) >
fde section offset 4120 0x00001018 cie offset for fde: 4124 0x0000101c
0 DW_CFA_advance_loc 8 (2 * 4)
1 DW_CFA_register r65 = r0
4 DW_CFA_offset r31 -8 (1 * -8)
6 DW_CFA_advance_loc 12 (3 * 4)
7 DW_CFA_def_cfa_offset 128
10 DW_CFA_offset_extended_sf r65 16 (-2 * -8)
13 DW_CFA_advance_loc 68 (17 * 4)
14 DW_CFA_remember_state
15 DW_CFA_def_cfa_offset 0
17 DW_CFA_advance_loc 16 (4 * 4)
18 DW_CFA_restore_extended r65
20 DW_CFA_restore r31
21 DW_CFA_advance_loc 8 (2 * 4)
22 DW_CFA_restore_state
23 DW_CFA_advance_loc 40 (10 * 4)
24 DW_CFA_remember_state
25 DW_CFA_def_cfa_offset 0
27 DW_CFA_advance_loc 16 (4 * 4)
28 DW_CFA_restore_extended r65
30 DW_CFA_restore r31
31 DW_CFA_advance_loc 8 (2 * 4)
32 DW_CFA_restore_state
33 DW_CFA_nop
34 DW_CFA_nop
/usr/src/contrib/libstdc++/libsupc++/eh_throw.cc has something
that /usr/src/contrib/libcxxrt/exception.cc does not have in
its error handling code path: a use of __cxa_begin_catch
in __cxxabiv1::__cxa_throw :
#ifdef _GLIBCXX_SJLJ_EXCEPTIONS
_Unwind_SjLj_RaiseException (&header->unwindHeader);
#else
_Unwind_RaiseException (&header->unwindHeader);
#endif
// Some sort of unwinding error. Note that terminate is a handler.
__cxa_begin_catch (&header->unwindHeader);
std::terminate ();
It looks to me like __cxa_begin_catch might do either of
terminate or _Unwind_Resume and that the (conditional)
_Unwind_Resume case is possibly needed here for the
normal execution.
There are also no other calls (bl's) before the terminate:
more directly indicated to not return.
I do not know if one or both of those points helped the code
unwind correctly or not. But it seems suggestive.
Other notes:
I've demonstrated the problem on my prior head
-r333594 environment that had been build via a
6.3 vintage of devel/powerpc64-gcc (but that was
then updated to ports -r469844 and so had 6.4 of
devel/powerpc64-gcc installed). Also: base/binutils
was of a 2.30 vintage and base/gcc was of a 6.3
vintage. (system-clang was not cc, base/gcc
provided system-cc.)
Compiling to produce the a.out via:
/usr/bin/powerpc64-unknown-freebsd12.0-g++
(and so via a 6.3 vintage g++) made no
difference. It still had the problem.
I have taken to having buildworld buildkernel
use -gdwarf-2 so that /usr/libexec/gdb has
access to the information in a format it is
(mostly) designed for. (/usr/local/bin/gdb is
broken by the thrown-C++-exception problem that
I'm investigating: gdb internally throws
exceptions in normal operation.)
===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)