Re: QA failures - holding the release

On Tue, Nov 15, 2016 at 7:09 AM, Ken McDonell <[email protected]> wrote:

> > 1069 python, pmrep segfaults
> > $ pmrep -s 1 --archive archives/20130706 -z -O 30m
> > disk.dev.read,,"'sda','sdb'",,,16
> >           d.d.read          d.d.read
> >                sda               sdb
> >            count/s           count/s
> >                N/A               N/A
> > Segmentation fault (core dumped)
> > must be segfaulting somewhere in the C libraries?
> >
>
> I have one segfault, and all the rest seem to have
> TypeError: non-empty format string passed to object.__format__
> in the failure traceback.
>

The TypeError/non-empty format string was a real PCP bug - only affecting
python3 on f24 and should be fixed with commit  d93dac6374722cad4.

The segfault seems to be a different issue - only f24 is affected (that I
know of) and it seems to be a python3 bug during Py_Finalize().
pmrep only sometimes triggers it when it exits. The stack traceback sort of
implicates a use-after-free because the refcnt on the PyObject involved is
unrealistic, see at the end of the gory details below. I don't think we can
fix this one with a PCP change but I may be wrong.

[qa]$ PCP_SYSCONF_DIR=/etc/pcp gdb --eval-command=run --eval-command=quit
--args /usr/bin/python3 /usr/bin/pmrep -s 3 disk.dev.read,,"'sda','sdb'"
GNU gdb (GDB) Fedora 7.11.1-86.fc24
...
Reading symbols from /usr/bin/python3...Reading symbols from
/usr/lib/debug/usr/bin/python3.5.debug...done.
done.
Starting program: /usr/bin/python3 /usr/bin/pmrep -s 3
disk.dev.read,,\'sda\',\'sdb\'
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Detaching after fork from child process 11667.
Detaching after fork from child process 11668.
Detaching after fork from child process 11669.
  d.d.read  d.d.read
       sda       sdb
   count/s   count/s
       N/A       N/A
     0.000     0.000
     0.000     0.000

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7d6aaa0 in PyUnicode_Type () from /lib64/libpython3.5m.so.1.0
...
(gdb) bt
#0  0x00007ffff7d6aaa0 in PyUnicode_Type () from /lib64/libpython3.5m.so.1.0
#1  0x00007ffff7a6c23f in visit_decref (op=<unknown at remote
0x7ffff7ee2620>,
    data=<optimized out>) at
/usr/src/debug/Python-3.5.2/Modules/gcmodule.c:375
#2  0x00007ffff799e55b in list_traverse (o=0x7fffe8957a88,
    visit=0x7ffff7a6c220 <visit_decref>, arg=0x0)
    at /usr/src/debug/Python-3.5.2/Objects/listobject.c:2220
#3  0x00007ffff7a6b413 in subtract_refs (containers=<optimized out>)
    at /usr/src/debug/Python-3.5.2/Modules/gcmodule.c:402
#4  collect (generation=generation@entry=2, n_collected=n_collected@entry
=0x0,
    n_uncollectable=n_uncollectable@entry=0x0, nofail=nofail@entry=1)
    at /usr/src/debug/Python-3.5.2/Modules/gcmodule.c:959
#5  0x00007ffff7a6c601 in _PyGC_CollectNoFail ()
    at /usr/src/debug/Python-3.5.2/Modules/gcmodule.c:1613
#6  0x00007ffff7a44928 in PyImport_Cleanup ()
    at /usr/src/debug/Python-3.5.2/Python/import.c:481
#7  0x00007ffff7a4f144 in Py_Finalize ()
    at /usr/src/debug/Python-3.5.2/Python/pylifecycle.c:576
#8  0x00007ffff7a6a32e in Py_Main (argc=<optimized out>, argv=<optimized
out>)
    at /usr/src/debug/Python-3.5.2/Modules/main.c:788
#9  0x0000555555554b70 in main (argc=5, argv=<optimized out>)
    at /usr/src/debug/Python-3.5.2/Programs/python.c:65
(gdb) up
#1  0x00007ffff7a6c23f in visit_decref (op=<unknown at remote
0x7ffff7ee2620>,
    data=<optimized out>) at
/usr/src/debug/Python-3.5.2/Modules/gcmodule.c:375
375    if (PyObject_IS_GC(op)) {
(gdb) p op
$1 = <unknown at remote 0x7ffff7ee2620>
(gdb) whatis op
type = PyObject *
(gdb) p *op
$2 = {ob_refcnt = 140737352967392, ob_type = 0x7ffff7ee2a80}
(gdb)