Re: Mysqld 3.23.56 crash on concurrent ALTER TABLE and FLUSH TABLE

Fred van Engen <[email protected]>
Newsgroups gmane.comp.db.mysql.bugs
Message-ID <[email protected]> 
Dean,

On Thu, Jan 15, 2004 at 10:54:15AM -0600, Dean Ellis wrote:
> > I reproduced it on 3.23.56 with this script:
> 
> I am indeed seeing crashes (or lockups) with your script with several 
> versions, but:
>  
> > ALTER TABLE a UNION=($union);
> 
> This shows a problem with multiple threads altering the UNION for the 
> same MERGE table, which is not quite the problem you described.  
> 

Oops, my mistake. Even so, it shouldn't crash even with those queries.


> If I change this to:
> 
> ALTER TABLE $basetable UNION=($union);
> 
> then I am unable to reproduce any problem with 3.23.58.
> 
> Can you verify whether or not your connections are actually altering the 
> same MERGE table?  Also, you should try your test with the above change 
> and see if you are still seeing crashes when the connections are not in 
> fact altering the same MERGE set.
> 

With your change I am still seeing crashes within seconds. I started
with a clean database (i.e. no 'test' database), just to be sure.

Gdb shows two threads handling an ALTER TABLE for different tables:

GNU gdb 4.18
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "sparc-sun-solaris2.7"...
Core was generated by `/opt/mysql-3.23.56/libexec/mysqld --defaults-file=/tmp/fen/my.cnf --basedir=/op'.
Program terminated with signal 9, Killed.
Reading symbols from /usr/lib/libdl.so.1...done.
Reading symbols from /usr/lib/libpthread.so.1...done.
Reading symbols from /usr/lib/libthread.so.1...done.
Reading symbols from /usr/lib/libcrypt_i.so.1...done.
Reading symbols from /usr/lib/libgen.so.1...done.
Reading symbols from /usr/lib/libsocket.so.1...done.
Reading symbols from /usr/lib/libnsl.so.1...done.
Reading symbols from /usr/lib/libm.so.1...done.
Reading symbols from /usr/lib/libc.so.1...done.
Reading symbols from /usr/lib/libmp.so.2...done.
Reading symbols from /usr/platform/SUNW,Ultra-250/lib/libc_psr.so.1...done.
#0  0xff339968 in __sigprocmask () from /usr/lib/libthread.so.1
(gdb) info threads
  14 Thread 6 (LWP 15)  0xff339968 in __sigprocmask () from /usr/lib/libthread.so.1
  13 Thread 5          0xff197a78 in _lwp_sema_wait () from /usr/lib/libc.so.1
  12 Thread 4          0xff197a78 in _lwp_sema_wait () from /usr/lib/libc.so.1
  11 Thread 3          0xff197a78 in _lwp_sema_wait () from /usr/lib/libc.so.1
  10 Thread 2 (LWP 2)  0xff197448 in _signotifywait () from /usr/lib/libc.so.1
  9 Thread 1 (LWP 1)  0xff195c94 in _poll () from /usr/lib/libc.so.1
  8 LWP    6          0xff197a2c in ___lwp_cond_wait () from /usr/lib/libc.so.1
  7 LWP    5          0xff195020 in door_restart () from /usr/lib/libc.so.1
  6 LWP    4          0xff197a78 in _lwp_sema_wait () from /usr/lib/libc.so.1
  5 LWP    14          0xff197a78 in _lwp_sema_wait () from /usr/lib/libc.so.1
  4 LWP    2          0xff197448 in _signotifywait () from /usr/lib/libc.so.1
  3 LWP    1          0xff195c94 in _poll () from /usr/lib/libc.so.1
  2 LWP    8          0xff197a78 in _lwp_sema_wait () from /usr/lib/libc.so.1
* 1 LWP    15          0xff339968 in __sigprocmask () from /usr/lib/libthread.so.1
(gdb) info stack
#0  0xff339968 in __sigprocmask () from /usr/lib/libthread.so.1
#1  0xff32f1f0 in _resetsig () from /usr/lib/libthread.so.1
#2  0xff32e93c in _sigon () from /usr/lib/libthread.so.1
#3  0xff3317b4 in _thrp_kill () from /usr/lib/libthread.so.1
#4  0x13e38c in write_core (sig=10) at stacktrace.c:220
#5  0xc1948 in handle_segfault (sig=10) at mysqld.cc:1334
#6  0xff33b928 in __sighndlr () from /usr/lib/libthread.so.1
#7  <signal handler called>
#8  0xff329b30 in pthread_cond_signal () from /usr/lib/libthread.so.1
#9  0x2cdd58 in thr_abort_locks (lock=0xbc61e0) at thr_lock.c:936
#10 0xbd380 in mysql_lock_abort (thd=0xbc4410, table=0xbcc3a8) at lock.cc:287
#11 0x1238a8 in close_cached_table (thd=0xc3d3b0, table=0xbcc3a8) at sql_table.cc:780
#12 0x125fa0 in mysql_alter_table (thd=0xc3d3b0, new_db=0xbc4c18 "test", new_name=0xc3da98 "b",
    create_info=0xc3d6e0, table_list=0x0, fields=@0x0, keys=@0xc3d5e0, drop_list=@0xc3d568,
    alter_list=@0xc3d578, order=0x0, drop_primary=false, handle_duplicates=DUP_ERROR) at sql_table.cc:1643
#13 0xcad2c in mysql_execute_command () at sql_parse.cc:1556
#14 0xccc20 in mysql_parse (thd=0xc3d3b0, inBuf=0xc3da70 "ALTER TABLE b UNION=(b5,b6,b7,b8)", length=12834000)
    at sql_parse.cc:2386
#15 0xc8f94 in do_command (thd=0xc3d3b0) at sql_parse.cc:840
#16 0xc83bc in handle_one_connection (arg=0xc3d3bc) at sql_parse.cc:558
(gdb) thread 5
[Switching to thread 5 (LWP    14        )]
#0  0xff197a78 in _lwp_sema_wait () from /usr/lib/libc.so.1
(gdb) info stack
#0  0xff197a78 in _lwp_sema_wait () from /usr/lib/libc.so.1
#1  0xff32b04c in _park () from /usr/lib/libthread.so.1
#2  0xff329a00 in cond_wait () from /usr/lib/libthread.so.1
#3  0xff329924 in pthread_cond_wait () from /usr/lib/libthread.so.1
#4  0x2ce798 in safe_cond_wait (cond=0xbcf198, mp=0xbc61f0, file=0x343330 "thr_lock.c", line=383)
    at thr_mutex.c:148
#5  0x2cca58 in wait_for_lock (wait=0xbc6240, data=0xc56410, in_wait_list=0 '\000') at thr_lock.c:383
#6  0x2cd050 in thr_lock (data=0xc56410, lock_type=TL_WRITE_ALLOW_READ) at thr_lock.c:590
#7  0x2cda38 in thr_multi_lock (data=0xc53460, count=4) at thr_lock.c:830
#8  0xbcde8 in mysql_lock_tables (thd=0xbc2c80, tables=0xc413f0, count=1) at lock.cc:101
#9  0xdc21c in open_ltable (thd=0xbc2c80, table_list=0xc413d0, lock_type=TL_WRITE_ALLOW_READ) at sql_base.cc:1448
#10 0x12495c in mysql_alter_table (thd=0xbc2c80, new_db=0xc3d340 "test", new_name=0x0, create_info=0xbc2fb0,
    table_list=0xc413d0, fields=@0xbc2ec0, keys=@0xbc2eb0, drop_list=@0xbc2e38, alter_list=@0xbc2e48, order=0x0,
    drop_primary=false, handle_duplicates=DUP_ERROR) at sql_table.cc:1159
#11 0xcad2c in mysql_execute_command () at sql_parse.cc:1556
#12 0xccc20 in mysql_parse (thd=0xbc2c80, inBuf=0xc41380 "ALTER TABLE a UNION=(a10,a11,a12,a13)",
    length=12332448) at sql_parse.cc:2386
#13 0xc8f94 in do_command (thd=0xbc2c80) at sql_parse.cc:840
#14 0xc83bc in handle_one_connection (arg=0xbc2c8c) at sql_parse.cc:558

Here is the point where is crashed:

(gdb) thread 1
[Switching to thread 1 (LWP    15        )]
#0  0xff339968 in __sigprocmask () from /usr/lib/libthread.so.1
(gdb) up
#1  0xff32f1f0 in _resetsig () from /usr/lib/libthread.so.1
(gdb) up
#2  0xff32e93c in _sigon () from /usr/lib/libthread.so.1
(gdb) up
#3  0xff3317b4 in _thrp_kill () from /usr/lib/libthread.so.1
(gdb) up
#4  0x13e38c in write_core (sig=10) at stacktrace.c:220
220       pthread_kill(pthread_self(), sig);
Current language:  auto; currently c
(gdb) up
#5  0xc1948 in handle_segfault (sig=10) at mysqld.cc:1334
1334       write_core(sig);
Current language:  auto; currently c++
(gdb) up
#6  0xff33b928 in __sighndlr () from /usr/lib/libthread.so.1
(gdb) up
#7  <signal handler called>
(gdb) up
#8  0xff329b30 in pthread_cond_signal () from /usr/lib/libthread.so.1
(gdb) up
#9  0x2cdd58 in thr_abort_locks (lock=0xbc61e0) at thr_lock.c:936
936         pthread_cond_signal(data->cond);
Current language:  auto; currently c
(gdb) print *data
$1 = {thread = 2408550287, next = 0x8f8f8f8f, prev = 0x8f8f8f8f, lock = 0x8f8f8f8f, cond = 0x8f8f8f8f,
  type = TL_UNLOCK, thread_id = 2408550287, status_param = 0x8f8f8f8f}
(gdb)


I'm not sure that the stack dump helps much, because it is slightly
different each time. Whenever I checked the cause of the bus error, the
pattern 0x8f8f8f8f was always there though, so memory got corrupted.

Also, it always crashed in the ALTER TABLE. The other thread could be
handling another query (e.g.  an INSERT when I used that in my test.pl
script).

Can I send the core dump(s) somewhere? Are they of any use for you?

The server runs rock solid apart from this. Putting a GET_LOCK and
RELEASE_LOCK around the code prevents the crash, so it really seems to
be some concurrency/locking issue.


Regards,

Fred.

-- 
Fred van Engen                              XB Networks B.V.
email: [email protected]                Televisieweg 2
tel: +31 36 5462400                         1322 AC  Almere
fax: +31 36 5462424                         The Netherlands

-- 
MySQL Bugs Mailing List
For list archives: http://lists.mysql.com/bugs
To unsubscribe:    http://lists.mysql.com/[email protected]
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.