Re: Memory leak (possibly connected to postgis) leading to server crash
Roman Cervenak <[email protected]>
| Newsgroups | gmane.comp.db.postgresql.bugs |
|---|---|
| Message-ID | <CAGjExY0Kb0-h3MqWbiYry3e40nKbYyOW3-=BO7iKLJpo8R66aw@mail.gmail.com> |
Hi, I am trying to do (gdb) p MemoryContextStats(TopMemoryContext) on postgres backend process, but the response is: 'TopMemoryContext' has unknown type; cast it to its declared type I never debugged postgres (or anything under linux), and google is not helpful. Can you please advise? Full output is attached. Regards, Roman On Fri, Dec 6, 2019 at 1:41 PM Tomas Vondra <[email protected]> wrote: > On Fri, Dec 06, 2019 at 12:46:44PM +0100, Roman Cervenak wrote: > >Yes, it was killed by oom killer: > > > >[2037990.376427] > > >oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/system-postgresql.slice,task=postgres,pid=52059,uid=111 > >[2037990.376433] Out of memory: Kill process 52059 (postgres) score 294 or > >sacrifice child > >[2037990.384186] Killed process 52059 (postgres) total-vm:17508832kB, > >anon-rss:4309296kB, file-rss:108kB, shmem-rss:12641580kB > >[2037990.516504] oom_reaper: reaped process 52059 (postgres), now > >anon-rss:0kB, file-rss:0kB, shmem-rss:12641580kB > > > >(full dmesg.log attached, if it is interesting; there are more postgres > >backends visible, but they were inactive at the time) > > > > OK, so it allocated extra ~4.3GB or so (plus shared buffers). > > >I can try the gdb dump next time I will see it. But I cannot imagine > giving > >you reproducible case - it is 500 GB proprietary database, and without it, > >queries would be hardly useful, I presume? I can try to make "sample" with > >generated data and find out if I can reproduce the issue with my queries > >that way, but that will be quite time consuming. > > > > Yeah, I understand the data is proprietary. But we need to identify > where the issue is, somehow. And being able to reproduce it is likely > the best way to do that. One way to do that is to either generate > synthetic data with similar structure/features (geometries of similar > size etc.), and hope that it triggers the issue too. Or distill a subset > of data triggering the issue and anonymize it. Or something like that. > > Yes, it's going to be quite time consuming, but I don't have better > ideas. Maybe running it under valgrind would help - you'd have to > rebuild PostgreSQL with valgrind support and run the workload on it. The > queries would be much slower (possibly by an order of magnitude or so), > so you'd have to run it on a different machine, but that's machine time, > not time wasted by a human. > > Anyway, let's start with the easy stuff - try running it again and get > the memory context stats. Maybe it's a simple leak in PostgreSQL, in > which case it should be easier to investigate it. If that turns out to > be untrue, you can try this more complicated stuff (valgrind, ...). > > >Would it help to dump the memory of the backend process and deliver the > >dump (by some private channel) to somebody to identify who is consuming > all > >that memory? (that is the usual drill in windows) > > > > I don't think that'd be very helpful - if this really is memory leak in > one of the libraries, I have no idea how to spot that in the memory > dump. Also, if you claim the data is sensitive/proprietary, you should > not really be sharing the dumps. > > regards > > -- > Tomas Vondra http://www.2ndQuadrant.com > PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services >
postgres_gdb.txt
(text/plain, 9 KB)
root@myvm:~# gdb -p 26850 GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word". Attaching to process 26850 Reading symbols from /usr/lib/postgresql/12/bin/postgres...(no debugging symbols found)...done. Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols from /usr/lib/debug/.build-id/28/c6aade70b2d40d1f0f3d0a1a0cad1ab816448f.debug...done. done. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Reading symbols from /usr/lib/x86_64-linux-gnu/libxml2.so.2...(no debugging symbols found)...done. Reading symbols from /lib/x86_64-linux-gnu/libpam.so.0...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libssl.so.1.1...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2...(no debugging symbols found)...done. Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/librt-2.27.so...done. done. Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libdl-2.27.so...done. done. Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libm-2.27.so...done. done. Reading symbols from /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libicui18n.so.60...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libicuuc.so.60...(no debugging symbols found)...done. Reading symbols from /lib/x86_64-linux-gnu/libsystemd.so.0...(no debugging symbols found)...done. Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.27.so...done. done. Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.27.so...done. done. Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done. Reading symbols from /lib/x86_64-linux-gnu/liblzma.so.5...(no debugging symbols found)...done. Reading symbols from /lib/x86_64-linux-gnu/libaudit.so.1...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5.so.3...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libk5crypto.so.3...(no debugging symbols found)...done. Reading symbols from /lib/x86_64-linux-gnu/libcom_err.so.2...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5support.so.0...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2...(no debugging symbols found)...done. Reading symbols from /lib/x86_64-linux-gnu/libresolv.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libresolv-2.27.so...done. done. Reading symbols from /usr/lib/x86_64-linux-gnu/libsasl2.so.2...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libgssapi.so.3...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libgnutls.so.30...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done. Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libicudata.so.60...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/liblz4.so.1...(no debugging symbols found)...done. Reading symbols from /lib/x86_64-linux-gnu/libgcrypt.so.20...(no debugging symbols found)...done. Reading symbols from /lib/x86_64-linux-gnu/libcap-ng.so.0...(no debugging symbols found)...done. Reading symbols from /lib/x86_64-linux-gnu/libkeyutils.so.1...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libheimntlm.so.0...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5.so.26...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libasn1.so.8...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libhcrypto.so.4...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libroken.so.18...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libp11-kit.so.0...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libidn2.so.0...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libunistring.so.2...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libtasn1.so.6...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libnettle.so.6...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libhogweed.so.4...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libgmp.so.10...(no debugging symbols found)...done. Reading symbols from /lib/x86_64-linux-gnu/libgpg-error.so.0...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libwind.so.0...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libheimbase.so.1...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libhx509.so.5...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libsqlite3.so.0...(no debugging symbols found)...done. Reading symbols from /lib/x86_64-linux-gnu/libcrypt.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libcrypt-2.27.so...done. done. Reading symbols from /usr/lib/x86_64-linux-gnu/libffi.so.6...(no debugging symbols found)...done. Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_files-2.27.so...done. done. Reading symbols from /usr/lib/postgresql/12/lib/postgis-3.so...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libgeos_c.so.1...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libproj.so.12...(no debugging symbols found)...done. Reading symbols from /lib/x86_64-linux-gnu/libjson-c.so.3...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libprotobuf-c.so.1...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libSFCGAL.so.1...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libgeos-3.7.1.so...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libCGAL_Core.so.13...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.65.1...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libboost_serialization.so.1.65.1...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libCGAL.so.13...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libgmpxx.so.4...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libmpfr.so.6...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libboost_system.so.1.65.1...(no debugging symbols found)...done. Reading symbols from /usr/lib/postgresql/12/lib/plpgsql.so...(no debugging symbols found)...done. Reading symbols from /usr/lib/postgresql/12/lib/llvmjit.so...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libLLVM-6.0.so.1...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libedit.so.2...(no debugging symbols found)...done. Reading symbols from /lib/x86_64-linux-gnu/libtinfo.so.5...(no debugging symbols found)...done. Reading symbols from /usr/lib/postgresql/12/lib/utf8_and_iso8859.so...(no debugging symbols found)...done. 0x00007f4d3f073b77 in epoll_wait (epfd=5, events=0x55f735ff1260, maxevents=1, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 30 ../sysdeps/unix/sysv/linux/epoll_wait.c: No such file or directory. (gdb) p MemoryContextStats(TopMemoryContext) 'TopMemoryContext' has unknown type; cast it to its declared type (gdb)