[RFC 0/3] Deferred, or on-demand, debuginfod downloading

Hi!

A while ago some patches were posted that added deferred debug info
downloading to GDB.  The latest iteration of these patches can be
found here:

  https://inbox.sourceware.org/gdb-patches/[email protected]

There was some great work done in the first cut of this work, however,
the series never really got much review attention, and the work ended
up on hold.

I've spent some time recently trying to move these patches forward to
a place where they might be ready to merge.

Progress seemed to be going OK, except for one issue which, right now,
seems like a truely nasty problem, my current fix for which does not
feel good.  So I wanted to share the current work with the community
to see if anyone has any ideas for a better solution.

But before we get into the problems, a quick summary of what changed
since the previous version.

The biggest change is testing.  The original series has a single test
which checked the standard basic use case.  That test was fine, but
there were a lot of changes in the patch which were not covered by
that test.  So I worked through the patch trying to ensure that every
single change was covered by a test.  I cannot guarantee 100%
coverage, but I do think that most stuff is now covered by a test
case.

Doing this meant I found some small pieces which were not needed,
probably left overs from earlier iterations.

Having written all the tests it was much easier to do some minor
refactoring to move checks into better locations, which allowed some
redundancy to be removed.

But these were all pretty minor clean ups.

The core idea behind the patch remains unchanged.  When we find a
objfile without debug information we ask debuginfod for the .gdb_index
section from the debug info file.  If the section arrives then we
attach this to the original objfile and treat this as a debug index.

Later when we get a "hit" in this .gdb_index we download the full
debug info file, attach this new objfile to GDB, and then repeat the
lookup within the new objfile, this can then find the full debug info
entry (if possible).  Finally, we remove the .gdb_index quick function
lookup from the original objfile, any future lookups will naturally be
serviced by the debug info objfile that was just added.

The original series also included a third patch that made is possible
for GDB to do an early download of .debug_line too, this could be used
for creating file/line breakpoints without the need for the full debug
info.  I have not yet incorporated those changes into this series.

So, what's the problem?

The issue is when the debug info objfile is added to GDB.  Adding a
new objfile triggers the Python new_objfile event listener, which can
then run user code.  User code can easily trigger a frame cache flush.

This is a problem because the deferred debug info downloading almost
always happens while GDB is building the backtrace.

That is where patch #1 of this series comes from.  It's an initial
attempt at allowing GDB to handle frame cache flushes while building
the backtrace.  This works by wrapping a bunch of functions with a
loop.  The functions all take a frame_info_ptr object.  If while
executing the body of the function the frame cache is flushed then GDB
will spot this and go back around the loop.  The hope is that any new
debug info will be added in the first iteration, triggering cache
flushes, then on the second iteration there will be no new debug info
added, so GDB will manage to exit the loop.

Clearly we don't want GDB looping forever, so if the frame cache does
end up being flushed a second time, triggering a third loop iteration,
then GDB will throw an error.

This approach works well enough, but something about it doesn't "feel"
right with me.  For one thing, it feels like there are likely other
places in GDB where this sort of bug might reside, but we'll likely
not find them until some random point in the future where we encounter
the right combination of deferred debug download and new objfile observer.

What feedback do I want?

I'd love people to give their thoughts on the general appraoch taken,
both in patch #1, and in patch #3.

Also, do people have any ideas for better approaches?

Basically, all feedback is welcome.

Thanks,
Andrew



---

Aaron Merey (2):
  gdb: use basic_safe_range to iterate over objfile::qf
  gdb/debuginfod: support on-demand debuginfo downloading

Andrew Burgess (1):
  gdb: limited support for frame cache flushing while building stack

 gdb/ctfread.c                                 |   4 +-
 gdb/dwarf2/index-cache.c                      |  22 +-
 gdb/dwarf2/index-cache.h                      |  11 +
 gdb/dwarf2/public.h                           |  12 +
 gdb/dwarf2/read-gdb-index.c                   | 185 +++++++++++++
 gdb/dwarf2/read-gdb-index.h                   |   9 +
 gdb/dwarf2/read.c                             | 185 ++++++++++++-
 gdb/dwarf2/read.h                             |   9 +
 gdb/elfread.c                                 |   3 +-
 gdb/frame.c                                   | 176 ++++++++++---
 gdb/jit.c                                     |   4 +-
 gdb/objfile-flags.h                           |   4 +
 gdb/objfiles.h                                |  51 +++-
 gdb/symfile-debug.c                           |  43 +--
 gdb/symfile.c                                 |   4 +-
 .../gdb.debuginfod/deferred-auto-load-lib.c   |  26 ++
 .../deferred-auto-load-py-script.h            |  41 +++
 .../gdb.debuginfod/deferred-auto-load.c       |  27 ++
 .../gdb.debuginfod/deferred-auto-load.exp     | 223 ++++++++++++++++
 .../gdb.debuginfod/deferred-download-lib1.c   |  43 +++
 .../gdb.debuginfod/deferred-download-lib2.c   |  37 +++
 .../gdb.debuginfod/deferred-download.c        |  29 ++
 .../gdb.debuginfod/deferred-download.exp      | 158 +++++++++++
 .../gdb.debuginfod/deferred-dwz-common.h      |  41 +++
 .../gdb.debuginfod/deferred-dwz-lib1.c        |  24 ++
 .../gdb.debuginfod/deferred-dwz-lib2.c        |  24 ++
 gdb/testsuite/gdb.debuginfod/deferred-dwz.c   |  30 +++
 gdb/testsuite/gdb.debuginfod/deferred-dwz.exp | 247 ++++++++++++++++++
 .../deferred-expand-warning-lib.c             |  36 +++
 .../gdb.debuginfod/deferred-expand-warning.c  |  25 ++
 .../deferred-expand-warning.exp               | 177 +++++++++++++
 .../gdb.debuginfod/deferred-expand-warning.py |  61 +++++
 .../gdb.debuginfod/deferred-frame-cache-lib.c |  25 ++
 .../gdb.debuginfod/deferred-frame-cache.c     |  28 ++
 .../gdb.debuginfod/deferred-frame-cache.exp   | 180 +++++++++++++
 .../gdb.debuginfod/deferred-frame-cache.py    |  68 +++++
 .../deferred-no-debuginfo-lib.c               |  22 ++
 .../gdb.debuginfod/deferred-no-debuginfo.c    |  25 ++
 .../gdb.debuginfod/deferred-no-debuginfo.exp  | 194 ++++++++++++++
 .../deferred-observer-symbols.exp             | 161 ++++++++++++
 .../deferred-observer-symbols.py              |  62 +++++
 .../gdb.debuginfod/deferred-pending.exp       | 172 ++++++++++++
 .../deferred-select-frame-lib1.c              |  26 ++
 .../deferred-select-frame-lib2.c              |  26 ++
 .../gdb.debuginfod/deferred-select-frame.c    |  61 +++++
 .../gdb.debuginfod/deferred-select-frame.exp  | 240 +++++++++++++++++
 .../gdb.debuginfod/deferred-select-frame.py   |  44 ++++
 .../gdb.python/py-frame-cache-flushing-lib1.c |  26 ++
 .../gdb.python/py-frame-cache-flushing-lib2.c |  26 ++
 .../gdb.python/py-frame-cache-flushing.c      |  55 ++++
 .../gdb.python/py-frame-cache-flushing.exp    | 115 ++++++++
 .../gdb.python/py-frame-cache-flushing.py     |  52 ++++
 gdb/testsuite/lib/debuginfod-support.exp      |  31 ++-
 gdb/testsuite/lib/gdb.exp                     |   8 +-
 gdbsupport/traits.h                           |   7 +
 55 files changed, 3557 insertions(+), 68 deletions(-)
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-auto-load-lib.c
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-auto-load-py-script.h
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-auto-load.c
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-auto-load.exp
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-download-lib1.c
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-download-lib2.c
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-download.c
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-download.exp
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-dwz-common.h
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-dwz-lib1.c
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-dwz-lib2.c
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-dwz.c
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-dwz.exp
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-expand-warning-lib.c
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-expand-warning.c
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-expand-warning.exp
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-expand-warning.py
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-frame-cache-lib.c
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-frame-cache.c
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-frame-cache.exp
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-frame-cache.py
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-no-debuginfo-lib.c
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-no-debuginfo.c
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-no-debuginfo.exp
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-observer-symbols.exp
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-observer-symbols.py
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-pending.exp
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-select-frame-lib1.c
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-select-frame-lib2.c
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-select-frame.c
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-select-frame.exp
 create mode 100644 gdb/testsuite/gdb.debuginfod/deferred-select-frame.py
 create mode 100644 gdb/testsuite/gdb.python/py-frame-cache-flushing-lib1.c
 create mode 100644 gdb/testsuite/gdb.python/py-frame-cache-flushing-lib2.c
 create mode 100644 gdb/testsuite/gdb.python/py-frame-cache-flushing.c
 create mode 100644 gdb/testsuite/gdb.python/py-frame-cache-flushing.exp
 create mode 100644 gdb/testsuite/gdb.python/py-frame-cache-flushing.py


base-commit: cef1fbf3b02ff6e5bc62e1b8a7d30c0a32b8b9f7
-- 
2.25.4