Re: Loading some symbols, when, and index-cache

On 2025-04-07 08:21, Lluís Batlle i Rossell via Gdb wrote:
> On Mon, Apr 07, 2025 at 09:05:10AM -0300, Guinevere Larsen via Gdb wrote:
>> $ time ./gdb gdb --batch -ex "complete list def" #Basically the same as you
>> did
>> ./gdb gdb --batch -ex "complete list def"  42.40s user 1.19s system 120% cpu
>> 36.243 total
>> $ time ./gdb gdb --batch --readnow -ex "complete list def"
>> ./gdb gdb --batch --readnow -ex "complete list def"  58.32s user 1.79s
>> system 100% cpu 1:00.08 total
>>
>> So even with the slower expansion, GDB is still faster than if we had all
>> symbols being read at the start, and this isn't even taking into account the
>> memory usage.
> 
> There are two points important:
> 
> The cache should allow having a big file on disk that it's just read into
> memory with zero work and then all symbols are ready to search. But
> apparently this happens only with the "minimal" set of symbols, which is
> far from enough for a tab completion.
> 
> And 2nd, at gdb ELF file loading, thread workers are launched to read the
> symbols from the CUs. Again, these seem to load only the "minimal" set of
> symbols. In your example of 1 minute that load of CUs runs single thread.
> 
> I can't even tell that the code behaves correctly. One could say that the
> initial load of CUs multithread should load ALL symbols, and same about
> caching. Or at least that could be an option. Otherwise, the things they
> are meant to run faster become quite limited, while the really-slow
> usual completions go single-thread uncached, taking for you 1 minute.
> 
> Thanks,
> Lluís.


Hi Lluís,

I think you misunderstand the role of what's called the index-cache
(meaning it's perhaps not documented properly).  I will summarize the
process that GDB takes from start to being able to use a symbol.  In
this situation no index is present in the binary file nor in the index
cache.

 1. You do "gdb myprogram".
 2. GDB reads the ELF symbols into what it calls "minimal symbols"
    internally.
 3. In parallel, GDB demangles the names Ada/C++/whatever symbol names
 4. GDB notices there is some DWARF info, so it opens it and lists the
    compilation units by hopping from header to header
 5. In parallel (the background workers you referred to), GDB scans each
    compilation unit to create an in-memory index.  What this index
    essentially consists of is a mapping of names to which compilation
    unit contains that name.  We only need the names of variables,
    functions and types that the user could possibly refer to in an
    expression.  While traversing the debug info to create that index,
    GDB skips most the of debug info, making it quite fast.
 6. You type "print foo" in the CLI, or another expression
    containing a symbol name.
 7. The expression parser asks the DWARF subsystem: expand all
    compilation units that contain a variable or function named "foo".
    The index is looked up to identify the candidate compilation units.
 8. Serially, the DWARF subsystem fully reads those compilation units
    to create some "symtab"s, which is a detailed internal representation
    of everything that could be found in the debug info.
 9. The core is then able to look up "foo" in the symtabs, find
    the relevant symbol, and continue the expression evaluation.

To avoid doing the work in step #5, it is possible to ask compilers to
generate a name index that ressembles the name -> compilation unit index
that GDB would produce.  Alternatively, GDB is able to add that index to
a binary that doesn't have it (see the gdb-add-index command).  The
index cache is a third way to access that pre-computed index, which
requires no user intervention (other than toggling the index-cache on).
If the index-cache enabled, and no pre-computed index is present
already, GDB will save it in ~/.cache/gdb, allowing it to read it back
later.  It's exactly the same data that gdb-add-index would add to the
binary.

The index cache was added because historically, using an index was
_much_ faster than having GDB generate its in-memory index (it used
what's called internally "partial symbols", it no longer uses them for
DWARF).  So it was useful, because the second time you loaded a binary
was much faster than the first time.  Nowadays, with the new-ish
parallelized scanner I described in step #5, the time difference between
using a pre-computed index or generating it on the fly is not that big.

The -readnow option that Guinevere talked about skips the in-memory
index generation (or skips reading the pre-computed index if there is
one) and goes directly to expanding all compilation units into symtabs
right away.  I would typically not recommend using this day-to-day,
other than maybe if you need to work around an indexer bug.

When you type "list def<TAB>" and it's very slow, it's likely that a lot
of compilation units have a symbol that starts with "def", so a lot of
compilation units get expanded into symtabs.  Only after that happens
can the core of GDB search generate completion list by searching the
symtabs.  That's not optimal, there's certainly room for improvement
here (I too get frustrated by very slow tab-completion, to the point
where I often avoid it).

I must point out that a pending patch series changes a bit how symbols
are looked up and symtabs expanded, which will probably make what I
described somewhat outdated:

  [PATCH v2 00/28] Search symbols via quick API
  https://inbox.sourceware.org/gdb-patches/[email protected]/T/#m737dd42bf8767f5719ffac7eb147977c4b0f2829

Simon