Re: [PATCH 0/3] debuginfod: speed up extraction from kernel debuginfo packages by 200x

On Thu, Jul 11, 2024 at 04:16:25PM -0400, Frank Ch. Eigler wrote:
> Hi, Omar -
> 
> Thanks.  I wish this sort of amazing kludge weren't necessary, but
> given that it helps, so be it.
> 
> I'd like to commend you on the effort needed to match your code up
> with the stylistic idiosyncracies of the debuginfod c++ code.  It
> looks just like the other code.  My only reservation is the schema
> change.  Reindexing some of our large repos takes WEEKS.  Here's a
> possible way to avoid that:
> 
> - Preserve the current BUILDID schema id and tables as is.
> 
> - Add a new table for the intra-archive coordinates.  Think of it like a cache.
>   Index it with archive-file-name and content-file-name (source0, source1 IIRC).
> 
> - During a fetch out of the archive-file-name, check whether the new
>   table has a record for that file.  If yes, cache hit, go through to
>   the xz extraction stuff, winner!
> 
> - If not, try the is_seekable() check on the archive.  If it is true, we have an
>   archive that should be seekable, but we don't have it in the intra-archive cache.
>   So take this opportunity to index that archive (only), populate the cache table,
>   as the archive is being extracted.  (No need to use the new cache data then, since
>   we've just paid the effort of decompressing/reading the whole thing already.)
> 
> - Need to confirm that during grooming, a disappeared
>   archive-file-name would also drop the corresponding intra-archive
>   rows.
> 
> - Heck, during grooming or scanning, maybe the tool could preemptively
>   do the intra-archive coordinate cache thing if it's not already
>   done, just to defeat the latency of doing it on demand.
> 
> 
> What do you think?

Hi, Frank,

I didn't realize how expensive reindexing could be, thank you for
pointing that out.  Your proposal makes sense to me, I'll rework this.

Thanks,
Omar