Re: `cpan` unusable on small-RAM machines

On Tue, 14 Nov 2023 14:17:37 +0000
"Paul \"LeoNerd\" Evans" <[email protected]> wrote:

> And on closer inspection, when it fails, it sits quite a while
> spinning CPU and (presumably) burning memory. It get as far as
> printing "Lockfile removed." nicely quickly, then waits aaaaages
> (over 10 seconds) before the OOM-killer kills it.
> 
>   All tests successful.
>   Files=8, Tests=43,  3 wallclock secs ( 0.21 usr  0.02 sys +  2.94
> cusr  0.34 csys =  3.51 CPU) Result: PASS
>   Lockfile removed.
>                       <== it waits for over 10 seconds here...
>   Killed

This issue came up again for me the other day, and I talked about it on
#p5p. With the help of some interesting debugging ideas from alh,
mauke, ilmari (and maybe some others whose names I have forgotten -
sorry if I missed you), we think we may have narrowed down what is
going on.

Fairly early on in its startup, the main `cpan` process loads the
entire metadata database into memory. It's somewhat big, but that's OK
- it still fits into memory and doesn't upset things too much (as
evidenced by the fact that the `cpan> look` command can run just fine,
for example). Next, it `fork`s itself, before invoking some
sub-process things via system(). This results in us having *two*
processes (the original parent and a child), both with that large
memory footprint. This is still fine - because of Linux copy-on-write.

But now we have two processes, *both* of which have an END block which
is going to do a lot of cleanup that touches a lot of memory, and we
believe here is the time that it gets OOM killed. Each process has to
suddenly grow its own copy of all those pages, because the SvREFCNT of
all those SVs is going to be decremented to throw them away.

Somewhat ironic that perl gets OOM killed during the process of it
trying to throw away all its memory.

Anyway, we don't have a firm idea of how to fix this yet, but we did
poke at a few possible thoughts:

  * Use POSIX::_exit instead of regular exit() from that inner process

  * Use exec() instead of system() to invoke commands from that inner
    process

  * fork() a sidecar process early in startup before loading that
    metadata database, and perform all the DB queries inside that via
    some pipes, to ensure the master controller process doesn't have a
    big memory footprint

I may have a look into some of these on my RPi machines.


However, I'm still not super inclined to actually continue maintaining
CPAN.pm *itself*, because every time I try to dive into the code I find
it a huge twisty nightmare of decades of accumulated cruft. It's hard
to work out what it is supposed to be doing or where or why. I still
feel that the time is well past due, to consider building a brand new
cleanroom small/neat/efficient "install me some modules from CPAN"
client to live in perl core. If I am going to put in a nontrivially
small amount of effort, it is that which I'd prefer to work on.

-- 
Paul "LeoNerd" Evans

[email protected]
http://www.leonerd.org.uk/  |  https://metacpan.org/author/PEVANS