Re: Memory usage question

On 05/08/2023 20.08, Duncan wrote:
> Robin Laing posted on Sat, 5 Aug 2023 12:19:52 -0600 as excerpted:
> 
>> On 05/08/2023 08.06, Detlef Graef via Pan-users wrote:
>>> Hi,
>>>
>>> Am 02.08.23 um 21:00 schrieb Robin Laing:
>>>
>>>> New to this list.
>>>>
>>>> I have been having issues with large groups.  I have seen the reports
>>>> and did some tests yesterday to see if I can find a cause for the
>>>> crash on my system.
>>>>
>>>> Fedora has not updated Pan from 0.149 so I cannot test with 0.150 at
>>>> this time.
>>>
>>> You can try to install Pan from here:
>>>
>>> https://copr.fedorainfracloud.org/coprs/dgraef/Pan/
>>>
>>> v0.154 is available there. You have to enable the repo. See the
>>> instructions there.
>>>
>>>> I am having issues trying to compile 0.150, which may be another
>>>> thread.
>>>>
>>>> Has there been any changes between 0.149 and 0.150 on how Pan uses
>>>> memory for headers?
> 
> The biggest "recent" (note the quotes but in this context 0.149 is...
> several years and several releases behind, and is thus somewhat "old") pan
> changes have been modernizing pan library dependencies.  Bugs aren't
> necessarily in pan itself, but can be in the libraries it depends on as
> well.  In particular, both pan and the gmime library it depends on for
> parsing headers (format-specified in the MIME RFCs) have had known header-
> parsing bugs within the given timeframe (fixed since 0.149 with its old
> gmime library dependencies).
> 
> AFAIK the direct effect of the bugs I'm aware of has been on header
> folding, but it's certainly possible that an indirect effect may have been
> memory leakage, which could certainly translate to *huge* (gigabytes, as
> seen below) leakage on huge enough groups (individual message counts in the
> millions... hundreds of thousands on 32-bit).
> 
>>>> In my tests yesterday, Pan was using 22.6G of ram according to TOP
>>>> which confirmed my issues of pan pushing my system to out of memory
>>>> state.
> 
>> --
> 
> Note that pan and similarly compliant internet messaging apps will
> interpret the specific above sequence (hyphen hyphen space, on a line by
> itself, of course quoted in my reply so it won't match) in accordance with
> RFCs and internet messaging tradition, as a signature separator.  (Some
> implementations, the most famous being certain MS implementations, used
> double-hyphen without the space as a sig-separator, tho that wasn't
> specifically compliant with the spec because it missed the ending space.
> But as a result, many implementations that actually send the correct space-
> terminated version also honor the version without it as a sig-separator as
> well.)
> 
> So your entire reply (after the quotes) was considered a signature!  Of
> course pan (which I use to read this list as a newsgroup via gmane.io)
> colored it accordingly, and until I realized that, it appeared you had
> posted an empty all-quote message.  Further, when I hit reply, it failed to
> include your "signature" in the quote, that of course being the entire new
> content!  (Specifically selecting that "signature" and hitting reply again
> did include it, properly quoted so I could cut and paste it back into this
> reply, where I had already replied to the versions discussion above.)
> 
> FWIW, when I want a "vertical delimitator", that I don't want to be
> mistaken for a sig-separator I'll use "---" (three hyphens without a
> terminating space, of course on its own line).  To my knowledge nothing
> interprets that as sig-separator (and it'd definitely be incorrect to do
> so).  Similarly, a single hyphen on a line by itself should work, but I'm
> more comfortable with three (where a single hyphen line delimitator would
> be appropriate I'll use a double empty line instead... or sometimes the
> tri-dot ellipsis, as just below).
> 

I know this and made the mistake.  100% missed it in my response.  May 
have been due to deleting my signature file but not the two dashes.

> ...
> 
> [Pan 0.154 from COPR]
> 
>> Installed now.  I noticed an issue though, if I do my normal update, the
>> old pan is listed as the update, not the COPR pan.  Will have to watch
>> for that.
> 
> I'm a gentooer so can't give you the fedora specifics, but most distros
> (including gentoo and presumably including fedora) have some way to "pin" a
> specific version, tho of course you have to be careful with that as you can
> miss security updates, etc.  But it should eliminate the trying to
> downgrade back to the old fedora version problem.
> 
> You might also look for a repo priority setting.  In theory, an equal
> priority repo should cause it to prioritize version numbers between those
> repos and upgrade but not downgrade, while the fedora main repo may beset
> to a higher priority now, so /any/ fedora version will want to overwrite
> the COPR version, and the reverse, setting COPR higher, would prioritize
> any version there over fedora (so a higher fedora version would be
> ignored).
> 
> Finally, some distros/package-managers have a (somewhat obscure as it's not
> as visible as name and version) package epoch setting, where a higher epoch
> overrides version number.  This allows "version number resets" if
> necessary, perhaps because someone mistakenly did say a version 33 instead
> of version 3, and now upstream has a version 4, that downstream wants to
> match instead of having to call it version 40 because of their previous 33
> mistake.  (One alternative to epoch is appending a number to the previous
> package name, treating it as part of the package name so it's really a
> different package, instead of the package version.  But that doesn't apply
> here as that'd be visible in the package name, pan2 instead of just pan.)
> If the fedora version has a higher epoch, I believe it'd try to replace the
> lower epoch COPR package, despite the COPR version being higher.
> 
> But that's in general.  Look for instructions specific to your distro and
> package and repo management system, as they might work slightly
> differently.  (Using gentoo with portage as an example because that's what
> I know, version is supreme within dependency constraints, so a higher
> version that fills all other dependencies always overrides repo, and repo
> priority only affects the same version number.  There's no epoc, but there
> is slot, which works a bit differently as it has a different dependency-
> related purpose, but with some work on other dependent packages if any, can
> be used in place of epoc if necessary.)

Something to learn.  I have not run into this before since almost all my 
software has been either updated on the normal servers of isn't carried 
by Fedora and I have compiled it myself.

It is something to think about but I am finding that I may be moving 
from Fedora for many little reasons.

I tried Gentoo years ago.  Started with Slackware.



> 
>>
>> Will test in the next couple of days for the memory issue.  Need to
>> finish some stuff to free up ram just in case I get the same memory
>> issues.
> 
> Been there; done that! =:^)
With 32g of ram, it shouldn't be an issue since most, if lucky, have 16g.

> 
>> I have tried to compile but get a vague message
>>
>> Can't exec "autopoint": No such file or directory at
>> /usr/share/autoconf/Autom4te/FileUtils.pm line 293.
> 
> A quick lookup on my gentoo system (where build-from-source is the default
> so it's setup for it) says /usr/bin/autopoint belongs to the gettext
> package.
> 
> So if you don't have a gettext package installed (and possibly gettext-dev
> or gettext-devel, for a default-binary distro with split development
> packages, as fedora and most common distros are, but gentoo is not), try
> installing it and see if the error goes away.
> 
>> Now to search through copr for other software that I want to use.
> 
> =:^)
> 

I made my comment after a short time of testing but not looking into the 
problem in detail.  I did have the gettext package installed but the 
gettext-devel package was the one needed.  Thanks for the pointer.

Over the decades, I have programmed in various languages so I have some 
idea of how things work.  Just not good at it.  It is on my learning 
plan as I need to do more programming.  Have never used automake.

I am finding many issues with Pan that are related to memory which I 
will expand on in a different email before submitting any bug reports. 
Even with 0.154 I am seeing the same issues as I saw with 0.149.

Robin




_______________________________________________
Pan-users mailing list
[email protected]
https://lists.nongnu.org/mailman/listinfo/pan-users