Re: [Fink-users] extremely slow 'Scanning dists/stable/main/binary-darwin-x86_64' step on 10.13 with HFS

On 2 Dec 2017, at 1:05 am, Alexander Hansen <[email protected]> wrote:
> 
>> On Dec 1, 2017, at 15:18, Derek Homeier <[email protected]> wrote:
>> 
>> On 14 Nov 2017, at 4:38 pm, Derek Homeier <[email protected]> wrote:
>>> 
>>> On 13 Nov 2017, at 8:44 pm, Jack Howarth <[email protected]> wrote:
>>>> 
>>>> Actually. after wiping my HFS+ volume and doing a clean install of
>>>> 10.13.1, the issue seems to have disappeared under a fresh bootstrap
>>>> of fink 0.42.0.
>>> 
>>> So you are indeed on HFS+? Then it’s probably not APFS-related; I am rather not looking
>>> forward to wiping my entire boot volume and reinstall the OS, but I will give the bootstrap
>>> Fink install a try (can’t seem to pull a JRE installation off via remote login, so this will have to
>>> wait for me to get back at the machine). I also noticed I probably did not upgrade to 0.42.0
>>> before the OS upgrade but somehow managed to complete the upgrade without following
>>> the exact instructions on http://www.finkproject.org; this may have left some things in disorder.
>>> OTOH I did not notice any other problems arising from the missing Packages.gz file so far.
>> 
>> Uh, as an update/follow-up on this:
>> 
>> I did not do a complete OS reinstall, but performed a fresh bootstrap installation of Fink as
>> described above, on an APFS volume here. I am still in the process of producing new builds
>> of all packages from my old installation (currently at ~50%), but while the scanning process
>> sped up reasonably right after setting up the new Fink, it started moving back to several minutes
>> after updating more packages. The Packages.gz file at least is rebuilt correctly, but the extreme
>> sluggishness appears in a different light after I found this in /sw/var/lib/fink:
>> 
>> -rw-r--r--   1 root  wheel  37425774592 Dec  2 00:08 scanpackages.db
>> 
>> - compared to a ~11 MB file in the old installation with about twice as many packages.
>> For perspective, this is over 12x the total size of all .debs or 4x the total installed size of the
>> Fink system. It’s just been completely rebuilt (taking O(40 min)) after deleting the previous instance
>> (and had almost blown up my Time Machine before I thought of excluding /sw/var from backups…).
>> 
>> I don’t know much about the Berkeley DB format or the internals of the Fink bookkeeping system,
>> has anyone got an idea what might be going on there, or could confirm if they are seeing similar
>> on APFS?
>> 
>> Cheers,
>> 				Derek
>> 
> 
> I’lll leave the crosspost for now, but don’t do that in the future, whomever started it.
> 
Sorry for the proliferation, I decided to keep both lists in since it looks clearly like a problem
for the developers, but found the db file growing out of proportion can cause serious troubles
if one is not aware of it…
For compensation, this time the reply didn’t make it out of my outbox at all for 3 days.

> There’s a bit of information in “man Fink::Scanpackages”, pulled from /sw/lib/perl5/Fink/Scanpackages.pm . 
> 
> As a simple thing to look at you might try “strings /sw/var/lib/fink/scanpackages.db” and see if you have verbatim repetition of entries.  Your results suggest that the database is getting duplicated rather than just appended to.
> 
Rather to the contrary, it looked like the new database contained fewer entries - some 6000 string lines
in total compared to the 200000+ in the old 11 MB one. But it seems strings does not handle large files
properly - apparently it’s only reading in or parsing the first (filesize modulo 2**32) bytes.
After splitting the scanpackages.db into 4 GB chunks with dd, I could verify that the total no. of string
entries is still > 200000.
However the listing looks something like this:

> strings -o /sw/var/lib/fink/scanpackages.db | head                                                                                                                        
16842652 1510877606mtime:dists/stable/main/binary-darwin-x86_64/devel/automake1.15_1.15.1-1_darwin-x86_64.deb
21036963 1512164197mtime:dists/stable/main/binary-darwin-x86_64/devel/cmake_3.10.0-1_darwin-x86_64.deb
29425564 1510877606mtime:dists/stable/main/binary-darwin-x86_64/devel/automake1.15_1.15.1-1_darwin-x86_64.deb
33618365 /Chris Zubrzycki <[email protected]>
33618417 Maintainer
33618428 
         automake1.15
33618445 Source
33618453 darwin-x86_64
33618469 
         Architecture
33618484 devel

It continues like this, with groups of some 10 string lines separated by several MB of void…

Looks like seek() or whatever the DB_File module or library is using is seriously broken on APFS.
I was able to work around this for the moment by moving the entire /sw/var directory onto the machine’s
hard disk, which is still HFS+, and creating a symlink into /sw, which brought scanpackages back to its
old size and performance (neither index.db nor proxies.db are showing any similar issues btw.).
Just worried for now that Fink might recreate the var subdir in its install location on the SSD at some point…

Cheers,
					Derek


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Fink-devel mailing list
[email protected]
List archive:
https://sourceforge.net/p/fink/mailman/fink-devel
Subscription management:
https://lists.sourceforge.net/lists/listinfo/fink-devel