Re: NVMe

On Sun, 13 Oct 2019 19:39:52 -0400 (EDT), Robert Krawitz wrote:
> Unfortunately, I'm not getting a lot of benefit from use of an NVMe;
> it looks like I'm hitting other limits right away (MD5 checksumming
> and thumbnail extraction) even with everything cached in memory.
>
> With increasing thread counts, it would be very uesful to be able to
> farm out checksumming and thumbnail generation.  Checksum generation
> shouldn't be much of a problem; it could be computed by the scout
> thread and stored in a hash, and only computed by the loader if it
> doesn't exist.  The good news (subject to verifying that it did the
> checksum correctly) is that even 3-way parallelism (3 scout threads)
> got me to 1.8 GB/sec I/O rate, something like 200 files/sec.
> Unfortunately, this then gets ahead of thumbnail generation, with the
> result that images have size (-1, -1) until their thumbnails get
> created.  Still need to figure out how to deal with that.

I've created a parallel-md5 branch that prototypes this.  With fast
backing store, such as the Inland Premium 2TB NVMe, image loading
really flies; I'm getting about 1.9 GB/sec loading images, limited by
MD5 checksum generation on a processor that's not especially fast by
recent standards (Xeon E3-1505M, 4x2 Skylake at 2.8/3.7 GHz).  I'm
using 4 scout threads to get there, with the scouts doing the MD5
calculation.  With RAID gen4 NVMe on a Threadripper or higher thread
count Epyc the results would be interesting.

Thumbnail generation of course lags badly on my hardware.  The result
is that I'm actually doing about 2x as much total I/O, but the user
gets control back very quickly.  I managed to get the image size
during preload, so the -1 problem went away.

The hard part's going to be figuring out how to autotune the number of
scout threads.
-- 
Robert Krawitz                                     <rlk-FrUbXkNCsVf2fBVCVOL8/[email protected]>

***  MIT Engineers   A Proud Tradition   http://mitathletics.com  ***
Member of the League for Programming Freedom  --  http://ProgFree.org
Project lead for Gutenprint   --    http://gimp-print.sourceforge.net

"Linux doesn't dictate how I work, I dictate how Linux works."
--Eric Crampton