Re: [Fwd: Re: [Fwd: [CPS-devel] directorystorage and ext3?]]

Toby Dickenson wrote:

>On Saturday 14 May 2005 14:57, Jean-Marc Orliaguet wrote:
>  
>
>>Hi!
>>
>>according to the benchmarks I'd done so far, ext3 does not seem to scale
>>well after 3 or 4GB of data (corresponding to ~ 400.000 small files in
>>the storage)
>>
>>I've simply uploaded documents:
>>- 1000 x 2MB image attachment
>>- 1000 x 80Kb pdf file (this also includes a transformation from pdf to
>>text)
>>
>>and measured the time that it takes to upload each document:
>>
>>1000x 2MB image
>>- http://www.medic.chalmers.se/~jmo/dokarkiv/perf-2.pdf
>>
>>+ 1000 x 80kb pdf file
>>- http://www.medic.chalmers.se/~jmo/dokarkiv/perf-3.pdf
>>    
>>
>
>Interesting graphs. thanks for sharing.
>
>A benchmark performing non-stop writes may not be a good indicator of 
>performance under a mixed read/write or read-heavy workload. During a write 
>transaction DS writes the new data into the journal directory. This is quick, 
>to reduce latency of the write transaction. When sufficient write 
>transactions have completed it asynchronously moves them all from the journal 
>into the main data directory in a batch. This is known as 'journal flushing', 
>and is relatively slow.
>
>Under sustained heavy write pressure the journal flushing process becomes the 
>bottleneck. Eventually one write transaction has to wait for the previous 
>journal flush to complete (to prevent the journal getting too large), causing 
>spikes exactly like I see in that graph. That seriously slows down 
>performance, but Ive only ever seen it happen in benchmarks.
>
>I wouldnt expect to see those spikes if the benchmark was to repeatedly upload 
>1 document then read 10 old ones. 
>
>  
>
>>It seems that directory storage does a transaction flush every ten
>>document or so, which knocks ext3 out.
>>    
>>
>
>There are some parameters in the [journal] section of the configuration file 
>which control journal flushing. Controlling when it starts flushing a batch 
>of transactions, and the number of outstanding batches before a write 
>transaction gets blocked. Im sure they could benefit from some tweaking to 
>improve performance in this write-heavy operating mode. 
>  
>

Hi Toby!

It is true that doing heavy writes probably pushes the limits, however,
after having written 2000 documents, there is another effect happening
now, it is that every write access takes a lot of time in ext3 while
it's under control with reiserfs.

these new benchmarks show the uploading of a 80 kb file 20 times in the
storages (with 3-4GB of data already stored). The flushing occurs twice
(at 8 and 16). The zope and zeo servers having been restarted I would
not consider these as "heavy" writes, though, it's only 80kb of data.

http://www.medic.chalmers.se/~jmo/dokarkiv/perf-4.pdf

you can see the 2 peaks with reiserfs (apart from them, it takes 1-2 sec
to write a document).
However with ext3 each write access now takes 10 to 20 seconds. So there
is clearly a scalability issue apart from the flushing factor.

I'll see how the [journal] parameters can be tweaked.

/JM




-------------------------------------------------------
This SF.Net email is sponsored by Oracle Space Sweepstakes
Want to be the first software developer in space?
Enter now for the Oracle Space Sweepstakes!
http://ads.osdn.com/?ad_id=7393&alloc_id=16281&op=click