Re: ext3 with maildir++ = huge disk latency and high load
Andrey <[email protected]> Thu, 29 Sep 2011 19:13:48 +0400
| Newsgroups | gmane.comp.file-systems.ext3.user |
|---|---|
| Message-ID | <[email protected]> |
Hello, This is standard Debian Squeeze(6.0.2) kernel: # uname -a Linux debian 2.6.32-5-686 #1 SMP Fri Sep 9 20:51:05 UTC 2011 i686 GNU/Linux 29.09.2011 18:44, Eric Sandeen пишет: > On 9/29/11 9:36 AM, Andrey wrote: >> Let me to share some testing RAID 5 results with bonnie++: > > What kernel version was this tested on? > > Thanks, > -Eric > >> ext3 (defaults,noatime): >> >> Version 1.96 ------Sequential Output------ --Sequential Input- --Random- >> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- >> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP >> debian 2G 242 96 22458 10 8826 2 1854 98 120985 11 317.1 3 >> Latency 211ms 896ms 720ms 22258us 18733us 622ms >> Version 1.96 ------Sequential Create------ --------Random Create-------- >> debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- >> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP >> 16 12857 33 +++++ +++ 15377 34 13585 33 +++++ +++ 15404 35 >> Latency 12284us 992us 1029us 432us 140us 76us >> >> ext3 (-T small,defaults,noatime): >> >> Version 1.96 ------Sequential Output------ --Sequential Input- --Random- >> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- >> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP >> debian 2G 229 98 4989 5 3862 1 1762 97 91111 9 266.6 6 >> Latency 79046us 22858ms 2577ms 19253us 12120us 767ms >> Version 1.96 ------Sequential Create------ --------Random Create-------- >> debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- >> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP >> 16 6422 16 +++++ +++ 10319 25 8934 21 +++++ +++ 10347 26 >> Latency 9968us 977us 964us 482us 144us 178us >> >> ext3 (-T news,defaults,noatime): >> Version 1.96 ------Sequential Output------ --Sequential Input- --Random- >> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- >> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP >> debian 2G 237 95 22807 11 8807 2 1897 99 121893 11 324.6 5 >> Latency 223ms 808ms 523ms 13765us 11049us 831ms >> Version 1.96 ------Sequential Create------ --------Random Create-------- >> debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- >> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP >> 16 12826 33 +++++ +++ 15900 35 14548 36 +++++ +++ 15460 35 >> Latency 417us 984us 1024us 430us 140us 175us >> >> ext4 (defaults,noatime): >> >> Version 1.96 ------Sequential Output------ --Sequential Input- --Random- >> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- >> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP >> debian 2G 256 98 21495 6 9896 2 1771 99 125775 11 349.7 5 >> Latency 37738us 992ms 3490ms 10811us 12045us 495ms >> Version 1.96 ------Sequential Create------ --------Random Create-------- >> debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- >> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP >> 16 14766 43 +++++ +++ 18026 46 16094 46 +++++ +++ 17428 45 >> Latency 424us 982us 1023us 367us 139us 174us >> >> xfs(defaults,noatime,logbufs=8,logbsize=131072): >> >> Version 1.96 ------Sequential Output------ --Sequential Input- --Random- >> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- >> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP >> debian 2G 476 96 35129 9 12524 3 1417 99 124716 12 445.9 9 >> Latency 19798us 420ms 721ms 14122us 9394us 131ms >> Version 1.96 ------Sequential Create------ --------Random Create-------- >> debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- >> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP >> 16 1552 8 +++++ +++ 1705 11 1675 9 +++++ +++ 1346 8 >> Latency 104ms 291us 48604us 109ms 45us 227ms >> >> It seems that latency is big in whole results, best is for XFS. It is tempting me to think that there are some RAID 5 issues here. It's really strange that block writing for SCSI server disks in RAID5 is no more than 30MB/sec(XFS). I guess I should consider XFS file system or different RAID configuration. May be someone can comment this strange benchmark result? Will very appreciate that. >> >> With regards, Andrey. >> >> 23.09.2011 11:31, Janne Pikkarainen пишет: >>> Hello, >>> >>> On 09/23/2011 08:51 AM, Andrey wrote: >>>> Hello, >>>> >>>> I have a production mail server with maildir++ structure and about >>>> 250GB (~10 millions) of files on the ext3 partition on RAID5. It's >>>> mounted with noatime option. These mail server is responsible to local >>>> delivery and storing mail messages. >>>> >>>> System has Debian Squeeze installed and Exim as MDA + Dovecot as >>>> IMAP+POP3 server. >>>> >>>> Bonnie results are terrible. Sequential output for Block and Rewrite >>>> are 10722ms and 9232ms. So if there is a 1000 messages in the mail >>>> queue load is extremely high, delivery time is very big and server can >>>> hang. I did not see such problems with UFS on FreeBSD server. >>>> >>>> As I understand ext3 file system is really bad for such configurations >>>> with Maildir++ (many smaill files)? Is there a way to decrease disk >>>> latency on ext3 or speed up it? >>>> >>>> With regards, Andrey >>>> >>>> ___ >>> >> >> _______________________________________________ >> Ext3-users mailing list >> [email protected] >> https://www.redhat.com/mailman/listinfo/ext3-users > > _______________________________________________ Ext3-users mailing list [email protected] https://www.redhat.com/mailman/listinfo/ext3-users