slow pipes chewing up system time (and real time)?

Lloyd Parkes <[email protected]> Wed, 17 Oct 2012 09:12:05 +1300
Newsgroups gmane.os.netbsd.devel.performance
Message-ID <[email protected]>
Hi,
I have finally got around to modifying the Mercurial CVS conversion tool =
so that it handles the NetBSD CVS repositories and I would like to run =
it on NetBSD to provide daily incremental repository updates, but I =
can't because the NetBSD performance is so bad.

I did all my development work on OS X because that's the biggest box I =
own and converting the src repository requires about 13GB of RAM. When I =
moved to running the tool on NetBSD, I started with the othersrc =
repository because that makes testing somewhat more manageable. My =
problem is that the conversion is five times slower on NetBSD than it is =
on OS X in real time even though CPU usage appears to be comparable.=20

My Mac is 3.4GHz Intel Core i7 with 4 cores and one extra virtual hyper =
thread core for each real core. The OS X 10.8.2 is run from an external =
thunderbolt attached RAID 0 disk pair and the data I'm working with is =
on the internal WD Caviar Black formatted with a case sensitive =
journaled HFS+ file system. The cvs server uses a RAM disk for it's =
temporary storage.

My NetBSD box is a 3.3GHz AMD Phenom II  with 6 cores. NetBSD 6.0 is run =
from an AHCI attached SSD (an old SSD) and the data I'm working with is =
variously on tmpfs or the SSD. The SSD file systems are formatted with =
FFSv2 without logging, with noatime, and for the designated destination =
file system, with async. The use of tmpfs or async doesn't make an =
obvious change to performance.

"time -l" on the Mac and NetBSD give me similar user CPU and system CPU =
numbers for the conversion of othersrc (half a minute and five seconds =
respectively), but real time on the Mac is 68s and on NetBSD it is 359s. =
Even though the reported system time is only five seconds, when I run =
top on NetBSD, one of the CPUs just sits there at 100% system.

I don't think the horrible wall clock time is caused by hardware =
constraints (disk performance etc.) because I have tested this on tmpfs =
with no change and after running ktrace I'm a bit suspicious of the pipe =
performance. I can't see a way to find out how long each system call is =
taking, so I don't know for sure though. The application talks to the =
cvs server with a stop and wait implementation of the cvs protocol and =
the workload is a lot of small requests. I see support in NetBSD for =
pipes handling lots of large requests and I'm wondering is anyone knows =
of any problems with NetBSD pipes handling lots of small requests.

Cheers,
Lloyd