Re: Re: can enbd server run in kernel?
Ming Zhang <[email protected]>
| Newsgroups | gmane.linux.enbd.general |
|---|---|
| Message-ID | <[email protected]> |
On Sun, 2004-11-28 at 11:32, Peter T. Breuer wrote:
> "Also sprach Ming Zhang:"
> > i wrote a small code to test the mmap performance. and i get some
> > results different from you. i guess the kernel version may have impact
>
> Well, it will depend on the speed of the cpu and memory too! I am
> using a P3 700MHz.
i am using PM 1.3GHZ. You did not buy a new box this black friday? I
think u p3 box is so slow for u. try to change a faster one, you can
bring out stuff faster, and we all can benefit from that. :P
>
> > on it or because of some parameters.
>
> I really doubt it's very significant - I am using 2.6.3.
>
one difference here i saw is i have mmap2 in strace output. but u do not
have that.
> > what kernel i use, from a fedora core 2 box
> > [mingz@whoami mmap]$ uname -a
> > Linux whoami.localdomain 2.6.8-1.521 #1 Mon Aug 16 09:01:18 EDT 2004
> > i686 i686 i386 GNU/Linux
> >
> > test c file attached. compile with
> > gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64
> > test.c
> >
> > change the len from 1024768 to 1024768*100 do two tests. dd a 600MB file
> > 'test' as the data file.
> >
> > 1) with len = 1MB
> >
> > [mingz@whoami mmap]$ strace -c ./a.out
> > execve("./a.out", ["./a.out"], [/* 35 vars */]) = 0
> > % time seconds usecs/call calls errors syscall
> > ------ ----------- ----------- --------- --------- ----------------
> > 26.86 0.000047 9 5 old_mmap
>
> How big are the mmap's? I see that sort of speed (to within a factor of
> 2) for 4KB!
>
> > 25.14 0.000044 11 4 1 open
> > 8.57 0.000015 8 2 mprotect
> > 8.57 0.000015 15 1 mmap2
mmap2 is the real call for data file on my system. the old_mmap is only
for system library.
> > 6.86 0.000012 12 1 munmap
> > 5.71 0.000010 5 2 fstat64
> > 5.14 0.000009 9 1 read
> > 4.57 0.000008 4 2 close
> > 3.43 0.000006 6 1 uname
> > 2.86 0.000005 5 1 set_thread_area
> > 2.29 0.000004 4 1 brk
> > ------ ----------- ----------- --------- --------- ----------------
> > 100.00 0.000175 21 1 total
>
> Well, I ran the test and got this:
>
> execve("./a.out", ["./a.out"], [/* 65 vars */]) = 0
> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 61.66 0.000119 30 4 1 open
> 18.65 0.000036 7 5 old_mmap
> 5.18 0.000010 10 1 read
> 5.18 0.000010 10 1 munmap
> 3.11 0.000006 6 1 mprotect
> 2.07 0.000004 2 2 fstat
> 1.55 0.000003 2 2 close
> 1.55 0.000003 3 1 brk
> 1.04 0.000002 2 1 getpid
> ------ ----------- ----------- --------- --------- ----------------
> 100.00 0.000193 18 1 total
>
> So you are slow :). Something is wrong with the code.
>
> >
> > 2) with len = 100MB
> >
> > [mingz@whoami mmap]$ strace -c ./a.out
> > execve("./a.out", ["./a.out"], [/* 35 vars */]) = 0
> > % time seconds usecs/call calls errors syscall
> > ------ ----------- ----------- --------- --------- ----------------
> > 26.63 0.000049 12 4 1 open
> > 26.63 0.000049 10 5 old_mmap
>
> That's simply impossible.
>
> > 9.24 0.000017 17 1 mmap2
> > 8.15 0.000015 8 2 mprotect
> > 6.52 0.000012 12 1 munmap
> > 5.43 0.000010 5 2 fstat64
> > 4.89 0.000009 9 1 read
> > 4.35 0.000008 4 2 close
> > 3.26 0.000006 6 1 uname
> > 2.72 0.000005 5 1 set_thread_area
> > 2.17 0.000004 4 1 brk
> > ------ ----------- ----------- --------- --------- ----------------
> > 100.00 0.000184 21 1 total
> >
> > so there is no big difference on total time.
>
> Uh - the pages are not used. Copy on write! I get the same. The newly
> made test file (I used O_CREAT) is zero length.
i create file externally via dd. though u method should be ok as well.
>
> ---xr-s--x 1 ptb 0 Nov 28 16:58 test*
>
> OK, fixing to make the file correctly:
>
> -rw-r--r-- 1 ptb 1024768 Nov 28 17:05 test
>
> I also added a write to the length of the file via memset, then munmap
> it. Note that the munmap took time to write to disk ;-).
>
> % strace -c ./a.out
> execve("./a.out", ["./a.out"], [/* 65 vars */]) = 0
> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 44.70 0.000253 127 2 munmap
> 21.55 0.000122 122 1 write
> 21.55 0.000122 31 4 1 open
> 5.48 0.000031 6 5 old_mmap
> 2.30 0.000013 13 1 read
> 1.41 0.000008 8 1 mprotect
> 0.88 0.000005 3 2 fstat
> 0.71 0.000004 2 2 close
> 0.53 0.000003 3 1 brk
> 0.53 0.000003 3 1 _llseek
> 0.35 0.000002 2 1 getpid
> ------ ----------- ----------- --------- --------- ----------------
> 100.00 0.000566 21 1 total
>
>
> Let's try with 100MB
>
> % strace -c ~/lang/c/nbd/nbd-2.4.32/a.out
> execve("/usr/oboe/ptb/lang/c/nbd/nbd-2.4.32/a.out",
> ["/usr/oboe/ptb/lang/c/nbd/nbd-2.4.32/a.out"], [/* 65 vars */]) = 0
> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 99.08 0.017001 8501 2 munmap
> 0.41 0.000071 18 4 1 open
> 0.22 0.000038 8 5 old_mmap
> 0.09 0.000016 8 2 close
> 0.07 0.000012 12 1 read
> 0.04 0.000007 7 1 mprotect
> 0.03 0.000006 3 2 fstat
> 0.02 0.000003 3 1 brk
> 0.02 0.000003 3 1 _llseek
> 0.01 0.000001 1 1 write
> 0.01 0.000001 1 1 getpid
> ------ ----------- ----------- --------- --------- ----------------
> 100.00 0.017159 21 1 total
>
> Again, very reasonable. Now I'll try an msync ms_invalidate before the
> munmap. It should take no time out of the write to disk. At 1MB:
>
> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 67.87 0.000919 184 5 old_mmap
> 15.81 0.000214 214 1 msync
> 10.78 0.000146 73 2 munmap
> 2.95 0.000040 10 4 1 open
> 0.81 0.000011 11 1 read
> 0.52 0.000007 7 1 mprotect
> 0.44 0.000006 3 2 fstat
> 0.30 0.000004 2 2 close
> 0.22 0.000003 3 1 brk
> 0.15 0.000002 2 1 _llseek
> 0.07 0.000001 1 1 write
> 0.07 0.000001 1 1 getpid
> ------ ----------- ----------- --------- --------- ----------------
> 100.00 0.001354 22 1 total
>
> Interesting - the msync took time out of the munmap when it should not
> have. I didn't ask to sync to disk, just to invalidate other pages
> pointing at the same place from VMS.
>
so we may misunderstand the usage of msync, it does both, write to disk
and invalidate pages. or some tricky flag?
>
> At 100MB, qualitatively the same:
>
> % strace -c ~/lang/c/nbd/nbd-2.4.32/a.out
> execve("/usr/oboe/ptb/lang/c/nbd/nbd-2.4.32/a.out",
> ["/usr/oboe/ptb/lang/c/nbd/nbd-2.4.32/a.out"], [/* 65 vars */]) = 0
> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 57.17 0.018991 18991 1 msync
> 42.15 0.014002 7001 2 munmap
> 0.23 0.000075 75 1 mprotect
> 0.14 0.000045 11 4 1 open
> 0.12 0.000040 8 5 old_mmap
> 0.09 0.000031 31 1 write
> 0.05 0.000017 17 1 read
> 0.01 0.000004 2 2 close
> 0.01 0.000004 4 1 brk
> 0.01 0.000004 2 2 fstat
> 0.01 0.000003 3 1 _llseek
> 0.01 0.000002 2 1 getpid
> ------ ----------- ----------- --------- --------- ----------------
> 100.00 0.033218 22 1 total
>
> I guess this gives me the chance to check how fast writing to a mmap
> is, compared to writing to a file descriptor. Here it is at 1MB:
>
> % strace -c ~/lang/c/nbd/nbd-2.4.32/a.out
> execve("/usr/oboe/ptb/lang/c/nbd/nbd-2.4.32/a.out",
> ["/usr/oboe/ptb/lang/c/nbd/nbd-2.4.32/a.out"], [/* 65 vars */]) = 0
> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 91.98 0.014155 56 252 write
> 6.48 0.000998 333 3 _llseek
> 0.77 0.000118 30 4 1 open
> 0.21 0.000032 6 5 old_mmap
> 0.21 0.000032 16 2 munmap
> 0.08 0.000012 12 1 read
> 0.08 0.000012 4 3 close
> 0.07 0.000011 3 4 brk
> 0.05 0.000008 3 3 fstat
> 0.05 0.000007 7 1 mprotect
> 0.02 0.000003 3 1 fcntl
> 0.01 0.000002 2 1 getpid
> ------ ----------- ----------- --------- --------- ----------------
> 100.00 0.015390 280 1 total
>
> I suppose those mmaps are of shared libraries. At 100MB:
>
> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 99.95 0.413549 17 25020 write
> 0.01 0.000041 10 4 1 open
> 0.01 0.000033 17 2 munmap
> 0.01 0.000032 6 5 old_mmap
> 0.01 0.000028 9 3 fstat
> 0.00 0.000017 4 4 1 close
> 0.00 0.000015 15 1 read
> 0.00 0.000010 3 4 brk
> 0.00 0.000008 8 1 mprotect
> 0.00 0.000007 2 3 _llseek
> 0.00 0.000004 4 1 fcntl
> 0.00 0.000003 3 1 getpid
> ------ ----------- ----------- --------- --------- ----------------
> 100.00 0.413747 25049 2 total
>
>
> (I wrote 4K at a time).
>
>
> >
>
> > fstat64(3, {st_mode=S_IFREG|0755, st_size=1459344, ...}) = 0
> > old_mmap(0xbee000, 1162188, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) =
>
> That's 1MB.
>
> > 0xbee000
> > old_mmap(0xd04000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED,
> > 3, 0x116000) = 0xd04000
>
> That's 16KB.
>
> > old_mmap(0xd08000, 7116, PROT_READ|PROT_WRITE,
>
> And less.
>
> > MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xd08000
> > close(3) = 0
> > old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
> > -1, 0) = 0xf6ff0000
> > mprotect(0xd04000, 4096, PROT_READ) = 0
> > mprotect(0xbea000, 4096, PROT_READ) = 0
> > set_thread_area({entry_number:-1 -> 6, base_addr:0xf6ff0300,
> > limit:1048575, seg_32bit:1, contents:0, read_exec_only:0,
> > limit_in_pages:1, seg_not_present:0, useable:1}) = 0
> > munmap(0xf6ff1000, 58108) = 0
> > open("test", O_RDWR|O_LARGEFILE) = 3
> > mmap2(NULL, 102476800, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) =
> > 0xf0e35000
> > exit_group(-253538304) = ?
> >
> > see this mmap2 usage.
> >
> > not sure if these will change when multiple processes open it.
>
> They will, but I only have one process in the tests.
>
> #define _LARGEFILE_SOURCE 1
> #define _LARGEFILE64_SOURCE 1
> #define _FILE_OFFSET_BITS 64
>
>
> #include <stdio.h>
> #include <stdlib.h>
> #include <sys/mman.h>
> #include <sys/types.h>
> #include <sys/stat.h>
> #include <fcntl.h>
> #include <unistd.h>
>
> int main(int argc, char **argv)
> {
> int fd;
> char *t;
> int len = 1024768 * 100;
> //int len = 1024768;
> char *path = "test";
>
> fd = open(path, O_RDWR|O_CREAT|O_LARGEFILE, 0644);
> if (fd < 0)
> printf("fail to open file\n");
> if (lseek(fd, len - 1, SEEK_SET) != len - 1)
> printf("fail to lseek to %d\n", len - 1);
> if (write(fd, " ", 1) != 1)
> printf("fail to write 1 byte at %d\n", len - 1);
> if (0) {
> t = mmap(0, len, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
> if ((long)t == -1)
> printf("fail to mmap");
> memset(t, 0, len);
> msync(t, len, MS_INVALIDATE);
> munmap(t, len);
> } else {
> FILE *f;
> lseek(fd, 0, SEEK_SET);
> f = fdopen(fd, "r+");
> if (!f)
> printf("fail to open file");
> while (len > 0) {
> fprintf(f, "%-*s", 4096, "");
> len -= 4096;
> }
> fclose(f);
> close(fd);
> }
>
> }