Re: Re: can enbd server run in kernel?
"Peter T. Breuer" <[email protected]>
| Newsgroups | gmane.linux.enbd.general |
|---|---|
| Message-ID | <[email protected]> |
"Also sprach Ming Zhang:"
> i wrote a small code to test the mmap performance. and i get some
> results different from you. i guess the kernel version may have impact
Well, it will depend on the speed of the cpu and memory too! I am
using a P3 700MHz.
> on it or because of some parameters.
I really doubt it's very significant - I am using 2.6.3.
> what kernel i use, from a fedora core 2 box
> [mingz@whoami mmap]$ uname -a
> Linux whoami.localdomain 2.6.8-1.521 #1 Mon Aug 16 09:01:18 EDT 2004
> i686 i686 i386 GNU/Linux
>
> test c file attached. compile with
> gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64
> test.c
>
> change the len from 1024768 to 1024768*100 do two tests. dd a 600MB file
> 'test' as the data file.
>
> 1) with len = 1MB
>
> [mingz@whoami mmap]$ strace -c ./a.out
> execve("./a.out", ["./a.out"], [/* 35 vars */]) = 0
> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 26.86 0.000047 9 5 old_mmap
How big are the mmap's? I see that sort of speed (to within a factor of
2) for 4KB!
> 25.14 0.000044 11 4 1 open
> 8.57 0.000015 8 2 mprotect
> 8.57 0.000015 15 1 mmap2
> 6.86 0.000012 12 1 munmap
> 5.71 0.000010 5 2 fstat64
> 5.14 0.000009 9 1 read
> 4.57 0.000008 4 2 close
> 3.43 0.000006 6 1 uname
> 2.86 0.000005 5 1 set_thread_area
> 2.29 0.000004 4 1 brk
> ------ ----------- ----------- --------- --------- ----------------
> 100.00 0.000175 21 1 total
Well, I ran the test and got this:
execve("./a.out", ["./a.out"], [/* 65 vars */]) = 0
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
61.66 0.000119 30 4 1 open
18.65 0.000036 7 5 old_mmap
5.18 0.000010 10 1 read
5.18 0.000010 10 1 munmap
3.11 0.000006 6 1 mprotect
2.07 0.000004 2 2 fstat
1.55 0.000003 2 2 close
1.55 0.000003 3 1 brk
1.04 0.000002 2 1 getpid
------ ----------- ----------- --------- --------- ----------------
100.00 0.000193 18 1 total
So you are slow :). Something is wrong with the code.
>
> 2) with len = 100MB
>
> [mingz@whoami mmap]$ strace -c ./a.out
> execve("./a.out", ["./a.out"], [/* 35 vars */]) = 0
> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 26.63 0.000049 12 4 1 open
> 26.63 0.000049 10 5 old_mmap
That's simply impossible.
> 9.24 0.000017 17 1 mmap2
> 8.15 0.000015 8 2 mprotect
> 6.52 0.000012 12 1 munmap
> 5.43 0.000010 5 2 fstat64
> 4.89 0.000009 9 1 read
> 4.35 0.000008 4 2 close
> 3.26 0.000006 6 1 uname
> 2.72 0.000005 5 1 set_thread_area
> 2.17 0.000004 4 1 brk
> ------ ----------- ----------- --------- --------- ----------------
> 100.00 0.000184 21 1 total
>
> so there is no big difference on total time.
Uh - the pages are not used. Copy on write! I get the same. The newly
made test file (I used O_CREAT) is zero length.
---xr-s--x 1 ptb 0 Nov 28 16:58 test*
OK, fixing to make the file correctly:
-rw-r--r-- 1 ptb 1024768 Nov 28 17:05 test
I also added a write to the length of the file via memset, then munmap
it. Note that the munmap took time to write to disk ;-).
% strace -c ./a.out
execve("./a.out", ["./a.out"], [/* 65 vars */]) = 0
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
44.70 0.000253 127 2 munmap
21.55 0.000122 122 1 write
21.55 0.000122 31 4 1 open
5.48 0.000031 6 5 old_mmap
2.30 0.000013 13 1 read
1.41 0.000008 8 1 mprotect
0.88 0.000005 3 2 fstat
0.71 0.000004 2 2 close
0.53 0.000003 3 1 brk
0.53 0.000003 3 1 _llseek
0.35 0.000002 2 1 getpid
------ ----------- ----------- --------- --------- ----------------
100.00 0.000566 21 1 total
Let's try with 100MB
% strace -c ~/lang/c/nbd/nbd-2.4.32/a.out
execve("/usr/oboe/ptb/lang/c/nbd/nbd-2.4.32/a.out",
["/usr/oboe/ptb/lang/c/nbd/nbd-2.4.32/a.out"], [/* 65 vars */]) = 0
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
99.08 0.017001 8501 2 munmap
0.41 0.000071 18 4 1 open
0.22 0.000038 8 5 old_mmap
0.09 0.000016 8 2 close
0.07 0.000012 12 1 read
0.04 0.000007 7 1 mprotect
0.03 0.000006 3 2 fstat
0.02 0.000003 3 1 brk
0.02 0.000003 3 1 _llseek
0.01 0.000001 1 1 write
0.01 0.000001 1 1 getpid
------ ----------- ----------- --------- --------- ----------------
100.00 0.017159 21 1 total
Again, very reasonable. Now I'll try an msync ms_invalidate before the
munmap. It should take no time out of the write to disk. At 1MB:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
67.87 0.000919 184 5 old_mmap
15.81 0.000214 214 1 msync
10.78 0.000146 73 2 munmap
2.95 0.000040 10 4 1 open
0.81 0.000011 11 1 read
0.52 0.000007 7 1 mprotect
0.44 0.000006 3 2 fstat
0.30 0.000004 2 2 close
0.22 0.000003 3 1 brk
0.15 0.000002 2 1 _llseek
0.07 0.000001 1 1 write
0.07 0.000001 1 1 getpid
------ ----------- ----------- --------- --------- ----------------
100.00 0.001354 22 1 total
Interesting - the msync took time out of the munmap when it should not
have. I didn't ask to sync to disk, just to invalidate other pages
pointing at the same place from VMS.
At 100MB, qualitatively the same:
% strace -c ~/lang/c/nbd/nbd-2.4.32/a.out
execve("/usr/oboe/ptb/lang/c/nbd/nbd-2.4.32/a.out",
["/usr/oboe/ptb/lang/c/nbd/nbd-2.4.32/a.out"], [/* 65 vars */]) = 0
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
57.17 0.018991 18991 1 msync
42.15 0.014002 7001 2 munmap
0.23 0.000075 75 1 mprotect
0.14 0.000045 11 4 1 open
0.12 0.000040 8 5 old_mmap
0.09 0.000031 31 1 write
0.05 0.000017 17 1 read
0.01 0.000004 2 2 close
0.01 0.000004 4 1 brk
0.01 0.000004 2 2 fstat
0.01 0.000003 3 1 _llseek
0.01 0.000002 2 1 getpid
------ ----------- ----------- --------- --------- ----------------
100.00 0.033218 22 1 total
I guess this gives me the chance to check how fast writing to a mmap
is, compared to writing to a file descriptor. Here it is at 1MB:
% strace -c ~/lang/c/nbd/nbd-2.4.32/a.out
execve("/usr/oboe/ptb/lang/c/nbd/nbd-2.4.32/a.out",
["/usr/oboe/ptb/lang/c/nbd/nbd-2.4.32/a.out"], [/* 65 vars */]) = 0
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
91.98 0.014155 56 252 write
6.48 0.000998 333 3 _llseek
0.77 0.000118 30 4 1 open
0.21 0.000032 6 5 old_mmap
0.21 0.000032 16 2 munmap
0.08 0.000012 12 1 read
0.08 0.000012 4 3 close
0.07 0.000011 3 4 brk
0.05 0.000008 3 3 fstat
0.05 0.000007 7 1 mprotect
0.02 0.000003 3 1 fcntl
0.01 0.000002 2 1 getpid
------ ----------- ----------- --------- --------- ----------------
100.00 0.015390 280 1 total
I suppose those mmaps are of shared libraries. At 100MB:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
99.95 0.413549 17 25020 write
0.01 0.000041 10 4 1 open
0.01 0.000033 17 2 munmap
0.01 0.000032 6 5 old_mmap
0.01 0.000028 9 3 fstat
0.00 0.000017 4 4 1 close
0.00 0.000015 15 1 read
0.00 0.000010 3 4 brk
0.00 0.000008 8 1 mprotect
0.00 0.000007 2 3 _llseek
0.00 0.000004 4 1 fcntl
0.00 0.000003 3 1 getpid
------ ----------- ----------- --------- --------- ----------------
100.00 0.413747 25049 2 total
(I wrote 4K at a time).
>
> fstat64(3, {st_mode=S_IFREG|0755, st_size=1459344, ...}) = 0
> old_mmap(0xbee000, 1162188, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) =
That's 1MB.
> 0xbee000
> old_mmap(0xd04000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED,
> 3, 0x116000) = 0xd04000
That's 16KB.
> old_mmap(0xd08000, 7116, PROT_READ|PROT_WRITE,
And less.
> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xd08000
> close(3) = 0
> old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
> -1, 0) = 0xf6ff0000
> mprotect(0xd04000, 4096, PROT_READ) = 0
> mprotect(0xbea000, 4096, PROT_READ) = 0
> set_thread_area({entry_number:-1 -> 6, base_addr:0xf6ff0300,
> limit:1048575, seg_32bit:1, contents:0, read_exec_only:0,
> limit_in_pages:1, seg_not_present:0, useable:1}) = 0
> munmap(0xf6ff1000, 58108) = 0
> open("test", O_RDWR|O_LARGEFILE) = 3
> mmap2(NULL, 102476800, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) =
> 0xf0e35000
> exit_group(-253538304) = ?
>
> see this mmap2 usage.
>
> not sure if these will change when multiple processes open it.
They will, but I only have one process in the tests.
#define _LARGEFILE_SOURCE 1
#define _LARGEFILE64_SOURCE 1
#define _FILE_OFFSET_BITS 64
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
int main(int argc, char **argv)
{
int fd;
char *t;
int len = 1024768 * 100;
//int len = 1024768;
char *path = "test";
fd = open(path, O_RDWR|O_CREAT|O_LARGEFILE, 0644);
if (fd < 0)
printf("fail to open file\n");
if (lseek(fd, len - 1, SEEK_SET) != len - 1)
printf("fail to lseek to %d\n", len - 1);
if (write(fd, " ", 1) != 1)
printf("fail to write 1 byte at %d\n", len - 1);
if (0) {
t = mmap(0, len, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
if ((long)t == -1)
printf("fail to mmap");
memset(t, 0, len);
msync(t, len, MS_INVALIDATE);
munmap(t, len);
} else {
FILE *f;
lseek(fd, 0, SEEK_SET);
f = fdopen(fd, "r+");
if (!f)
printf("fail to open file");
while (len > 0) {
fprintf(f, "%-*s", 4096, "");
len -= 4096;
}
fclose(f);
close(fd);
}
}