Re: AW: Re: WG: AW: AW: AW: WG: Re: read performance much slow er than write

"Also sprach Hoffmann Christian:"
> All tests are using 4 enbd-servers with raid5:
> - on server side without option -n, on client side without merge_requests:
>   read speed about 36MB/s
> 
> - on server side with option -n, on client side without merge_requests:
>   read speed about 36MB/s

It's probable that all 4KB blocks (interleaved with small acks) going
across the network card is the limiting factor here. 

> - on server side without option -n, on client side with merge_requests=512:
>   read speed about 24MB/s (As meusured already in other tests before)

This needs information on the average block size, as measured by the
"Spectrum" output from /proc/ndbdinfo. Let's assume it's about 32KB
(you'll have to tell me).

> - on server side with option -n, on client side with merge_requests=512:
>   read speed about 46MB/s (As meusured already in other tests before)

This is conclusive - the only change made is in the mode that the
server opens the resource in. O_DIRECT or no O_DIRECT. I don't know of
any other change that is made. The server's internal flag for dealing
with this is "F_DIRECT", and it is mostly used in file.c (the unit that
presents the resource to the server code in a generic way):

       #ifdef O_DIRECT
       if (self->flags & F_DIRECT)
           openflags |= O_DIRECT;
       #endif

The ONE other thing it does is take care to do aligned reads when that flag
is set:

           if ((self->flags & F_DIRECT) && ((long)buf & (self->blksize - 1))) {
               res = aligned_read(fd, buf, len, self->blksize);
           } else {
               res = read(fd, buf, len);
           }


You might want to see if forcing the use of aligned_read() instead of
simple read() does the trick, independently of whether or not O_DIRECT
is in the open flags.

> It seems that merge_requests slows down read performance significantly if
> the option -n on the server is not used. If it is used it speeds up the
> performance.

I wouldn't interpret things that way. Allowing the client kernel to
aggregate requests together if it wants to is what "merge_requests"
does. You'd have to show the Spectrum output from /proc/nbdinfo
to see what as really happening. Larger requests should be better
in general, because they  reduce network overhead significantly. If one
graphs request size against network loading (and transfer speed) one
usually sees a logarithmic or asymptotic graph tending to a limit at
the largest size available.

> >I imagine a huge cache and a slow processor might be nasty. What was
> >the output from "free" like during slow reads? Or meminfo? ANy
> >noticable difference wrt fast reads under direct i/o?
> 
> free on servers during the test without -n on servers with merge
> requests=512 on client with raid5 24,5MB/s read speed:
> server1:
>              total       used       free     
> Mem:       1036524     114496     922028          
> -/+ buffers/cache:      30012    1006512
                          ^^^^^^ very little. That's improbable.
> Swap:      1999992          0    1999992
> 
> shared    buffers     cached
>      0      48616      35868
> 
> server2:
>              total       used       free     
> Mem:       1036524     120364     916160          
> -/+ buffers/cache:      32452    1004072
                          ^^^^^^ also little, also improbable.
> Swap:      1999992          0    1999992

These results are not likely - reading the resource ought to put all of
it into the kernel cache. That has not happened. What happens when you
just dd the resource into /dev/null? Does the cache grow any larger?


> >You can compare speed with/without direct i/o on the server side using a
> >simple c program. I'll write it if you like.
> 
> Yes, please since I am not so firm with c.

Change "transfer_size" to suit. It is the unit in which the read is
done.  CHange "resource" and play with "openflags".

 main() {

    char * resource = "/tmp/foo";
    int openflags = O_RDONLY|O_DIRECT;
    int transfer_size = 1024 * 128;
    int block_size = 4096;

    int fd;
    int tot = 0;
    static char * basebuf, * buf;

    if (!basebuf) {
        basebuf = malloc(transfer_size + block_size);
        if (!basebuf) {
            return ENOMEM;
        }
        buf = (char *) (((long) &basebuf[block_size - 1]) & ~(block_size - 1));
    }

    fd = open(resource, openflags);
    if (fd < 0) {
         return errno;
    }

    while (1) {
        int n = 0;
        while (n < transfer_size) {
            int m = read(fd, buf, transfer_size - n);
            if (m <= 0)
                goto end;
            if (m & (block_size - 1) != 0) {
                n += m;
                goto end;
            }
            n += m;
        }
        tot += n;
    }
end:
    printf("%d bytes transfered\n", tot); 
    return 0;
 }

Untested. Uncompiled. Etc.


> >How can you use "2 clients"?
> 
> Oh, sorry I was not clear, I meant 2 different server machines (computers)
> running enbd-clients both running with the same setup but using different
> partions on the servers.

OK. So you are simply running two sets of experiments at the same time,
on the same machines and network. And you see that total thruput is a
constant?

Well, that is very significant. In principle it rules out enbd as being
the immediate cause.

Any two instances of enbd servers know nothing at all about each other.
If you run into a total performance limit for them, then the limit must
act instead through the common resources they access, which are:

     1) the network,
     2) the disks (kernel, etc.).

You see this as a serverside problem? Then the enbd driver is not
involved either. It looks as though the server kernel has trouble 
reading two different disks/partitions simultaneously.

You should be able to see the effect without enbd. I can modify the
C code above to simulate raid-5 patterning? What is the stripe size?
It should really be 4K, I think.


> >Maybe Arne can tell you how to make such measurements.
> 
> Could you please connect me to him so I can ask him to help me out with
> those tests?

He probably is listening! I'll do so in a while.

Peter