Re: Extreme CPU usage by enbd-client?

Peter T. Breuer wrote:
> "Also sprach Bas van Schaik:"
> 
>>Quote (Denis Bonnenfant):
>>
>>>Perfs are pretty good. Without any tweaking,  36 MB/s during
>>>reconstruction over GbE link, and bonnie between 12-20 MB/s Writes / 20-50 MB/s
>>> reads. It's a little bit less for writes thant with direct access to disk, and
>>>slightly more for reads.
>>
>>My eyes dropped out after reading this: 35MB/s during reconstruction?!
> 
> 
> I suppose he might have been using FR1. But higher rates are perfectly
> possible.
It keeps getting better :), I was happy with the 5MB/s I had on the
other machine, but currently (after setting the merge_requests to 9999)
the sync speed is 10MB/s! The total bandwith used now (on my network
interface), is around 32MB/s. I don't exactly know where Denis was
talking about (sync speed or total bandwith on his NIC), but I now
understand that finetuning enbd can really speed things up!


> 
> Are you aggregating requests? What does /proc/nbdinfo show for the
> request sizes? Over a Gfast ethernet you want to aggregate requests
> if you can. Echo something to nbdinfo (I forget what) or echo something
> to a proc/sys area ...
> 
>    % cat /proc/sys/dev/enbd/devices/a/merge_requests
>    0
> 
> Set that to some-large-number.
Done that, it helps a lot! However, I don't really know what those
"request sizes" are exactly, and how to find them in nbdinfo. There's
nothing like "request size" in there?


>>I've tried multiple block sizes, multiple combinations of servers (to
> 
> 
> Well, larger blocksizes should show something!
Didn't show anything at all. What block size would you advise on a
GBit-network?



>>test if one of the servers is the bottleneck), but the difference
>>between a P4 2.8GHz client and a P4 2.6GHz client makes me think the
>>problem lies within the client.
> 
> ??
> 
I tried to explain, that I didn't think the "storage servers" were the
bottleneck, but the client, since the speed dropped from 5MB/s to
3.5MB/s after changing client hardware.


>>One thing which attracts my attention, is the CPU usage on the client.
> 
> Look at nbdinfo for data.
> 
>>FYI, the top output during the reconstruction:
>>
>>>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>> 3072 root      15   0  1932  988 1500 S 10.0  0.2   1:07.01 enbd-client
> 
> 
> Not terribly high, but you seem to have a lot of them!
Maybe I'm making a terrible mistake here, but I've got four different
machines acting as a "storage server", each exporting one block device.
There's one "frontend" machine which connects to the four servers and
imports the block devices. Per block device (= per server), two channels
are used (default). So why are you surprised about the amount of
clients? Am I doing something fundamentally wrong?

--Bas