Re: NetBSD Raid5, slow write speeds, using big disks?!

On 6/30/26 11:10 PM, Robert Elz wrote
> That's pointless as you are just going to use those filesystems again
> (I presume) in your raid0.conf file (if had unique names for the two,
> you could use NAME=island_1 and NAME=island_2 - or whatever you choose,
> so you're not tied to dkN names, which might be different tomorrow)
Thank you, good point! Will apply for my raid5 now also. Had used 
/dev/dk7 in my fstab file.
> :
>    | gpt add -a 1m -b 1m -t ffs -l island raid0
>
>    | dkctl raid0 addwedge island3 1048576 4191152128 ffs
>    | newfs -O2 -a 1m -b 64k -s -1g /dev/dk9
>    |
>    | Where for both raid1 and raid0 i used
>    | # sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level_0
>    | 128 1 1 0/1 depeding on raid type
>
> Do you mean raid1 and raid0 or raid1 and raid2 ?   Or all 3?
>
> It shouldn't matter for mirrors, using a 64KB block size though.
> And everything appears aligned, so it isn't surprising that:
>
>    | 4194304000 bytes transferred in 8.255 secs (508092549 bytes/sec) CHAZAAAM!
>
> goes fairly fast, but that's too fast I suspect, you might want to try
> that with a bigger than 4GB transfer, the chances are that lots of that
> might be just writing into your buffer cache (it isn't clear there whether
> you're doing a write onto the bare drive, or a file in a filesystem on the
> drive).   If it is the latter, just repeat that several times, with different
> output file names (irrelevant what the input is).  Just to make sure it
> is really sustaining that rate when you're well beyond what the kernel can
> just buffer.   Your 1TB (almost, 1000GB) spaces should make a filesystem
> big enough for quite a lot of those files!
>
>    | The best i got for raid5 so far, is:
>    | 4194304000 bytes transferred in 107.170 secs (39136922 bytes/sec)
>
> That looks a bit slower than I would expect for locally connected drives
> (your drives might not be quite as fast as mine, bigger ones tend to
> transfer faster as they squeeze more bytes into each revolution, but not
> that much slower).
>
>    | In raid5.conf you suggest: 16 1 1 5 that would have me use these
>    | commands while doing the disks and raid
>    | gpt add -t raid -l raid5@wd0 -b 64k -a 32k -s 1000g wd0 ****
>    | gpt add -a 32k -b 64k -t ffs -l island raid5
>    | newfs -O2 -a 32k -b 64k -s -1g /dev/dk7
>
> All looks OK to me.  Show me the gpt labels for the raid5@wdN drives
> (or at least 1 of them, if they are all the same, at least all the same
> block numbers, they should have different labels, but if they don't, that's
> harmless for now) - just "gpt show -t wd0" (etc) should be enough.
>
> And then the label for the raid5 (your dk7 there).
>
> You should drop the "-a 32k" on the newfs, that doesn't do what you think
> it does, I believe it probably does nothing at all - but there's an outside
> chance that it might be the problem if it really is limiting I/O to being 32K
> (which was once its purpose, I think it should be obsolete now - it certainly
> has nothing to do with alignment).
>
> kre
>
>
CASE CLOSED! :)
i think the 16 sectpersu and 32k align was the secret sauce. Now my 
small 3 drive raid5 is speedy, like 400mb/s writes!
So i will skip raid10 for now, and see if adding the 4th drive to the 
raid5 makes difference. I had no -t option to gpt show, and remove the 
align for newfs... then a last mail with the final config of the whole 
disks...
Writes below, were done right after eachother...

Soooo many thanks! atleast the ai was correct in that, i was not crazy :D

nas1# gpt show wd0
         start         size  index  contents
             0            1         PMBR (active)
             1            1         Pri GPT header
             2           32         Pri GPT table
            34        65502         Unused
         65536   2097152000      1  GPT part - NetBSD RAIDFrame component
    2097217536  13530835599         Unused
   15628053135           32         Sec GPT table
   15628053167            1         Sec GPT header


nas1# gpt create -Af raid5
nas1# gpt add -a 32k -b 64k -t ffs -l island raid5
/dev/rraid5d: Partition 1 added: 49f48d5a-b10e-11dc-b99b-0019d1879648 
65536 4194238272
nas1# dkctl raid5 addwedge island 65536 4194238272 ffs
dk7 created successfully.
nas1# newfs -O2 -a 32k -b 64k -s -1g /dev/dk7
/dev/rdk7: 2046943.9MB (4192141120 sectors) block size 65536, fragment 
size 8192
         using 689 cylinder groups of 2970.94MB, 47535 blks, 94464 inodes.
super-block backups (for fsck_ffs -b #) at:
256, 6084736, 12169216, 18253696, 24338176, 30422656, 36507136, 
42591616, 48676096, 54760576, 60845056, 66929536, 73014016, 79098496, 
85182976, 91267456, 97351936, 103436416, 109520896, 115605376, 
121689856, 127774336, 133858816, 139943296, 146027776, 152112256, 158196736,
.......................................................................................................................................................................................................................................................................................
nas1# mount /dev/dk7 /mnt/island
nas1# dd if=/dev/zero bs=1024k count=1000 of=/mnt/island/test.txt
1048576000 bytes transferred in 1.805 secs (580928531 bytes/sec)
nas1# dd if=/dev/zero bs=4096k count=1000 of=/mnt/island/test.txt
4194304000 bytes transferred in 8.762 secs (478692535 bytes/sec)
nas1# dd if=/dev/zero bs=64k count=100000 of=/mnt/island/test.txt
6553600000 bytes transferred in 14.928 secs (439013933 bytes/sec)
nas1# dd if=/dev/zero bs=64k count=100000 of=/mnt/island/test1.txt
6553600000 bytes transferred in 15.075 secs (434733001 bytes/sec)
nas1# dd if=/dev/zero bs=64k count=100000 of=/mnt/island/test2.txt
6553600000 bytes transferred in 14.967 secs (437869980 bytes/sec)
nas1# dd if=/dev/zero bs=4096k count=1000 of=/mnt/island/test3.txt
4194304000 bytes transferred in 9.063 secs (462794218 bytes/sec)
nas1# dd if=/dev/zero bs=4096k count=1000 of=/mnt/island/test4.txt
4194304000 bytes transferred in 9.267 secs (452606453 bytes/sec)
nas1# dd if=/dev/zero bs=8196k count=1000 of=/mnt/island/test5.txt
8392704000 bytes transferred in 19.717 secs (425658264 bytes/sec)
nas1# dd if=/dev/zero bs=10m count=1000 of=/mnt/island/test6.txt
10485760000 bytes transferred in 24.011 secs (436706509 bytes/sec)