Re: NetBSD Raid5, slow write speeds, using big disks?! - Resolved

On 7/1/26 1:07 AM, smurfd wrote:
> On 6/30/26 11:10 PM, Robert Elz wrote
>> That's pointless as you are just going to use those filesystems again
>> (I presume) in your raid0.conf file (if had unique names for the two,
>> you could use NAME=island_1 and NAME=island_2 - or whatever you choose,
>> so you're not tied to dkN names, which might be different tomorrow)
> Thank you, good point! Will apply for my raid5 now also. Had used 
> /dev/dk7 in my fstab file.
>> :
>>    | gpt add -a 1m -b 1m -t ffs -l island raid0
>>
>>    | dkctl raid0 addwedge island3 1048576 4191152128 ffs
>>    | newfs -O2 -a 1m -b 64k -s -1g /dev/dk9
>>    |
>>    | Where for both raid1 and raid0 i used
>>    | # sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level_0
>>    | 128 1 1 0/1 depeding on raid type
>>
>> Do you mean raid1 and raid0 or raid1 and raid2 ?   Or all 3?
>>
>> It shouldn't matter for mirrors, using a 64KB block size though.
>> And everything appears aligned, so it isn't surprising that:
>>
>>    | 4194304000 bytes transferred in 8.255 secs (508092549 bytes/sec) 
>> CHAZAAAM!
>>
>> goes fairly fast, but that's too fast I suspect, you might want to try
>> that with a bigger than 4GB transfer, the chances are that lots of that
>> might be just writing into your buffer cache (it isn't clear there 
>> whether
>> you're doing a write onto the bare drive, or a file in a filesystem 
>> on the
>> drive).   If it is the latter, just repeat that several times, with 
>> different
>> output file names (irrelevant what the input is).  Just to make sure it
>> is really sustaining that rate when you're well beyond what the 
>> kernel can
>> just buffer.   Your 1TB (almost, 1000GB) spaces should make a filesystem
>> big enough for quite a lot of those files!
>>
>>    | The best i got for raid5 so far, is:
>>    | 4194304000 bytes transferred in 107.170 secs (39136922 bytes/sec)
>>
>> That looks a bit slower than I would expect for locally connected drives
>> (your drives might not be quite as fast as mine, bigger ones tend to
>> transfer faster as they squeeze more bytes into each revolution, but not
>> that much slower).
>>
>>    | In raid5.conf you suggest: 16 1 1 5 that would have me use these
>>    | commands while doing the disks and raid
>>    | gpt add -t raid -l raid5@wd0 -b 64k -a 32k -s 1000g wd0 ****
>>    | gpt add -a 32k -b 64k -t ffs -l island raid5
>>    | newfs -O2 -a 32k -b 64k -s -1g /dev/dk7
>>
>> All looks OK to me.  Show me the gpt labels for the raid5@wdN drives
>> (or at least 1 of them, if they are all the same, at least all the same
>> block numbers, they should have different labels, but if they don't, 
>> that's
>> harmless for now) - just "gpt show -t wd0" (etc) should be enough.
>>
>> And then the label for the raid5 (your dk7 there).
>>
>> You should drop the "-a 32k" on the newfs, that doesn't do what you 
>> think
>> it does, I believe it probably does nothing at all - but there's an 
>> outside
>> chance that it might be the problem if it really is limiting I/O to 
>> being 32K
>> (which was once its purpose, I think it should be obsolete now - it 
>> certainly
>> has nothing to do with alignment).
>>
>> kre
>>
>>
> CASE CLOSED! :)
> i think the 16 sectpersu and 32k align was the secret sauce. Now my 
> small 3 drive raid5 is speedy, like 400mb/s writes!
> So i will skip raid10 for now, and see if adding the 4th drive to the 
> raid5 makes difference. I had no -t option to gpt show, and remove the 
> align for newfs... then a last mail with the final config of the whole 
> disks...
> Writes below, were done right after eachother...
>
> Soooo many thanks! atleast the ai was correct in that, i was not crazy :D
>
> nas1# gpt show wd0
>         start         size  index  contents
>             0            1         PMBR (active)
>             1            1         Pri GPT header
>             2           32         Pri GPT table
>            34        65502         Unused
>         65536   2097152000      1  GPT part - NetBSD RAIDFrame component
>    2097217536  13530835599         Unused
>   15628053135           32         Sec GPT table
>   15628053167            1         Sec GPT header
>
>
> nas1# gpt create -Af raid5
> nas1# gpt add -a 32k -b 64k -t ffs -l island raid5
> /dev/rraid5d: Partition 1 added: 49f48d5a-b10e-11dc-b99b-0019d1879648 
> 65536 4194238272
> nas1# dkctl raid5 addwedge island 65536 4194238272 ffs
> dk7 created successfully.
> nas1# newfs -O2 -a 32k -b 64k -s -1g /dev/dk7
> /dev/rdk7: 2046943.9MB (4192141120 sectors) block size 65536, fragment 
> size 8192
>         using 689 cylinder groups of 2970.94MB, 47535 blks, 94464 inodes.
> super-block backups (for fsck_ffs -b #) at:
> 256, 6084736, 12169216, 18253696, 24338176, 30422656, 36507136, 
> 42591616, 48676096, 54760576, 60845056, 66929536, 73014016, 79098496, 
> 85182976, 91267456, 97351936, 103436416, 109520896, 115605376, 
> 121689856, 127774336, 133858816, 139943296, 146027776, 152112256, 
> 158196736,
> ....................................................................................................................................................................................................................................................................................... 
>
> nas1# mount /dev/dk7 /mnt/island
> nas1# dd if=/dev/zero bs=1024k count=1000 of=/mnt/island/test.txt
> 1048576000 bytes transferred in 1.805 secs (580928531 bytes/sec)
> nas1# dd if=/dev/zero bs=4096k count=1000 of=/mnt/island/test.txt
> 4194304000 bytes transferred in 8.762 secs (478692535 bytes/sec)
> nas1# dd if=/dev/zero bs=64k count=100000 of=/mnt/island/test.txt
> 6553600000 bytes transferred in 14.928 secs (439013933 bytes/sec)
> nas1# dd if=/dev/zero bs=64k count=100000 of=/mnt/island/test1.txt
> 6553600000 bytes transferred in 15.075 secs (434733001 bytes/sec)
> nas1# dd if=/dev/zero bs=64k count=100000 of=/mnt/island/test2.txt
> 6553600000 bytes transferred in 14.967 secs (437869980 bytes/sec)
> nas1# dd if=/dev/zero bs=4096k count=1000 of=/mnt/island/test3.txt
> 4194304000 bytes transferred in 9.063 secs (462794218 bytes/sec)
> nas1# dd if=/dev/zero bs=4096k count=1000 of=/mnt/island/test4.txt
> 4194304000 bytes transferred in 9.267 secs (452606453 bytes/sec)
> nas1# dd if=/dev/zero bs=8196k count=1000 of=/mnt/island/test5.txt
> 8392704000 bytes transferred in 19.717 secs (425658264 bytes/sec)
> nas1# dd if=/dev/zero bs=10m count=1000 of=/mnt/island/test6.txt
> 10485760000 bytes transferred in 24.011 secs (436706509 bytes/sec)
>
This problem is now resolved!!
If you have arrived here, with a similar problem (should be applicable 
for softraid for *BSD).
This is what, after alot of help in this mailthread, i did.
The secret i believe was sectPerSU=16 and a=32k. In the man page for 
raidctl, it states that
for raid5, use sectPerSU=16, but me thinking that this was outdated info 
or for smaller disks
was incorrect. Another thing that i believe bite me, was that the gpt 
destroy and raidctl -u
commands did not really clear the previous raid config. Im not sure 
about this, but, its a
feeling i have... so i started doing a dd zero to /dev/rraid5 before 
recreating a raid.

This is the /etc/raid5.conf file i use. While trying i had it as 
/etc/r5.conf so if i needed
to reboot, it would not start...

/etc/raid5.conf
START array
# numRow numCol numSpare
1 3 1

START disks
/dev/dk3
/dev/dk4
/dev/dk5

START spare
/dev/dk6

START layout
# sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level_5
16 1 1 5

START queue
fifo 127

These are the commands i ran:
---
# destroy possible previous raid5 config, also important while testing
gpt destroy raid5
raidctl -u raid5
dd if=/dev/zero of=/dev/rraid5 bs=1024k count=1  # i saw this was needed 
to really clear its parameters

gpt destroy wd0 **** # **** means for each of the 4 disks, wd0,wd1,wd2,wd3
gpt create -Af wd0 ***
gpt add -t raid -l raid5@wd0 -a 32k -b 64k wd0 **** # start with small 
disks until you get the speed (-s 1000g), then remove the size

raidctl -C /etc/raid5.conf raid5
raidctl -I 13371337 raid5
raidctl -iv raid5. # this takes many hours, maby between 8 and 72+ 
hours, depending on size

gpt create -Af raid5
gpt add -a 32k -b 64k -t ffs -l island raid5
dkctl raid5 addwedge island <last two values from above commands output> ffs
newfs -O2 -a 32k -b 64k -s -1g /dev/dk7  # reserve 1g for journal files
mount /dev/dk7 /mnt/island
---
Tried the above configs, but instead of 3 disk + 1 spare, i used 4 
disks. Speeds were alot slower, so i stuck to 3 + 1 disks
4 disks
# dd if=/dev/zero bs=1024k count=1000 of=/mnt/island/test.txt
1048576000 bytes transferred in 17.498 secs (59925477 bytes/sec)
# dd if=/dev/zero bs=4096k count=1000 of=/mnt/island/test.txt
4194304000 bytes transferred in 122.339 secs (34284275 bytes/sec)

3 disks
# dd if=/dev/zero bs=1024k count=1000 of=/mnt/island/test.txt
1048576000 bytes transferred in 1.695 secs (618628908 bytes/sec)
# dd if=/dev/zero bs=4096k count=1000 of=/mnt/island/test.txt
4194304000 bytes transferred in 8.912 secs (470635547 bytes/sec)
# dd if=/dev/zero bs=4096k count=3000 of=/mnt/island/test.txt
12582912000 bytes transferred in 28.801 secs (436891496 bytes/sec)

Tried the above dd commands a couple of times to verify that the speeds 
were similar, which they were.
Used different out-filenames

/etc/fstab add row, once done, since /dev/dk7 might change at reboot. 
Where NAME= is the same value you
use in the -l parameter for the raid.
NAME=island     /mnt/island     ffs     rw,-m1666,log     1 1

# df -h |grep dk7
/dev/dk7        14T    27G    14T   0% /mnt/island


Tips:
Check dmesg after each command. When you create the raid with the 
raidctl -C,
it might complain with fatal errors, but they are ignored. that seems to 
be okey!

Raidctl -iv should take time. If instant, something is wrong!

To save time on the raidctl -iv, start with smaller disks, like 1TB.

If you get this error, if i remember correctly when you create the gpt 
for raid5,
you can clear that with: dd if=/dev/zero of=/dev/rraid5 bs=1024k count=1
gpt: /dev/rraid5: map entry doesn't fit media: new start + new size < 
start + size

Again, super thanks!
ps. no commentary needed! :)
Br