Slow linstor queries after recent upgrade

Hi,

I'm using Linstor on 5 node Proxmox 7.x cluster. After recent upgrade 
I've got some performance issues with some linstor queries.

Upgrade: linstor-common:amd64 (1.20.2-1, 1.22.1-1), python-linstor:amd64 
(1.16.0-1, 1.18.0-1), linstor-client:amd64 (1.16.0-1, 1.18.0-1), 
linstor-controller:amd64 (1.20.2-1, 1.22.1-1), linstor-satellite:amd64 
(1.20.2-1, 1.22.1-1), linstor-proxmox:amd64 (6.1.0-1, 7.0.0-1)

Some queries work fast:

# time linstor node list
+------------------------------------------------------------------------------+
| Node   | NodeType| Addresse               s                   | State   |
|================================================|
| node1 | SATELLITE | xxx.xxx.xxx.xx1:3366 (PLAIN) | Online |
| node2 | SATELLITE | xxx.xxx.xxx.xx2:3366 (PLAIN) | Online |
| node3 | SATELLITE | xxx.xxx.xxx.xx3:3366 (PLAIN) | Online |
| node4 | SATELLITE | xxx.xxx.xxx.xx4:3366 (PLAIN) | Online |
| node5 | SATELLITE | xxx.xxx.xxx.xx5:3366 (PLAIN) | Online |
+------------------------------------------------------------------------------+

real    0m0.294s
user    0m0.259s
sys     0m0.033s

But some got very slow, up to a minute. I'm 100% sure it was a lot 
faster before this upgrade:

# time linstor volume list
+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Node     | Resource          | StoragePool               | VolNr | 
MinorNr | DeviceName     |  Allocated    | InUse    | State |
|=================================================================================================|
| node3   | linstor_db        | pool-drbd-mgmt       |         0 |     
1000 | /dev/drbd1000 | 136.80 MiB | InUse    |   UpToDate |
| node4   | linstor_db        | pool-drbd-mgmt       |         0 |       
1000 | /dev/drbd1000 | 141.82 MiB | Unused |   UpToDate |
| node5   | linstor_db        | pool-drbd-mgmt       |         0 |     
1000 | /dev/drbd1000 |   36.43 MiB | Unused |   UpToDate |
| node3   | vm-100-disk-2 | DfltDisklessStorPool |        0 | 1012 | 
/dev/drbd1012 |                      | Unused | TieBreaker |
| node4   | vm-100-disk-2 | pool-drbd-4-5            |        0 |     
1012 | /dev/drbd1012 |   29.90 GiB  | InUse    |   UpToDate |
| node5   | vm-100-disk-2 | pool-drbd-4-5            |        0 |       
1012 | /dev/drbd1012 |   29.90 GiB  | Unused |   UpToDate |
| node1   | vm-101-disk-3 | DfltDisklessStorPool |        0 | 1048 | 
/dev/drbd1048 |                      | Unused | TieBreaker |
| node4   | vm-101-disk-3 | pool-drbd-4-5            |        0 |       
1048 | /dev/drbd1048 | 146.09 GiB | InUse     |   UpToDate |
| node5   | vm-101-disk-3 | pool-drbd-4-5            |        0 |     
1048 | /dev/drbd1048 | 147.09 GiB | Unused |   UpToDate |
| node1   | vm-102-disk-2 | pool-drbd-1-2            |        0 |       
1021 | /dev/drbd1021 |        16 MiB | InUse     | UpToDate |
| node2   | vm-102-disk-2 | pool-drbd-1-2            |        0 |     
1021 | /dev/drbd1021 |        16 MiB | Unused |   UpToDate |
| node3   | vm-102-disk-2 | DfltDisklessStorPool |        0 | 1021 | 
/dev/drbd1021 |                      | Unused | TieBreaker |
| node1   | vm-102-disk-3 | pool-drbd-1-2            |        0 |       
1044 | /dev/drbd1044 |   10.25 GiB | InUse     |   UpToDate |
| node2   | vm-102-disk-3 | pool-drbd-1-2            |        0 |       
1044 | /dev/drbd1044 |   10.25 GiB | Unused |   UpToDate |
| node3   | vm-102-disk-3 | DfltDisklessStorPool |        0 | 1044 | 
/dev/drbd1044 |                      | Unused | TieBreaker |

[...]

+-------------------------------------------------------------------------------------------------------------------------------------------------------------+

real    0m58.888s
user    0m0.229s
sys     0m0.047s

There is 113 volumes (with TieBreakers):

# linstor volume list | grep -c /dev/drbd
113

Performance graphs in Proxmox are all scattered after upgrade, I suppose 
it's related to long query times. It looks like this - top image is 
before upgrade, bottom one is after upgrade: https://imgur.com/a/c5FrS0y

I've changed node on which linstor-controller runs but it didn't help.

Is there anything I can do to make it work like before this upgrade? I 
will gladly provide additional info if needed.

-- 
Best regards
Łukasz Wąsikowski

_______________________________________________
Star us on GITHUB: https://github.com/LINBIT
drbd-user mailing list
[email protected]
https://lists.linbit.com/mailman/listinfo/drbd-user