Re: Multiple CPU support for uvmm on virt-arm64 qemu machine

Philipp Eppelt <[email protected]> Mon, 4 Nov 2024 11:36:16 +0100
Newsgroups	gmane.comp.micro-kernel.l4.devel
Message-ID	<[email protected]>
Hi Mohamed,


Am 04.11.24 um 06:26 schrieb Mohamed Dawod:
> Hi Philipp,
> 
> I'm already using *start_vm()* and setting */cpu/* parameter to the values you 
> mentioned, but it is still working randomly (sometimes works and sometimes hangs 
> or one of the VM hangs).
> 
> Please find attached my ned script :
A note on the script:
You are using "console=hvc0" which is the virtio-console device. This is 
initialized only late in the linux boot process. For early console output I use 
"console=ttyS0 earlyprintk=serial,ttyS0" in the bootargs string.

Since you are using hvc0, I assume you have changed 
uvmm/configs/dts/virt-pc.dts, and enabled the virtio_uart node. I recommend to 
also change the uart8250 node to include the l4vmm,vcon_cap line. Then you also 
need to provide the capability named "uart" to the uvmm. You can do this via the 
ext_caps parameter. For example:

   ext_caps = { uart = vmm.loader.log_fab:create(L4.Proto.Log, "uart") }

Change the "uart" string in the create() function to something you like to 
distinguish the two uvmms.

> 
> Also I can not understand the effect of the */prio/* parameter!
> I have changed it among different values but nothing changed!
> What is the effect of the */prio/* parameter and when/how can I use it ?

I assume you are using the 2024-08 snapshot. In this version, both, the prio and 
cpus parameters are necessary to take effect. (A later version makes this more 
user friendly see [1].)

In more detail: To assign apps like the uvmm to specific cores, L4Re uses 
scheduling proxies, which ensure the application scheduled ontop of the 
scheduling proxy have only access to the resources managed by the proxy. Here 
resources means cores and priority range.

If given prio and cpus parameter, start_vm() creates a scheduling proxy with 
these parameters and limiting priority range to prio+10.
This means:
vm1 runs on cores 0xc with a priority range of [255, 255] (min/max).
vm2 runs on cores 0x3 with a priority range of [3, 13].

3 and 255 are the interesting numbers here: The base priority for l4re apps is 2 
and the given prio parameter just adds to this, so 2+1=3. 2+12345 is 255, 
because 255 is the maximum priority level.

This can be source of the slowdown behavior you are observing, since this 
priority level is the same as for services vm1 depends upon.
My recommendation would be prio=2 for vm1.

Cheers,
Philipp


p.s. I haven't forgotten about your arm64 PCI MSI question, I just need some 
time to set this up myself to be able to give a good answer.


[1] https://github.com/kernkonzept/uvmm/blob/master/configs/vmm.lua#L40


> 
> Thanks in advance,
> Regards
> 
> On Thu, Oct 31, 2024 at 8:47 PM Philipp Eppelt <[email protected] 
> <mailto:[email protected]>> wrote:
> 
>     Hi Mohamed,
> 
>     Am 31.10.24 um 14:30 schrieb Mohamed Dawod:
>      > Thanks Philipp,
>      >
>      > Multiple CPUs worked for virt-arm64 machine
>     Yippie!
> 
>      > I tried to launch 2 linux VMs on top of L4 using uvmm and assign 2 CPUs
>     to one
>      > of the two VMs and another 2 CPUs to the other one.
>      > I noticed that the linux booting process becomes slower and as more CPUs are
>      > added to qemu with -smp option and passed to the VMs as more as the VM
>     booting
>      > becomes more slower!
>      > Also VMs become working randomly (sometimes they work and sometimes they
>     hang or
>      > one of them hangs) >
>      > Why does this strange behaviour happen when using uvmm and 2 Linux VMs  ?
>     To make this easier please show me your ned script starting the VMs.
> 
>     If you use start_vm() please note that the `cpus` parameter takes a bitmap. So
>     make sure to start VM1 with `cpus=0x3` and VM2 with `cpus=0xc` to place them on
>     separate cores of a four core platform (e.g. QEMU with -smp 4).
> 
>     Cheers,
>     Philipp
> 
>      >
>      > Thanks,
>      > Regards
>      >
>      > On Wed, Oct 30, 2024 at 8:46 PM Philipp Eppelt
>     <[email protected] <mailto:[email protected]>
>      > <mailto:[email protected]
>     <mailto:[email protected]>>> wrote:
>      >
>      >     Hi Mohamed,
>      >
>      >     Am 29.10.24 um 10:43 schrieb Mohamed Dawod:
>      >      > Hello,
>      >      > I'm trying to provide multiple CPUs for a linux VM on top of L4.
>      >      > I'm using the qemu virt machine and building for aarch64. so I
>     used *-smp*
>      >      > option to provide more CPUs.
>      >      >
>      >      >     $ qemu-system-aarch64 -M virt,virtualization=true -cpu cortex-a57
>      >     -smp 4 -m
>      >      >     1024 -kernel ....etc....
>      >     I'm not sure which gic version qemu uses. Please try setting it
>     explicitly to
>      >     with gic-version=3 argument:  `-M virt,virtualization=true,gic-version=3`
>      >
>      >      >
>      >      > Unfortunately, This didn't work. I tried to add more CPU device
>     nodes to
>      >     the dts
>      >      > file *virt-arm_virt-64.dts *but it also didn't work. >
>      >      > I think that it's because of the provided interrupt-controller with
>      >      > *virt-arm_virt-64.dts* in /l4/pkg/uvmm/conf/dts/ which mentioned
>     that it
>      >      > supports only one CPU.
>      >      >
>      >      >     icsoc {
>      >      >              compatible = "simple-bus";
>      >      >              #address-cells = <2>;
>      >      >              #size-cells = <2>;
>      >      >              ranges;
>      >      >
>      >      >              /* Uvmm will adapt the compatible string depending on the
>      >     present gic
>      >      >               * version. It expects reg entries that provide
>     enough space
>      >     for the
>      >      >               * Cpu/Dist interface for gicv2 (at least 0x1000,
>     0x1000) or the
>      >      >               * Dist/Redist interface for gicv3 (0x10000, 0x20000 *
>      >     number of cpus).
>      >
>      >     I'm not an expert for ARM64, but judging from the line above I'd say you
>      >     have to
>      >     increase the size of the second reg entry. For example for four cores:
>      >              reg = <0 0x40000 0 0x10000>,
>      >                    <0 0x50000 0 0x80000>;
>      >
>      >     You should be able to just use the github version of this file, it
>     has a gic
>      >     node that is configured for 32 cores and comes with four CPU nodes.
>      >
>     https://github.com/kernkonzept/uvmm/blob/master/configs/dts/virt-arm_virt-64.dts <https://github.com/kernkonzept/uvmm/blob/master/configs/dts/virt-arm_virt-64.dts> <https://github.com/kernkonzept/uvmm/blob/master/configs/dts/virt-arm_virt-64.dts <https://github.com/kernkonzept/uvmm/blob/master/configs/dts/virt-arm_virt-64.dts>>
>      >
>      >
>      >      >               * *The entries provided here support any gicv2 setup
>     or a
>      >     gicv3 setup
>      >      >               * with one Cpu.*
>      >      >               */
>      >      >              gic: interrupt-controller {
>      >      >                  compatible = "arm,gic-400", "arm,cortex-a15-gic",
>      >      >     "arm,cortex-a9-gic";
>      >      >                  #interrupt-cells = <3>;
>      >      >                  #address-cells = <0>;
>      >      >                  interrupt-controller;
>      >      >                  reg = <0 0x40000 0 0x10000>,
>      >      >                        <0 0x50000 0 0x20000>;
>      >      >                  };
>      >      >          };
>      >      >
>      >      >
>      >      > My question now, is there any workaround to support multiple CPUs
>     for virt
>      >      > machine  on arm64  ?
>      >
>      >     Multiple CPUs should work. For SMP there are a couple of things to
>     consider:
>      >     - QEMU: -smp parameter
>      >     - Kernel configuration for SMP and the number of maximum cores
>      >     - The DTS defines the maximum number of cores for the uvmm will set
>     up. So
>      >     adding CPU device nodes is the correct path.
>      >     - The ned script defines the number of cores available at runtime to
>     uvmm. No
>      >     cpu parameter in the start_vm({}) call means the VM gets access to
>     all cpus.
>      >     - Linux must of course also support SMP, but that's very likely not
>     the problem
>      >     here ;-)
>      >
>      >     I hope this sheds some light.
>      >
>      >     Cheers,
>      >     Philipp
>      >
>      >     --
>      > [email protected] <mailto:[email protected]>
>     <mailto:[email protected]
>     <mailto:[email protected]>> -
>      >     Tel. 0351-41 883 221
>      > http://www.kernkonzept.com <http://www.kernkonzept.com>
>     <http://www.kernkonzept.com <http://www.kernkonzept.com>>
>      >
>      >     Kernkonzept GmbH.  Sitz: Dresden.  Amtsgericht Dresden, HRB 31129.
>      >     Geschäftsführer: Dr.-Ing. Michael Hohmuth
>      >     _______________________________________________
>      >     l4-hackers mailing list -- [email protected]
>     <mailto:[email protected]>
>      >     <mailto:[email protected]
>     <mailto:[email protected]>>
>      >     To unsubscribe send an email to [email protected]
>     <mailto:[email protected]>
>      >     <mailto:[email protected]
>     <mailto:[email protected]>>
>      >
>      >
>      >
>      > *Driving Innovation! Visit our website www.avelabs.com
>     <http://www.avelabs.com>
>      > <http://www.avelabs.com/ <http://www.avelabs.com/>>*, to read Avelabs
>     Confidentiality Notice, follow this
>      > link: http://www.avelabs.com/email/disclaimer.html
>     <http://www.avelabs.com/email/disclaimer.html>
>      > <http://www.avelabs.com/email/disclaimer.html
>     <http://www.avelabs.com/email/disclaimer.html>>
> 
>     -- 
>     [email protected] <mailto:[email protected]> -
>     Tel. 0351-41 883 221
>     http://www.kernkonzept.com <http://www.kernkonzept.com>
> 
>     Kernkonzept GmbH.  Sitz: Dresden.  Amtsgericht Dresden, HRB 31129.
>     Geschäftsführer: Dr.-Ing. Michael Hohmuth
> 
> 
> 
> *Driving Innovation! Visit our website www.avelabs.com 
> <http://www.avelabs.com/>*, to read Avelabs Confidentiality Notice, follow this 
> link: http://www.avelabs.com/email/disclaimer.html 
> <http://www.avelabs.com/email/disclaimer.html>

-- 
[email protected] - Tel. 0351-41 883 221
http://www.kernkonzept.com

Kernkonzept GmbH.  Sitz: Dresden.  Amtsgericht Dresden, HRB 31129.
Geschäftsführer: Dr.-Ing. Michael Hohmuth

_______________________________________________
l4-hackers mailing list -- [email protected]
To unsubscribe send an email to [email protected]
OpenPGP_signature.asc (application/pgp-signature, 840 B)