Re: [poky] Error while bitbaking core-image-minimal using Poky 5.0.9 (Scarthgap) on VMware

Thank you so much Peter, Alex and Quentin for quick and informative
responses.

I tried running bitbake world -c cleanall just to be sure to run a complete
fresh build. Then ran bitbake core-image-minimal and it finished without
any errors.
I also tried running bitbake -c cleansstate createrepo-c and then bitbake
createrepo-c. Everything went well, no errors.

Also, I changed my VM config to 8 cores, 16GB RAM and 16GB swap memory. But
still did not get any errors regarding do_compile() on gcc. I used
BB_NUMBER_THREADS=8 PARALLEL_MAKE=8 in this build and maybe that is why I
did not get any OOM errors this time



On Thu, May 15, 2025 at 3:01 PM Peter Marko via lists.yoctoproject.org
<[email protected]> wrote:

> RAM consumption under bitbake is a topic which you usually need to
> fine-tune for your setup.
> By default bitbake spawns number of build processes equal to number of
> cores and each of those processes will spawn the same number of compiler
> threads.
> So it's exponential, which is very problematic and oom-kills occur when
> several memory hungry build processes are run at the same time..
>
> The rule is - the more CPUs, the faster the build.
> However there is the RAM caveat - you have to have sufficient RAM for all
> the spawned processes.
> I personally gave to my (Debian) VM 24 cores and 48G RAM with default
> exponential settings and never observer OOM kill.
> My rule of thumb is to give it 2GB RAM for each CPU I assign.
>
> Of course this is not guaranteed to work in all cases.
> In worst case, you need cores*cores*x GB of RAM (e.g. for 16 cores you
> need 256*x GB).
> But the reality is better, since likelihood that all running bitbake
> processes are busy with super-memory-hungry compiler processes is very low.
> Since only few components are extremely memory hungry compilation
> processes, the average memory factor for each bitbake process is lower with
> higher cpu count.
> You also have to take into account that with low number of cores and RAM,
> also the system eats something.
>
> As example: you build image with 200 components, 8 of them are
> super-memory-hungry and usually there are some dependencies so they can't
> run all at the same time.
> If you give it two cores, probability that all running processes will be
> memory hungry is basically 100%, so give it 4GB per core.
> It you have 32 cores, you can't get to situation that spawned all
> processes are memory hungry, so even if some several at the same time, the
> sum will not be extreme, 2GB should be fine.
>
> Additionally, you can tweak compression settings (e.g. yocto default for
> xz is set to unbelievable 50% of whole RAM ... imagine that 8 components
> are using it at the same time)
> Also using BB_PRESSURE_MAX_{CPU|IO|MEMORY} configuration will help limit
> concurrent processes thus allow using less RAM.
> Also maybe don't use the default cores*cores formula and tweak
> BB_NUMBER_THREADS/PARALLEL_MAKE variables.
> And there are for sure other RAM relevant configurations/optimizations
> possible.
>
> ---
>
> Regarding the problem with quilt - quilt should be used from recipe
> sysroot (workdir), not from host.
> Per log below bitbake thinks it is installed but it is not found when
> needed.
> I think this may be a consequence of oom-kill.
> When oom-kill happens, I'd recommend deleting whole build directory
> because it is probably corrupted.
> Trying to perform a build in corrupted build directory leads to very
> strange build errors.
>
> ---
>
> Peter
>
> > -----Original Message-----
> > From: [email protected] <[email protected]> On
> Behalf Of
> > srivamsi_malladi via lists.yoctoproject.org
> > Sent: Thursday, May 15, 2025 13:02
> > To: [email protected]
> > Subject: [poky] Error while bitbaking core-image-minimal using Poky 5.0.9
> > (Scarthgap) on VMware
> >
> > Hello all,
> >
> > TL;DR: I ran into a couple of errors and fixed them with the help of
> Gemini. I would
> > like to know if there are any better solutions to solve these errors.
> >
> > I am absolutely new to the Yocto project and Poky, so I thought I will
> so a 'hello
> > world' kind of thing with Poky. For that, I setup a VMware with Ubuntu
> 22.04 LTS
> > and with 27.7GB RAM and 16 cores hardware config. My host computer is a
> i9-
> > 13th gen with 32GB RAM running Windows. I chose the really high (max
> actually)
> > RAM and cores thinking I would have a really fast VM, but due to the
> problems I
> > ran into, I realized more RAM, mores cores != more performance. So,
> following is
> > the problem I ran into.
> >
> > When I bitbake core-image-minimal in terminal, I got an error saying
> do_compile
> > failed on gcc 13.0.1 and when I looked at the log, it said error
> x86_64-poky-linux-
> > g++: fatal error: Killed signal terminated program cc1plus.
> > After spending some time on Google and some forums, I found that my VM
> might
> > be fighting with my host PC for resources and in this process, the tasks
> that
> > bitbake wants to run (compiling gcc in this case) are not able to get
> enough PC
> > resources due to which cc1plus got terminated which is needed in the
> process.
> > I tried reducing the RAM to 16GB and cores to 8 and then to 4, the error
> still
> > remained. After reducing the RAM to 8GB and cores to 2, gcc could finally
> > compile, but I ran into another error.
> >
> >
> > When I run bitbake core-image-minimal, I get an error and following are
> the
> > contents of the corresponding log file
> >
> >       DEBUG: Executing python function extend_recipe_sysroot
> >
> >       NOTE: Direct dependencies are
> > ['/home/ubuntu2204/work/poky/meta/recipes-devtools/quilt/quilt-
> > native_0.67.bb:do_populate_sysroot']
> >
> >       NOTE: Installed into sysroot: []
> >
> >       NOTE: Skipping as already exists in sysroot: ['quilt-native']
> >
> >       DEBUG: Python function extend_recipe_sysroot finished
> >
> >       DEBUG: Executing python function do_patch
> >
> >       DEBUG: Executing python function patch_do_patch
> >
> >       DEBUG: Searching for 0001-Do-not-set-PYTHON_INSTALL_DIR-by-
> > running-python.patch in paths:
> >
> >           /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-
> > c/createrepo-c-1.0.4/poky
> >
> >           /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-
> > c/createrepo-c/poky
> >
> >           /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-
> > c/files/poky
> >
> >           /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-
> > c/createrepo-c-1.0.4/
> >
> >           /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-
> > c/createrepo-c/
> >
> >
>  /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-c/files/
> >
> >           /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-
> > c/createrepo-c-1.0.4/x86-64
> >
> >           /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-
> > c/createrepo-c/x86-64
> >
> >           /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-
> > c/files/x86-64
> >
> >           /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-
> > c/createrepo-c-1.0.4/
> >
> >           /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-
> > c/createrepo-c/
> >
> >
>  /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-c/files/
> >
> >       DEBUG: Using /home/ubuntu2204/work/poky/meta/recipes-
> > devtools/createrepo-c/createrepo-c/0001-Do-not-set-PYTHON_INSTALL_DIR-by-
> > running-python.patch for 0001-Do-not-set-PYTHON_INSTALL_DIR-by-running-
> > python.patch
> >
> >       DEBUG: Searching for 0001-include-rpm-rpmstring.h.patch in paths:
> >
> >           /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-
> > c/createrepo-c-1.0.4/poky
> >
> >           /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-
> > c/createrepo-c/poky
> >
> >           /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-
> > c/files/poky
> >
> >           /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-
> > c/createrepo-c-1.0.4/
> >
> >           /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-
> > c/createrepo-c/
> >
> >
>  /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-c/files/
> >
> >           /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-
> > c/createrepo-c-1.0.4/x86-64
> >
> >           /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-
> > c/createrepo-c/x86-64
> >
> >           /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-
> > c/files/x86-64
> >
> >           /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-
> > c/createrepo-c-1.0.4/
> >
> >           /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-
> > c/createrepo-c/
> >
> >
>  /home/ubuntu2204/work/poky/meta/recipes-devtools/createrepo-c/files/
> >
> >       DEBUG: Using /home/ubuntu2204/work/poky/meta/recipes-
> > devtools/createrepo-c/createrepo-c/0001-include-rpm-rpmstring.h.patch
> for 0001-
> > include-rpm-rpmstring.h.patch
> >
> >       NOTE: Applying patch '0001-Do-not-set-PYTHON_INSTALL_DIR-by-
> > running-python.patch'
> (../meta/recipes-devtools/createrepo-c/createrepo-c/0001-
> > Do-not-set-PYTHON_INSTALL_DIR-by-running-python.patch)
> >
> >       ERROR: Applying patch '0001-Do-not-set-PYTHON_INSTALL_DIR-by-
> > running-python.patch' on target directory
> > '/home/ubuntu2204/work/poky/build/tmp/work/x86_64-linux/createrepo-c-
> > native/1.0.4/git'
> >
> >       CmdError('quilt --quiltrc
> > /home/ubuntu2204/work/poky/build/tmp/work/x86_64-linux/createrepo-c-
> > native/1.0.4/recipe-sysroot-native/etc/quiltrc push', 0, 'stdout:
> >
> >       stderr: /bin/sh: 1: quilt: not found
> >
> >       ')
> >
> >       DEBUG: Python function patch_do_patch finished
> >
> >       DEBUG: Python function do_patch finished
> >
> > I asked Gemini about the error and it suggested installing quilt. But it
> was already
> > installed.
> > Then I gave it the whole log file and then it explained that
> quilt-native is available in
> > sysroot but not in PATH and so asked me to create a new layer with
> > do_patch:prepend()where I should add the bindir (which should be
> /usr/bin in the
> > sysroot) to PATH.
> > This fixed the error.
> >
> > Now my questions are
> > 1. Regarding the first error, I had to decrease the RAM to 8GB and cores
> to 2
> > which cured the error but increased the build time. Is there a better
> way to handle
> > this without compromising on the build time?
> > 2. Regarding the second error, did this occur because of some problems
> in my
> > system? Or is this a known error with Poky Scarthgap?
> >
> > Thanks & Regards,
> > Srivamsi Malladi.
> >
> >
> >
> >
> >
> >
> >
> >
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#13621):
> https://lists.yoctoproject.org/g/poky/message/13621
> Unfollow This Topic: https://lists.yoctoproject.org/unft/113124703/9810001
> Group Owner: [email protected]
> Unsubscribe: https://lists.yoctoproject.org/g/poky/unsub [
> [email protected]]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>