Re: Toaster Error - Rocko

Hi David,

Thanks for looking into this.

OK, I will ignore 1) as you said that the messages are largely passive.

For 2) I have added following slightly modified code to _cached_get

    def _cached_get(self, clazz, **kwargs):
        """ This is a memory-cached get. We assume that the objects will
not change  in the database between gets.
        """
        assert issubclass(clazz, models.Model), "_cached_get needs to get
the class as first argument"

        key = ORMWrapper._build_key(**kwargs)
        dictname = "objects_%s" % clazz.__name__

        if not dictname in vars(self).keys():
            vars(self)[dictname] = {}

        if not key in vars(self)[dictname].keys():
            try:
                vars(self)[dictname][key] = clazz.objects.get(**kwargs)
            except (ObjectDoesNotExist, MultipleObjectsReturned) as e:
                logger.info("ERROR:buildinfohelper:_cached_get: '%s'
(%s)(%s)" % (e, key, kwargs))
                for obj in clazz.objects.filter(**kwargs):
                    logger.info("    FOUND:'%s'" % (obj))
                vars(self)[dictname][key] =
clazz.objects.filter(**kwargs).first()

        return vars(self)[dictname][key]

I see a lot of queries returning "get() returned more than one Target_File
-- it returned 2". Please see the log attached. Unfortunately there was
another error and bitbake still returned error code.

I am interested to know what could be the reason for this error? Can it
happen if a user starts a new build without first getting completed using
same Toaster databases? How can I reset the database entries? Do I need to
completely remove mysql database or there is something I can do within
current build directory?

Best Regards,
Shakeel

On Mon, Aug 27, 2018 at 2:14 PM Reyna, David <[email protected]>
wrote:

> Hi Shakeel,
>
> 1) “WARNING: Could not add dependency to the package libc6 because
> ld-linux.so.2 is an unknown package”
>
> Yes, I have seen those messages. I have not debugged them deeply since the
> errors are largely passive, but it have been on my list of things to look
> at because there may be some details in the build data that we could be
> missing out.
>
> 2) From the looks of the trace, it appears that a found target file that
> has _*two*_ registered parents, at least using the cache/key mechanism of
> ‘*_cached_get**’.*
>
> I have not myself seen this error.
>
> Since you have a reproducer, here is some sample code that you could in
> place of the existing “_cached_get” in
> “bitbake/lib/bb/ui/buildinfohelper.py”. It will:
>   (a) catch the exception,
>   (b) return the first found value so that the build can continue, and
>   (c) send to the log the value(s) that broke it.
>
> If you could insert this code (once it compiles), then we will have a
> better idea what happened.
>
> *    def _cached_get(self, clazz, **kwargs):*
> *        """ This is a memory-cached get. We assume that the objects will
> not change  in the database between gets.*
> *        """*
> *        assert issubclass(clazz, models.Model), "_cached_get needs to get
> the class as first argument"*
>
> *        key = ORMWrapper._build_key(**kwargs)*
> *        dictname = "objects_%s" % clazz.__name__*
>
> *        if not dictname in vars(self).keys():*
> *            vars(self)[dictname] = {}*
>
> *        if not key in vars(self)[dictname].keys():*
>             *try:*
>                 *vars(self)[dictname][key] = clazz.objects.get(**kwargs)*
>             *except KeyError as e:*
>                 *logger.debug(1, "ERROR:buildinfohelper:_cached_get: '%s'
> **(%s)**(%s)" % (e,**key,****kwargs))*
> *                for obj in **clazz.objects.filter(**kwargs)**:*
>                     *logger.debug(1, "**    FOUND:**'%s'" % (**obj**))*
>                 *vars(self)[dictname][key] =
> clazz.objects.filter(**kwargs)[0]*
>
> *        return vars(self)[dictname][key]*
>
> BTW: this is the call in the trace that led to the error:
>             parent_obj = self._cached_get(Target_File, target =
> target_obj, path = parent_path, inodetype = Target_File.ITYPE_DIRECTORY)
>
> The second thing we could try is to directly look in the database and scan
> for duplicates for the above values. Any duplicates would cause the
> observed crash.
>
> Thanks,
> David
>
>
> *From:* [email protected] [
> mailto:[email protected] <[email protected]>]
> *On Behalf Of *Muhammad Shakeel Khan
> *Sent:* Monday, August 27, 2018 1:18 AM
> *To:* [email protected]
> *Subject:* [Toaster] Toaster Error - Rocko
>
> Hi,
>
> We are running the Toaster production instance based on Rocko (2.4.3) and
> seeing a lot of warnings when the build is completed. Our setup is:
>
> 1) Start Toaster production instance in isolation (toaster user) with
> nobuild option, just to monitor command line builds
> 2) A different user then fetches 'poky' separately and then sets up the
> build environment
> 3) Start toaster: $ source toaster start noweb
> 4) Run build: $ bitbake core-image-minimal
> 5) Stop toaster: $ source toaster stop
>
> This result in following warnings/notes when the build is completed
> (complete log attached)
>
> WARNING: Could not add dependency to the package libc6 because
> ld-linux.so.2 is an unknown package
> NOTE: We did not find one recipe for theconfiguration data package libc6
> Recipe matching query does not exist.
> WARNING: Could not add dependency to the package libc6 because
> ld-linux.so.2 is an unknown package
>
> What might be going wrong here? The Toaster web UI shows 100% progress.
>
> Also in one other custom image build we see the following error at the end
> of the build:
>
> ERROR: get() returned more than one Target_File -- it returned 2!
> Traceback (most recent call last):
>   File
> "/home/mshakeel/build-areas/build-area-rocko/sources/poky/bitbake/lib/bb/ui/toasterui.py",
> line 416, in main
>     buildinfohelper.store_target_package_data(event)
>   File
> "/home/mshakeel/build-areas/build-area-rocko/sources/poky/bitbake/lib/bb/ui/buildinfohelper.py",
> line 1383, in store_target_package_data
>
> self.orm_wrapper.save_target_file_information(self.internal_state['build'],
> target, filedata)
>   File
> "/home/mshakeel/build-areas/build-area-rocko/sources/poky/bitbake/lib/bb/ui/buildinfohelper.py",
> line 536, in save_target_file_information
>     parent_obj = self._cached_get(Target_File, target = target_obj, path =
> parent_path, inodetype = Target_File.ITYPE_DIRECTORY)
>   File
> "/home/mshakeel/build-areas/build-area-rocko/sources/poky/bitbake/lib/bb/ui/buildinfohelper.py",
> line 123, in _cached_get
>     vars(self)[dictname][key] = clazz.objects.get(**kwargs)
>   File
> "/home/mshakeel/.local/lib/python3.6/site-packages/django/db/models/manager.py",
> line 85, in manager_method
>     return getattr(self.get_queryset(), name)(*args, **kwargs)
>   File
> "/home/mshakeel/.local/lib/python3.6/site-packages/django/db/models/query.py",
> line 384, in get
>     (self.model._meta.object_name, num)
> orm.models.MultipleObjectsReturned: get() returned more than one
> Target_File -- it returned 2!
>
> NOTE: No commit since BUILDHISTORY_COMMIT != '1'
> NOTE: Build exited with errorcode 0
> NOTE: ToasterUI build done, brbe: None
> WARNING: Return value is 1
>
> Any thoughts about this error? We are using mysql
> (django.db.backends.mysql) for Toaster database.
>
> Best Regards,
> Shakeel
>
>