Re: Toaster Error - Rocko

Hi Shakeel,

1. Since all of the offending objects are Target_File , you could change the log message so show the object’s actual value by adding “.path”.

                   logger.info("    FOUND:'%s'" % (obj.path))

That will show us to look inside of the Target_File objects and observe the actual multiple entries.

Since we are seeing a lot of upper/lower case similar files, I wonder if it is a case problem:
  (0-2-/usr/share/terminfo/a-18)({'target': <Target: Target object>, 'path': '/usr/share/terminfo/a', 'inodetype': 2})
  (0-2-/usr/share/terminfo/A-18)({'target': <Target: Target object>, 'path': '/usr/share/terminfo/A', 'inodetype': 2})

The only cost of these errors should be perhaps missing entries in the target file tree.

2. The second error set looks a lot nastier.

_mysql_exceptions.OperationalError: (1267, "Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation '='")

Where did the Swedish text come from?

Do you perhaps have files with Unicode in their names?

I am wondering if we should have Toaster normalize all file paths/names for TargetFile objects to UTF8 or something. That is certainly doable.

- David

From: Muhammad Shakeel Khan [mailto:[email protected]]
Sent: Monday, August 27, 2018 6:17 AM
To: Reyna, David
Cc: [email protected]
Subject: Re: [Toaster] Toaster Error - Rocko

Hi David,

Thanks for looking into this.

OK, I will ignore 1) as you said that the messages are largely passive.

For 2) I have added following slightly modified code to _cached_get

    def _cached_get(self, clazz, **kwargs):
        """ This is a memory-cached get. We assume that the objects will not change  in the database between gets.
        """
        assert issubclass(clazz, models.Model), "_cached_get needs to get the class as first argument"

        key = ORMWrapper._build_key(**kwargs)
        dictname = "objects_%s" % clazz.__name__

        if not dictname in vars(self).keys():
            vars(self)[dictname] = {}

        if not key in vars(self)[dictname].keys():
            try:
                vars(self)[dictname][key] = clazz.objects.get(**kwargs)
            except (ObjectDoesNotExist, MultipleObjectsReturned) as e:
                logger.info<http://logger.info/>("ERROR:buildinfohelper:_cached_get: '%s' (%s)(%s)" % (e, key, kwargs))
                for obj in clazz.objects.filter(**kwargs):
                    logger.info<http://logger.info/>("    FOUND:'%s'" % (obj))
                vars(self)[dictname][key] = clazz.objects.filter(**kwargs).first()

        return vars(self)[dictname][key]

I see a lot of queries returning "get() returned more than one Target_File -- it returned 2". Please see the log attached. Unfortunately there was another error and bitbake still returned error code.

I am interested to know what could be the reason for this error? Can it happen if a user starts a new build without first getting completed using same Toaster databases? How can I reset the database entries? Do I need to completely remove mysql database or there is something I can do within current build directory?

Best Regards,
Shakeel

On Mon, Aug 27, 2018 at 2:14 PM Reyna, David <[email protected]<mailto:[email protected]>> wrote:
Hi Shakeel,

1) “WARNING: Could not add dependency to the package libc6 because ld-linux.so.2 is an unknown package”

Yes, I have seen those messages. I have not debugged them deeply since the errors are largely passive, but it have been on my list of things to look at because there may be some details in the build data that we could be missing out.

2) From the looks of the trace, it appears that a found target file that has _two_ registered parents, at least using the cache/key mechanism of ‘_cached_get’.

I have not myself seen this error.

Since you have a reproducer, here is some sample code that you could in place of the existing “_cached_get” in “bitbake/lib/bb/ui/buildinfohelper.py”. It will:
  (a) catch the exception,
  (b) return the first found value so that the build can continue, and
  (c) send to the log the value(s) that broke it.

If you could insert this code (once it compiles), then we will have a better idea what happened.

    def _cached_get(self, clazz, **kwargs):
        """ This is a memory-cached get. We assume that the objects will not change  in the database between gets.
        """
        assert issubclass(clazz, models.Model), "_cached_get needs to get the class as first argument"

        key = ORMWrapper._build_key(**kwargs)
        dictname = "objects_%s" % clazz.__name__

        if not dictname in vars(self).keys():
            vars(self)[dictname] = {}

        if not key in vars(self)[dictname].keys():
            try:
                vars(self)[dictname][key] = clazz.objects.get(**kwargs)
            except KeyError as e:
                logger.debug(1, "ERROR:buildinfohelper:_cached_get: '%s' (%s)(%s)" % (e,key,**kwargs))
                for obj in clazz.objects.filter(**kwargs):
                    logger.debug(1, "    FOUND:'%s'" % (obj))
                vars(self)[dictname][key] = clazz.objects.filter(**kwargs)[0]

        return vars(self)[dictname][key]

BTW: this is the call in the trace that led to the error:
            parent_obj = self._cached_get(Target_File, target = target_obj, path = parent_path, inodetype = Target_File.ITYPE_DIRECTORY)

The second thing we could try is to directly look in the database and scan for duplicates for the above values. Any duplicates would cause the observed crash.

Thanks,
David


From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Muhammad Shakeel Khan
Sent: Monday, August 27, 2018 1:18 AM
To: [email protected]<mailto:[email protected]>
Subject: [Toaster] Toaster Error - Rocko

Hi,

We are running the Toaster production instance based on Rocko (2.4.3) and seeing a lot of warnings when the build is completed. Our setup is:

1) Start Toaster production instance in isolation (toaster user) with nobuild option, just to monitor command line builds
2) A different user then fetches 'poky' separately and then sets up the build environment
3) Start toaster: $ source toaster start noweb
4) Run build: $ bitbake core-image-minimal
5) Stop toaster: $ source toaster stop

This result in following warnings/notes when the build is completed (complete log attached)

WARNING: Could not add dependency to the package libc6 because ld-linux.so.2 is an unknown package
NOTE: We did not find one recipe for theconfiguration data package libc6 Recipe matching query does not exist.
WARNING: Could not add dependency to the package libc6 because ld-linux.so.2 is an unknown package

What might be going wrong here? The Toaster web UI shows 100% progress.

Also in one other custom image build we see the following error at the end of the build:

ERROR: get() returned more than one Target_File -- it returned 2!
Traceback (most recent call last):
  File "/home/mshakeel/build-areas/build-area-rocko/sources/poky/bitbake/lib/bb/ui/toasterui.py", line 416, in main
    buildinfohelper.store_target_package_data(event)
  File "/home/mshakeel/build-areas/build-area-rocko/sources/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 1383, in store_target_package_data
    self.orm_wrapper.save_target_file_information(self.internal_state['build'], target, filedata)
  File "/home/mshakeel/build-areas/build-area-rocko/sources/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 536, in save_target_file_information
    parent_obj = self._cached_get(Target_File, target = target_obj, path = parent_path, inodetype = Target_File.ITYPE_DIRECTORY)
  File "/home/mshakeel/build-areas/build-area-rocko/sources/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 123, in _cached_get
    vars(self)[dictname][key] = clazz.objects.get(**kwargs)
  File "/home/mshakeel/.local/lib/python3.6/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/mshakeel/.local/lib/python3.6/site-packages/django/db/models/query.py", line 384, in get
    (self.model._meta.object_name, num)
orm.models.MultipleObjectsReturned: get() returned more than one Target_File -- it returned 2!

NOTE: No commit since BUILDHISTORY_COMMIT != '1'
NOTE: Build exited with errorcode 0
NOTE: ToasterUI build done, brbe: None
WARNING: Return value is 1

Any thoughts about this error? We are using mysql (django.db.backends.mysql) for Toaster database.

Best Regards,
Shakeel