Re: Dexterity performance issues.

Hi,

some profiling visualizations examples about this for those who are
interested:

https://dl.dropboxusercontent.com/u/228224/test_reindexObjectSecurity_from_root_lrchange-20140517.png

https://dl.dropboxusercontent.com/u/228224/test_reindexObjectSecurity_from_root_lrchange-20140517.pdf

# bin/test --tests-pattern=benchmarks --profile=cProfile -t VanillaDX
# python gprof2dot.py
--root="benchmarks:238:test_reindexObjectSecurity_from_root_lrchange" -f
pstats parts/test/tests_profile*.prof|dot -Tpng -o profile.png

-Asko

Matt Hamilton wrote:
> The main  one we had problems with is __ac_local_roles_block__ in this case. We managed to speed the re-index operation up by about 20% or so by setting that as a class-level attribute on one of the base classes (OFS.SimpleItem I think it was). But that still doesn’t help with the general case. Also there is some short circuit code in Dexterity that exits with AttributeError if the attribute starts with an underscore. But I think by then it is too late.
>
> -Matt
>
> On 16 May 2014, at 20:13, Laurence Rowe <[email protected]> wrote:
>
>> Would setting the acquired attributes to Acquisition.Acquired be enough? That would avoid the call to __getattr__ completely. https://pypi.python.org/pypi/Acquisition#controlling-acquisition
>>
>> Laurence
>>
>> On 16 May 2014 09:20, David Glick (Plone) <david.glick-z4DKO/[email protected]> wrote:
>> Hmm, the benchmarking I did early in Dexterity's life suggested that it
>> was roughly comparable in performance to Archetypes. It's possible that
>> a regression has happened since then, or it's possible that that
>> benchmark didn't adequately cover this use case.
>>
>> Thanks for providing your benchmark code so others can investigate.
>>
>> On 5/16/14, 8:37 AM, Matt Hamilton wrote:
>>> After digging around for most of a day, it seems like the main culprit is the __get__ call in p.dexterity.content.FTIAwareSpecification:
>>> https://github.com/plone/plone.dexterity/blob/master/plone/dexterity/content.py#L69
>>>
>>> In short, this is a method that on AT types is in C, and in Dexterity is in Python (I think). The DX call also has a mutex around a module level schema cache which may not be helping things (although in this case there is only one thread). In this *specific* case, setting a class-level attribute of __ac_local_roles_block__ = None does help as is means that accessing that attribute on a DX object stops there and don’t trigger then custom __getattr__ code. But it is a wider problem. Anytime you call e.g. IFoo.providedBy(my_dx_object) then it takes a long time… and we do that *a lot*.
>>>
>> I have a branch I never finished up where I added a list of attributes
>> that are commonly missing or acquired to make sure that __getattr__
>> doesn't try to look them up as fields. Does this help at all?
>>
>> +MISSING_OR_ACQUIRED_ATTRIBUTES = frozenset([
>> +    '__allow_groups__',
>> +    '__annotations__',
>> +    '__bobo_traverse__',
>> +    '__cmp__',
>> +    '__conform__',
>> +    '__provides__',
>> +    '_dav_writelocks',
>> +    '_EtagSupport__etag',
>> +    '_isPortalRoot',
>> +    '_is_wrapperish',
>> +    '_owner',
>> +    '_plone.uuid',
>> +    '_subobject_permissions',
>> +    '_v__providedBy__',
>> +    'aq_inner',
>> +    'dav__title',
>> +    'default_page',
>> +    'displayContentsTab',
>> +    'document_src',
>> +    'externalEditorEnabled',
>> +    'im_self',
>> +    'index_html',
>> +    'isTemporary',
>> +    'folder_listing',
>> +    'getCurrentSkinName',
>> +    'getCustomNavQuery',
>> +    'getFolderContents',
>> +    'getRawRelatedItems',
>> +    'getURL',
>> +    'global_cache_settings',
>> +    'global_statusmessage',
>> +    'layout',
>> +    'kss_generic_macros',
>> +    'main_template',
>> +    'plone_utils',
>> +    'portal_actions',
>> +    'portal_catalog',
>> +    'portal_css',
>> +    'portal_factory',
>> +    'portal_javascripts',
>> +    'portal_kss',
>> +    'portal_languages',
>> +    'portal_membership',
>> +    'portal_placeful_workflow',
>> +    'portal_properties',
>> +    'portal_types',
>> +    'portal_url',
>> +    'portal_workflow',
>> +    'role_map',
>> +    'sort-on',
>> +    'sort-limit',
>> +    'sort-order',
>> +    'translation_service',
>> +    'view',
>> +    ])
>> +
>> +
>>   class DexterityContent(DAVResourceMixin, PortalContent,
>> DefaultDublinCoreImpl, Contained):
>>       """Base class for Dexterity content
>>       """
>> @@ -183,13 +244,15 @@ class DexterityContent(DAVResourceMixin,
>> PortalContent, DefaultDublinCoreImpl, C
>>       description = u''
>>
>>       def __getattr__(self, name):
>> -        # optimization: sometimes we're asked for special attributes
>> -        # such as __conform__ that we can disregard (because we
>> -        # wouldn't be in here if the class had such an attribute
>> -        # defined).
>> -        if name.startswith('__'):
>> +        """Return default values for fields that don't have a value
>> stored."""
>> +        # optimization: avoid checking the schema for names that
>> +        # are expected to be missing or acquired
>> +        if name in MISSING_OR_ACQUIRED_ATTRIBUTES \
>> +                or name.endswith('_Permission'):
>>               raise AttributeError(name)
>>
>> ------------------------------------------------------------------------------
>> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
>> Instantly run your Selenium tests across 300+ browser/OS combos.
>> Get unparalleled scalability from the best Selenium testing platform available
>> Simple to use. Nothing to install. Get started now for free."
>> http://p.sf.net/sfu/SauceLabs
>> _______________________________________________
>> Plone-developers mailing list
>> Plone-developers-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
>> https://lists.sourceforge.net/lists/listinfo/plone-developers
>>
>> ------------------------------------------------------------------------------
>> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
>> Instantly run your Selenium tests across 300+ browser/OS combos.
>> Get unparalleled scalability from the best Selenium testing platform available
>> Simple to use. Nothing to install. Get started now for free."
>> http://p.sf.net/sfu/SauceLabs_______________________________________________
>> Plone-developers mailing list
>> Plone-developers-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
>> https://lists.sourceforge.net/lists/listinfo/plone-developers
>

------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs