Re: Re: Multiple Reindexing Issue

Loading Zope, please stand by ./usr/lib/python2.4/whrandom.py:38: DeprecationWarning: the whrandom module is deprecated; please use the random module
  DeprecationWarning)
.. done (2.595s)
/home/sidnei/src/instance/pd21/Products/PortalTransforms/TransformEngine.py:18: DeprecationWarning: The module, 'Products.CMFCore.CMFCorePermissions' is a deprecated compatiblity alias for 'Products.CMFCore.permissions';  please use the new module instead.
  from Products.CMFCore import CMFCorePermissions
Installing CMFCore ... done (0.075s)
Installing CMFDefault ... done (0.087s)
Installing CMFCalendar ... done (0.064s)
Installing CMFTopic ... done (0.036s)
Installing DCWorkflow ... done (0.121s)
Installing CMFActionIcons ... done (0.049s)
Installing CMFQuickInstallerTool ... /home/sidnei/src/instance/pd21/Products/CMFQuickInstallerTool/__init__.py:36: DeprecationWarning: The product_name parameter of ToolInit is deprecated and will be ignored in CMF 2.0: QuickInstallerTool
  icon='tool.gif'
done (0.078s)
Installing CMFFormController ... /home/sidnei/src/instance/pd21/Products/CMFCore/utils.py:770: DeprecationWarning: format_stx() will be removed in CMF 2.0. Please use StructuredText.StructuredText.HTML instead.
  DeprecationWarning)
/home/sidnei/src/instance/pd21/Products/CMFFormController/__init__.py:25: DeprecationWarning: The product_name parameter of ToolInit is deprecated and will be ignored in CMF 2.0: CMFFormController
  product_name='CMFFormController', icon='tool.gif',
done (0.330s)
Installing ResourceRegistries ... /home/sidnei/src/instance/pd21/Products/ResourceRegistries/__init__.py:20: DeprecationWarning: The product_name parameter of ToolInit is deprecated and will be ignored in CMF 2.0: ResourceRegistries
  icon = 'tool.gif',
done (0.071s)
Installing GroupUserFolder ... /home/sidnei/src/instance/pd21/Products/GroupUserFolder/__init__.py:99: DeprecationWarning: The product_name parameter of ToolInit is deprecated and will be ignored in CMF 2.0: GroupUserFolder
  , icon="tool.gif"
done (0.073s)
Installing ZCTextIndex ... done (0.065s)
Installing ExtendedPathIndex ... done (0.075s)
Installing SecureMailHost ... done (0.065s)
Installing CMFPlone ... /home/sidnei/src/instance/pd21/Products/CMFPlone/__init__.py:190: DeprecationWarning: The product_name parameter of ToolInit is deprecated and will be ignored in CMF 2.0: CMFPlone
  , icon='tool.gif'
done (0.741s)
Installing Archetypes ... /home/sidnei/src/instance/pd21/Products/Archetypes/__init__.py:90: DeprecationWarning: The product_name parameter of ToolInit is deprecated and will be ignored in CMF 2.0: Archetypes
  icon="tool.gif",
done (0.904s)
/home/sidnei/src/instance/pd21/Products/MimetypesRegistry/__init__.py:34: DeprecationWarning: The product_name parameter of ToolInit is deprecated and will be ignored in CMF 2.0: MimetypesRegistry
  icon="tool.gif",
/home/sidnei/src/instance/pd21/Products/PortalTransforms/__init__.py:35: DeprecationWarning: The product_name parameter of ToolInit is deprecated and will be ignored in CMF 2.0: PortalTransforms
  icon="tool.gif",
Installing ATContentTypes ... /home/sidnei/src/instance/pd21/Products/ATContentTypes/__init__.py:82: DeprecationWarning: The product_name parameter of ToolInit is deprecated and will be ignored in CMF 2.0: ATContentTypes
  icon='tool.gif', ).initialize(context)
done (2.844s)
Installing ATReferenceBrowserWidget ... done (0.051s)
Installing ArchetypesTestUpdateSchema ... NOT FOUND
Adding Plone Site ... /home/sidnei/src/instance/pd21/Products/CMFPlone/Portal.py:181: DeprecationWarning: Please switch to the new format for typeinfo names "product_id: type_id (meta_type)", the old spelling will disappear in CMF 2.0
  typeinfo_name='CMFPlone: Plone Folder')
/home/sidnei/src/instance/pd21/Products/CMFPlone/Portal.py:184: DeprecationWarning: Please switch to the new format for typeinfo names "product_id: type_id (meta_type)", the old spelling will disappear in CMF 2.0
  typeinfo_name='CMFPlone: Large Plone Folder')
/home/sidnei/src/instance/pd21/Products/CMFCore/TypesTool.py:270: DeprecationWarning: getActionById() is deprecated and will be removed in CMF 2.0. Please use getActionInfo()['url'] if you need an URL or queryMethodID() if you need a method ID.
  DeprecationWarning)
/home/sidnei/src/instance/pd21/Products/CMFPlone/migrations/v2/plone2_base.py:268: DeprecationWarning: Please switch to the new format for typeinfo names "product_id: type_id (meta_type)", the old spelling will disappear in CMF 2.0
  typeinfo_name='CMFCore: Portal Folder')
/home/sidnei/src/instance/pd21/Products/CMFPlone/migrations/v2/plone2_base.py:419: DeprecationWarning: Please switch to the new format for typeinfo names "product_id: type_id (meta_type)", the old spelling will disappear in CMF 2.0
  typeinfo_name='CMFPlone: Plone Site')
/home/sidnei/src/instance/pd21/Products/Archetypes/Extensions/utils.py:254: DeprecationWarning: Please switch to the new format for typeinfo names "product_id: type_id (meta_type)", the old spelling will disappear in CMF 2.0
  typeinfo_name=typeinfo_name)
done (21.648s)
Adding CMFFormController ... done (0.005s)
Adding Archetypes ... done (0.298s)
F
======================================================================
FAIL: Doctest: reindex_sanity.txt
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/sidnei/src/zope/2.7/lib/python/Testing/ZopeTestCase/zopedoctest/doctest.py", line 2154, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for reindex_sanity.txt
  File "/home/sidnei/src/instance/pd21/Products/Archetypes/tests/reindex_sanity.txt", line 0

----------------------------------------------------------------------
File "/home/sidnei/src/instance/pd21/Products/Archetypes/tests/reindex_sanity.txt", line 81, in reindex_sanity.txt
Failed example:
    response = self.publish(
        '%s/%s' % (self.folder_path,
                   'invokeFactory?type_name=DDocument&id=new_doc'),
        self.basic_auth)
Differences (ndiff with -expected +actual):
    + Catalog:  /plone/uid_catalog
    + Indexing: Members/test_user_1_/new_doc []
    + Callers:  <string>:4:addDDocument
    +           OFS.ObjectManager:276:_setObject
    +           Archetypes.BaseContent:37:manage_afterAdd
    +           Archetypes.BaseObject:176:manage_afterAdd
    +           Archetypes.Referenceable:216:manage_afterAdd
    +           Archetypes.Referenceable:197:_updateCatalog
    +           Archetypes.Referenceable:284:_catalogUID
    +           Archetypes.ReferenceEngine:445:catalog_object
    + ------------------------------------------------------------
    + Catalog:  /plone/portal_catalog
    + Indexing: /plone/Members/test_user_1_/new_doc []
    + Callers:  CMFCore.TypesTool:335:constructInstance
    +           CMFCore.TypesTool:566:_constructInstance
    +           <string>:4:addDDocument
    +           OFS.ObjectManager:276:_setObject
    +           Archetypes.BaseContent:38:manage_afterAdd
    +           CMFCore.CMFCatalogAware:198:manage_afterAdd
    +           Archetypes.CatalogMultiplex:26:indexObject
    +           CMFPlone.CatalogTool:412:catalog_object
    + ------------------------------------------------------------
    + Catalog:  /plone/uid_catalog
    + Indexing: Members/test_user_1_/new_doc []
    + Callers:  CMFCore.TypesTool:566:_constructInstance
    +           <string>:4:addDDocument
    +           OFS.ObjectManager:276:_setObject
    +           Archetypes.BaseContent:38:manage_afterAdd
    +           CMFCore.CMFCatalogAware:198:manage_afterAdd
    +           Archetypes.CatalogMultiplex:28:indexObject
    +           Archetypes.Referenceable:284:_catalogUID
    +           Archetypes.ReferenceEngine:445:catalog_object
    + ------------------------------------------------------------
    + Catalog:  /plone/portal_catalog
    + Indexing: /plone/Members/test_user_1_/new_doc ['review_state']
    + Callers:  CMFCore.TypesTool:925:constructContent
    +           CMFCore.TypesTool:337:constructInstance
    +           CMFCore.TypesTool:349:_finishConstruction
    +           CMFCore.CMFCatalogAware:141:notifyWorkflowCreated
    +           CMFCore.WorkflowTool:355:notifyCreated
    +           CMFCore.WorkflowTool:697:_reindexWorkflowVariables
    +           Archetypes.CatalogMultiplex:65:reindexObject
    +           CMFPlone.CatalogTool:412:catalog_object
    + ------------------------------------------------------------
    + Catalog:  /plone/uid_catalog
    + Indexing: Members/test_user_1_/new_doc []
    + Callers:  CMFCore.TypesTool:337:constructInstance
    +           CMFCore.TypesTool:349:_finishConstruction
    +           CMFCore.CMFCatalogAware:141:notifyWorkflowCreated
    +           CMFCore.WorkflowTool:355:notifyCreated
    +           CMFCore.WorkflowTool:697:_reindexWorkflowVariables
    +           Archetypes.CatalogMultiplex:66:reindexObject
    +           Archetypes.Referenceable:284:_catalogUID
    +           Archetypes.ReferenceEngine:445:catalog_object
    + ------------------------------------------------------------
    + Catalog:  /plone/portal_catalog
    + Indexing: /plone/Members/test_user_1_/new_doc ['allowedRolesAndUsers']
    + Callers:  CMFCore.TypesTool:337:constructInstance
    +           CMFCore.TypesTool:349:_finishConstruction
    +           CMFCore.CMFCatalogAware:141:notifyWorkflowCreated
    +           CMFCore.WorkflowTool:355:notifyCreated
    +           CMFCore.WorkflowTool:701:_reindexWorkflowVariables
    +           CMFCore.CMFCatalogAware:128:reindexObjectSecurity
    +           CMFCore.CatalogTool:353:reindexObject
    +           CMFPlone.CatalogTool:412:catalog_object
    + ------------------------------------------------------------
    + Catalog:  /plone/portal_catalog
    + Indexing: /plone/Members/test_user_1_/new_doc []
    + Callers:  ZPublisher.Publish:39:call_object
    +           ATContentTypes.lib.constraintypes:274:invokeFactory
    +           CMFCore.PortalFolder:408:invokeFactory
    +           CMFCore.TypesTool:925:constructContent
    +           CMFCore.TypesTool:337:constructInstance
    +           CMFCore.TypesTool:351:_finishConstruction
    +           Archetypes.CatalogMultiplex:65:reindexObject
    +           CMFPlone.CatalogTool:412:catalog_object
    + ------------------------------------------------------------
    + Catalog:  /plone/uid_catalog
    + Indexing: Members/test_user_1_/new_doc []
    + Callers:  ATContentTypes.lib.constraintypes:274:invokeFactory
    +           CMFCore.PortalFolder:408:invokeFactory
    +           CMFCore.TypesTool:925:constructContent
    +           CMFCore.TypesTool:337:constructInstance
    +           CMFCore.TypesTool:351:_finishConstruction
    +           Archetypes.CatalogMultiplex:66:reindexObject
    +           Archetypes.Referenceable:284:_catalogUID
    +           Archetypes.ReferenceEngine:445:catalog_object
    + ------------------------------------------------------------


----------------------------------------------------------------------
Ran 1 test in 0.906s

FAILED (failures=1)

Reindex Sanity
==============

Those tests make sure that the catalog is only called as necessary.

  >>> from os.path import sep
  >>> from Products.ZCatalog.ZCatalog import ZCatalog
  >>> from Products.CMFCore.utils import getToolByName
  >>> from Products.Archetypes.utils import wrap_method
  >>> from Products.Archetypes.utils import unwrap_method
  >>> from Products.Archetypes.utils import call_original as call
  >>> from Products.Archetypes.tests.attestcase import default_user
  >>> from Products.Archetypes.tests.atsitetestcase import portal_owner

  >>> self.setRoles(['Manager'])
  >>> self.folder_url = self.folder.absolute_url()
  >>> self.folder_path = '/%s' % self.folder.absolute_url(1)
  >>> self.basic_auth = '%s:secret' % default_user
  >>> self.owner_auth = '%s:secret' % portal_owner

  >>> def callers(n=1):
  ...     stack = []
  ...     import inspect
  ...     frames = inspect.getouterframes(inspect.currentframe())
  ...     for i in range(n+1, 1, -1):
  ...         try:
  ...             frame = frames[i]
  ...         except IndexError:
  ...             continue
  ...         fname, ln, func = frame[1:4]
  ...         fname = fname.split('python')[-1]
  ...         fname = fname.split('site-packages')[-1]
  ...         fname = fname.split('Products')[-1]
  ...         fname = fname.split('.py')[0]
  ...         fname = '.'.join(fname.strip(sep).split(sep))
  ...         stack.append(':'.join((fname, str(ln), func)))
  ...     return stack

Wrap ZCatalog methods to let us know about why and when they were called:

  >>> pattern = '__at_reindex_sanity_%s__'

  >>> def catalog_object(self, *args, **kw):
  ...     rv = call(self, 'catalog_object', pattern, *args, **kw)
  ...     print 'Catalog: ', '/'.join(self.getPhysicalPath())
  ...     uid = kw.get('uid', None)
  ...     if not uid:
  ...         if len(args) > 1:
  ...             uid = args[1]
  ...         else:
  ...             uid = '/'.join(args[0].getPhysicalPath())
  ...     idxs = kw.get('idxs', [])
  ...     if not idxs:
  ...         if len(args) > 2:
  ...             idxs = args[2]
  ...     print 'Indexing:', uid, idxs
  ...     print 'Callers: ', '\n          '.join(callers(8))
  ...     print '-' * 60
  ...     return rv

  >>> def uncatalog_object(self, *args, **kw):
  ...     rv = call(self, 'uncatalog_object', pattern, *args, **kw)
  ...     print 'Catalog:   ', '/'.join(self.getPhysicalPath())
  ...     print 'UnIndexing:', kw.get('uid', None) or args[0]
  ...     print 'Callers:   ', '\n             '.join(callers(8))
  ...     print '-' * 60
  ...     return rv

  >>> wrap_method(ZCatalog, 'catalog_object',
  ...             catalog_object, pattern=pattern)

  >>> wrap_method(ZCatalog, 'uncatalog_object',
  ...             uncatalog_object, pattern=pattern)

Disable Portal Factory:

  >>> pf = getToolByName(self.portal, 'portal_factory', None)
  >>> if pf is not None:
  ...    pf.manage_setPortalFactoryTypes(listOfTypeIds=[])

  >>> response = self.publish(
  ...     '%s/%s' % (self.folder_path,
  ...                'invokeFactory?type_name=DDocument&id=new_doc'),
  ...     self.basic_auth)

Restore ZCatalog methods:

  >>> unwrap_method(ZCatalog, 'catalog_object')
  >>> unwrap_method(ZCatalog, 'uncatalog_object')