Re: silva 2.3 - pylibmc issues

Op 29 jul 2011, om 19:20 heeft Andrew Altepeter het volgende geschreven:

> Greetings!
> 

  Hello,

> I attempted to upgrade our Silva 2.1 instance to Silva 2.3 on Wednesday.
> Before the upgrade was scheduled, everything was tested and functioning
> properly.
> 
> The upgrade went smoothly, but the resultant system was immediately
> unstable.  We're using apache/mod_wsgi in production.  After starting up,
> the server would lock up when serving the assets for the first web page.
> This lock-up would happen whether running under mod_wsgi, or using paster
> serve and either debug or deploy scripts.  As a result, we aborted the
> upgrade.  I don't know why we did not see this issue earlier.
> 
> What follows in this email is my troubleshooting steps and results, and what
> we're thinking of doing to address this issue.
> 
> I've been troubleshooting the past two days, and I think I've found the
> problem.  First, I needed to inspect a thread dump.  Zope 2, pre-wsgi had
> DeadlockDebugger.  For wsgi, I found a nice thread dumper for 'paster
> deploy', and adjusted the 'debug.ini' script as follows:
> 

   You have one integrated to infrae.wsgi. You have a view debugzope.html that does it.
  (Need manager rights).

   [...]


> 
> Using the thread dumper, I was able to see where the threads were locking
> up.  Since we are using ZEO, we are also using memcached for caching
> storage.  Every thread was locking up in a specific section of code, having
> to deal with Beaker either retrieving values from memcached, or see if they
> exist.
> 
> In my development environment (opensuse 11.3, system python 2.6.5, pylibmc
> 1.1.1 and 1.1.2, libmemcached-0.47), I actually get tracebacks instead of
> hung threads. Here is the tail end of one:
> 
>  File
> "/home/aaltepet/.buildout/eggs/Products.PluggableAuthService-1.7.2-py2.6.egg/Products/PluggableAuthService/PluggableAuthService.py",
> line 226, in validate
>    user_ids = self._extractUserIds(request, plugins)
>  File
> "/home/aaltepet/.buildout/eggs/Products.PluggableAuthService-1.7.2-py2.6.egg/Products/PluggableAuthService/PluggableAuthService.py",
> line 599, in _extractUserIds
>    credentials )
>  File
> "/home/aaltepet/silva/local-Silva-2.3/src/silva.pas.base/src/silva/pas/base/plugins/cookie.py",
> line 178, in authenticateCredentials
>    if session.get('secret', None) == client_secret:
>  File
> "/home/aaltepet/silva/local-Silva-2.3/src/silva.core.cache/src/silva/core/cache/store.py",
> line 31, in get
>    return self.__backend.get(self._real_key(key))
>  File
> "/home/aaltepet/.buildout/eggs/Beaker-1.5.4-py2.6.egg/beaker/cache.py", line
> 214, in get
>    return self._get_value(key, **kw).get_value()
>  File
> "/home/aaltepet/.buildout/eggs/Beaker-1.5.4-py2.6.egg/beaker/container.py",
> line 252, in get_value
>    has_value = self.has_value()
>  File
> "/home/aaltepet/.buildout/eggs/Beaker-1.5.4-py2.6.egg/beaker/container.py",
> line 214, in has_value
>    return self.namespace.has_key(self.key)
>  File
> "/home/aaltepet/.buildout/eggs/Beaker-1.5.4-py2.6.egg/beaker/ext/memcached.py",
> line 61, in has_key
>    return key in self
>  File
> "/home/aaltepet/.buildout/eggs/Beaker-1.5.4-py2.6.egg/beaker/ext/memcached.py",
> line 57, in __contains__
>    value = self.mc.get(self._format_key(key))
> MemcachedError: error 15 from memcached_get: STORED
> 


 [...]

> I mentioned that our older Silva cms has some custom memcached caching.
> We're using python-memcached for this, which is a pure python memcached
> client.  I adjusted the Silva 2.3 buildout to install python-memcached and
> NOT install pylibmc, since pylibmc is preferred if found.  I turned
> memcached storage for auth sessions back on, and Silva 2.3 is now stable.
> 
> I can only think that libmemcached is preferred for performance reasons, and
> nothing else.  Bethel's current ZEO setup is quite responsive using
> python-memcached, and since it appears stable I think we'll be launching
> with that memcached client instead.



  Yes, some people got the same troubles with pylibmc and by so used
  python-memcached instead. And it worked great.

  I recommended pylibmc since when I first tested relstorage I had trouble with python-memcached
  and relstorage. Asking to the relstorage people they recomand pylibmc for it. But actually
  in production, we reverted back to python-memcached for I think the same reason and didn't
  experience the problems we had in test.

  Regards,

  Sylvain

-- 
Sylvain Viollon -- Infrae
t +31 10 243 7051 -- http://infrae.com
Hoevestraat 10 3033GC Rotterdam -- The Netherlands