[ZCM] [ZC] 2306/ 5 Comment "Pdata objects can not be Pickled once the chain is longer than 500"

Issue #2306 Update (Comment) "Pdata objects can not be Pickled once the chain is longer than 500"
 Status Pending, Zope/bug medium
To followup, visit:
  http://www.zope.org/Collectors/Zope/2306

==============================================================
= Comment - Entry #5 by esrever_otua on Sep 2, 2007 5:43 pm

Folks, I'm gobsmacked.

Fine, the testcase is a PloneTestCase - I will create a ZopeTestCase when I have a spare moment. Honestly, though, this is a trivial problem to duplicate. Upload a big file TTW, then try to run some code that pickles it.

Have either of you *actually run* the attached testcase? Does it fail? Can you offer some *constructive input* into why it is perfectly acceptable that it should fail?

tseaver:
You are using Zope 2.10 - the bug is logged against Zope 2.8 and Zope 2.9 - maybe it's fixed in 2.10, who knows? Also, you have dumped the data without using the Pickler class - maybe this affects it?
________________________________________
= Comment - Entry #4 by ajung on Sep 2, 2007 1:27 pm

I also have some doubts. We're storing large objects up to 150MB using Pdata chunking within the ZODB...never had any issues so far.


________________________________________
= Comment - Entry #3 by tseaver on Sep 2, 2007 7:19 am

I can't reproduce the described behavior at all, using the Python
API of File::

    $ cd ~/projects/Zope-CVS/Zope-2.10-branch
    $ PYTHONPATH=lib/python ../bin/python
    Python 2.4.4 (#1, Apr 30 2007, 11:13:06) 
    [GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> x = 'X' * 1024 * 1024
    >>> data = open('/tmp/issue_2306', 'wb')
    >>> for i in range(34):
    ...     data.write(x)
    ... 
    >>> data.flush()
    >>> data.close()
    >>> data = open('/tmp/issue_2306', 'rb')
    >>> from OFS.Image import File
    >>> f = File('issue_2306', '', '')
    >>> f.manage_upload(data)
    >>> f.size
    35651584L
    >>> from cPickle import dump, load
    >>> target = open('/tmp/pickle', 'wb')
    >>> dump(f, target)
    >>> target.flush()
    >>> target.close()
    >>> source = open('/tmp/pickle', 'rb')
    >>> f2 = load(source)
    >>> f2 == f
    False
    >>> f2.data == f.data
    False
    >>> f2.size
    35651584L
    >>> str(f2.data) == str(f.data)
    True


________________________________________
= Comment - Entry #2 by ajung on Sep 1, 2007 5:05 am

Please submit a *simple* testcase on top of ZopeTestCase but not on top of PloneTestCase with lots of different dependencies from other products.
________________________________________
= Request - Entry #1 by esrever_otua on Apr 5, 2007 6:19 am


Uploaded:  "pdata_failing_testcase.tar.gz"
 - http://www.zope.org/Collectors/Zope/2306/pdata_failing_testcase.tar.gz/view
In OFS.Image there is a 'File' class and 'Pdata' wrapper class. Pdata objects are designed to operate efficiently with large amounts of data by allowing themselves to be chained together in a C linked-list, while presenting an interface as if to a single object at the head of the list.
The 'File' class (which is used for, EG, ATFile, CMFFile, etc) stores its data internally in a Pdata object. It provides a 'manage_upload()' interface for modifying or supplying data to be stored in the File. This function splits the data into 64KB chunks and assigns each chunk to a new Pdata object in the C linked-list Pdata chain.
Once a file more than 32MB has been stored via manage_upload(), the File object can no longer be dumped with the Pickler() class.
EG, cPickle.Pickler.dump(OFS.Image.File) fails when the data stored in OFS.Image.File is greater than 32MB and was stored via the supplied manage_upload() interface.
The error is a RuntimeError for maximum recursion being hit.

Attached is a simple ZopeTestCase that demonstrates the problem. 
==============================================================