Re: Pyro 4.20 serpent serializer issues

On 8-6-2013 21:54, Vernon D. Cole wrote:
> I ran my remote database access test, which basically throws every data type known to
> man across the network.  There were three errors under Pyro4 2.20 which do not appear
> under 2.18. 
> It seems to be having trouble sending array.array('B') and datetime.date(), and is
> incorrectly receiving decimal.Decimal()
> 
> Do you need simple test cases, or will this traceback give you the clues you need?

Serpent is the new serialization default instead of pickle. Serpent is not equivalent to
pickle though (and json and marshal even less so). Like json and marshal it is limited
in what types it can serialize, although it does a better job than json and marshal.

What you receive on the other side may not be exactly what you put into it in the
sender, because serpent converts stuff into Python literal expressions and only a
handful of classes are converted back automatically.

For instance:

>>> import serpent
>>> import decimal
>>> x=serpent.dumps(decimal.Decimal("555555555555.6666666666666666666666"))
>>> x
b"# serpent utf-8 python3.2\n'555555555555.6666666666666666666666'"
>>> serpent.loads(x)
'555555555555.6666666666666666666666'
>>>

Decimals are encoded as a literal string (because otherwise you will lose precision).
The receiver however needs to 'know' this and will have to convert the string back to a
decimal.Decimal. Similarly, datetime.datetime are encoded as a ISO formatted date+time
string, and the receiver will have to convert it back explicitly (if desired).

If you want to change/enhance serpents behavior you can try to register custom
serialization hooks via serpent.register_class

For more info see the serpent library itself. It doesn't have extensive documentation
yet though. Serpent's source file itself is very small and should be fairly
self-explanatory, but I'll be happy to provide help if needed.


The key issue here is that when not using pickle, you probably will have to redesign
your remote method contracts if they depend a lot on custom types on both ends. If you
stick to Python literals mainly, the pain will be a lot less or even non-existent...

That being said, I'm open to suggestions for improvements on the current implementation.


Regards
Irmen de Jong


------------------------------------------------------------------------------
How ServiceNow helps IT people transform IT departments:
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j