Re: [dev] RFC: Transaction meta data

> On Nov 14, 2016, at 13:41, Jim Fulton <[email protected]> wrote:

> Recently, I tried to clean up transaction meta data:
...
> This opened a small can of worms. 

I've been going around in circles on this one. My tl;dr is that I don't think I like having encoding/decoding responsibility be so asymmetrical (high-level ZODB would be responsible for encoding, but who would be responsible for decoding?). I also have some backwards compatibility concerns, and I wonder, if we're going to be breaking compatibility, maybe it would be better to thread text/unicode all the way down to the storage, instead of stopping at a layer above it, or even just specify them as bytes for maximum flexibility.


> The existing storage APIs are vague about whether the user and description field are text/unicode or bytes.  Historically, they've been bytes, but storage implementations are defensive and allow either, although they're also conservative and, if given text, encode using an ASCII encoding.

RelStorage 1.x (which was only Python 2) used `str(user)` to do the encoding, which (almost always) encoded as ASCII when given text. However, it was explicitly tested that bytes arguments could round-trip the entire 256 possible values; there was a user bug filed for this[1]. According to the commit comment, this was because "according to the transaction interface, transaction metadata is always a string, but may have encoded characters" (where "string" here is python 2's bytes)[2].

For RelStorage 2 (currently in beta---so this could change---and supporting Python 2 and 3), the ASCII encoding was expanded to be Latin-1/ISO-8859-1, thus ensuring that, no matter the platform, and no matter whether text or bytes, the first 256 values round-trip in the same form, while prohibiting other values:

  py> u'\x80'.encode('latin-1') # ord 128
  '\x80


[1] https://github.com/zodb/relstorage/commit/f4b5370033f1e248ab2a0cc35acb99262a119047
[2] https://github.com/zodb/relstorage/commit/01f3414bd0ab2dcc9d86f884cf02cb5389dcab8f

> 
> For a while I've realized that storages should be asked to do less than they do now, and that more should be done by ZODB.
> 
> In that spirit, I'm inclined to say that transaction user and description should be encoded by ZODB, using UTF-8 and that storages should only deal with bytes.  Eventually, storages can be less defensive.


So long as the high-level API (ZODB.DB, ZODB.Connection) doesn't expose any way to get those bytes back out as text, that should be OK. That is, if the IStorage interfaces (.iterator(), .history(), .undoLog()---are there more?) deal in bytes, and that's the only way to get that data back out, that works out from a conceptual level (IStorage deals just in bytes) and a practical level (there are existing RelStorage's out there that have non-UTF-8 data in 'user' and 'description', and, given some hypothetical text-returning API, while RelStorage could potentially catch UnicodeDecodeError on UTF-8 and try again as latin-1, there's no real reason why some higher level ZODB API should be expected to know to do that).

On the other hand, we're still left with asymmetry, ambiguity and guessing, if those IStorage interfaces are the only way to get those byte strings back out. There are tools like zodbbrowser that will display them, and we've simply passed the decoding task on to them. There's a disconnect between where the values---text---are encoded (somewhere in ZODB.DB) and where you can get them back out---as bytes---the storage. For example, the version of zodbbrowser I had handy displays those values like this (I think more recent versions delegate to the PageTemplate to do that):

         str(time.strftime('%Y-%m-%d %H:%M:%S',
                          time.localtime(d['time']))) 
                     + " "
                     + d['user_name'] + " "
                     + d['description'])

On Python 2, `time.strftime ` returns bytes, so everything "just works" at this level. But on Python 3, it returns text and this line will raise a TypeError. And the developer will have to guess at what the byte encoding is (or use Latin-1, which can decode *anything*, although not always sensibly). 

Now, we could sidestep this by defining the bytes encoding to be utf-8 and enshrining that in the API documentation for IStorage. 

This leads to my biggest concern, which is that there are existing ("legacy") storages, where the user has been in full control of the byte encoding this entire time. There's no guarantee that those bytes are in utf-8, latin-1, or other.  A generic tool would have to look something like this:

   try:
      desc = d['description'].decode('utf-8')
   except UnicodeError:
      # This may not display correctly, but is guaranteed to decode.
      desc = d['description'].decode('latin-1') 

That doesn't really change. But currently, custom tools would just be doing something like `d[description'].decode('big5')`, and they would have to change similarly. The issue, other than the repetition, is that I'm pretty sure it's not possible to catch improper decoding in every codec, so some mojibake will inevitably happen.

An additional small loss I notice in defining these values to be text is that applications lose the ability to store mixed data or otherwise custom bytes in these fields. I'm not sure what the use of that would be, but it has been possible. 

Here are three different possibilities to consider:

In all of them, I'm using the same types at the `transaction` level as at the `IStorage` level, for consistency.

If instead we define these values as "native strings", i.e., text on Python 3, bytes on Python 2, just like WSGI does, then overall there's the least amount of work to do in terms of avoiding TypeErrors. Presumably this is largely the de facto situation. Things may not always look right (on Python 2), but Python 2 client applications don't have any work to do either (assuming storages continue to be permissive in allowing unicode values that they choose an encoding for; latin-1 for Python 2/3 compatibility, utf-8 for maximum encoding). Python 3 would not have the ability to store arbitrary bytes data in these fields, unless the storages were permissive. The responsibility for choosing an appropriate storage format is on each storage, which seems fair (some RDBMSs can natively store Unicode text, for example, and RelStorage might choose to use that on Python 3).

Alternatively, we could define the transaction attributes to be bytes everywhere. That matches the implied python 2 contract. If storages continue to still accept and encode text data (presumably as utf-8), no clients have to change. ZODB could still do the encoding for incoming text values to make sure that storages consistently use the same encoding, if desired. Tools that want to attempt to display the data as text could try to decode it, but they wouldn't have any guarantees; that shifts some of the work onto them, but also allows users to continue to use their preferred encodings (or stuff arbitrary data there). 

Lastly, we could define these values as text everywhere. This has the same problems for Python 2 applications as the code currently committed to transaction (unless some layer wants to attempt a conversion for them, which may not always be possible), but it widens the implicitly permitted character set. Storages still have to determine how they want to store unicode text, but they could stick to just encoding it, which is what they probably do now, just with UTF-8 instead of ASCII. It probably results in tool changes, too, even on a single version of Python.

I'm not sure which, if any, of these is the most improvement over either the old situation or the current master situation, although I think I'm personally leaning towards the native string approach. I suppose it depends on the relative importance and use of "browser" type tools, non-ASCII data in existing databases, and the need to encode/decode data in separate places. 

I think having this conversation, and at least getting things better documented, is a good thing!


> I've updated the ZODB interfaces on a branch:
> 
> https://github.com/zopefoundation/ZODB/compare/storage-byte-appreciation#diff-881ceb274f9e538d4144950eefce8682
> 
> This defines an interface, IStorageTransactionMetaData, that specifies these data as bytes.  ZODB will build an object implementing this interface as pass it to storages rather than passing transaction objects.
> 
> I've also defined IStorageTransactionMetaDataExtensionBytes, which has serialized extension data. This can be used to take responsibility for serializing extension data off storage hands.  (I need this for byteserver, but I think it will be beneficial in a small way for other storages.)


Storages are still responsible for de-serializing extension data, though, so this doesn't strike me as a complete solution to the problem. At least .history() and .undoLog() require that storages de-serialize the extension data and include it in the returned dictionary. That means that the storage needs to continue to be in charge of the serialization (so it knows how to deserialize, which I guess just means not using this attribute), or that the interface document the serialization format used by IStorageTransactionMetaDataExtensionBytes (for the same reason), *or* that an API is provided by ZODB to do the deserialization. 

Jason


-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.