Re: [dev] RFC: Transaction meta data

I'm going to work on some other things while I wait for a resolution on
this.

Silence is assent to option 1 after a day or two. :)

Jim

On Mon, Nov 14, 2016 at 6:08 PM, Jim Fulton <[email protected]> wrote:

> I'm making a high-level response and a low-level response. It might be
> best to just pay attention to the high-level response. :)  (The low-level
> response is all just in the weeds.)
>
> You wrote so many words because I wasn't clear. Sorry.  When I said that
> ZODB would be responsible for encoding, I thought it would be clear that
> "encoding" encompasses encoding and decoding.  I never meant to imply any
> asymmetry. I said "storages should only deal with bytes", which to my
> mind means they don't decode, because that would be dealing with text.
> <shrug>
>
> My proposal was that ZODB be the boundary between text and bytes.  Text
> above, bytes below.
>
> FileStorage encodes text as ascii, and accepts bytes.  It sounds like
> RelStorage 1 did the same (unless someone changed the Python default
> encoding in which case they get what they deserve ;).  The iterator APIs
> don't specify (or specify str, which is the same as not specifying) and
> most or implementations return bytes, same for undo logs and history I
> expect.
>
> I really don't think people have put non-encoded text in the user or
> description fields. (If they have, they get what they deserve. :))
> Extension data makes storing binary data easy enough.  If we have people
> with Latin-1 data, then we could make the encoding used by ZODB
> configurable.
>
> I don't like the native string idea because it just pushes the problem off
> to Python 3.  It's inherently ambiguous.  Ultimately, it means that
> storages have to make encoding decisions and I don't want that.
>
> I can see 2 viable approaches (both of which are symmetrical :)):
>
>
>    1. Text above bytes below ZODB. That's the current proposal.  We can
>    make the encoding used by ZODB configurable to support legacy encoding.
>    IMO, this is the most application friendly approach, because modern
>    applications use unicode to represent text.
>
>    2. Bytes everywhere.  In this case we push encoding decisions up to
>    applications.  Transactions would encode text using ASCII to force clients
>    to decide.
>
> I still prefer option 1.
>
> I really appreciate the time you took to respond. Thank you very much.
>
> Jim
>
>
> On Mon, Nov 14, 2016 at 4:59 PM, Jason Madden <
> [email protected]> wrote:
>
>>
>> > On Nov 14, 2016, at 13:41, Jim Fulton <[email protected]> wrote:
>>
>> > Recently, I tried to clean up transaction meta data:
>> ...
>> > This opened a small can of worms.
>>
>> I've been going around in circles on this one. My tl;dr is that I don't
>> think I like having encoding/decoding responsibility be so asymmetrical
>> (high-level ZODB would be responsible for encoding, but who would be
>> responsible for decoding?).
>
>
> When I said that ZODB would be responsible for encoding, I meant to imply
> that it would be responsible for decoding too.  The asymmetry you mention
> wouldn't make sense. :)
>
>
>
>
>> I also have some backwards compatibility concerns, and I wonder, if we're
>> going to be breaking compatibility, maybe it would be better to thread
>> text/unicode all the way down to the storage, instead of stopping at a
>> layer above it, or even just specify them as bytes for maximum flexibility.
>>
>
> That would produce lots of backward compatibility around the restore APIs.
> Iterators don't specify the types of the data, but most or all
> implementations return bytes.
>
>
>>
>>
>> > The existing storage APIs are vague about whether the user and
>> description field are text/unicode or bytes.  Historically, they've been
>> bytes, but storage implementations are defensive and allow either, although
>> they're also conservative and, if given text, encode using an ASCII
>> encoding.
>>
>> RelStorage 1.x (which was only Python 2) used `str(user)` to do the
>> encoding, which (almost always) encoded as ASCII when given text.
>
>
> Almost always?
>
> FWIW, FileStorage is conservative and uses an ascii encoding.
>
>
>
>> However, it was explicitly tested that bytes arguments could round-trip
>> the entire 256 possible values; there was a user bug filed for this[1].
>> According to the commit comment, this was because "according to the
>> transaction interface, transaction metadata is always a string, but may
>> have encoded characters" (where "string" here is python 2's bytes)[2].
>>
>
> Well, it said str, but of course, that means unicode on Python 3. That was
> the ambiguity I was trying to address.
>
>
>>
>> For RelStorage 2 (currently in beta---so this could change---and
>> supporting Python 2 and 3), the ASCII encoding was expanded to be
>> Latin-1/ISO-8859-1,
>
>
> Ugh.
>
>
>> thus ensuring that, no matter the platform, and no matter whether text or
>> bytes, the first 256 values round-trip in the same form, while prohibiting
>> other values:
>>
>>   py> u'\x80'.encode('latin-1') # ord 128
>>   '\x80
>>
>
> That doesn't seem to me to be much of a win.  Why did you go with Latin-1
> and not UTF-8?
>
> I really don't want storages to make this decision.
>
>
>>
>>
>> [1] https://github.com/zodb/relstorage/commit/f4b5370033f1e248ab
>> 2a0cc35acb99262a119047
>> [2] https://github.com/zodb/relstorage/commit/01f3414bd0ab2dcc9d
>> 86f884cf02cb5389dcab8f
>
> >
>> > For a while I've realized that storages should be asked to do less than
>> they do now, and that more should be done by ZODB.
>> >
>> > In that spirit, I'm inclined to say that transaction user and
>> description should be encoded by ZODB, using UTF-8 and that storages should
>> only deal with bytes.  Eventually, storages can be less defensive.
>>
>>
>> So long as the high-level API (ZODB.DB, ZODB.Connection) doesn't expose
>> any way to get those bytes back out as text, that should be OK.
>
>
> Except that they should.
>
> In fact, I was going to start working on that soon. :)
>
>
>
>> That is, if the IStorage interfaces (.iterator(), .history(),
>> .undoLog()---are there more?) deal in bytes,
>
>
> Yep
>
>
>
>> and that's the only way to get that data back out,
>
>
> Nope.
>
> that works out from a conceptual level (IStorage deals just in bytes) and
>> a practical level (there are existing RelStorage's out there that have
>> non-UTF-8 data in 'user' and 'description', and, given some hypothetical
>> text-returning API, while RelStorage could potentially catch
>> UnicodeDecodeError on UTF-8 and try again as latin-1, there's no real
>> reason why some higher level ZODB API should be expected to know to do
>> that).
>>
>
> You're making a legacy argument for something that's in beta.
>
>
>
>>
>> On the other hand, we're still left with asymmetry, ambiguity and guessing
>
>
> No. ZODB should decode. I wasn't clear.
> ...
>
>
>>
>> Now, we could sidestep this by defining the bytes encoding to be utf-8
>> and enshrining that in the API documentation for IStorage.
>>
>
> If we enshrine it at the ZODB level, we don't need to mess with existing
> storages.  Mostly we just have to clarify the interface.
>
>
>
>>
>> This leads to my biggest concern, which is that there are existing
>> ("legacy") storages, where the user has been in full control of the byte
>> encoding this entire time.
>
>
> I find it a little hard to believe that people have exercised control.
>
>
>> There's no guarantee that those bytes are in utf-8, latin-1, or other.  A
>> generic tool would have to look something like this:
>>
>>    try:
>>       desc = d['description'].decode('utf-8')
>>    except UnicodeError:
>>       # This may not display correctly, but is guaranteed to decode.
>>       desc = d['description'].decode('latin-1')
>>
>
> That's what would be the case if this was left undefined or the storage's
> responsibility.
>
>
>>
>> That doesn't really change. But currently, custom tools would just be
>> doing something like `d[description'].decode('big5')`, and they would
>> have to change similarly. The issue, other than the repetition, is that I'm
>> pretty sure it's not possible to catch improper decoding in every codec, so
>> some mojibake will inevitably happen.
>>
>> An additional small loss I notice in defining these values to be text is
>> that applications lose the ability to store mixed data or otherwise custom
>> bytes in these fields. I'm not sure what the use of that would be, but it
>> has been possible.
>>
>
> I would hope such applications would have used extended info (aka
> _extension).
>
>
>
>>
>> Here are three different possibilities to consider:
>>
>> In all of them, I'm using the same types at the `transaction` level as at
>> the `IStorage` level, for consistency.
>>
>> If instead we define these values as "native strings", i.e., text on
>> Python 3, bytes on Python 2, just like WSGI does, then overall there's the
>> least amount of work to do in terms of avoiding TypeErrors.
>
>
> I don't think that's true, especially for Python 3. I also think that's
> moot because up to now, we've only allowed ascii unicode strings.
>
>
>> Presumably this is largely the de facto situation.
>
>
> The de facto situation is ascii for most people, which is insufficient.
>
>
>
>> Things may not always look right (on Python 2), but Python 2 client
>> applications don't have any work to do either (assuming storages continue
>> to be permissive in allowing unicode values that they choose an encoding
>> for; latin-1 for Python 2/3 compatibility, utf-8 for maximum encoding).
>> Python 3 would not have the ability to store arbitrary bytes data in these
>> fields, unless the storages were permissive. The responsibility for
>> choosing an appropriate storage format is on each storage, which seems fair
>> (some RDBMSs can natively store Unicode text, for example, and RelStorage
>> might choose to use that on Python 3).
>>
>
> That sounds like a nightmare to me.
>
>
>>
>> Alternatively, we could define the transaction attributes to be bytes
>> everywhere. That matches the implied python 2 contract. If storages
>> continue to still accept and encode text data (presumably as utf-8),
>
>
> I don't think any storages encode as utf-8 now. Most encode as ASCII. (I
> wonder what NEO does.)
>
>
>
>> no clients have to change. ZODB could still do the encoding for incoming
>> text values to make sure that storages consistently use the same encoding,
>> if desired.
>
>
> No, because data come in via transactions and IIUC, you're saying that
> transaction meta data should be bytes.
>
>
>
>> Tools that want to attempt to display the data as text could try to
>> decode it, but they wouldn't have any guarantees; that shifts some of the
>> work onto them, but also allows users to continue to use their preferred
>> encodings (or stuff arbitrary data there).
>>
>
> This doesn't sound desirable to me.
>
>
>>
>> Lastly, we could define these values as text everywhere. This has the
>> same problems for Python 2 applications as the code currently committed to
>> transaction (unless some layer wants to attempt a conversion for them,
>> which may not always be possible), but it widens the implicitly permitted
>> character set. Storages still have to determine how they want to store
>> unicode text, but they could stick to just encoding it, which is what they
>> probably do now, just with UTF-8 instead of ASCII. It probably results in
>> tool changes, too, even on a single version of Python.
>>
>
> This would cause lots of breakage.
>
>
>>
>> I'm not sure which, if any, of these is the most improvement over either
>> the old situation or the current master situation, although I think I'm
>> personally leaning towards the native string approach. I suppose it depends
>> on the relative importance and use of "browser" type tools, non-ASCII data
>> in existing databases, and the need to encode/decode data in separate
>> places.
>>
>> I think having this conversation, and at least getting things better
>> documented, is a good thing!
>>
>>
>> > I've updated the ZODB interfaces on a branch:
>> >
>> > https://github.com/zopefoundation/ZODB/compare/storage-byte-
>> appreciation#diff-881ceb274f9e538d4144950eefce8682
>> >
>> > This defines an interface, IStorageTransactionMetaData, that specifies
>> these data as bytes.  ZODB will build an object implementing this interface
>> as pass it to storages rather than passing transaction objects.
>> >
>> > I've also defined IStorageTransactionMetaDataExtensionBytes, which has
>> serialized extension data. This can be used to take responsibility for
>> serializing extension data off storage hands.  (I need this for byteserver,
>> but I think it will be beneficial in a small way for other storages.)
>>
>>
>> Storages are still responsible for de-serializing extension data, though,
>> so this doesn't strike me as a complete solution to the problem.
>
>
> No.  It would need to be expanded to deal with history and undoLog in some
> way.
>
>
>
>> At least .history() and .undoLog() require that storages de-serialize the
>> extension data and include it in the returned dictionary.
>
>
> Technically history does but undoLog doesn't promise.
>
>
>> That means that the storage needs to continue to be in charge of the
>> serialization (so it knows how to deserialize, which I guess just means not
>> using this attribute), or that the interface document the serialization
>> format used by IStorageTransactionMetaDataExtensionBytes (for the same
>> reason), *or* that an API is provided by ZODB to do the deserialization.
>>
>
> Or allow storages to return strings.  Note that even if this isn't a win
> for history and undo, which are rarely used, it is a win for ZEO :),
> because it pushes takes work off the server. Currently, ZEO pickles the
> extension data to send it over the wire, then the server then unpickles it
> when interpreting the message, and then pickles it again (in FileStorage).
> This is mostly a ZEO issue, but I would need FileStorage to play along to
> get this win. But all of this is too complicated for this level.
>
> I retract this part of the proposal. :)
>
> Jim
>
> --
> Jim Fulton
> http://jimfulton.info
>



-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.