Re: [dev] RFC: Transaction meta data

> On Nov 14, 2016, at 17:08, Jim Fulton <[email protected]> wrote:
> 
> You wrote so many words because I wasn't clear. Sorry.  When I said that ZODB would be responsible for encoding, I thought it would be clear that "encoding" encompasses encoding and decoding.  I never meant to imply any asymmetry. I said "storages should only deal with bytes", which to my mind means they don't decode, because that would be dealing with text. <shrug>

Ah, well, yes, that changes things then :)

> My proposal was that ZODB be the boundary between text and bytes.  Text above, bytes below.


>> So long as the high-level API (ZODB.DB, ZODB.Connection) doesn't expose any way to get those bytes back out as text, that should be OK.
> 
> Except that they should.
> 
> In fact, I was going to start working on that soon. :)

I'm curious what you have in mind with regards to that. Will there be new APIs that tools like zodbbrowser can use to get history, except these values will be pre-decoded? That is, the IStorage APIs would all be defined in terms of bytes (as they implicitly are now), and a new wrapper API would be defined that called through to an IStorage for history, but which did the decoding, and tools would change to use that API?


> 
> I really don't think people have put non-encoded text in the user or description fields. (If they have, they get what they deserve. :)) Extension data makes storing binary data easy enough.  If we have people with Latin-1 data, then we could make the encoding used by ZODB configurable.

I'm not sure what you mean by "non-encoded text." I do know that there are people that pass in non-ASCII-decodable `bytes` values currently (or at least there were in 2009). I have no idea what those are encoded as (assuming they represent text; and you're right, if they don't represent text, that's a you-get-what-you-get scenario). Some quick googling of the name of the person that reported the issues suggests the encoding could either be UTF8 *or* CP1251 for Cyrillic.


> I don't like the native string idea because it just pushes the problem off to Python 3.  It's inherently ambiguous.  Ultimately, it means that storages have to make encoding decisions and I don't want that.

If there will be a centralized API to handle the encoding/decoding, then that makes sense to me too. DRY FTW. (In domains like WSGI, there are standards that specify the encoding used, so I guess it's less of a problem there.)

> 
> I can see 2 viable approaches (both of which are symmetrical :)):
> 
> 	• Text above bytes below ZODB. That's the current proposal.  We can make the encoding used by ZODB configurable to support legacy encoding.  IMO, this is the most application friendly approach, because modern applications use unicode to represent text.
> 
> 	• Bytes everywhere.  In this case we push encoding decisions up to applications.  Transactions would encode text using ASCII to force clients to decide.  
> I still prefer option 1.

I'm in general agreement with option 1. I just have one nit about the current implementation.

Even parts of modern frameworks like Pyramid (anything that uses WebOb) are still going to be native strings (e.g, Request.path and Request.url) and those are likely to end up in the transaction metadata. However, under Python 2, the auto-encoding that happens in the transaction setters (val + u'') will often mask that fact---until you get a TypeError. On traversal-using applications, where URLs are constructed dynamically, this may not happen until production; heck, because URLs coming in are under the control of a remote person, even a static URL scheme is vulnerable to this.

So, if we're going to go with transaction metadata being text, can we either (1) make it more strict and `assert isinstance(val, text)` or (2) make it more lenient and `val.decode('utf-8')`, possibly with a fallback to latin-1? Option (1) probably breaks lots of Python 2 code (I know it would ours). (We could even make this configurable by giving TransactionManager a `transaction_factory` attribute that it uses to instantiate the Transaction object, and have two Transaction subclasses that handle it both ways.)

Jason

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.