Re: [dev] RFC: Transaction meta data

On Tue, Nov 15, 2016 at 11:01 AM, Jason Madden <[email protected]
<https://mail.google.com/mail/?view=cm&fs=1&tf=1&[email protected]>
> wrote:

>
> > On Nov 14, 2016, at 17:08, Jim Fulton <[email protected]
> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&[email protected]>>
> wrote:
> >
> > You wrote so many words because I wasn't clear. Sorry.  When I said that
> ZODB would be responsible for encoding, I thought it would be clear that
> "encoding" encompasses encoding and decoding.  I never meant to imply any
> asymmetry. I said "storages should only deal with bytes", which to my mind
> means they don't decode, because that would be dealing with text. <shrug>
>
> Ah, well, yes, that changes things then :)
>
> > My proposal was that ZODB be the boundary between text and bytes.  Text
> above, bytes below.
>
>
> >> So long as the high-level API (ZODB.DB, ZODB.Connection) doesn't expose
> any way to get those bytes back out as text, that should be OK.
> >
> > Except that they should.
> >
> > In fact, I was going to start working on that soon. :)
>
> I'm curious what you have in mind with regards to that. Will there be new
> APIs that tools like zodbbrowser can use to get history, except these
> values will be pre-decoded?


DB has always (as far as I can remember) provided history and undo APIs.
IDK why these aren't documented in IDatabase.  Perhaps this was just an
oversight (but ``undo`` is there).  I really don't expect application code
to interact with storages directly.

So this wouldn't be a new API, but rather just a change to existing APIs.
I guess I'd add then to IDatabase while I was at it.



> That is, the IStorage APIs would all be defined in terms of bytes (as they
> implicitly are now), and a new wrapper API would be defined that called
> through to an IStorage for history, but which did the decoding, and tools
> would change to use that API?
>

The existing DB apis would be updated to decode user and description. I'll
document them in IDatabase as a bonus.  For example, when application code
wants an object's history, it would call the ``history`` method on the
database. (This is what I've more or less always done.)  And in the data it
got back, the user and description would have been decoded.


>
>
> >
> > I really don't think people have put non-encoded text in the user or
> description fields. (If they have, they get what they deserve. :))
> Extension data makes storing binary data easy enough.  If we have people
> with Latin-1 data, then we could make the encoding used by ZODB
> configurable.
>
> I'm not sure what you mean by "non-encoded text."


I meant to write non-encoded-text and meant non-(encoded-text). I should
have said "data that isn't encoded text". :)


> I do know that there are people that pass in non-ASCII-decodable `bytes`
> values currently (or at least there were in 2009). I have no idea what
> those are encoded as (assuming they represent text; and you're right, if
> they don't represent text, that's a you-get-what-you-get scenario). Some
> quick googling of the name of the person that reported the issues suggests
> the encoding could either be UTF8 *or* CP1251 for Cyrillic.
>

So perhaps databases should grow an encoding option that defaults to
utf-8.  I'd rather not add another knob for something so minor though.


>
> > I don't like the native string idea because it just pushes the problem
> off to Python 3.  It's inherently ambiguous.  Ultimately, it means that
> storages have to make encoding decisions and I don't want that.
>
> If there will be a centralized API to handle the encoding/decoding, then
> that makes sense to me too. DRY FTW. (In domains like WSGI, there are
> standards that specify the encoding used, so I guess it's less of a problem
> there.)
>
> >
> > I can see 2 viable approaches (both of which are symmetrical :)):
> >
> >       • Text above bytes below ZODB. That's the current proposal.  We
> can make the encoding used by ZODB configurable to support legacy
> encoding.  IMO, this is the most application friendly approach, because
> modern applications use unicode to represent text.
> >
> >       • Bytes everywhere.  In this case we push encoding decisions up to
> applications.  Transactions would encode text using ASCII to force clients
> to decide.
> > I still prefer option 1.
>
> I'm in general agreement with option 1. I just have one nit about the
> current implementation.
>
> Even parts of modern frameworks like Pyramid (anything that uses WebOb)
> are still going to be native strings (e.g, Request.path and Request.url)
> and those are likely to end up in the transaction metadata. However, under
> Python 2, the auto-encoding that happens in the transaction setters (val +
> u'') will often mask that fact---until you get a TypeError.


I don't understand. If values are non-ascii bytes, the error will happen
when an attempt is made to set the data on a transaction.  That seems like
the right time.


> On traversal-using applications, where URLs are constructed dynamically,
> this may not happen until production; heck, because URLs coming in are
> under the control of a remote person, even a static URL scheme is
> vulnerable to this.


So, basically, you're worried about applications that aren't carefully
written.


>


> So, if we're going to go with transaction metadata being text, can we
> either (1) make it more strict and `assert isinstance(val, text)`


That would cause lots of breakage, but might be for the best. IDK. I wonder
what other folks think.


> or (2) make it more lenient and `val.decode('utf-8')`, possibly with a
> fallback to latin-1?


That feels too guessy to me, although I could live with utf-8.



> Option (1) probably breaks lots of Python 2 code (I know it would ours).



But I'm guessing the code it breaks is pretty centralized. A framework that
adds the path to the transaction note during traversal is likely do this in
one place that's easily fixed.



> (We could even make this configurable by giving TransactionManager a
> `transaction_factory` attribute that it uses to instantiate the Transaction
> object, and have two Transaction subclasses that handle it both ways.)
>

We could, but that's another knob for a pretty minor feature.

Jim

-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.