Re: PyYAML multi-line strings

Hello Kirill,

On 21/01/13 01:54, Kirill Simonov wrote:
> Use yaml.dump(..., allow_unicode=True)
> 
> By default, pyyaml always escape non-ASCII characters, allow_unicode
> reverses this behavior.

The above served us well, until recently, when someone had trouble with
accented characters, I did some further tweaks and now that Postgres 9.3
has been released the results are different when run on Python 3 vs.
Python 2.

I'd like to summarize some basic facts about how Pyrseas uses PyYAML.

1. We retrieve data from Postgres catalogs using psycopg2.

2. The data are mostly text strings, stuff like table and column names
(typically stored in UTF-8).

3. Under Python 2, we use psycopg2 register_type(UNICODE) which has the
effect of decoding the strings so that Pyrseas handles them as
'unicode'.  The same effect is achieved under Python 3, but of course
there they are type 'str'.  [Note: the first part is one of the recent
tweaks]

4. In order to take advantage of the nicer formatting of multi-line
strings, a contributor wrote a MultiLineStr_presenter class which is
found here:
https://github.com/jmafc/Pyrseas/blob/master/pyrseas/yamlutil.py . This
allows a text string such as the following:

  definition: " SELECT t1.c1, \n    t1.c3 * 2 AS mc3\n   FROM t1;"

to be represented instead as

  definition: |2-
     SELECT t1.c1,
        t1.c3 * 2 AS mc3
       FROM t1;

We do have to "cheat" a little because Postgres 9.3 for some odd reason
adds a space before that first newline, which means the nice formatting
could not be used.  So we strip trailing blanks before newlines, from
texts such as the above (they're more common in datababase function code).

5. Before we added the tweak in step 3, a string containing non-ASCII
text (e.g., a database function to translate accented characters to
non-accented) caused a decoding error under Python 2.  Fetching Unicode
fixed that but that had the YAML side-effect of displaying
!!python/unicode just about everywhere in the output YAML.

6. To fix the latter problem, it was suggested that we use
yaml.safe_dump instead of yaml.dump, which I did (as you can see in the
yamlutil.py referred to above).  However, that breaks the
Multiline_representer because pyyaml raises

yaml.representer.RepresenterError: cannot represent an object:  SELECT
t1.c1,
    t1.c3 * 2 AS mc3
   FROM t1;

Bottom line: under Python 3, everything works fine.  Under Python 2, it
appears we have to opt between (a) using safe_dump and not using the
multi-line representer (and the nice formatting) or (b) using yaml.dump
and somehow eliminating the !!python/unicode tags.

Can we have our cake and eat it under Python 2?

Best regards,

Joe

------------------------------------------------------------------------------
How ServiceNow helps IT people transform IT departments:
1. Consolidate legacy IT systems to a single system of record for IT
2. Standardize and globalize service processes across IT
3. Implement zero-touch automation to replace manual, redundant tasks
http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk