Re: Unicode user and file names (and v2.7.1)
Jeff Allen <[email protected]>
| Newsgroups | gmane.comp.lang.jython.devel |
|---|---|
| Message-ID | <[email protected]> |
Thanks all. +1 on the RC. Nearly there with my bit. I have fixed the test_runpy failure James reported. It's not Linux-specific, just I had to quieten the unlink() error to see it on Windows. Bonus: we now pass the standard CPython test_runpy. The regrtest has been running one last time as I typed. I've pushed to https://bitbucket.org/tournesol/jython-utf8 just now. I will next merge into the Jython trunk. That may not be totally smooth because of the pervasive change. And now I think about it, it's worth a note in NEWS. My time is a little limited today, so it could be much later today or tomorrow evening. Jeff Jeff Allen On 20/05/2017 19:48, Jim Baker wrote: > +100 > > On Sat, May 20, 2017 at 12:33 PM, Darjus Loktevic <[email protected] > <mailto:[email protected]>> wrote: > > Agreed regarding not blocking on 2487. That whole area needs a > rewrite and we could potentially utilize libraries available for > Java 8. > > Let's get Jeff's work in and do an RC? > > Darjus > > On Fri, May 19, 2017 at 7:51 PM Jim Baker <[email protected] > <mailto:[email protected]>> wrote: > > I don't necessarily see http://bugs.jython.org/issue2487 > <http://bugs.jython.org/issue2487> as a blocker, but it would > be nice. It just hasn't come up in real usage, unlike the > earlier iteration of the bug which Darjus hacked around by > busy waiting. I did spend some time on trying to get the > publication to work without racing, using the approach I > detail in that bug, but no luck yet. (But mostly because an > utter lack of time to spend on the issue.) > > Merging in Jeff's recent work on Unicode is important and we > should get it in. I haven't had a chance to test myself, but > given Jeff's amazing attention to detail, I'm sure it's ready. > > The blocker for the RC - because we lost OSX support of > setuptools support of installed executables - is fixed, as I > just finally confirmed: http://bugs.jython.org/issue2570 > <http://bugs.jython.org/issue2570> > > > On Fri, May 19, 2017 at 8:26 PM, Stefan Richthofer > <[email protected] <mailto:[email protected]>> > wrote: > > AFAIK every release happens by having a successful RC that > is renamed to 'release' after a while. So, per definition > another RC is inevitable. > That said, I suppose we should get > http://bugs.jython.org/issue2487 > <http://bugs.jython.org/issue2487> fixed before we can > release. I guess Jeff's work will be ready until then. At > least that decision can be postponed until an RC is > actually doable. > *Gesendet:* Samstag, 20. Mai 2017 um 02:35 Uhr > *Von:* "Darjus Loktevic" <[email protected] > <mailto:[email protected]>> > *An:* "Jeff Allen" <[email protected] > <mailto:[email protected]>>, "Jython Developers" > <[email protected] > <mailto:[email protected]>> > *Betreff:* Re: [Jython-dev] Unicode user and file names > (and v2.7.1) > > Hey Jeff, > > Sounds good. Let's do another rc but to be honest I'm not > even sure the RC matters much if there aren't people > trying it except us. > > Thoughts? > Darjus > > On Fri, May 19, 2017, 1:19 AM Jeff Allen > <[email protected] <mailto:[email protected]>> wrote: > > Hi Darjus. > > On inclusion, I'm happy to go with the community view, > as always. On one of the related tickets > (http://bugs.jython.org/issue1839 > <http://bugs.jython.org/issue1839>), Jim said we'd get > it in if timing allowed and there was some user support. > > I'm very keen to see a 2.7.1 too. The last (soft) RC > was unsuccessful, and we're still making changes, so I > assume we're talking about another RC first rather > than a release? > > The UTF-8 work is nearly there, but not quite: one > Linux defect to fix, as noted on the same issue by > James against the "latin-1" version. After all the > additions in the last couple of weeks (to get full BMP > support), I'm happy to find from my Linux laptop that > it is still the only thing I have to do. It looks > trivial. I've been unable code at all for a few days, > so haven't looked into a solution, but now I'm back I > expect to nail it for us today or tomorrow. > > I can, of course, merge all this myself and will. I > shared your hesitancy initially, hence the fork > repository, but it's turned out so well I feel it's > now low risk, as long as we still have a few days. > > I will now dive under the desk and wire up my Linux > dev box. > > Jeff Allen > > On 16/05/2017 21:46, Darjus Loktevic wrote: > > Hey Jeff, > It seems your last commit to this branch is of > three days ago. Is this ready for review? BTW, > your changes look good to me. > I'm a little hesitant to merge this since we've > had an RC and REALLY have to release 2.7.1 It's > miles better than 2.7.0. > Cheers, > Darjus > On Mon, May 1, 2017 at 6:34 AM Jeff Allen > <[email protected] <mailto:[email protected]>> > wrote: > > I went for sys.getfilesystemencoding() == > 'utf-8' and it works pretty > well. Rather than just push directly I have > published to here: > > https://bitbucket.org/tournesol/jython-utf8 > <https://bitbucket.org/tournesol/jython-utf8> > > I write to ask for a second or third pair of > eyes on it. Please tell me > you can see it and whether it breaks things > you care about. > > I touched a lot of files in the core and > import system: quite a lot of > tricky stuff with loaders and search paths has > been adjusted. I think it > a good sign that I changed hardly anything in > the standard library we > inherit from CPython, that we hadn't already > specialised. > > By "works pretty well" above, I mean that the > regression tests run > cleanly for me when my user name is "Épreuve", > where previously Jython > died horribly. The launcher works from a > Chinese user name too, as long > as I localise Windows to China (CPython 2.7 > feature). I can use the > prompt and runs some tests with that setup, > but I can't run the > regression test yet, and printing a stack dump > is fatal, so there's a > bit more to do for Chinese. > > I think this means we have solid support for > "latin-1" languages, but > there are still places where we fatally assume > bytes are Unicode code > points. > > Jeff Allen > > On 05/04/2017 08:57, Jeff Allen wrote: > > I've been working on > http://bugs.jython.org/issue2356 > <http://bugs.jython.org/issue2356> which I'd > like to > > get in 2.7.1 -- it seems rather poor that > Jython simply does not run for > > users whose names have an un-American > character ;). I know this issue is > > not a blocker in most minds. > > > > I've made pretty good progress by allowing > file names to be unicode > > objects more often than they would be in > CPython 2, which usually > > returns them as bytes in some encoding that > we may not know. I've got > > the launcher to work properly, and > straightened the logic in our > > printing of trace-backs and exceptions from > Java. Unicode file names > > seems the way to go for Jython because: > > > > 1. Java gives us competently decoded > unicode file names, from > > java.io.File, etc.. Re-encoding the result > will be a pain (and > > overlooked). > > 2. We appear not to have the codec we need > ('mbcs'), that CPython > > reports on Windows via > sys.getfilesystemencoding(). > > 3. We do this already. In 2.7.0, > os.getcwd() returns unicode if necessary. > > > > Most regression tests pass. However, I'm > struggling with test_doctest. > > Problems arise when mixing unicode and bytes > when one byte is 128 and > > over. This happens in ''.join(list) and > formatted output like "%s %s" % > > (ustr, bstr). The behaviour of these is > identical with CPython: they > > raise UnicodeDecodeError because the bytes > are promoted to characters > > with a strict ascii interpretation. This > happens a lot in doctest.py and > > traceback.py, for example, where file paths > and stack dumps that include > > them, are now frequently unicode, while > other inputs are byte data > > containing file paths presented in the > console encoding. > > > > I can beat this into submission with enough > customisation of the stdlib > > modules, but that always makes me > uncomfortable. I usually see that as a > > hint that user code might also need to > change. This may be unfounded. I > > can probably ensure no impact to users of > only ascii paths, and the > > others seem unable to run Jython at all (in > the scope of this issue). > > However, I'm seriously wondering if I should > pursue the approach where > > file names from Java are re-encoded to bytes > (maybe as utf-8 > > everywhere), but that's grim. > > > > Thoughts? > > > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of > the world's most > engaging tech sites, Slashdot.org! > http://sdm.link/slashdot > _______________________________________________ > Jython-dev mailing list > [email protected] > <mailto:[email protected]> > https://lists.sourceforge.net/lists/listinfo/jython-dev > <https://lists.sourceforge.net/lists/listinfo/jython-dev> > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's > most engaging tech sites, Slashdot.org! > http://sdm.link/slashdot_______________________________________________ > <http://sdm.link/slashdot_______________________________________________> > Jython-dev mailing list [email protected] > <mailto:[email protected]> > https://lists.sourceforge.net/lists/listinfo/jython-dev > <https://lists.sourceforge.net/lists/listinfo/jython-dev> > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's > most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > Jython-dev mailing list > [email protected] > <mailto:[email protected]> > https://lists.sourceforge.net/lists/listinfo/jython-dev > <https://lists.sourceforge.net/lists/listinfo/jython-dev> > > > > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > > _______________________________________________ > Jython-dev mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/jython-dev ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Jython-dev mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/jython-dev