Closing out 2.7 beta 4

tl;dr

Jython 2.7.0 has been under active development since the PyCon sprints in March 2011. It’s about time we brought this to a close! Therefore I would like to propose that the beta 4 work be completed by the end of this month (Friday October 31), or at the very least by one week later (November 7).

Under this schedule, because beta 4 is our last beta, in 3 to 4 weeks we will be absolutely done with new features and will only focus on stabilization and high priority bug fixes for a release candidate well before the end of the year.

In addition, I would like to suggest that we provide a beta 4 preview by using the soft beta process, just so people can try out what is now available, such as better Django support, without having to build their own release with ant. Let’s have this done by the end of the week.

What follows is a long email, but I believe it’s best to capture the relevant details in one place. I would very much appreciate any feedback!

What is now in trunk since beta 3

There were a total of five beta 3 releases, thanks to our use of soft betas and the desire to provide timely access to new functionality and bug fixes. Excessive or not, a lot was done after the first beta 3 release, so I will discuss this in conjunction with what was done in the work leading up to beta 4:

Memory leaks and other system robustness

Initialization is more robust. PySystemState no longer attempts to initialize constants in the Py namespace, which should stablilize initialization. Instead, we use a lazy singleton holder pattern to ensure that initialization proceeds in the correct order. This also makes it possible to make these constants actually be final. Additionally, because we were free to break our API because of moving to 2.7, PySystemState no longer uses mutable statics, but instead uses instance variables.

Thread locals in Jython (via ThreadState) provide necessary context for moving back and forth between Java and Jython’s runtime. However, in another breaking API change, we have make this more robust by maintaining a hybrid strategy where ThreadState is set to a thread local upon entry into Python context, so lookup costs remain low, but is then removed. sys.recursionlimit support is retained, although it’s now only approximate given that it doesn’t take in account ops like __add__. Lastly, we use Object[1] indirection to avoid having a thread maintaining hard references to classes, thereby causing a ClassLoader resource leak.

Other ClassLoader resource leaks and subclass audit issues are avoided by making RefReaperThread and ShutdownCloser implement Runnable instead of extending Thread. PySystemState and PythonInterpreter now both implement AutoCloseable and Closeable.

Java integration

Except for one outstanding and to be fixed bug (that should not impact any real usage), Jython now supports standard Python semantics for proxied Java objects that implement java.util.Map. Such maps may or may not implement ConcurrentMap. There is one quite imporant change: d[key_not_present] raises KeyError instead of returning None. So this means its behavior now matches standard dict semantics. However, it may impact your code! In particular, we did have to fix some usage expecting these nonstandard Python semantics in Jython’s stdlib.

Classes implementing java.util.Map, List, and Set are now treated as subclasses of the corresponding Python abstract base classes MutableMapping, MutableSequence, and MutableSet.

Bound methods can now be used as arguments for methods taking a interface implementing a single method.

Finally, there’s a new jythonlib module to support libraries in stdlib that are written in Python (examples: _socket, threading) that need more precise integration from Java and Jython’s runtime, as well as other Jython-specific integration points. In particular, jythonlib now supports building dicts with arbitrary ConcurrentMap backing, especially for Google Guava’s MapMaker, without seeing boxing/unboxing issues moving between Java and Python code.

Finalizers

Finalizers see support in two ways:

__del__ for newstyle classes now works properly, but with Python 3.4 (and Java) semantics that the finalizer is only called once, even if the object is resurrected by the finalizer. This is almost certainly the behavior users would want. However, it’s possible to restore an object’s finalizer via __ensure_finalizer__.

JyNI has specific hook support via FinalizeTrigger

Miscellaneous bug fixes and upgrades for 2.7

Metaprogramming support was improved by some simple fixes to support the new 2.7 (and 3.x) names for function attributes, as well as fixing a bug in supporting meta path importers. With these improvements we now fully support the six module for supporting Python 2 and 3 with a common codebase. Another fix that has been long missing is supporting inspect’s ability to introspect function parameters that are anonymous tuples. CPython does this by looking at the Python bytecode, but it’s accessible via another path in Jython.

All the functions in itertools, via their supporting classes, now extend PyIterator instead of PyObject so they be chained together when the given itertools object is returned by an __iter__ method ina user-defined class. This also enables itertools classes to be directly iterated over by Java code. Related iterators enumerate and xrange were updated to support new functionality in Python 2.7.

For collections, collections.deque now includes support for bounded queues, and deques are now threadsafe like other collections in Jython. defaultdict now properly supports being subclassed.

String formatting has been seen significant improvements, including code size reduction.

PyUnicode now uniformly forbids lone surrogates, while speeding up index and slice support when working with Unicode strings not in the basic multilingual plane (> 0xFFFF for the codepoint).

Socket support had a bug on when to shut down the Netty child worker thread pool. We have also made the regrtest more robust with respect to sockets and corresponding thread pools. Note there remains an outstanding issue with some threads not terminating for the thread pools associated with server sockets, but hopefully this will go away once we move to Netty 4.1 (currently in beta).

Underlying platform functionality has been improved, including inherited handles for subprocesses, subprocess termination, and symlink support in shutil.rmtree.

The bz2 module now always attempts to read the number of requested bytes, even if the underlying TextIOBase may return less. This appeared as a difference between Java and Python conventions to what read should do when given a requested number of bytes. Jython now also ensures that bz2 decompression has enough data to decode.

Other changes

The site module is now imported by default on all entry points to Jython, including PythonInterpreter. This may impact your code! You can turn off via the importSite option.

Encoding support

Jython now supports Java encodings (via java.nio.charset.Charset) if the Python equivalent is not available/does not load with codecs.lookup. This means we now support CJK codecs like big5, shift_jis, and johab, but some codecs like hz and euc_jisx0213 remain unsupported due to their lack of availability in Java.

Similar codec support is available in the parser, which allows at least some programs to run where the console encoding is not one supported by Jython with a Python codec.

Additional anticipated features in beta 4

Substitute JyJDBC for zxJDBC. We have had great success with zxJDBC since being included as part of Jython 2.1, however, zxJDBC lacks sufficient unit testing and is therefore hard for us to maintain. However, we now have a great substitute in JyJDBC, which is a pure Jython JDBC dbapi driver - that is, it’s written in Python, but imports Java libraries including JDBC.

Chris Clark and his employer Actian (formerly Ingres) are contributing JyJDBC under the Python License, so we can maintain directly as part of Jython going forward, and thereby remove zxJDBC.

Note that the bundled version will be imported via a stub such that from com.ziclix.python.sql import zxJDBC will actually import JyJDBC. Or you can import it under the jyjdbc name.

xml.sax.expatreader, because of its use by Django.

Java implementation of json. This library is a perfect example of the sort of code that can see a significant improvement when written in Java. In this particular case, rather than using something like Jackson, in an experimental patch, I chose to port the C speedup functions from CPython because the json module makes some, ummm, interesting design choices in its API. Unfortunately, my initial effort failed only one test, but a very significant test, which was decoding from str and how it verifies the decodes. However, it also saw 100x speedup for some typical JSON docs when decoding.

Other committers - and some newly nominated ones - have been doing great work, so please provide feedback on what they they can expect to get in for beta 4, given the proposed cutoff date.

Triage existing skips

We now should be including all tests (at the file level) in regrtest that are applicable to Jython, with a triage occurring during the beta 3 cycle. However, a number of skips were added to these regrtest tests, most relatively early in the 2.7.0 development cycle. These skips themselves need to be triaged. One significant example is test_cmath, which tests complex math support in the cmath module.

Failing tests

In general, the most important problems to be fixed are those that directly tested by the regrtest. Here’s is the current list of failures, from regrtest itself:

     [exec] 16 fails unexpected:
     [exec]     test_asynchat test_asyncore test_dict_jy test_email
     [exec]     test_email_renamed test_gzip test_import test_inspect
     [exec]     test_isinstance test_json test_socket test_ssl test_strtod
     [exec]     test_tarfile test_univnewlines test_xmlrpc
These tests fail for a variety of reasons: difference due to the underlying OS (test_socket fails on Linux but not on my OS X laptop); being flakey (test_strtod), albeit with one or two least significant bits of float numeric precision; or will not fail if run separately (test_xmlrpc). We will almost certainly triage some of these tests for 2.7.1 or later, such as edge case CJK support seen in test_email and test_email_renamed. Skips as necessary will be added for this triage so that we can document these expected problems and note our plan to fix in a subsequent release (perhaps some of the missing functionality covered by test_ssl and most likely test_asynchat and test_asyncore which are rather subtle failures).

In the absolute certainty category, expect to see both test_dict_jy (currently fails because of a need to define inequality comparisons on duck-typed java.util.Map objects as Mappable) and test_instance fixed.

I really hope we can fix test_import (which is almost certainly related to some import bugs that are marked with high priority), however, this may be challenging given the complexity of import semantics.

Bugs by priority

I have labeled bugs that may still get in 2.7.0 with high priority in bugs.jython.org; normal priority means nice to have; low priority means 2.7.1 for sure; unprioritized generally means the same as low for this purpose, but also that we may have insufficient understanding of the problem. High priority bugs are either really easy or are likely to impact Python codebases. Low priority bugs tend to be corner cases, such as fixing some Java integration issue.

Some of these bugs have corresponding patches and pull requests, although often we do not have associated tests.

Again, this is an initial triage - I would like to see most of the high priority items fixed, and thoses with patches/PRs likely will be, but if not, I expect we can defer to 2.7.1 or later.

Outstanding pull requests

There are a number of outstanding pull requests for both the Bitbucket and GitHub mirrors. Let’s face it - we need to be more pro active in reviewing these PRs, ensuring they are well tested and solve the problem appropriately, and then getting them merged in. I don’t believe I have seen a single PR that didn’t have something useful to contribute. Sorry about that! I’m hopeful that after 2.7.0 is released we will do a better job.

Deferred to 2.7.1 (or later)

Note that there’s a small chance, depending on pull requests/patches (!) that we will implement the following functionality for 2.7.0. However, it seems much more realistic that this will have to slip to 2.7.1:

Upgrade to JLine2

sqlite3 There was a more-or-less complete patch (with BSD license) that was submitted about 7 years ago using javasqlite http://bugs.jython.org/issue1682864 However, it needs to be updated to use Xerial, which only requires Java, because we do not want to maintain code outside of JNR with direct JNI dependencies and corresponding shared C libraries. Note that it’s possible we could support by using CFFI instead, but that code also needs to be written.

I am disappointed that we will not have sqlite3 support because it is essential for such projects as IPython.

ensurepip and venv. Both of these would be backports from Python 3.4 and 3.3 respectively. There are significant advantages in having these supported in terms of providing standard PyPI ecosystem support to Jython; venv in particular could be adapted to know about singlejar layouts.

Please note, some of this functionality may slip to even releases beyond 2.7.1. For example, in this release cycle, we have been fixing a number of bugs that we had hoped to complete for 2.5.1 (!).

2.7.1 Release Schedule?

Since we didn’t include your favorite feature or fix a challenging bug, what is our plan for 2.7.1 and beyond? I think we should start moving to a regular release cycle after 2.7.0 is released so that we can adopt a time-based release schedule instead of one based on features. There are two reasons for this:

Python 2.7 as a language will see both continued and widespread support for many years to come.

Such schedules are inherently easier to work with as developers. Now that we have completed the core of the work for the language, standard libraries, and Python ecosystem, we can focus on making Jython better in a number of ways.

My initial thought is that we should count on having a release of 2.7.1 about six months after, then possibly adjust to a yearly schedule after that. I would love to hear others’ opinions on this!

Jython 3.5 development plans

We plan to kickoff Jython 3.5 development at the PyCon 2015 sprints in Montreal; as with previous versioning, Jython 3.5 will implement Python 3.5. (The CPython reference implementation is currently un development and is scheduled to be released in September 2015.) In many ways, Jython 3.5 will be a better fit for the underlying JVM and its libraries than Jython 2.x. Consider for example that java.lang.String is a natural fit for Python’s str type, without the awkwardness of using unicode. There are numerous other examples. I’m particularly interested in how the proposed optional static type annotations can help Jython better integrate with Java. Another motivating reason even for a subset of functionality - just supporting the Python 3 language - is supporting educational usage. Jython is a popular choice for a number of teaching programming environments (such as TigerJython), and in talking to my academic colleagues who teach introductory Python, they would like to see such Python 3 support for Jython.

Jython 3.5 should target a minimum of Java 8; and we should upgrade to using Antlr 4 for our parser support.



-- 
- Jim
Jim Baker

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://p.sf.net/sfu/Zoho

_______________________________________________
Jython-dev mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/jython-dev