Re: unicode test failures urllib tests etc.

It's 2.4MB, which is big but not huge, but then there is an automatic 
adaptation to version in test_normalization, and we'd have to remember 
to do that. Also, it becomes another obstacle to reverting 
test_normalization to the stdlib version, which is a standing objective 
for every test.

But this is solving the wrong problem.

The stack dump Ray exhibits is from a different test, at a different 
site, where the access is not guarded by "urlfetch". In fact I think 
we're talking about rejection by the FTP server itself, for which 
accessing a file is not a substitute test.

Jeff Allen

On 02/11/2018 07:48, Adam Burke wrote:
> The resource system in the regrtests is sensible. But it seems a shame 
> to make the network resource a prerequisite for what is essentially a 
> static, versioned file?
>
> Cheers
> Adam
>
> On Fri, 2 Nov 2018 at 17:42, Jeff Allen <[email protected] 
> <mailto:[email protected]>> wrote:
>
>     The tests use a system of "resources" to switch on or off tests.
>     There is a "network" resource that if not enabled (regrtest -u) is
>     intended to switch off those tests that make external reference
>     and also a "urlfetch" resource, and this is what
>     test_normalization uses. There is a test_support.requires_resource
>     decorator applicable test by test.
>
>     If Travis blocks external reference, then we ought to run tests
>     there without that resource so we don't try. Or is that the rate
>     at which we launch them just too high? We could consider defining
>     another, as long as we're clear what we mean and document it. But
>     using "urlfetch" in a few more places may express what we really
>     mean: existing is always better than bespoke.
>
>     On the substantive issue, we decided not to support lone
>     surrogates in Jython and then changed our minds to allow the
>     (ab)use by which CPython smuggle bytes in file paths that exist
>     apart from their intended encoding. Some design is based on the
>     former state. I would say that when a lone surrogate is
>     encountered it will still depend on the context what it means.
>     (ISTR it's only 0xDC00..0xDCFF that's at issue.)
>
>     Jeff
>
>     Jeff Allen
>
>     On 02/11/2018 03:32, Adam Burke wrote:
>>     Interesting. Apart from the unintentional DOS, I feel unit / regr
>>     test time dependencies on remote resources should be limited anyway.
>>
>>     How are you thinking of resolving it? One way would be to keep a
>>     local copy of the relevant file, and declare the dependency
>>     similarly to a library, perhaps in the gradle build files. The
>>     metadata that this is a copy of someone else’s golden source
>>     seems valuable.
>>
>>     Cheers
>>     Adam
>>
>>     在 2018年10月26日，上午4:20，Raymond Ferguson via Jython-dev
>>     <[email protected]
>>     <mailto:[email protected]>> 写道：
>>
>>>      Finally managed to get some verbose output.
>>>
>>>     ./Lib/test/test_urllib2net.py:   FTP_HOST =
>>>     "ftp://ftp.mirror.nl/pub/gnu/"
>>>
>>>     We're probably participating in an unintentional DOS attack via
>>>     github->travis and just being blocked.
>>>
>>>          [exec] Re-running failed tests in verbose mode
>>>          [exec] Re-running test 'test_urllib2net' in verbose mode
>>>          [exec] test test_urllib2net failed -- Traceback (most
>>>     recent call last):
>>>          [exec]  File
>>>     "/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py",
>>>     line 312, in test_ftp_no_timeout
>>>          [exec]    u = _urlopen_with_retry(self.FTP_HOST, timeout=None)
>>>          [exec]  File
>>>     "/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py",
>>>     line 27, in wrapped
>>>          [exec]    return _retry_thrice(func, exc, *args, **kwargs)
>>>          [exec]  File
>>>     "/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py",
>>>     line 23, in _retry_thrice
>>>          [exec]    raise last_exc
>>>          [exec] URLError: <urlopen error ftp error: 425 Security:
>>>     Bad IP connecting.>
>>>     *Ray Ferguson*
>>>     *Enterprise Solutions Developer*
>>>     *Devendor Tech LLC*
>>>     *[email protected]*
>>>     <mailto:[email protected]>
>>>     *W: 608-960-8090C: 608-516-9391*
>>>
>>>
>>>     ᐧ
>>>
>>>     On Thu, Oct 25, 2018 at 1:14 PM Raymond Ferguson
>>>     <[email protected]
>>>     <mailto:[email protected]>> wrote:
>>>
>>>         There is this little bit of code in __builtin__.unichr that
>>>         creates a hole in unicode values for utf-16 encoding.
>>>
>>>          if (i >= 0xD800 && i <= 0xDFFF) {
>>>           throw Py.ValueError("unichr() arg is a lone surrogate in
>>>         range (0xD800, 0xDFFF) (Jython UTF-16 encoding)");
>>>         }
>>>
>>>         Anyway... I don't see any way that test_normalization can
>>>         pass given that sys.max_unicode > 0xd800 and one of the
>>>         tests does a for range(sys.maxunicode) unichr(...).
>>>
>>>         Strangely though, this does pass from travis-ci.org
>>>         <http://travis-ci.org>. Apparently there is some sort of
>>>         reactive blocking in place that will drop connections to
>>>         www.unicode.org <http://www.unicode.org> from your IP if you
>>>         trigger something, probably rate of get requests for test
>>>         files.  The test in question grabs a file from
>>>         www.unicode.org <http://www.unicode.org> and skips the test
>>>         on timeout, so it doesn't hit the part of the test that will
>>>         always fail.
>>>
>>>         Anyway, after troubleshooting for a while, I managed to
>>>         trigger this and suddenly can't reach unicode.org
>>>         <http://unicode.org> from my LAN, but if I switch my phone
>>>         over to mobile data for a new source address, clearly the
>>>         site is still up.  I can even ping it but tcp syn is dropped.
>>>
>>>         I've also noticed that urllib2 tests work fine from my
>>>         network, but fail on travis-ci.  My guess is that the at
>>>         least one of the test targets are taking steps to reduce
>>>         impact of being hard coded into so many CI tests.
>>>
>>>         *Ray Ferguson*
>>>         *Enterprise Solutions Developer*
>>>         *Devendor Tech LLC*
>>>         *[email protected]*
>>>         <mailto:[email protected]>
>>>         *W: 608-960-8090C: 608-516-9391*
>>>
>

_______________________________________________
Jython-dev mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/jython-dev