Re: Performance of PyUnicode Issue #2100

Jeff,
Two tools that I've seen that have a reasonable reputation for being able
to create reliable micro benchmarks on the JVM are JMH [1] and Caliper [2].
I've not used either but my understanding is that they are suited to that
sort of benchmarking and handle the vagaries of the JIT compiler.

[1] http://openjdk.java.net/projects/code-tools/jmh/
[2] https://code.google.com/p/caliper/


On 17 August 2014 18:04, Jeff Allen <[email protected]> wrote:

> In addressing #2100 I'm keen to avoid increasing the execution time of
> operations on unicode objects that *don't* contain supplementary
> characters, and to minimise time overall. I believe I know how to do
> that, but I want to be able to prove it, so I've devised a benchmark.
>
> This may be of interest to others. When benchmarking Java it seems
> almost impossible to get repeatable results. It is common practice to
> run each test multiple times and take a minimum (see timeit module), to
> compensate for transient scheduling disturbances.  This hasn't worked
> for me with the JVM, which will step in when it feels like it, and
> changes the code being measured. Two repeat-blocks running the same test
> may yield radically different times, because the JVM adds a permanent
> optimisation some time during the second block. I seem to have found an
> approach that works. I repeat the entire sequence of tests, and take
> corresponding minimum times. It means the code is likely to be in the
> same state for successive tests, rather than adapting to the test
> currently being repeated. Now I think about it, randomising the order
> ought also to help too.
>
> Turning off the JIT compiler also produces fairly consistent times, but
> they're about a forty times slower. These are not the conditions under
> which we most need to understand our code.
>
> My benchmark is at:
>
> https://bitbucket.org/tournesol/jython-ja/commits/aed06364c44d7089d58affa92c0502f477274b8f#chg-tests/python/unicode_index_times.py
> . I'll add this to the central repo IDC: I'm just trying to avoid
> creating merges.
>
> The figures are quite interesting, and also the comparison with
> CPython.  I'm doing this on WIndows 7x64, occupying a single AMD CPU at
> 2.9GHz). This is for Jython (microseconds per operation):
>
>  >%JT%\dist\bin\jython unicode_index_times.py
> time (us)         len         [i]     [i:i+n]     find(c) find(str)
> rfind(c)  rfind(str)
> single bmp          1       0.269       3.086 0.279       0.122
> 0.228       0.121
> single              1       0.259       2.381 0.275       0.122
> 0.226       0.122
> short bmp          10       0.155       0.418 0.279       0.159
> 0.216       0.145
> short 50%          10       0.188       0.591 0.290       0.160
> 0.213       0.149
> medium bmp        100       0.144       0.195 0.255       0.250
> 0.172       0.284
> medium 10%        100       0.316       0.718 0.274       0.274
> 0.168       0.260
> long bmp         1000       0.143       0.211 0.259       1.229
> 0.152       1.218
> long 1%          1000       1.390       4.261 0.275       1.225
> 0.151       1.240
> long 10%         1000       1.935       2.976 0.291       1.264
> 0.144       1.249
> long 10% low     1000       1.600       2.931 0.285       1.267
> 0.145       1.241
> long 10% high    1000       1.385       2.534 0.282       1.223
> 0.145       1.302
> long 50%         1000       2.450       3.737 0.301       1.321
> 0.124       1.367
> huge bmp        10000       0.139       0.464 0.235       7.111
> 0.142       7.661
> huge 10%        10000      11.113      39.374 0.247      11.305
> 0.136      11.358
>
> The percentages are percentages of non-BMP characters in the string, and
> low and high refers to concentrating them in the first or last quarter
> of the string. (I expect that to make a difference in one implementation
> I have in mind.) The presence of non-BMP characters makes a striking
> difference to index and slice operations in long strings, because they
> have to count along the string to convert character (code point) index
> to the implementation (UTF-16) index. The operations find() and rfind()
> don't show such a big difference, but they also get the wrong answer by
> failing to perform the reverse translation, having found the correct
> UTF-16 index. find(c) is roughly independent of string length because
> the operation timed is the incremental find of a single letter, and
> there's nearly always a match about 26 cells away. It's odd that a
> reverse find should be noticeably quicker.
>
> The following is for CPython 2.7.6. On Windows this means we're confined
> to basic plane characters. Index and slice operations are a little
> slower than the Jython version (for BMP strings, and once the JIT has
> done its work):
>
>  >python unicode_index_times.py
> Narrow build: all characters from BMP
> time (us)         len         [i]     [i:i+n]     find(c) find(str)
> rfind(c)  rfind(str)
> single bmp          1       0.447       1.465 0.633       0.502
> 0.674       0.501
> single              1       0.444       1.457 0.635       0.501
> 0.671       0.501
> short bmp          10       0.204       0.382 0.669       0.549
> 0.667       0.555
> short 50%          10       0.203       0.379 0.670       0.552
> 0.669       0.558
> medium bmp        100       0.179       0.233 0.708       0.606
> 0.627       0.610
> medium 10%        100       0.180       0.231 0.714       0.588
> 0.626       0.586
> long bmp         1000       0.185       0.268 0.743       1.020
> 0.629       1.034
> long 1%          1000       0.186       0.271 0.744       1.010
> 0.630       1.062
> long 10%         1000       0.186       0.264 0.745       1.223
> 0.630       1.165
> long 10% low     1000       0.186       0.265 0.758       1.084
> 0.630       1.024
> long 10% high    1000       0.186       0.264 0.745       1.018
> 0.628       1.135
> long 50%         1000       0.186       0.263 0.746       1.243
> 0.630       1.149
> huge bmp        10000       0.186       0.390 0.750       5.097
> 0.622       4.394
> huge 10%        10000       0.186       0.394 0.747       8.275
> 0.626       7.291
>
> And this is for CPython 3.4.1 (where we have the full Unicode range):
>
>  >python unicode_index_times.py
> time (us)         len         [i]     [i:i+n]     find(c) find(str)
> rfind(c)  rfind(str)
> single bmp          1       0.703       2.101 0.901       0.707
> 0.932       0.693
> single              1       0.701       2.098 0.891       0.708
> 0.933       0.696
> short bmp          10       0.310       0.613 0.942       0.783
> 0.907       0.777
> short 50%          10       0.337       0.650 1.095       1.115
> 1.043       1.070
> medium bmp        100       0.271       0.423 1.036       0.853
> 0.792       0.833
> medium 10%        100       0.288       0.460 1.404       1.153
> 1.009       1.119
> long bmp         1000       0.300       0.479 1.162       1.396
> 0.789       1.302
> long 1%          1000       0.311       0.575 1.635       1.709
> 1.081       1.725
> long 10%         1000       0.302       0.562 1.610       1.919
> 1.041       1.769
> long 10% low     1000       0.309       0.589 1.597       1.787
> 1.054       1.479
> long 10% high    1000       0.316       0.610 1.605       1.764
> 1.060       1.642
> long 50%         1000       0.373       0.573 1.481       1.819
> 0.925       1.735
> huge bmp        10000       0.301       0.577 1.144       5.596
> 0.800       4.464
> huge 10%        10000       0.315       0.817 1.645       9.053
> 0.985       6.550
>
> Now onwards and upwards, to do some good to PyUnicode itself.
>
> --
> Jeff Allen
>
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Jython-dev mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/jython-dev
>



-- 
Indra Talip

------------------------------------------------------------------------------

_______________________________________________
Jython-dev mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/jython-dev