Re: PEAK-Rules and PyPy

--14dae93411f5f55dc404cc351b72
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

On Tue, Oct 16, 2012 at 12:48 PM, Alain Poirier
<alain.poirier-HPmRmcG/[email protected]> wrote:
> Le 16 oct. 2012 =E0 18:26, Marcin Tustin <[email protected]> a =E9c=
rit :
>
>> Please see: http://docs.python.org/library/__builtin__.html?highlight=3D=
__builtins__
>
> Thanks Marcin. I see from the 'CPython implementation detail' note, '__bu=
iltins__'
> can be a module or a dict. Ok, so I added a check to use '__builtins__.__=
dict__' if
> '__builtins__' is a module and now I've got:
>
>>>>> from peak.rules import predicates
> Traceback (most recent call last):
>   File "<console>", line 1, in <module>
>   File "/private/tmp/p/site-packages/DecoratorTools-1.8-py2.7.egg/peak/ut=
il/decorators.py", line 617, in tracer
>     frm.f_locals[k] =3D callback(frm,k,v,old_locals)
>   File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/core.py", line=
 276, in callback
>     register_for_class(None)
>   File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/core.py", line=
 270, in register_for_class
>     _register_rule(f, pred, context, cls)
>   File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/core.py", line=
 395, in _register_rule
>     rules.add(parse_rule(Dispatching(gf).engine, pred, context, cls))
>   File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/core.py", line=
 685, in parse_rule
>     def parse_rule(engine, predicate, context, cls):
>   File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/core.py", line=
 953, in parse_upgrade
>     predicate, context, cls
>   File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/core.py", line=
 685, in parse_rule
>     def parse_rule(engine, predicate, context, cls):
>   File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/predicates.py"=
, line 606, in _parse_string
>     maybe_bind(ctx.body, bindings), expr, ctx.actiontype, ctx.sequence
> UnboundLocalError: local variable 'expr' referenced before assignment
>
> which is strange because 'expr' is just defined some lines above.
>
>> CPython implementation detail: Most modules have the name __builtins__ (=
note the 's') made available as part of their globals. The value of__builti=
ns__ is normally either this module or the value of this modules=92s __dict=
__ attribute. Since this is an implementation detail, it may not be used by=
 alternate implementations of Python.
>>
>> On Tue, Oct 16, 2012 at 12:22 PM, PJ Eby <pje-Wh6+Hckhi6HFNGf7iClzIwC/[email protected]> wrote:
>> On Tue, Oct 16, 2012 at 11:18 AM, Alain Poirier
>> <alain.poirier-HPmRmcG/[email protected]> wrote:
>> >   File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/predicates.=
py", line 595, in _parse_string
>> >     b =3D CriteriaBuilder(engine.arguments, ctx.localdict, ctx.globald=
ict, __builtins__)
>> >   File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/codegen.py"=
, line 334, in __init__
>> >     dict([(k,self.Const(v)) for k,v in ns.iteritems()]) for ns in name=
spaces
>> > AttributeError: 'module' object has no attribute 'iteritems'
>> >
>> >
>> > Do you think it could be possible to have a working PyPy version of Pe=
ak Rules?
>>
>> It depends. It looks like the problem above is that __builtins__ in
>> PyPy might be a module instead of a dictionary?  If so, that's
>> probably a bug in PyPy that needs to be fixed.
>>
>> There are likely to be other problems besides this one, but let's take
>> them one at a time.  ;-)


There are bigger fish to fry - the AddOns package (which is a
dependency) fails all its tests because PyPy doesn't support
non-string keys in type dictionaries.  DecoratorTools' test suite also
fails, but AFAICT it's all due to changes in repr() of various
built-in types (unless PyPy doesn't support classic classes), and
should be shallow.  AddOns, however, is quite heavily used by
PEAK-Rules.

An important BytecodeAssembler test also fails: PyPy has a
slightly-incompatible bytecode interpreter, and PEAK-Rules abuses a
weak link in CPython's bytecode to implement a "computed goto"
operation in generated bytecode.  My guess is that the UnboundLocal
errors are being caused by how PyPy handles (or more precisely,
doesn't handle) the computed goto.

The specific issue is this: in CPython, the END_FINALLY bytecode takes
a "why" value on top of the stack, plus an extra value as a jump
offset.  PyPy, however, wants one value on the stack: a special
interpreter-owned value (that's AFAICT can't be created directly in
Python) that wraps the reason and the jump offset together.

Unless there's some way to create these special SContinueLoop objects
from Python (and I'm guessing there's not), I'd have to write a
replacement code generator for PyPy that uses a linear search instead
-- which ironically may make PyPy's predicate dispatch slower than
CPython's in some cases, unless the JIT can optimize it out enough.

[pause for some hacking]

After hacking around a bit, I have a quick and dirty patch (attached)
that makes most of the tests pass, by fixing up the __builtins__, and
handling the END_FINALLY issue by using a linear search in the
innermost dispatch loops.  It isn't suitable for release at the
moment, because it doesn't check for whether it's running under PyPy,
and there are still some tests that fail due to shallow minor issues
like the PyPy repr() differences and hash iteration differences.  Let
me know if it works for you, and maybe I'll clean it up for release.

Technically, AddOns and BytecodeAssembler need some fixes too, but the
parts of them that don't work the same under PyPy don't actually get
used by PEAK-Rules, at least not in the test suite.

--14dae93411f5f55dc404cc351b72
Content-Type: application/octet-stream; name="quick.patch"
Content-Disposition: attachment; filename="quick.patch"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_h8dm15580

SW5kZXg6IHBlYWsvcnVsZXMvY29kZWdlbi5weQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBwZWFrL3J1bGVzL2Nv
ZGVnZW4ucHkJKHJldmlzaW9uIDI3MDQpCisrKyBwZWFrL3J1bGVzL2NvZGVnZW4ucHkJKHdvcmtp
bmcgY29weSkKQEAgLTIsNiArMiw3IEBACiBmcm9tIHBlYWsudXRpbC5zeW1ib2xzIGltcG9ydCBT
eW1ib2wKIGZyb20gcGVhay5ydWxlcy5jb3JlIGltcG9ydCBnZW5fYXJnLCBjbG9uZV9mdW5jdGlv
bgogZnJvbSBhc3RfYnVpbGRlciBpbXBvcnQgYnVpbGQsIHBhcnNlX2V4cHIKK2Zyb20gdHlwZXMg
aW1wb3J0IE1vZHVsZVR5cGUKIGltcG9ydCBzeXMKIHRyeToKICAgICBzZXQKQEAgLTMxLDE0ICsz
MiwxMyBAQAogICAgIHJldHVybiBjb2RlLlNMSUNFXzAoKQogCiAKK2RlZiBtb2R1bGVfdG9fbnMo
bnMpOgorICAgIGlmIGlzaW5zdGFuY2UobnMsIE1vZHVsZVR5cGUpOgorICAgICAgICByZXR1cm4g
bnMuX19kaWN0X18KKyAgICByZXR1cm4gbnMKIAogCiAKLQotCi0KLQotCiBub2RldHlwZShmbXQ9
IiVzOiVzOiVzIikKIGRlZiBCdWlsZFNsaWNlKHN0YXJ0PVBhc3MsIHN0b3A9UGFzcywgc3RyaWRl
PVBhc3MsIGNvZGU9Tm9uZSk6CiAgICAgaWYgY29kZSBpcyBOb25lOgpAQCAtMTc0LDcgKzE3NCw3
IEBACiAgICAgICAgIHNlbGYuY29kZSA9IGNvZGUgPSBDU0VDb2RlLmZyb21fZnVuY3Rpb24oZnVu
YykgIywgY29weV9saW5lbm89VHJ1ZSkKICAgICAgICAgc2VsZi5hY3Rpb25zID0ge30KICAgICAg
ICAgc2VsZi5mdW5jID0gZnVuYwotICAgICAgICBsb29wX3RvcCwgZXhpdCwgYmFkX2FjdGlvbiwg
ZmFrZSA9IExhYmVsKCksIExhYmVsKCksIExhYmVsKCksIExhYmVsKCkKKyAgICAgICAgZmlyc3Rf
YWN0aW9uLCBsb29wX3RvcCwgZXhpdCwgYmFkX2FjdGlvbiwgZmFrZSA9IExhYmVsKCksIExhYmVs
KCksIExhYmVsKCksIExhYmVsKCksIExhYmVsKCkKICAgICAgICAgYXJncywgc3RhciwgZHN0YXIs
IGRlZmF1bHRzID0gaW5zcGVjdC5nZXRhcmdzcGVjKGZ1bmMpCiAgICAgICAgIGFjdGlvbnMsIHNl
bGYuYWN0aW9uc19jb25zdCA9IHNlbGYubWFrZV9jb25zdCh7fSkKICAgICAgICAgc3RhcnRfbm9k
ZSwgc2VsZi5zdGFydG5vZGVfY29uc3QgPSBzZWxmLm1ha2VfY29uc3Qob2JqZWN0KCkpCkBAIC0x
ODgsOCArMTg4LDggQEAKICAgICAgICAgICAgIGJhZF9hY3Rpb24uSlVNUF9JRl9GQUxTRV9PUl9Q
T1AsCiAgICAgICAgICAgICBDb2RlLlJPVF9UV08sICAgIyBhcmd1bWVudCwgYWN0aW9uCiAgICAg
ICAgICAgICBzZWxmLlNFVF9BUkcsICAgIyBhY3Rpb24KLSAgICAgICAgICAgIGZha2UuU0VUVVBf
TE9PUCwgc2VsZi5XSFlfQ09OVElOVUUsIENvZGUuRU5EX0ZJTkFMTFksCi0gICAgICAgICAgICBD
b2RlLlBPUF9CTE9DSywgZmFrZSwgUmV0dXJuKFBhc3MpLCAgIyA8LSBhbGwgZGVhZCBjb2RlLCBu
ZXZlciBydW5zCisgICAgICAgICAgICBmaXJzdF9hY3Rpb24uSlVNUF9GT1JXQVJELCAjZmFrZS5T
RVRVUF9MT09QLCBzZWxmLldIWV9DT05USU5VRSwgQ29kZS5FTkRfRklOQUxMWSwKKyAgICAgICAg
ICAgICNDb2RlLlBPUF9CTE9DSywgZmFrZSwgUmV0dXJuKFBhc3MpLCAgIyA8LSBhbGwgZGVhZCBj
b2RlLCBuZXZlciBydW5zCiAgICAgICAgIGV4aXQsCiAgICAgICAgICAgICBDb2RlLlBPUF9UT1As
ICAgICAgICMgZHJvcCBhY3Rpb24sIGxlYXZpbmcgYXJndW1lbnQKICAgICAgICAgICAgIFJldHVy
bigKQEAgLTE5Nyw3ICsxOTcsOCBAQAogICAgICAgICAgICAgKSwKICAgICAgICAgYmFkX2FjdGlv
biwKICAgICAgICAgICAgIENvZGUuUE9QX1RPUCwKLSAgICAgICAgICAgIFJldHVybihDYWxsKENv
bnN0KHNlbGYuYmFkX2FjdGlvbiksKENvZGUuUk9UX1RIUkVFLCBDb2RlLlJPVF9UV08pKSkKKyAg
ICAgICAgICAgIFJldHVybihDYWxsKENvbnN0KHNlbGYuYmFkX2FjdGlvbiksKENvZGUuUk9UX1RI
UkVFLCBDb2RlLlJPVF9UV08pKSksCisgICAgICAgIGZpcnN0X2FjdGlvbiwKICAgICAgICAgKQog
ICAgICAgICBzZWxmLk5FWFRfU1RBVEUgPSBsb29wX3RvcC5KVU1QX0FCU09MVVRFCiAgICAgICAg
IHNlbGYubWF5YmVfY2FjaGUgPSBjb2RlLm1heWJlX2NhY2hlCkBAIC0yMjEsMTAgKzIyMiwxNyBA
QAogICAgICAgICAgICAgcmV0dXJuIHNlbGYuYWN0aW9uc1tleHByZXNzaW9uXQogICAgICAgICBl
eGNlcHQgS2V5RXJyb3I6CiAgICAgICAgICAgICBhY3Rpb24gPSBzZWxmLmFjdGlvbnNbZXhwcmVz
c2lvbl0gPSBzZWxmLmNvZGUuaGVyZSgpCi0gICAgICAgICAgICBzZWxmLmNvZGUuc3RhY2tfc2l6
ZSA9IDAKLSAgICAgICAgICAgIHNlbGYuY29kZShleHByZXNzaW9uKQorICAgICAgICAgICAgc2Vs
Zi5jb2RlLnN0YWNrX3NpemUgPSAxCisgICAgICAgICAgICBuZXh0X2FjdGlvbiA9IExhYmVsKCkK
KyAgICAgICAgICAgIHNlbGYuY29kZSgKKyAgICAgICAgICAgICAgICBDb21wYXJlKENvZGUuRFVQ
X1RPUCwgKCgnPT0nLCBhY3Rpb24pLCkpLAorICAgICAgICAgICAgICAgIG5leHRfYWN0aW9uLkpV
TVBfSUZfRkFMU0VfT1JfUE9QLAorICAgICAgICAgICAgICAgIENvZGUuUE9QX1RPUCwgZXhwcmVz
c2lvbiwKKyAgICAgICAgICAgICkKKyAgICAgICAgICAgICNzZWxmLmNvZGUoZXhwcmVzc2lvbikK
ICAgICAgICAgICAgIGlmIHNlbGYuY29kZS5zdGFja19zaXplIGlzIG5vdCBOb25lOgogICAgICAg
ICAgICAgICAgIHNlbGYuY29kZShzZWxmLk5FWFRfU1RBVEUpCisgICAgICAgICAgICBzZWxmLmNv
ZGUobmV4dF9hY3Rpb24sIENvZGUuUE9QX1RPUCkKICAgICAgICAgICAgIHJldHVybiBhY3Rpb24K
IAogICAgIGRlZiBiYWRfYWN0aW9uKHNlbGYsIGFjdGlvbiwgYXJndW1lbnQpOgpAQCAtMjM4LDEy
ICsyNDYsNiBAQAogCiAKIAotCi0KLQotCi0KLQogQ0FDSEUgPSBMb2NhbCgnJENTRUNhY2hlJykK
IFNFVF9DQUNIRSA9IGxhbWJkYSBjb2RlOiBjb2RlLlNUT1JFX0ZBU1QoQ0FDSEUubmFtZSkKIApA
QCAtMzI0LDE0ICszMjYsMTIgQEAKICAgICAgICAgICAgIGVsc2U6CiAgICAgICAgICAgICAgICAg
c2NhbGwob2IpCiAKLQotCiBjbGFzcyBFeHByQnVpbGRlcjoKICAgICAiIiJFeHByZXNzaW9uIGJ1
aWxkZXIgcmV0dXJuaW5nIGJ5dGVjb2RlLWFibGUgQVNUIG5vZGVzIiIiCiAKICAgICBkZWYgX19p
bml0X18oc2VsZixhcmd1bWVudHMsKm5hbWVzcGFjZXMpOgogICAgICAgICBzZWxmLmJpbmRpbmdz
ID0gWwotICAgICAgICAgICAgZGljdChbKGssc2VsZi5Db25zdCh2KSkgZm9yIGssdiBpbiBucy5p
dGVyaXRlbXMoKV0pIGZvciBucyBpbiBuYW1lc3BhY2VzCisgICAgICAgICAgICBkaWN0KFsoayxz
ZWxmLkNvbnN0KHYpKSBmb3Igayx2IGluIG1vZHVsZV90b19ucyhucykuaXRlcml0ZW1zKCldKSBm
b3IgbnMgaW4gbmFtZXNwYWNlcwogICAgICAgICBdCiAgICAgICAgIHNlbGYucHVzaChhcmd1bWVu
dHMpOyBzZWxmLnB1c2goKQogCg==
--14dae93411f5f55dc404cc351b72
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
PEAK mailing list
[email protected]
http://www.eby-sarna.com/mailman/listinfo/peak
--14dae93411f5f55dc404cc351b72--