Re: Input transformation rework - input requested
Jason Grout <[email protected]> Mon, 28 May 2018 08:41:53 -0700
| Newsgroups | gmane.comp.python.ipython.devel |
|---|---|
| Message-ID | <CAPDWZHxcM42yTSXVApkZgATQG0yb6QDZ+eN7AuBoYWAE5cS9CA@mail.gmail.com> |
--===============1402391022551455852== Content-Type: multipart/alternative; boundary="000000000000d1b25b056d45f56d" --000000000000d1b25b056d45f56d Content-Type: text/plain; charset="UTF-8" It would be good to post separately to the Sage list. The code has changed since I was last in it, but I think these are the relevant bits: Register transforms: https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/ipython_extension.py#L504-L513 Preparsing: Defined at https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/interpreter.py#L379, actual implementation at https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/preparse.py Prompts: https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/interpreter.py#L416 There's also another transformation, it seems: https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/interpreter.py#L468 Thanks, Jason On Mon, May 28, 2018 at 5:27 AM Thomas Kluyver <[email protected]> wrote: > Thanks Aaron. If working with valid Python syntax is sufficient, the API > will probably just be functions that take and return a string, leaving it > up to you how you tokenise/parse the code. We only need to do something > more complex if third-party transformations need to integrate with our own > ones to produce valid Python. > > Does anyone know what Sage does? Jason mentioned that they might also use > input transformations. > > On 28 May 2018 at 08:13, Aaron Meurer <[email protected]> wrote: > >> I believe you're right. >> >> I suppose another potential transform would be to allow x^2 to >> represent x**2. This would have be done at the token level since ^ has >> a different precedence than **. >> >> Only operating on valid Python seems fine. If someone wants to >> implement a DSL they should just write a kernel. >> >> Have you considered using something like parso, which can manipulate >> an AST and tokens losslessly? >> >> Aaron Meurer >> >> On Sun, May 27, 2018 at 11:50 PM, Thomas Kluyver <[email protected]> >> wrote: >> > Thanks Aaron. I am aware that Sympy uses input transformation, but it >> would >> > be good to check some more details of exactly how: >> > >> > 1. I believe wrapping integers and defining symbols can both be done >> with >> > AST transformations. Let me know if I'm missing something. >> > 2. Wrapping floats can't, because the string form is discarded before >> we get >> > the AST. But this is still an operation that changes one syntactically >> valid >> > piece of Python code to another, right? I'm considering limiting the >> API to >> > only operate on already valid Python code, and using private APIs to >> handle >> > IPython's special syntax (like %magics). >> > 3. Are there any other transformations that Sympy does (or wants to do) >> > besides these three? >> > >> > On 28 May 2018 at 00:02, Aaron Meurer <[email protected]> wrote: >> >> >> >> I'm sure you're already aware, but we use them for SymPy for some >> >> (optional) transformers that make Python more symbolic friendly >> >> (auto-replace undefined variables with Symbols, wrap integer literals >> >> with Integer so that exact rational numbers work, and so on). The AST >> >> transformer is nice but limiting. We can't use it to wrap float >> >> literals with higher precision for instance because Python drops the >> >> precision of floats in the AST. >> >> >> >> There are already some open issues about some limitations we've seen >> >> (https://github.com/sympy/sympy/issues/14440 and >> >> https://github.com/ipython/ipython/issues/10893). I'm not fully >> >> updated on the specifics of what the limitations were, but from what I >> >> remember, there were issues with IPython doing things line-by-line. >> >> >> >> I don't know how the new code is architected, but it would be nice if >> >> the transformers could just access the raw input from IPython, and >> >> various transformers could be plugged in to that to make things >> >> simpler if desired (the AST transformer could be one such instance). >> >> >> >> At the very least, if you want to test the new design, you could look >> >> at translating our transformers in SymPy (including the work in >> >> progress float transformer at >> >> https://github.com/sympy/sympy/pull/13300). >> >> >> >> Aaron Meurer >> >> >> >> On Sun, May 27, 2018 at 2:50 PM, Brian Granger <[email protected]> >> >> wrote: >> >> > Thomas this is great news! I think that part of the code base could >> >> > benefit >> >> > from being simpler and more extensible. Thanks for tackling this! >> >> > >> >> > Sent from my iPhone >> >> > >> >> > On May 27, 2018, at 1:19 PM, Thomas Kluyver <[email protected]> >> wrote: >> >> > >> >> > Hi all, >> >> > >> >> > For IPython 7, I'm planning a rework of the input transformation >> >> > framework: >> >> > https://github.com/ipython/ipython/pull/11041 >> >> > >> >> > The new framework is - I hope - simpler than the one it replaces, and >> >> > will >> >> > hopefully have fewer weird corner cases. But it's still a pretty >> complex >> >> > beast, and I don't feel like it's a good platform for people to add >> more >> >> > transformations to. One option to improve this is to make a smaller >> >> > extensible API, restricting the kind of things that third party code >> can >> >> > do. >> >> > >> >> > I don't think there's ever been much third party code extending >> >> > IPython's >> >> > input transformation, but I'd like to find out about the code that >> does. >> >> > So >> >> > if you're aware of a project that does use our input transformation >> API: >> >> > >> >> > - What project? >> >> > - What does it want to transform? >> >> > - Can you point me to the code? >> >> > - What fallback options would it have if IPython didn't support the >> >> > transformation it wanted? >> >> > >> >> > This is specifically about text transformations; AST transformations >> are >> >> > not >> >> > changed. >> >> > >> >> > Thanks, >> >> > Thomas >> >> > >> >> > _______________________________________________ >> >> > IPython-dev mailing list >> >> > [email protected] >> >> > https://mail.python.org/mailman/listinfo/ipython-dev >> >> > >> >> > >> >> > _______________________________________________ >> >> > IPython-dev mailing list >> >> > [email protected] >> >> > https://mail.python.org/mailman/listinfo/ipython-dev >> >> > >> >> _______________________________________________ >> >> IPython-dev mailing list >> >> [email protected] >> >> https://mail.python.org/mailman/listinfo/ipython-dev >> > >> > >> > >> > _______________________________________________ >> > IPython-dev mailing list >> > [email protected] >> > https://mail.python.org/mailman/listinfo/ipython-dev >> > >> _______________________________________________ >> IPython-dev mailing list >> [email protected] >> https://mail.python.org/mailman/listinfo/ipython-dev >> > > _______________________________________________ > IPython-dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/ipython-dev > --000000000000d1b25b056d45f56d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>It would be good to post separately to the Sage list.= The code has changed since I was last in it, but I think these are the rel= evant bits:</div><div><br></div><div>Register transforms: <a href=3D"https:= //github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/sr= c/sage/repl/ipython_extension.py#L504-L513">https://github.com/sagemath/sag= e/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/ipython_exten= sion.py#L504-L513</a></div><div><br></div><div>Preparsing: Defined at <a hr= ef=3D"https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d52= 017e044f8/src/sage/repl/interpreter.py#L379">https://github.com/sagemath/sa= ge/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/interpreter.= py#L379</a>, actual implementation at <a href=3D"https://github.com/sagemat= h/sage/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/preparse= .py">https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d520= 17e044f8/src/sage/repl/preparse.py</a></div><div><br></div><div>Prompts: <a= href=3D"https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7= d52017e044f8/src/sage/repl/interpreter.py#L416">https://github.com/sagemath= /sage/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/interpret= er.py#L416</a></div><div><br></div><div>There's also another transforma= tion, it seems: <a href=3D"https://github.com/sagemath/sage/blob/854f9764d1= 4236110b8d7f7b35a7d52017e044f8/src/sage/repl/interpreter.py#L468">https://g= ithub.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/src/s= age/repl/interpreter.py#L468</a></div><div><br></div><div>Thanks,</div><div= ><br></div><div>Jason</div><div><br></div><div><br></div><div><br></div></d= iv><br><div class=3D"gmail_quote"><div dir=3D"ltr">On Mon, May 28, 2018 at = 5:27 AM Thomas Kluyver <<a href=3D"mailto:[email protected]">takowl@gmail= .com</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"mar= gin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr= "><div>Thanks Aaron. If working with valid Python syntax is sufficient, the= API will probably just be functions that take and return a string, leaving= it up to you how you tokenise/parse the code. We only need to do something= more complex if third-party transformations need to integrate with our own= ones to produce valid Python.</div><div><br></div><div>Does anyone know wh= at Sage does? Jason mentioned that they might also use input transformation= s.<br></div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote"= >On 28 May 2018 at 08:13, Aaron Meurer <span dir=3D"ltr"><<a href=3D"mai= lto:[email protected]" target=3D"_blank">[email protected]</a>></span>= wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bor= der-left:1px #ccc solid;padding-left:1ex">I believe you're right.<br> <br> I suppose another potential transform would be to allow x^2 to<br> represent x**2. This would have be done at the token level since ^ has<br> a different precedence than **.<br> <br> Only operating on valid Python seems fine. If someone wants to<br> implement a DSL they should just write a kernel.<br> <br> Have you considered using something like parso, which can manipulate<br> an AST and tokens losslessly?<br> <span class=3D"m_1443657635040616590HOEnZb"><font color=3D"#888888"><br> Aaron Meurer<br> </font></span><div class=3D"m_1443657635040616590HOEnZb"><div class=3D"m_14= 43657635040616590h5"><br> On Sun, May 27, 2018 at 11:50 PM, Thomas Kluyver <<a href=3D"mailto:tako= [email protected]" target=3D"_blank">[email protected]</a>> wrote:<br> > Thanks Aaron. I am aware that Sympy uses input transformation, but it = would<br> > be good to check some more details of exactly how:<br> ><br> > 1. I believe wrapping integers and defining symbols can both be done w= ith<br> > AST transformations. Let me know if I'm missing something.<br> > 2. Wrapping floats can't, because the string form is discarded bef= ore we get<br> > the AST. But this is still an operation that changes one syntactically= valid<br> > piece of Python code to another, right? I'm considering limiting t= he API to<br> > only operate on already valid Python code, and using private APIs to h= andle<br> > IPython's special syntax (like %magics).<br> > 3. Are there any other transformations that Sympy does (or wants to do= )<br> > besides these three?<br> ><br> > On 28 May 2018 at 00:02, Aaron Meurer <<a href=3D"mailto:asmeurer@g= mail.com" target=3D"_blank">[email protected]</a>> wrote:<br> >><br> >> I'm sure you're already aware, but we use them for SymPy f= or some<br> >> (optional) transformers that make Python more symbolic friendly<br= > >> (auto-replace undefined variables with Symbols, wrap integer liter= als<br> >> with Integer so that exact rational numbers work, and so on). The = AST<br> >> transformer is nice but limiting. We can't use it to wrap floa= t<br> >> literals with higher precision for instance because Python drops t= he<br> >> precision of floats in the AST.<br> >><br> >> There are already some open issues about some limitations we'v= e seen<br> >> (<a href=3D"https://github.com/sympy/sympy/issues/14440" rel=3D"no= referrer" target=3D"_blank">https://github.com/sympy/sympy/issues/14440</a>= and<br> >> <a href=3D"https://github.com/ipython/ipython/issues/10893" rel=3D= "noreferrer" target=3D"_blank">https://github.com/ipython/ipython/issues/10= 893</a>). I'm not fully<br> >> updated on the specifics of what the limitations were, but from wh= at I<br> >> remember, there were issues with IPython doing things line-by-line= .<br> >><br> >> I don't know how the new code is architected, but it would be = nice if<br> >> the transformers could just access the raw input from IPython, and= <br> >> various transformers could be plugged in to that to make things<br= > >> simpler if desired (the AST transformer could be one such instance= ).<br> >><br> >> At the very least, if you want to test the new design, you could l= ook<br> >> at translating our transformers in SymPy (including the work in<br= > >> progress float transformer at<br> >> <a href=3D"https://github.com/sympy/sympy/pull/13300" rel=3D"noref= errer" target=3D"_blank">https://github.com/sympy/sympy/pull/13300</a>).<br= > >><br> >> Aaron Meurer<br> >><br> >> On Sun, May 27, 2018 at 2:50 PM, Brian Granger <<a href=3D"mail= to:[email protected]" target=3D"_blank">[email protected]</a>><br> >> wrote:<br> >> > Thomas this is great news! I think that part of the code base= could<br> >> > benefit<br> >> > from being simpler and more extensible. Thanks for tackling t= his!<br> >> ><br> >> > Sent from my iPhone<br> >> ><br> >> > On May 27, 2018, at 1:19 PM, Thomas Kluyver <<a href=3D"ma= ilto:[email protected]" target=3D"_blank">[email protected]</a>> wrote:<br= > >> ><br> >> > Hi all,<br> >> ><br> >> > For IPython 7, I'm planning a rework of the input transfo= rmation<br> >> > framework:<br> >> > <a href=3D"https://github.com/ipython/ipython/pull/11041" rel= =3D"noreferrer" target=3D"_blank">https://github.com/ipython/ipython/pull/1= 1041</a><br> >> ><br> >> > The new framework is - I hope - simpler than the one it repla= ces, and<br> >> > will<br> >> > hopefully have fewer weird corner cases. But it's still a= pretty complex<br> >> > beast, and I don't feel like it's a good platform for= people to add more<br> >> > transformations to. One option to improve this is to make a s= maller<br> >> > extensible API, restricting the kind of things that third par= ty code can<br> >> > do.<br> >> ><br> >> > I don't think there's ever been much third party code= extending<br> >> > IPython's<br> >> > input transformation, but I'd like to find out about the = code that does.<br> >> > So<br> >> > if you're aware of a project that does use our input tran= sformation API:<br> >> ><br> >> > - What project?<br> >> > - What does it want to transform?<br> >> > - Can you point me to the code?<br> >> > - What fallback options would it have if IPython didn't s= upport the<br> >> > transformation it wanted?<br> >> ><br> >> > This is specifically about text transformations; AST transfor= mations are<br> >> > not<br> >> > changed.<br> >> ><br> >> > Thanks,<br> >> > Thomas<br> >> ><br> >> > _______________________________________________<br> >> > IPython-dev mailing list<br> >> > <a href=3D"mailto:[email protected]" target=3D"_blank">I= [email protected]</a><br> >> > <a href=3D"https://mail.python.org/mailman/listinfo/ipython-d= ev" rel=3D"noreferrer" target=3D"_blank">https://mail.python.org/mailman/li= stinfo/ipython-dev</a><br> >> ><br> >> ><br> >> > _______________________________________________<br> >> > IPython-dev mailing list<br> >> > <a href=3D"mailto:[email protected]" target=3D"_blank">I= [email protected]</a><br> >> > <a href=3D"https://mail.python.org/mailman/listinfo/ipython-d= ev" rel=3D"noreferrer" target=3D"_blank">https://mail.python.org/mailman/li= stinfo/ipython-dev</a><br> >> ><br> >> _______________________________________________<br> >> IPython-dev mailing list<br> >> <a href=3D"mailto:[email protected]" target=3D"_blank">IPytho= [email protected]</a><br> >> <a href=3D"https://mail.python.org/mailman/listinfo/ipython-dev" r= el=3D"noreferrer" target=3D"_blank">https://mail.python.org/mailman/listinf= o/ipython-dev</a><br> ><br> ><br> ><br> > _______________________________________________<br> > IPython-dev mailing list<br> > <a href=3D"mailto:[email protected]" target=3D"_blank">IPython-de= [email protected]</a><br> > <a href=3D"https://mail.python.org/mailman/listinfo/ipython-dev" rel= =3D"noreferrer" target=3D"_blank">https://mail.python.org/mailman/listinfo/= ipython-dev</a><br> ><br> _______________________________________________<br> IPython-dev mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">IPython-dev@pyt= hon.org</a><br> <a href=3D"https://mail.python.org/mailman/listinfo/ipython-dev" rel=3D"nor= eferrer" target=3D"_blank">https://mail.python.org/mailman/listinfo/ipython= -dev</a><br> </div></div></blockquote></div><br></div> _______________________________________________<br> IPython-dev mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">IPython-dev@pyt= hon.org</a><br> <a href=3D"https://mail.python.org/mailman/listinfo/ipython-dev" rel=3D"nor= eferrer" target=3D"_blank">https://mail.python.org/mailman/listinfo/ipython= -dev</a><br> </blockquote></div> --000000000000d1b25b056d45f56d-- --===============1402391022551455852== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPython-dev mailing list [email protected] https://mail.python.org/mailman/listinfo/ipython-dev --===============1402391022551455852==--