Re: Input transformation rework - input requested
Thomas Kluyver <[email protected]> Tue, 29 May 2018 15:19:13 +0200
| Newsgroups | gmane.comp.python.ipython.devel |
|---|---|
| Message-ID | <CAOvn4qinoQ9LxYPf9aof3nOg3RxXrO7c2iBwktL1wdvX88c4Yw@mail.gmail.com> |
--===============6016329491209657203== Content-Type: multipart/alternative; boundary="0000000000006d19dd056d5816ed" --0000000000006d19dd056d5816ed Content-Type: text/plain; charset="UTF-8" Looking at those links, Sage definitely does not restrict itself to valid Python syntax. However, if there's only one downstream doing that, I think it's reasonable for us to simplify the API and let Sage handle its own preparsing. Does anyone know of any other projects which use input transformations? More specifically, I propose that we offer three transformation APIs: 1. Before handling our special syntax, to strip formatting like prompts (accepts a cell as a string, returns transformed string) 2. After handling our special syntax, when the result should be valid Python syntax, for things like wrapping floats (string -> string) 3. AST transformations (AST -> AST) If projects like Sage want to, they could still transform invalid tokens in hook 2, but their special syntax may interact unpredictably with our own. Alternatively, they could copy and modify our transformation machinery to handle their own special syntax. Either way, it avoids us having to maintain a complex transformation interface which we may need to break again. Can I ask someone who's got a foot in the Sage camp to ping them about this on their own lists? Thomas On 28 May 2018 at 17:41, Jason Grout <[email protected]> wrote: > It would be good to post separately to the Sage list. The code has changed > since I was last in it, but I think these are the relevant bits: > > Register transforms: https://github.com/sagemath/sage/blob/ > 854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/ > ipython_extension.py#L504-L513 > > Preparsing: Defined at https://github.com/sagemath/sage/blob/ > 854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/interpreter.py#L379, > actual implementation at https://github.com/sagemath/sage/blob/ > 854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/preparse.py > > Prompts: https://github.com/sagemath/sage/blob/ > 854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/interpreter.py#L416 > > There's also another transformation, it seems: > https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d5 > 2017e044f8/src/sage/repl/interpreter.py#L468 > > Thanks, > > Jason > > > > > On Mon, May 28, 2018 at 5:27 AM Thomas Kluyver <[email protected]> wrote: > >> Thanks Aaron. If working with valid Python syntax is sufficient, the API >> will probably just be functions that take and return a string, leaving it >> up to you how you tokenise/parse the code. We only need to do something >> more complex if third-party transformations need to integrate with our own >> ones to produce valid Python. >> >> Does anyone know what Sage does? Jason mentioned that they might also use >> input transformations. >> >> On 28 May 2018 at 08:13, Aaron Meurer <[email protected]> wrote: >> >>> I believe you're right. >>> >>> I suppose another potential transform would be to allow x^2 to >>> represent x**2. This would have be done at the token level since ^ has >>> a different precedence than **. >>> >>> Only operating on valid Python seems fine. If someone wants to >>> implement a DSL they should just write a kernel. >>> >>> Have you considered using something like parso, which can manipulate >>> an AST and tokens losslessly? >>> >>> Aaron Meurer >>> >>> On Sun, May 27, 2018 at 11:50 PM, Thomas Kluyver <[email protected]> >>> wrote: >>> > Thanks Aaron. I am aware that Sympy uses input transformation, but it >>> would >>> > be good to check some more details of exactly how: >>> > >>> > 1. I believe wrapping integers and defining symbols can both be done >>> with >>> > AST transformations. Let me know if I'm missing something. >>> > 2. Wrapping floats can't, because the string form is discarded before >>> we get >>> > the AST. But this is still an operation that changes one syntactically >>> valid >>> > piece of Python code to another, right? I'm considering limiting the >>> API to >>> > only operate on already valid Python code, and using private APIs to >>> handle >>> > IPython's special syntax (like %magics). >>> > 3. Are there any other transformations that Sympy does (or wants to do) >>> > besides these three? >>> > >>> > On 28 May 2018 at 00:02, Aaron Meurer <[email protected]> wrote: >>> >> >>> >> I'm sure you're already aware, but we use them for SymPy for some >>> >> (optional) transformers that make Python more symbolic friendly >>> >> (auto-replace undefined variables with Symbols, wrap integer literals >>> >> with Integer so that exact rational numbers work, and so on). The AST >>> >> transformer is nice but limiting. We can't use it to wrap float >>> >> literals with higher precision for instance because Python drops the >>> >> precision of floats in the AST. >>> >> >>> >> There are already some open issues about some limitations we've seen >>> >> (https://github.com/sympy/sympy/issues/14440 and >>> >> https://github.com/ipython/ipython/issues/10893). I'm not fully >>> >> updated on the specifics of what the limitations were, but from what I >>> >> remember, there were issues with IPython doing things line-by-line. >>> >> >>> >> I don't know how the new code is architected, but it would be nice if >>> >> the transformers could just access the raw input from IPython, and >>> >> various transformers could be plugged in to that to make things >>> >> simpler if desired (the AST transformer could be one such instance). >>> >> >>> >> At the very least, if you want to test the new design, you could look >>> >> at translating our transformers in SymPy (including the work in >>> >> progress float transformer at >>> >> https://github.com/sympy/sympy/pull/13300). >>> >> >>> >> Aaron Meurer >>> >> >>> >> On Sun, May 27, 2018 at 2:50 PM, Brian Granger <[email protected]> >>> >> wrote: >>> >> > Thomas this is great news! I think that part of the code base could >>> >> > benefit >>> >> > from being simpler and more extensible. Thanks for tackling this! >>> >> > >>> >> > Sent from my iPhone >>> >> > >>> >> > On May 27, 2018, at 1:19 PM, Thomas Kluyver <[email protected]> >>> wrote: >>> >> > >>> >> > Hi all, >>> >> > >>> >> > For IPython 7, I'm planning a rework of the input transformation >>> >> > framework: >>> >> > https://github.com/ipython/ipython/pull/11041 >>> >> > >>> >> > The new framework is - I hope - simpler than the one it replaces, >>> and >>> >> > will >>> >> > hopefully have fewer weird corner cases. But it's still a pretty >>> complex >>> >> > beast, and I don't feel like it's a good platform for people to add >>> more >>> >> > transformations to. One option to improve this is to make a smaller >>> >> > extensible API, restricting the kind of things that third party >>> code can >>> >> > do. >>> >> > >>> >> > I don't think there's ever been much third party code extending >>> >> > IPython's >>> >> > input transformation, but I'd like to find out about the code that >>> does. >>> >> > So >>> >> > if you're aware of a project that does use our input transformation >>> API: >>> >> > >>> >> > - What project? >>> >> > - What does it want to transform? >>> >> > - Can you point me to the code? >>> >> > - What fallback options would it have if IPython didn't support the >>> >> > transformation it wanted? >>> >> > >>> >> > This is specifically about text transformations; AST >>> transformations are >>> >> > not >>> >> > changed. >>> >> > >>> >> > Thanks, >>> >> > Thomas >>> >> > >>> >> > _______________________________________________ >>> >> > IPython-dev mailing list >>> >> > [email protected] >>> >> > https://mail.python.org/mailman/listinfo/ipython-dev >>> >> > >>> >> > >>> >> > _______________________________________________ >>> >> > IPython-dev mailing list >>> >> > [email protected] >>> >> > https://mail.python.org/mailman/listinfo/ipython-dev >>> >> > >>> >> _______________________________________________ >>> >> IPython-dev mailing list >>> >> [email protected] >>> >> https://mail.python.org/mailman/listinfo/ipython-dev >>> > >>> > >>> > >>> > _______________________________________________ >>> > IPython-dev mailing list >>> > [email protected] >>> > https://mail.python.org/mailman/listinfo/ipython-dev >>> > >>> _______________________________________________ >>> IPython-dev mailing list >>> [email protected] >>> https://mail.python.org/mailman/listinfo/ipython-dev >>> >> >> _______________________________________________ >> IPython-dev mailing list >> [email protected] >> https://mail.python.org/mailman/listinfo/ipython-dev >> > > _______________________________________________ > IPython-dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/ipython-dev > > --0000000000006d19dd056d5816ed Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Looking at those links, Sage definitely does not rest= rict itself to valid Python syntax. However, if there's only one downst= ream doing that, I think it's reasonable for us to simplify the API and= let Sage handle its own preparsing. Does anyone know of any other projects= which use input transformations?</div><div><br></div><div>More specificall= y, I propose that we offer three transformation APIs:</div><div><br></div><= div>1. Before handling our special syntax, to strip formatting like prompts= (accepts a cell as a string, returns transformed string)</div><div>2. Afte= r handling our special syntax, when the result should be valid Python synta= x, for things like wrapping floats (string -> string)</div><div>3. AST t= ransformations (AST -> AST)<br></div><div><br></div><div>If projects lik= e Sage want to, they could still transform invalid tokens in hook 2, but th= eir special syntax may interact unpredictably with our own. Alternatively, = they could copy and modify our transformation machinery to handle their own= special syntax. Either way, it avoids us having to maintain a complex tran= sformation interface which we may need to break again.</div><div><br></div>= <div>Can I ask someone who's got a foot in the Sage camp to ping them a= bout this on their own lists?<br></div><div><br></div><div>Thomas<br></div>= </div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On 28 May 2= 018 at 17:41, Jason Grout <span dir=3D"ltr"><<a href=3D"mailto:jason@jas= ongrout.org" target=3D"_blank">[email protected]</a>></span> wrote:<b= r><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:= 1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div>It would be good to = post separately to the Sage list. The code has changed since I was last in = it, but I think these are the relevant bits:</div><div><br></div><div>Regis= ter transforms: <a href=3D"https://github.com/sagemath/sage/blob/854f9764d1= 4236110b8d7f7b35a7d52017e044f8/src/sage/repl/ipython_extension.py#L504-L513= " target=3D"_blank">https://github.com/sagemath/<wbr>sage/blob/<wbr>854f976= 4d14236110b8d7f7b35a7d5<wbr>2017e044f8/src/sage/repl/<wbr>ipython_extension= .py#L504-L513</a></div><div><br></div><div>Preparsing: Defined at <a href= =3D"https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d5201= 7e044f8/src/sage/repl/interpreter.py#L379" target=3D"_blank">https://github= .com/sagemath/<wbr>sage/blob/<wbr>854f9764d14236110b8d7f7b35a7d5<wbr>2017e0= 44f8/src/sage/repl/<wbr>interpreter.py#L379</a>, actual implementation at <= a href=3D"https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a= 7d52017e044f8/src/sage/repl/preparse.py" target=3D"_blank">https://github.c= om/sagemath/<wbr>sage/blob/<wbr>854f9764d14236110b8d7f7b35a7d5<wbr>2017e044= f8/src/sage/repl/<wbr>preparse.py</a></div><div><br></div><div>Prompts: <a = href=3D"https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d= 52017e044f8/src/sage/repl/interpreter.py#L416" target=3D"_blank">https://gi= thub.com/sagemath/<wbr>sage/blob/<wbr>854f9764d14236110b8d7f7b35a7d5<wbr>20= 17e044f8/src/sage/repl/<wbr>interpreter.py#L416</a></div><div><br></div><di= v>There's also another transformation, it seems: <a href=3D"https://git= hub.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/src/sag= e/repl/interpreter.py#L468" target=3D"_blank">https://github.com/sagemath/<= wbr>sage/blob/<wbr>854f9764d14236110b8d7f7b35a7d5<wbr>2017e044f8/src/sage/r= epl/<wbr>interpreter.py#L468</a></div><div><br></div><div>Thanks,</div><div= ><br></div><div>Jason</div><div><br></div><div><br></div><div><br></div></d= iv><div class=3D"HOEnZb"><div class=3D"h5"><br><div class=3D"gmail_quote"><= div dir=3D"ltr">On Mon, May 28, 2018 at 5:27 AM Thomas Kluyver <<a href= =3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>> wro= te:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;b= order-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div>Thanks Aa= ron. If working with valid Python syntax is sufficient, the API will probab= ly just be functions that take and return a string, leaving it up to you ho= w you tokenise/parse the code. We only need to do something more complex if= third-party transformations need to integrate with our own ones to produce= valid Python.</div><div><br></div><div>Does anyone know what Sage does? Ja= son mentioned that they might also use input transformations.<br></div></di= v><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On 28 May 2018 = at 08:13, Aaron Meurer <span dir=3D"ltr"><<a href=3D"mailto:asmeurer@gma= il.com" target=3D"_blank">[email protected]</a>></span> wrote:<br><bloc= kquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #cc= c solid;padding-left:1ex">I believe you're right.<br> <br> I suppose another potential transform would be to allow x^2 to<br> represent x**2. This would have be done at the token level since ^ has<br> a different precedence than **.<br> <br> Only operating on valid Python seems fine. If someone wants to<br> implement a DSL they should just write a kernel.<br> <br> Have you considered using something like parso, which can manipulate<br> an AST and tokens losslessly?<br> <span class=3D"m_3182807902186703932m_1443657635040616590HOEnZb"><font colo= r=3D"#888888"><br> Aaron Meurer<br> </font></span><div class=3D"m_3182807902186703932m_1443657635040616590HOEnZ= b"><div class=3D"m_3182807902186703932m_1443657635040616590h5"><br> On Sun, May 27, 2018 at 11:50 PM, Thomas Kluyver <<a href=3D"mailto:tako= [email protected]" target=3D"_blank">[email protected]</a>> wrote:<br> > Thanks Aaron. I am aware that Sympy uses input transformation, but it = would<br> > be good to check some more details of exactly how:<br> ><br> > 1. I believe wrapping integers and defining symbols can both be done w= ith<br> > AST transformations. Let me know if I'm missing something.<br> > 2. Wrapping floats can't, because the string form is discarded bef= ore we get<br> > the AST. But this is still an operation that changes one syntactically= valid<br> > piece of Python code to another, right? I'm considering limiting t= he API to<br> > only operate on already valid Python code, and using private APIs to h= andle<br> > IPython's special syntax (like %magics).<br> > 3. Are there any other transformations that Sympy does (or wants to do= )<br> > besides these three?<br> ><br> > On 28 May 2018 at 00:02, Aaron Meurer <<a href=3D"mailto:asmeurer@g= mail.com" target=3D"_blank">[email protected]</a>> wrote:<br> >><br> >> I'm sure you're already aware, but we use them for SymPy f= or some<br> >> (optional) transformers that make Python more symbolic friendly<br= > >> (auto-replace undefined variables with Symbols, wrap integer liter= als<br> >> with Integer so that exact rational numbers work, and so on). The = AST<br> >> transformer is nice but limiting. We can't use it to wrap floa= t<br> >> literals with higher precision for instance because Python drops t= he<br> >> precision of floats in the AST.<br> >><br> >> There are already some open issues about some limitations we'v= e seen<br> >> (<a href=3D"https://github.com/sympy/sympy/issues/14440" rel=3D"no= referrer" target=3D"_blank">https://github.com/sympy/<wbr>sympy/issues/1444= 0</a> and<br> >> <a href=3D"https://github.com/ipython/ipython/issues/10893" rel=3D= "noreferrer" target=3D"_blank">https://github.com/ipython/<wbr>ipython/issu= es/10893</a>). I'm not fully<br> >> updated on the specifics of what the limitations were, but from wh= at I<br> >> remember, there were issues with IPython doing things line-by-line= .<br> >><br> >> I don't know how the new code is architected, but it would be = nice if<br> >> the transformers could just access the raw input from IPython, and= <br> >> various transformers could be plugged in to that to make things<br= > >> simpler if desired (the AST transformer could be one such instance= ).<br> >><br> >> At the very least, if you want to test the new design, you could l= ook<br> >> at translating our transformers in SymPy (including the work in<br= > >> progress float transformer at<br> >> <a href=3D"https://github.com/sympy/sympy/pull/13300" rel=3D"noref= errer" target=3D"_blank">https://github.com/sympy/<wbr>sympy/pull/13300</a>= ).<br> >><br> >> Aaron Meurer<br> >><br> >> On Sun, May 27, 2018 at 2:50 PM, Brian Granger <<a href=3D"mail= to:[email protected]" target=3D"_blank">[email protected]</a>><br> >> wrote:<br> >> > Thomas this is great news! I think that part of the code base= could<br> >> > benefit<br> >> > from being simpler and more extensible. Thanks for tackling t= his!<br> >> ><br> >> > Sent from my iPhone<br> >> ><br> >> > On May 27, 2018, at 1:19 PM, Thomas Kluyver <<a href=3D"ma= ilto:[email protected]" target=3D"_blank">[email protected]</a>> wrote:<br= > >> ><br> >> > Hi all,<br> >> ><br> >> > For IPython 7, I'm planning a rework of the input transfo= rmation<br> >> > framework:<br> >> > <a href=3D"https://github.com/ipython/ipython/pull/11041" rel= =3D"noreferrer" target=3D"_blank">https://github.com/ipython/<wbr>ipython/p= ull/11041</a><br> >> ><br> >> > The new framework is - I hope - simpler than the one it repla= ces, and<br> >> > will<br> >> > hopefully have fewer weird corner cases. But it's still a= pretty complex<br> >> > beast, and I don't feel like it's a good platform for= people to add more<br> >> > transformations to. One option to improve this is to make a s= maller<br> >> > extensible API, restricting the kind of things that third par= ty code can<br> >> > do.<br> >> ><br> >> > I don't think there's ever been much third party code= extending<br> >> > IPython's<br> >> > input transformation, but I'd like to find out about the = code that does.<br> >> > So<br> >> > if you're aware of a project that does use our input tran= sformation API:<br> >> ><br> >> > - What project?<br> >> > - What does it want to transform?<br> >> > - Can you point me to the code?<br> >> > - What fallback options would it have if IPython didn't s= upport the<br> >> > transformation it wanted?<br> >> ><br> >> > This is specifically about text transformations; AST transfor= mations are<br> >> > not<br> >> > changed.<br> >> ><br> >> > Thanks,<br> >> > Thomas<br> >> ><br> >> > ______________________________<wbr>_________________<br> >> > IPython-dev mailing list<br> >> > <a href=3D"mailto:[email protected]" target=3D"_blank">I= [email protected]</a><br> >> > <a href=3D"https://mail.python.org/mailman/listinfo/ipython-d= ev" rel=3D"noreferrer" target=3D"_blank">https://mail.python.org/<wbr>mailm= an/listinfo/ipython-dev</a><br> >> ><br> >> ><br> >> > ______________________________<wbr>_________________<br> >> > IPython-dev mailing list<br> >> > <a href=3D"mailto:[email protected]" target=3D"_blank">I= [email protected]</a><br> >> > <a href=3D"https://mail.python.org/mailman/listinfo/ipython-d= ev" rel=3D"noreferrer" target=3D"_blank">https://mail.python.org/<wbr>mailm= an/listinfo/ipython-dev</a><br> >> ><br> >> ______________________________<wbr>_________________<br> >> IPython-dev mailing list<br> >> <a href=3D"mailto:[email protected]" target=3D"_blank">IPytho= [email protected]</a><br> >> <a href=3D"https://mail.python.org/mailman/listinfo/ipython-dev" r= el=3D"noreferrer" target=3D"_blank">https://mail.python.org/<wbr>mailman/li= stinfo/ipython-dev</a><br> ><br> ><br> ><br> > ______________________________<wbr>_________________<br> > IPython-dev mailing list<br> > <a href=3D"mailto:[email protected]" target=3D"_blank">IPython-de= [email protected]</a><br> > <a href=3D"https://mail.python.org/mailman/listinfo/ipython-dev" rel= =3D"noreferrer" target=3D"_blank">https://mail.python.org/<wbr>mailman/list= info/ipython-dev</a><br> ><br> ______________________________<wbr>_________________<br> IPython-dev mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">IPython-dev@pyt= hon.org</a><br> <a href=3D"https://mail.python.org/mailman/listinfo/ipython-dev" rel=3D"nor= eferrer" target=3D"_blank">https://mail.python.org/<wbr>mailman/listinfo/ip= ython-dev</a><br> </div></div></blockquote></div><br></div> ______________________________<wbr>_________________<br> IPython-dev mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">IPython-dev@pyt= hon.org</a><br> <a href=3D"https://mail.python.org/mailman/listinfo/ipython-dev" rel=3D"nor= eferrer" target=3D"_blank">https://mail.python.org/<wbr>mailman/listinfo/ip= ython-dev</a><br> </blockquote></div> </div></div><br>______________________________<wbr>_________________<br> IPython-dev mailing list<br> <a href=3D"mailto:[email protected]">[email protected]</a><br> <a href=3D"https://mail.python.org/mailman/listinfo/ipython-dev" rel=3D"nor= eferrer" target=3D"_blank">https://mail.python.org/<wbr>mailman/listinfo/ip= ython-dev</a><br> <br></blockquote></div><br></div> --0000000000006d19dd056d5816ed-- --===============6016329491209657203== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPython-dev mailing list [email protected] https://mail.python.org/mailman/listinfo/ipython-dev --===============6016329491209657203==--