Re: Updated dataflow semantics for RTT
Johannes Meyer <[email protected]> Tue, 29 Sep 2015 15:12:14 +0200
| Newsgroups | gmane.science.robotics.orocos.devel |
|---|---|
| Message-ID | <CABowFJFD58Y1zdf2YNk9GCfTLi+D9zSnAAVm6SkB2PUvfkEPvA@mail.gmail.com> |
--===============1384777085== Content-Type: multipart/alternative; boundary=001a1141e4c8380d900520e2916a --001a1141e4c8380d900520e2916a Content-Type: text/plain; charset=UTF-8 On Tue, Sep 29, 2015 at 1:37 PM, Herman Bruyninckx < [email protected]> wrote: > On Mon, 28 Sep 2015, Johannes Meyer wrote: > > Hi, >> thanks for all your input and ideas and I agree with Herman that the >> "one-framework-for-everything" approach of RTT >> is probably not how you would start a new project nowadays. >> > > > Some inputs from a discussion on the ROS mailinglist which is very much > related to this RTT-centric thread: > <http://lists.ros.org/pipermail/ros-users/2015-September/069663.html> > <http://lists.ros.org/pipermail/ros-users/2015-September/069664.html> > > These are the critical remarks of two experienced people towards the > "one size fits all" approach of communication in ROS2.0; at the same time, > they also provide concrete use cases and trade-offs that are relevant for > the envisaged update of comunication in RTT... > > Herman > > PS I am of course more than a bit biased: both messages reflect also my > personal view on this issue of "communication middleware", and are not just > random samples from that ROS thread. > > PS2 Geoff Biggs is another RTT-knowledgeable person who _did_ intervene in > that ROS thread, as well as in this RTT thread. Are there others on this > list who, like Geoff and myself, are following both threads? Because, > together, we might "mine" both threads to extract very interesting inputs > from potential users, with widely varying expectations and contexts... I also follow the ros-users and ros-sig-ng-ros mailing lists and the discussions you mentioned. We will hopefully learn a lot about the current state of ROS 2.0 development and why certain decisions have been made at ROSCON next weekend in Hamburg. As I said, I completely agree that RTT tries to solve too many problems at once, given that a lot of established solutions exists nowadays for most of the building blocks that could be integrated in a way more flexible way. However, I do not necessarily share the opinion that dividing a complex problem into as small as possible domains and re-use existing concepts and tools for each of them in a modular way will *always* result in a better and less error-prone solution. There are also a lot of pitfalls in the necessary "gluing" to adapt APIs, to really make the domains interchangeable and to avoid performance pitfalls, and at the end it will still be a complex solution for a complex problem. It was never the intention of this thread to start a discussion about "the next big version of RTT" - even if we considered to increment the major version number to 3 for the case the new behavior cannot be made in a backwards-compatible way. It is all about relatively small updates and bugfixes that hopefully will not disrupt existing applications and most likely will not open RTT too a much broader community either. On the other hand, there are some more patches that should be discussed and merged sooner or later, that have the potential to break things and are well worth a new version number: - CORBA aliases (https://github.com/orocos-toolchain/rtt/pull/66) - TLSF memory corruption detection ( https://github.com/orocos-toolchain/rtt/pull/77) - Reintroduce v1 cmd syntax (https://github.com/orocos-toolchain/rtt/pull/84 ) - Include directories reordering ( https://github.com/orocos-toolchain/rtt/pull/85) - Master update hook vs callback queue ( https://github.com/orocos-toolchain/rtt/pull/91) - CORBA efficieny patches (no pull request yet, see https://github.com/orocos-toolchain/rtt/compare/master...meyerj:corba-efficiency-patches ) On Tue, Sep 29, 2015 at 1:44 AM, Sylvain Joyeux <[email protected]> wrote: > Hi Johannes, thanks for the long explanation. > > On the implementation side, this whole discussion revived an idea I > have had a while ago, and is pretty much in line with the discussion > Herman started. > > Basically, one could get rid of (a.k.a. "deprecate" of course) the > connection API on the ports themselves, and replace it by a separate > object (let's call it "ConnectionManager") which would be in charge of > managing all these buffers and create all these connections and > implement the . In its simplest form, each process would have one of > these objects. > > The basic operations it would have support are: > - create/remove storage object (data object / buffer object) > - connect output port to storage object > - connect storage object to input port > That is already quite close to what the ConnFactory class is doing in RTT 2: building data objects, building channel input and output and finally creating connections between ports or ports and a transport by using this basic operations. All of those are governed by the ConnPolicy object which describes the connection's internals. But the ConnFactory is considered as internal API and the ports' connection methods are only simple wrappers that call ConnFactory methods. The ConnFactory class is the one that has been touched most in order to implement the proposed v3 dataflow semantics and even without changing the semantics it was worth the effort to do a cleanup here and clearly separate those basic operations. At least with my proposed patch, once a connection is established, all channel elements involved (buffers, remote channels, transport-specific elements, ...) are managed by shared pointers and cleaned up automatically if not needed anymore. The ConnectionManager instance, which currently belongs to each port, is only required to introspect its connections, but it completely lost its role in data processing and could also be dropped, replaced by a global connection manager or some other tool. RTT port implementations are already quite independent of the dataflow model and channels could implement anything that supports write and/or read operations from and to a reference to the sample. > > A "pull" connection would involve the ConnectionManager of the output > port. A "push" connection the ConnectionManager of the receiving side. > How one gets shared or private (or mixed) connections is obvious. > Flags would be split between the two halves (which is good !). For > instance, 'init' only affects the output->buffer side. > If you replace ConnectionManager by ConnFactory here, that basically describes what is going on in RTT 2 internally and what I think is even separated much better now. > > In principle -- but that's a principle that even I take with a big > grain of salt given that I did not seriously touch that stuff for > years -- the current underlying API (creation of the two channel > halves) would be reused. In other words, only where the connections > are created would change, not how they are created. > > On top of that, it's trivial to create higher-level, easier-to-use > APIs, and also implement the current connection API so as to not break > existing code. > > It would open the path to a lot of interesting things: > - finally have a decent way to monitor the data objects (i.e. buffer > fill rates, "data loss", data counters, that kind of thing). It would > also make obvious how one could replace the RTT "base connection > manager" by something else without touching the components. It would > spell how wonderfully relevant RTT still is to those who don't see it > ;-) > - if we add the possibility of inter-ConnectionManager connections > would allow us to move the keepSample functionalities of OutputPort > out of it, which would also make the whole init functionality a lot > more transparent. It would allow to expose (and monitor ...) the CORBA > dispatchers. > I do not fully understand these last two points and why these things would not already be possible today with RTT 2, perhaps by adding some simple, non API-breaking helper methods to port and ConnectionManager classes. It would be quite straight-forward to implement a store-and-forward channel element that can be installed right after the channel input by the ConnFactory and that replaces the current init functionality out of the OutputPort implementation, but I do not really see why this would be beneficial or open new possibilities for now. > > Now, I do understand that it might sound "blue sky" and that you might > not have the time for that ... just sayin' ;-) > > Sylvain Johannes --001a1141e4c8380d900520e2916a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><br><div class=3D"gmail_extra"><br><div class=3D"gmail_quo= te">On Tue, Sep 29, 2015 at 1:37 PM, Herman Bruyninckx <span dir=3D"ltr">&l= t;<a href=3D"mailto:[email protected]" target=3D"_blank">H= [email protected]</a>></span> wrote:<br><blockquote clas= s=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;b= order-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"= ><span class=3D"">On Mon, 28 Sep 2015, Johannes Meyer wrote:<br> <br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> Hi,<br> thanks for all your input and ideas and I agree with Herman that the "= one-framework-for-everything" approach of RTT<br> is probably not how you would start a new project nowadays.<br> </blockquote> <br> <br></span> Some inputs from a discussion on the ROS mailinglist which is very much<br> related to this RTT-centric thread:<br> =C2=A0<<a href=3D"http://lists.ros.org/pipermail/ros-users/2015-Septembe= r/069663.html" rel=3D"noreferrer" target=3D"_blank">http://lists.ros.org/pi= permail/ros-users/2015-September/069663.html</a>><br> =C2=A0<<a href=3D"http://lists.ros.org/pipermail/ros-users/2015-Septembe= r/069664.html" rel=3D"noreferrer" target=3D"_blank">http://lists.ros.org/pi= permail/ros-users/2015-September/069664.html</a>><br> <br> These are the critical remarks of two experienced people towards the<br> "one size fits all" approach of communication in ROS2.0; at the s= ame time,<br> they also provide concrete use cases and trade-offs that are relevant for<b= r> the envisaged update of comunication in RTT...<br> <br> Herman<br> <br> PS I am of course more than a bit biased: both messages reflect also my<br> personal view on this issue of "communication middleware", and ar= e not just<br> random samples from that ROS thread.<br> <br> PS2 Geoff Biggs is another RTT-knowledgeable person who _did_ intervene in<= br> that ROS thread, as well as in this RTT thread. Are there others on this<br= > list who, like Geoff and myself, are following both threads? Because,<br> together, we might "mine" both threads to extract very interestin= g inputs<br> from potential users, with widely varying expectations and contexts...</blo= ckquote><div><br></div><div><br></div><div>I also follow the ros-users and = ros-sig-ng-ros mailing lists and the discussions you mentioned. We will hop= efully learn a lot about the current state of ROS 2.0 development and why c= ertain decisions have been made at ROSCON next weekend in Hamburg. As I sai= d, I completely agree that RTT tries to solve too many problems at once, gi= ven that a lot of established solutions exists nowadays for most of the bui= lding blocks that could be integrated in a way more flexible way. However, = I do not necessarily share the opinion that dividing a complex problem into= as small as possible domains and re-use existing concepts and tools for ea= ch of them in a modular way will *always* result in a better and less error= -prone solution. There are also a lot of pitfalls in the necessary "gl= uing" to adapt APIs, to really make the domains interchangeable and to= avoid performance pitfalls, and at the end it will still be a complex solu= tion for a complex problem.<br></div><div><br></div><div>It was never the i= ntention of this thread to start a discussion about "the next big vers= ion of RTT" - even if we considered to increment the major version num= ber to 3 for the case the new behavior cannot be made in a backwards-compat= ible way. It is all about relatively small updates and bugfixes that hopefu= lly will not disrupt existing applications and most likely will not open RT= T too a much broader community either. On the other hand, there are some mo= re patches that should be discussed and merged sooner or later, that have t= he potential to break things and are well worth a new version number:</div>= <div><br></div><div>- CORBA aliases (<a href=3D"https://github.com/orocos-t= oolchain/rtt/pull/66">https://github.com/orocos-toolchain/rtt/pull/66</a>)<= /div><div>- TLSF memory corruption detection (<a href=3D"https://github.com= /orocos-toolchain/rtt/pull/77">https://github.com/orocos-toolchain/rtt/pull= /77</a>)</div><div>- Reintroduce v1 cmd syntax (<a href=3D"https://github.c= om/orocos-toolchain/rtt/pull/84">https://github.com/orocos-toolchain/rtt/pu= ll/84</a>)<br></div><div>- Include directories reordering (<a href=3D"https= ://github.com/orocos-toolchain/rtt/pull/85">https://github.com/orocos-toolc= hain/rtt/pull/85</a>)</div><div>-=C2=A0Master update hook vs callback queue= (<a href=3D"https://github.com/orocos-toolchain/rtt/pull/91">https://githu= b.com/orocos-toolchain/rtt/pull/91</a>)</div><div>- CORBA efficieny patches= (no pull request yet, see=C2=A0<a href=3D"https://github.com/orocos-toolch= ain/rtt/compare/master...meyerj:corba-efficiency-patches">https://github.co= m/orocos-toolchain/rtt/compare/master...meyerj:corba-efficiency-patches</a>= )<br></div><div><br></div><div><br></div><div><div class=3D"gmail_quote">On= Tue, Sep 29, 2015 at 1:44 AM, Sylvain Joyeux=C2=A0<span dir=3D"ltr"><<a= href=3D"mailto:[email protected]" target=3D"_blank">sylvain.joyeux@m4= x.org</a>></span>=C2=A0wrote:<br><blockquote class=3D"gmail_quote" style= =3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(20= 4,204,204);border-left-style:solid;padding-left:1ex">Hi Johannes, thanks fo= r the long explanation.<br><br>On the implementation side, this whole discu= ssion revived an idea I<br>have had a while ago, and is pretty much in line= with the discussion<br>Herman started.<br><br>Basically, one could get rid= of (a.k.a. "deprecate" of course) the<br>connection API on the p= orts themselves, and replace it by a separate<br>object (let's call it = "ConnectionManager") which would be in charge of<br>managing all = these buffers and create all these connections and<br>implement the . In it= s simplest form, each process would have one of<br>these objects.<br><br>Th= e basic operations it would have support are:<br>=C2=A0- create/remove stor= age object (data object / buffer object)<br>=C2=A0- connect output port to = storage object<br>=C2=A0- connect storage object to input port<br></blockqu= ote><div><br></div><div>That is already quite close to what the ConnFactory= class is doing in RTT 2: building data objects, building channel input and= output and finally creating connections between ports or ports and a trans= port by using this basic operations. All of those are governed by the ConnP= olicy object which describes the connection's internals. But the ConnFa= ctory is considered as internal API and the ports' connection methods a= re only simple wrappers that call ConnFactory methods. The ConnFactory clas= s is the one that has been touched most in order to implement the proposed = v3 dataflow semantics and even without changing the semantics it was worth = the effort to do a cleanup here and clearly separate those basic operations= .</div><div><br></div><div>At least with my proposed patch, once a connecti= on is established, all channel elements involved (buffers, remote channels,= transport-specific elements, ...) are managed by shared pointers and clean= ed up automatically if not needed anymore. The ConnectionManager instance, = which currently belongs to each port, is only required to introspect its co= nnections, but it completely lost its role in data processing and could als= o be dropped, replaced by a global connection manager or some other tool.</= div><div><br></div><div>RTT port implementations are already quite independ= ent of the dataflow model and channels could implement anything that suppor= ts write and/or read operations from and to a reference to the sample.</div= ><div><br></div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D= "margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,2= 04,204);border-left-style:solid;padding-left:1ex"><br>A "pull" co= nnection would involve the ConnectionManager of the output<br>port. A "= ;push" connection the ConnectionManager of the receiving side.<br>How = one gets shared or private (or mixed) connections is obvious.<br>Flags woul= d be split between the two halves (which is good !). For<br>instance, '= init' only affects the output->buffer side.<br></blockquote><div><br= ></div><div>If you replace ConnectionManager by ConnFactory here, that basi= cally describes what is going on in RTT 2 internally and what I think is ev= en separated much better now.</div><div><br></div><div>=C2=A0<br></div><blo= ckquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left= -width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;paddi= ng-left:1ex"><br>In principle -- but that's a principle that even I tak= e with a big<br>grain of salt given that I did not seriously touch that stu= ff for<br>years -- the current underlying API (creation of the two channel<= br>halves) would be reused. In other words, only where the connections<br>a= re created would change, not how they are created.<br><br>On top of that, i= t's trivial to create higher-level, easier-to-use<br>APIs, and also imp= lement the current connection API so as to not break<br>existing code.<br><= br>It would open the path to a lot of interesting things:<br>=C2=A0- finall= y have a decent way to monitor the data objects (i.e. buffer<br>fill rates,= "data loss", data counters, that kind of thing). It would<br>als= o make obvious how one could replace the RTT "base connection<br>manag= er" by something else without touching the components. It would<br>spe= ll how wonderfully relevant RTT still is to those who don't see it<br>;= -)<br>=C2=A0- if we add the possibility of inter-ConnectionManager connecti= ons<br>would allow us to move the keepSample functionalities of OutputPort<= br>out of it, which would also make the whole init functionality a lot<br>m= ore transparent. It would allow to expose (and monitor ...) the CORBA<br>di= spatchers.<br></blockquote><div><br></div><div>I do not fully understand th= ese last two points and why these things would not already be possible toda= y with RTT 2, perhaps by adding some simple, non API-breaking helper method= s to port and ConnectionManager classes. It would be quite straight-forward= to implement a store-and-forward channel element that can be installed rig= ht after the channel input by the ConnFactory and that replaces the current= init functionality out of the OutputPort implementation, but I do not real= ly see why this would be beneficial or open new possibilities for now.</div= ><div><br></div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D= "margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,2= 04,204);border-left-style:solid;padding-left:1ex"><br>Now, I do understand = that it might sound "blue sky" and that you might<br>not have the= time for that ... just sayin' ;-)<br><span class=3D""><font color=3D"#= 888888"><br>Sylvain</font></span></blockquote></div></div><div><br></div><d= iv><br></div><div>Johannes</div><div><br></div></div><div class=3D"gmail_si= gnature"></div> </div></div> --001a1141e4c8380d900520e2916a-- --===============1384777085== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline -- Orocos-Dev mailing list [email protected] http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev --===============1384777085==--