Re: Updated dataflow semantics for RTT
Willy Lambert <[email protected]> Sun, 27 Sep 2015 22:43:02 +0200
| Newsgroups | gmane.science.robotics.orocos.devel |
|---|---|
| Message-ID | <CAKvQZ_3k5YYxa1aTcsAMyzYZBAA8bCNWBAW7mhsE-yWXVV1cLw@mail.gmail.com> |
--===============0759051834== Content-Type: multipart/alternative; boundary=001a113418c6856e740520c0a017 --001a113418c6856e740520c0a017 Content-Type: text/plain; charset=UTF-8 2015-09-27 14:10 GMT+02:00 Herman Bruyninckx < [email protected]>: > On Mon, 21 Sep 2015, Johannes Meyer wrote: > > Dear Orocos community, >> during the last few weeks we have been working on the concept and >> implementation of a major redesign of the Orocos >> RTT dataflow, that we would like to share and ask for your valuable >> feedback. There was already a previous >> discussion on the mailing list about the topic earlier this year [1], and >> the following proposal tries to address >> all the issues mentioned there. >> >> With the version 2 of the Orocos toolchain some years ago, the previous >> data flow architecture, which was based on a >> single data object and buffer shared between all ports participating at >> the same connection, was replaced by a >> connection architecture based on data flow, where each pair of output and >> input ports (or streams) is connected >> through an individual channel, each having its own data object or buffer. >> While there was a well substantiated >> motivation for this update and it solved some of the shortcomings in RTT >> version 1 [2], it also introduced some new, >> sometimes hidden, problems and broke other use cases. One example is an >> input port with multiple connections, where >> due to the ambiguity in which channel to read from the result is >> sometimes unexpected to the user, who expects that >> each write overwrites previous samples in case of simple data connections. >> >> We assembled a document (see attachment, or [3]) which introduces the >> version 1 and version 2 data flow semantics >> for those who are not aware of the details, three example use cases and >> presents the new dataflow architecture we >> have in mind, which ideally works for a superset of use cases that are >> supported by RTT version 1 and 2. >> >> We do not want to go into details here, so only a short summary for those >> who do not want to read the full document: >> We propose to add two more boolean flags to the ConnPolicy struct that >> describes the type of connection between two >> ports: "shared" and "mandatory". Furthermore the existing "pull" flag, >> which was only meaningful for remote >> connections, now has a big influence for local connections. >> >> "push" vs. "pull" >> >> > Thanks for this nice effort. I think it is really time to consider a next > generation of Orocos RTT... > > If you allow me to bring in my two cents... It's possible that you won't > like the > message, or find it off-topic, but I think I should share it with you > anyway... > > The best document I have read about communication that connects > high-performance with flexibility and configurability is this one: > < > https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying > > > > It cleanly separates the communication of "events" from (i) what they mean, > (ii) who is interested in them, and (iii) with what policy they are > consumed. > > In our current research at KU Leuven, this is going to be _the_ basis for > all interaction between asynchronous activities. > The other basis will be the separation between (i) functional requirements > of the application, (ii) "roles and responsibilities" of "components", > (iii) software components to implement both, and (iv) deployment of these > components on your hardware and OS platforms. > > All of the above are not present in Orocos RTT, which I consider design > flaws. > So solving these problems is definitely worthwhile. But, and this is a big > but, my view on solving them (which is currently being transformed into > a first set of models and code and tools) implies that we do not need > Orocos RTT at all anymore. The reasons why we started RTT more than fifteen > years ago have not disappeared, but have all been solved in more > "mainstream" > software projects than Orocos. Fifteen years ago, what we where after was a > "node-port-connector" modelling-and-coding interface, where nodes could be > deployed in realtime activities on an operating system. It was the time of > single core CPUs, with only proprietary solutions to fast networking. Those > days have gone, together with the need to use RTT as a software solution. > > More in particular, the "communication" aspect (which is the core of the > document you distributed) is a lot better understood and supported in > mainstream than 15 years ago. Taking the model of the event log and the > corresponding log processing around it means that all the policies that > your document wants to introduce as "API" methods can (should...) better be > realised by configurations of the event log mechanism in your _application > architecture_; there is no need to provide APIs in "middleware", > "frameworks" or "libraries". > > _The_ core functionalities of RTT that still survive after 15 years are: > - its introspection capabilities; > - the lock-free data buffer. > +1, this is my case. > The latter is just a special case ("configuration") of the event log > mechanism, that provides all other policies that you mention (and more!) > just through other configurations. The introspection is a form of "model > based" implementation, which > considers models as first-class citizens in the software; this is great, > and should be extended, starting with modelling all the relevant event log > policies explicity. (KU Leuven is working on that; it takes a lot of > efforts...) Lock-free data buffers are the core of event log processing, > and are currently better known under the new buzzword "immutable data". > > The other "good-intuition-that-turned-out-to-be-badly-implemented" of > fifteen years ago is that of "stored procedures" (known under various > names, such as "microservices", proxy computations, etc.). RTT provides a > very complex and ambiguous API to let one component execute code for > another component. Again, that functionality should not be provided by an > "RPC API", but by models and tools of _deployment_. > > The added value of Orocos as a project could shift from providing code to > providing professional quality documentation and tutorials for "best > practices" and "patterns", some specific realtime-optimized > implementations, or patches to the large amount of existing libraries that > support alul of the above. > > It is up to the Orocos community to decide whether to go that direction or > not. I would be glad to cooperate in that directoion, because otherwise we > will be forced to start a "competing" project. > > Note that this message does not contain threats to "fork" or whatever, but > just my insights after working with Orocos RTT for about a decade, and > having Orocos RTT work against the architectural requirements in our > robotics projects for the subsequent five years... > > I frankly think it is time to reinvent the project. Big time. > > The easiest way is to _deprecate_ things, to make the _core_ things simpler > for users, throwing out more and more code and start using other already > existing software (ZeroMQ, libevent, the lock-free buffer library, > RethinkDB,...) instead of trying to squeeze in their functionalities within > NIH APIs and implementations. In the end, RTT will be empty, and the > development efforts can then finally shift to give Sylvain a much bigger > hand with ROCK :-) > (Just make sure to refactor ROCK in order to decouple the _models_ from the > Ruby and RTT _implementations_ :-)) > > Best regards, > > Herman Bruyninckx I have never been an active part of the project (testing and using apart), so I can not comment all this, but : _ thanks Hermann for sharing _ today, if I had to choose a solution for my professionnal project, I would not choose Orocos(Herman has well explained why) and the miss would be the lock free communication and introspection tools. _ I would also migrate from pure Orocos to Rock if my application wasn't considered as "frozen". I'm not sure it's the good post to chat on that. > > > Like it is already the case for remote connections via CORBA, the "pull" >> flag will decide whether the connection's >> data object or buffer is installed at the writer's side (pull) or the >> reader's side (push) of the connection. The >> first case is equivalent to the current version 2 connection model and >> readers have to select the connection/output >> port to read from, while for push connections there will be only one >> single buffer per input port. We think that the >> push connection should be the default connection type for local ports, >> but this might break existing applications >> which rely on having one buffer per connection now. >> >> "private" vs. "shared" >> >> The shared flag reintroduces the shared data connection model that was >> standard in RTT version 1. For shared >> connections multiple input ports read from the same data object or >> consume samples from the same buffer instance, so >> their read operations can influence each other. Shared connections are >> especially useful for producer/consumer >> applications with multiple consumers. >> >> "mandatory" >> >> The introduction of the "mandatory" flag was necessary to decide on the >> return value of the write() operation on an >> output port. In RTT version 2 it was not possible anymore to get a >> feedback of whether a write operation was >> successful or failed, e.g. because of a full buffer or broken remote >> connection. Only connections with the mandatory >> flag set would be considered for the overall result of write() >> operations, while failures of non-mandatory >> connections will be ignored. >> >> >> For those who want to take a look at the current state of the >> implementation, this is the link to the working branch >> on GitHub: >> Tree: >> https://github.com/orocos-toolchain/rtt/tree/updated-dataflow-semantics >> Diff to master: >> https://github.com/orocos-toolchain/rtt/compare/master...updated-dataflow-semantics >> >> The implementation is not fully working yet for non-standard transports >> (like Corba and mqueue), but especially the >> updated test ports_test.cpp already shows what would change from a user >> perspective. >> >> Obviously, such a major refactoring cannot be done without changes in the >> public API. However, we tried to keep the >> required changes in user code as minimal as possible. >> The new dataflow semantics are targeted for an upcoming version 2.9 >> release (or even name it 3.0 to communicate this >> potentially breaking change?). >> >> We are planning to also come up with some comparative performance >> measurements during this week, as people have >> asked for that before [1]. >> >> >> Best regards, >> Johannes and Peter >> >> [1] >> http://www.orocos.org/forum/rtt/rtt-dev/limitations-v2-port-implementation >> [2] http://www.orocos.org/wiki/rtt/rtt-2.0/dataflow >> [3] >> https://docs.google.com/document/d/1zDnPPz4SiCVvfEFxYFUZBcVbXKj33o4KaotpCtXM4E0/pub >> >> >> > -- > Orocos-Dev mailing list > [email protected] > http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev > > --001a113418c6856e740520c0a017 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><br><div class=3D"gmail_extra"><br><div class=3D"gmail_quo= te">2015-09-27 14:10 GMT+02:00 Herman Bruyninckx <span dir=3D"ltr"><<a h= ref=3D"mailto:[email protected]" target=3D"_blank">Herman.= [email protected]</a>></span>:<br><blockquote class=3D"gmail_q= uote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1e= x"><span>On Mon, 21 Sep 2015, Johannes Meyer wrote:<br> <br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Dear Orocos community,<br> during the last few weeks we have been working on the concept and implement= ation of a major redesign of the Orocos<br> RTT dataflow, that we would like to share and ask for your valuable feedbac= k. There was already a previous<br> discussion on the mailing list about the topic earlier this year [1], and t= he following proposal tries to address<br> all the issues mentioned there.<br> <br> With the version 2 of the Orocos toolchain some years ago, the previous dat= a flow architecture, which was based on a<br> single data object and buffer shared between all ports participating at the= same connection, was replaced by a<br> connection architecture based on data flow, where each pair of output and i= nput ports (or streams) is connected<br> through an individual channel, each having its own data object or buffer. W= hile there was a well substantiated<br> motivation for this update and it solved some of the shortcomings in RTT ve= rsion 1 [2], it also introduced some new,<br> sometimes hidden, problems and broke other use cases. One example is an inp= ut port with multiple connections, where<br> due to the ambiguity in which channel to read from the result is sometimes = unexpected to the user, who expects that<br> each write overwrites previous samples in case of simple data connections.<= br> <br> We assembled a document (see attachment, or [3]) which introduces the versi= on 1 and version 2 data flow semantics<br> for those who are not aware of the details, three example use cases and pre= sents the new dataflow architecture we<br> have in mind, which ideally works for a superset of use cases that are supp= orted by RTT version 1 and 2.=C2=A0<br> <br> We do not want to go into details here, so only a short summary for those w= ho do not want to read the full document:<br> We propose to add two more boolean flags to the ConnPolicy struct that desc= ribes the type of connection between two<br> ports: "shared" and "mandatory". Furthermore the existi= ng "pull" flag, which was only meaningful for remote<br> connections, now has a big influence for local connections.<br> <br> "push" vs. "pull"<br> <br> </blockquote> <br></span> Thanks for this nice effort. I think it is really time to consider a next<b= r> generation of Orocos RTT...<br> <br> If you allow me to bring in my two cents... It's possible that you won&= #39;t like the<br> message, or find it off-topic, but I think I should share it with you anywa= y...<br> <br> The best document I have read about communication that connects<br> high-performance with flexibility and configurability is this one:<br> =C2=A0<<a href=3D"https://engineering.linkedin.com/distributed-systems/l= og-what-every-software-engineer-should-know-about-real-time-datas-unifying"= rel=3D"noreferrer" target=3D"_blank">https://engineering.linkedin.com/dist= ributed-systems/log-what-every-software-engineer-should-know-about-real-tim= e-datas-unifying</a>><br> <br> It cleanly separates the communication of "events" from (i) what = they mean,<br> (ii) who is interested in them, and (iii) with what policy they are consume= d.<br> <br> In our current research at KU Leuven, this is going to be _the_ basis for<b= r> all interaction between asynchronous activities.<br> The other basis will be the separation between (i) functional requirements<= br> of the application, (ii) "roles and responsibilities" of "co= mponents",<br> (iii) software components to implement both, and (iv) deployment of these<b= r> components on your hardware and OS platforms.<br> <br> All of the above are not present in Orocos RTT, which I consider design fla= ws.<br> So solving these problems is definitely worthwhile. But, and this is a big<= br> but, my view on solving them (which is currently being transformed into<br> a first set of models and code and tools) implies that we do not need<br> Orocos RTT at all anymore. The reasons why we started RTT more than fifteen= <br> years ago have not disappeared, but have all been solved in more "main= stream"<br> software projects than Orocos. Fifteen years ago, what we where after was a= <br> "node-port-connector" modelling-and-coding interface, where nodes= could be<br> deployed in realtime activities on an operating system. It was the time of<= br> single core CPUs, with only proprietary solutions to fast networking. Those= <br> days have gone, together with the need to use RTT as a software solution.<b= r> <br> More in particular, the "communication" aspect (which is the core= of the<br> document you distributed) is a lot better understood and supported in<br> mainstream than 15 years ago. Taking the model of the event log and the<br> corresponding log processing around it means that all the policies that<br> your document wants to introduce as "API" methods can (should...)= better be<br> realised by configurations of the event log mechanism in your _application<= br> architecture_; there is no need to provide APIs in "middleware",<= br> "frameworks" or "libraries".<br> <br> _The_ core functionalities of RTT that still survive after 15 years are:<br= > - its introspection capabilities;<br> - the lock-free data buffer.<br></blockquote><div><br></div><div>+1, this i= s my case.<br></div><div>=C2=A0</div><blockquote class=3D"gmail_quote" styl= e=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> The latter is just a special case ("configuration") of the event = log<br> mechanism, that provides all other policies that you mention (and more!)<br= > just through other configurations. The introspection is a form of "mod= el based" implementation, which<br> considers models as first-class citizens in the software; this is great,<br= > and should be extended, starting with modelling all the relevant event log<= br> policies explicity. (KU Leuven is working on that; it takes a lot of<br> efforts...) Lock-free data buffers are the core of event log processing,<br= > and are currently better known under the new buzzword "immutable data&= quot;.<br> <br> The other "good-intuition-that-turned-out-to-be-badly-implemented"= ; of<br> fifteen years ago is that of "stored procedures" (known under var= ious<br> names, such as "microservices", proxy computations, etc.). RTT pr= ovides a<br> very complex and ambiguous API to let one component execute code for<br> another component. Again, that functionality should not be provided by an<b= r> "RPC API", but by models and tools of _deployment_.<br> <br> The added value of Orocos as a project could shift from providing code to<b= r> providing professional quality documentation and tutorials for "best<b= r> practices" and "patterns", some specific realtime-optimized<= br> implementations, or patches to the large amount of existing libraries that<= br> support alul of the above.<br> <br> It is up to the Orocos community to decide whether to go that direction or<= br> not. I would be glad to cooperate in that directoion, because otherwise we<= br> will be forced to start a "competing" project.<br> <br> Note that this message does not contain threats to "fork" or what= ever, but<br> just my insights after working with Orocos RTT for about a decade, and<br> having Orocos RTT work against the architectural requirements in our<br> robotics projects for the subsequent five years...<br> <br> I frankly think it is time to reinvent the project. Big time.<br> <br> The easiest way is to _deprecate_ things, to make the _core_ things simpler= <br> for users, throwing out more and more code and start using other already<br= > existing software (ZeroMQ, libevent, the lock-free buffer library,<br> RethinkDB,...) instead of trying to squeeze in their functionalities within= <br> NIH APIs and implementations.=C2=A0 In the end, RTT will be empty, and the<= br> development efforts can then finally shift to give Sylvain a much bigger<br= > hand with ROCK :-)<br> (Just make sure to refactor ROCK in order to decouple the _models_ from the= <br> =C2=A0Ruby and RTT _implementations_ :-))<br> <br> Best regards,<br> <br> Herman Bruyninckx</blockquote><div><br></div><div>I have never been an acti= ve part of the project (testing and using apart), so I can not comment all = this, but=C2=A0 :<br>_ thanks Hermann for sharing<br>_ today, if I had to c= hoose a solution for my professionnal project, I would not choose Orocos(He= rman has well explained why) and the miss would be the lock free communicat= ion and introspection tools.<br></div><div>_ I would also migrate from pure= Orocos to Rock if my application wasn't considered as "frozen&quo= t;.<br></div><div><br></div><div>I'm not sure it's the good post to= chat on that. <br><br></div><div></div><div>=C2=A0</div><blockquote class= =3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd= ing-left:1ex"><div><div><br> <br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Like it is already the case for remote connections via CORBA, the "pul= l" flag will decide whether the connection's<br> data object or buffer is installed at the writer's side (pull) or the r= eader's side (push) of the connection. The<br> first case is equivalent to the current version 2 connection model and read= ers have to select the connection/output<br> port to read from, while for push connections there will be only one single= buffer per input port. We think that the<br> push connection should be the default connection type for local ports, but = this might break existing applications<br> which rely on having one buffer per connection now.<br> <br> "private" vs. "shared"<br> <br> The shared flag reintroduces the shared data connection model that was stan= dard in RTT version 1. For shared<br> connections multiple input ports read from the same data object or consume = samples from the same buffer instance, so<br> their read operations can influence each other. Shared connections are espe= cially useful for producer/consumer<br> applications with multiple consumers.<br> <br> "mandatory"<br> <br> The introduction of the "mandatory" flag was necessary to decide = on the return value of the write() operation on an<br> output port. In RTT version 2 it was not possible anymore to get a feedback= of whether a write operation was<br> successful or failed, e.g. because of a full buffer or broken remote connec= tion. Only connections with the mandatory<br> flag set would be considered for the overall result of write() operations, = while failures of non-mandatory<br> connections will be ignored.<br> <br> <br> For those who want to take a look at the current state of the implementatio= n, this is the link to the working branch<br> on GitHub:<br> Tree: <a href=3D"https://github.com/orocos-toolchain/rtt/tree/updated-dataf= low-semantics" rel=3D"noreferrer" target=3D"_blank">https://github.com/oroc= os-toolchain/rtt/tree/updated-dataflow-semantics</a><br> Diff to master:=C2=A0<a href=3D"https://github.com/orocos-toolchain/rtt/com= pare/master...updated-dataflow-semantics" rel=3D"noreferrer" target=3D"_bla= nk">https://github.com/orocos-toolchain/rtt/compare/master...updated-datafl= ow-semantics</a><br> <br> The implementation is not fully working yet for non-standard transports (li= ke Corba and mqueue), but especially the<br> updated test ports_test.cpp already shows what would change from a user per= spective.<br> <br> Obviously, such a major refactoring cannot be done without changes in the p= ublic API. However, we tried to keep the<br> required changes in user code as minimal as possible.<br> The new dataflow semantics are targeted for an upcoming version 2.9 release= (or even name it 3.0 to communicate this<br> potentially breaking change?).<br> <br> We are planning to also come up with some comparative performance measureme= nts during this week, as people have<br> asked for that before [1].<br> <br> <br> Best regards,<br> Johannes and Peter<br> <br> [1] <a href=3D"http://www.orocos.org/forum/rtt/rtt-dev/limitations-v2-port-= implementation" rel=3D"noreferrer" target=3D"_blank">http://www.orocos.org/= forum/rtt/rtt-dev/limitations-v2-port-implementation</a><br> [2]=C2=A0<a href=3D"http://www.orocos.org/wiki/rtt/rtt-2.0/dataflow" rel=3D= "noreferrer" target=3D"_blank">http://www.orocos.org/wiki/rtt/rtt-2.0/dataf= low</a><br> [3] <a href=3D"https://docs.google.com/document/d/1zDnPPz4SiCVvfEFxYFUZBcVb= XKj33o4KaotpCtXM4E0/pub" rel=3D"noreferrer" target=3D"_blank">https://docs.= google.com/document/d/1zDnPPz4SiCVvfEFxYFUZBcVbXKj33o4KaotpCtXM4E0/pub</a><= br> <br> <br> </blockquote> </div></div><br>--<br> Orocos-Dev mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">Oroc= [email protected]</a><br> <a href=3D"http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev" rel= =3D"noreferrer" target=3D"_blank">http://lists.mech.kuleuven.be/mailman/lis= tinfo/orocos-dev</a><br> <br></blockquote></div><br></div></div> --001a113418c6856e740520c0a017-- --===============0759051834== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline -- Orocos-Dev mailing list [email protected] http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev --===============0759051834==--