Re: Updated dataflow semantics for RTT
S Roderick <[email protected]> Tue, 22 Sep 2015 18:11:34 -0400
| Newsgroups | gmane.science.robotics.orocos.devel |
|---|---|
| Message-ID | <[email protected]> |
--===============1953219044== Content-type: multipart/alternative; boundary="Apple-Mail=_4A0C3AA2-EBA0-4533-9FDE-56EAC227572B" --Apple-Mail=_4A0C3AA2-EBA0-4533-9FDE-56EAC227572B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 > On Sep 22, 2015, at 09:32, Janosch Machowinski = <[email protected]> wrote: >=20 > Hi, > here are my remarks: > First of all, I like the idea of the input buffer (push case). But I = also see some > problems with it. The restriction on the same buffer size and type in = the=20 > connection policy will be annoying (In my mind I can see the runtime = errors pop up).=20 > It will be hard to track this in bigger systems. Do I get it = correctly, that you only > want to preserve arrival order and runtime here ? I=92ll have to disagree here. We have several hundred connections in our = systems, and we track the policy type amongst all. > If you are already changing the Connection implementation, I would = recommend, to > put a new thread in every connection in remote case. We experienced = the issue, that > bad Wifi connections, slowed our systems down, as it would hang on the = write call. Not my first choice. I agree with Sylvain - this should be part of the = transport layer. > For the Feedback on the write I would recommend to create a second = enum, and not > to use FlowStatus. Sometimes I use switch/case to react to the read = FlowStatus. If you > Introduce new members, this will result in compile warnings, because = of unhandled > entrie. Yep, backwards compatibility is a concern. :-( > For the creation of shared connections, I would recommend an new API. = The create call > should contain all involved members. This spares us the headache of = tracking buffer sizes > and types around in the code. Also this will simplify the code logic, = as we don't need the lookup. > e.g.=20 > out.connectShared(b.in, c.in, d.in, policy) ; In the above are =93b=94, =93c=94, and =93d=94, all ports? If so, what = about cases were not all port connections are made at the same time = (e.g. XML deployments)? Cheers S > Greetings > Janosch >=20 >=20 >=20 > Am 21.09.2015 um 15:15 schrieb Johannes Meyer: >> Dear Orocos community, >>=20 >> during the last few weeks we have been working on the concept and = implementation of a major redesign of the Orocos RTT dataflow, that we = would like to share and ask for your valuable feedback. There was = already a previous discussion on the mailing list about the topic = earlier this year [1], and the following proposal tries to address all = the issues mentioned there. >>=20 >> With the version 2 of the Orocos toolchain some years ago, the = previous data flow architecture, which was based on a single data object = and buffer shared between all ports participating at the same = connection, was replaced by a connection architecture based on data = flow, where each pair of output and input ports (or streams) is = connected through an individual channel, each having its own data object = or buffer. While there was a well substantiated motivation for this = update and it solved some of the shortcomings in RTT version 1 [2], it = also introduced some new, sometimes hidden, problems and broke other use = cases. One example is an input port with multiple connections, where due = to the ambiguity in which channel to read from the result is sometimes = unexpected to the user, who expects that each write overwrites previous = samples in case of simple data connections. >>=20 >> We assembled a document (see attachment, or [3]) which introduces the = version 1 and version 2 data flow semantics for those who are not aware = of the details, three example use cases and presents the new dataflow = architecture we have in mind, which ideally works for a superset of use = cases that are supported by RTT version 1 and 2.=20 >>=20 >> We do not want to go into details here, so only a short summary for = those who do not want to read the full document: >> We propose to add two more boolean flags to the ConnPolicy struct = that describes the type of connection between two ports: "shared" and = "mandatory". Furthermore the existing "pull" flag, which was only = meaningful for remote connections, now has a big influence for local = connections. >>=20 >> "push" vs. "pull" >>=20 >> Like it is already the case for remote connections via CORBA, the = "pull" flag will decide whether the connection's data object or buffer = is installed at the writer's side (pull) or the reader's side (push) of = the connection. The first case is equivalent to the current version 2 = connection model and readers have to select the connection/output port = to read from, while for push connections there will be only one single = buffer per input port. We think that the push connection should be the = default connection type for local ports, but this might break existing = applications which rely on having one buffer per connection now. >>=20 >> "private" vs. "shared" >>=20 >> The shared flag reintroduces the shared data connection model that = was standard in RTT version 1. For shared connections multiple input = ports read from the same data object or consume samples from the same = buffer instance, so their read operations can influence each other. = Shared connections are especially useful for producer/consumer = applications with multiple consumers. >>=20 >> "mandatory" >>=20 >> The introduction of the "mandatory" flag was necessary to decide on = the return value of the write() operation on an output port. In RTT = version 2 it was not possible anymore to get a feedback of whether a = write operation was successful or failed, e.g. because of a full buffer = or broken remote connection. Only connections with the mandatory flag = set would be considered for the overall result of write() operations, = while failures of non-mandatory connections will be ignored. >>=20 >>=20 >> For those who want to take a look at the current state of the = implementation, this is the link to the working branch on GitHub: >> Tree: = https://github.com/orocos-toolchain/rtt/tree/updated-dataflow-semantics = <https://github.com/orocos-toolchain/rtt/tree/updated-dataflow-semantics> >> Diff to master: = https://github.com/orocos-toolchain/rtt/compare/master...updated-dataflow-= semantics = <https://github.com/orocos-toolchain/rtt/compare/master...updated-dataflow= -semantics> >>=20 >> The implementation is not fully working yet for non-standard = transports (like Corba and mqueue), but especially the updated test = ports_test.cpp already shows what would change from a user perspective. >>=20 >> Obviously, such a major refactoring cannot be done without changes in = the public API. However, we tried to keep the required changes in user = code as minimal as possible. >> The new dataflow semantics are targeted for an upcoming version 2.9 = release (or even name it 3.0 to communicate this potentially breaking = change?). >>=20 >> We are planning to also come up with some comparative performance = measurements during this week, as people have asked for that before [1]. >>=20 >>=20 >> Best regards, >> Johannes and Peter >>=20 >>=20 >>=20 >> [1] = http://www.orocos.org/forum/rtt/rtt-dev/limitations-v2-port-implementation= = <http://www.orocos.org/forum/rtt/rtt-dev/limitations-v2-port-implementatio= n> >> [2] http://www.orocos.org/wiki/rtt/rtt-2.0/dataflow = <http://www.orocos.org/wiki/rtt/rtt-2.0/dataflow> >> [3] = https://docs.google.com/document/d/1zDnPPz4SiCVvfEFxYFUZBcVbXKj33o4KaotpCt= XM4E0/pub = <https://docs.google.com/document/d/1zDnPPz4SiCVvfEFxYFUZBcVbXKj33o4KaotpC= tXM4E0/pub> >>=20 >>=20 >>=20 >=20 >=20 > --=20 > Dipl. Inf. Janosch Machowinski > SAR- & Sicherheitsrobotik >=20 > Universit=E4t Bremen > FB 3 - Mathematik und Informatik > AG Robotik > Robert-Hooke-Stra=DFe 1 > 28359 Bremen, Germany > =20 > Zentrale: +49 421 178 45-6611 > =20 > Besuchsadresse der Nebengesch=E4ftstelle:=20 > Robert-Hooke-Stra=DFe 5 > 28359 Bremen, Germany > =20 > Tel.: +49 421 178 45-6614 > Empfang: +49 421 178 45-6600 > Fax: +49 421 178 45-4150 > E-Mail: [email protected] = <mailto:[email protected]> >=20 > Weitere Informationen: http://www.informatik.uni-bremen.de/robotik = <http://www.informatik.uni-bremen.de/robotik> > --=20 > Orocos-Dev mailing list > [email protected] > http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev --Apple-Mail=_4A0C3AA2-EBA0-4533-9FDE-56EAC227572B Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dwindows-1252"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div><blockquote type=3D"cite" class=3D""><div class=3D"">On = Sep 22, 2015, at 09:32, Janosch Machowinski <<a = href=3D"mailto:[email protected]" = class=3D"">[email protected]</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""> =20 <meta content=3D"text/html; charset=3Dwindows-1252" = http-equiv=3D"Content-Type" class=3D""> =20 <div text=3D"#000000" bgcolor=3D"#FFFFFF" class=3D""> <div class=3D"moz-cite-prefix">Hi,<br class=3D""> here are my remarks:<br class=3D""> First of all, I like the idea of the input buffer (push case). But I also see some<br class=3D""> problems with it. The restriction on the same buffer size and type in the <br class=3D""> connection policy will be annoying (In my mind I can see the runtime errors pop up). <br class=3D""> It will be hard to track this in bigger systems. Do I get it correctly, that you only<br class=3D""> want to preserve arrival order and runtime here ?<br = class=3D""></div></div></div></blockquote><div><br class=3D""></div>I=92ll= have to disagree here. We have several hundred connections in our = systems, and we track the policy type amongst all.</div><div><br = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div = text=3D"#000000" bgcolor=3D"#FFFFFF" class=3D""><div = class=3D"moz-cite-prefix"> =20 If you are already changing the Connection implementation, I would recommend, to<br class=3D""> put a new thread in every connection in remote case. We experienced the issue, that<br class=3D""> bad Wifi connections, slowed our systems down, as it would hang on the write call.<br = class=3D""></div></div></div></blockquote><div><br class=3D""></div>Not = my first choice. I agree with Sylvain - this should be part of the = transport layer.</div><div><br class=3D""><blockquote type=3D"cite" = class=3D""><div class=3D""><div text=3D"#000000" bgcolor=3D"#FFFFFF" = class=3D""><div class=3D"moz-cite-prefix"> =20 For the Feedback on the write I would recommend to create a second enum, and not<br class=3D""> to use FlowStatus. Sometimes I use switch/case to react to the read FlowStatus. If you<br class=3D""> Introduce new members, this will result in compile warnings, because of unhandled<br class=3D""> entrie.<br class=3D""></div></div></div></blockquote><div><br = class=3D""></div>Yep, backwards compatibility is a concern. = :-(</div><div><br class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D""><div text=3D"#000000" bgcolor=3D"#FFFFFF" class=3D""><div = class=3D"moz-cite-prefix"> =20 For the creation of shared connections, I would recommend an new API. The create call<br class=3D""> should contain all involved members. This spares us the headache of tracking buffer sizes<br class=3D""> and types around in the code. Also this will simplify the code logic, as we don't need the lookup.<br class=3D""> e.g. <br class=3D""> out.connectShared(b.in, c.in, d.in, policy) ;<br = class=3D""></div></div></div></blockquote><div><br class=3D""></div>In = the above are =93b=94, =93c=94, and =93d=94, all ports? If so, what = about cases were not all port connections are made at the same time = (e.g. XML deployments)?</div><div><br = class=3D""></div><div>Cheers</div><div>S</div><div><br = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div = text=3D"#000000" bgcolor=3D"#FFFFFF" class=3D""><div = class=3D"moz-cite-prefix"> Greetings<br class=3D""> Janosch<br class=3D""> <br class=3D""> <br class=3D""> <br class=3D""> Am 21.09.2015 um 15:15 schrieb Johannes Meyer:<br class=3D""> </div> <blockquote = cite=3D"mid:[email protected]= il.com" type=3D"cite" class=3D""> <div dir=3D"ltr" class=3D"">Dear Orocos community, <div class=3D""><br class=3D""> </div> <div class=3D"">during the last few weeks we have been working = on the concept and implementation of a major redesign of the Orocos RTT dataflow, that we would like to share and ask for your valuable feedback. There was already a previous discussion on the mailing list about the topic earlier this year [1], and the following proposal tries to address all the issues mentioned there.</div> <div class=3D""><br class=3D""> </div> <div class=3D"">With the version 2 of the Orocos toolchain some = years ago, the previous data flow architecture, which was based on a single data object and buffer shared between all ports participating at the same connection, was replaced by a connection architecture based on data flow, where each pair of output and input ports (or streams) is connected through an individual channel, each having its own data object or buffer. While there was a well substantiated motivation for this update and it solved some of the shortcomings in RTT version 1 [2], it also introduced some new, sometimes hidden, problems and broke other use cases. One example is an input port with multiple connections, where due to the ambiguity in which channel to read from the result is sometimes unexpected to the user, who expects that each write overwrites previous samples in case of simple data connections.</div> <div class=3D""><br class=3D""> </div> <div class=3D"">We assembled a document (see attachment, or [3]) = which introduces the version 1 and version 2 data flow semantics for those who are not aware of the details, three example use cases and presents the new dataflow architecture we have in mind, which ideally works for a superset of use cases that are supported by RTT version 1 and 2. </div> <div class=3D""><br class=3D""> </div> <div class=3D"">We do not want to go into details here, so only = a short summary for those who do not want to read the full = document:<br class=3D""> </div> <div class=3D"">We propose to add two more boolean flags to the = ConnPolicy struct that describes the type of connection between two ports: "shared" and "mandatory". Furthermore the existing "pull" flag, which was only meaningful for remote connections, now has a big influence for local connections.</div> <div class=3D""><br class=3D""> </div> <div class=3D""><b class=3D"">"push" vs. "pull"</b></div> <div class=3D""><b class=3D""><br class=3D""> </b></div> <div class=3D"">Like it is already the case for remote = connections via CORBA, the "pull" flag will decide whether the connection's data object or buffer is installed at the writer's side (pull) or the reader's side (push) of the connection. The first case is equivalent to the current version 2 connection model and readers have to select the connection/output port to read from, while for push connections there will be only one single buffer per input port. We think that the push connection should be the default connection type for local ports, but this might break existing applications which rely on having one buffer per connection now.</div> <div class=3D""><br class=3D""> </div> <div class=3D""><b class=3D"">"private" vs. "shared"</b></div> <div class=3D""><br class=3D""> </div> <div class=3D"">The shared flag reintroduces the shared data = connection model that was standard in RTT version 1. For shared connections multiple input ports read from the same data object or consume samples from the same buffer instance, so their read operations can influence each other. Shared connections are especially useful for producer/consumer applications with multiple consumers.</div> <div class=3D""><br class=3D""> </div> <div class=3D""><b class=3D"">"mandatory"</b></div> <div class=3D""><br class=3D""> </div> <div class=3D"">The introduction of the "mandatory" flag was = necessary to decide on the return value of the write() operation on an output port. In RTT version 2 it was not possible anymore to get a feedback of whether a write operation was successful or failed, e.g. because of a full buffer or broken remote connection. Only connections with the mandatory flag set would be considered for the overall result of write() operations, while failures of non-mandatory connections will be = ignored.</div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">For those who want to take a look at the current = state of the implementation, this is the link to the working branch on GitHub:</div> <div class=3D"">Tree: <a moz-do-not-send=3D"true" = href=3D"https://github.com/orocos-toolchain/rtt/tree/updated-dataflow-sema= ntics" = class=3D"">https://github.com/orocos-toolchain/rtt/tree/updated-dataflow-s= emantics</a><br class=3D""> </div> <div class=3D"">Diff to master: <a moz-do-not-send=3D"true" = href=3D"https://github.com/orocos-toolchain/rtt/compare/master...updated-d= ataflow-semantics" = class=3D"">https://github.com/orocos-toolchain/rtt/compare/master...update= d-dataflow-semantics</a></div> <div class=3D""><br class=3D""> </div> <div class=3D"">The implementation is not fully working yet for non-standard transports (like Corba and mqueue), but especially the updated test ports_test.cpp already shows what would change from a user perspective.</div> <div class=3D""><br class=3D""> </div> <div class=3D"">Obviously, such a major refactoring cannot be = done without changes in the public API. However, we tried to keep the required changes in user code as minimal as possible.</div> <div class=3D"">The new dataflow semantics are targeted for an = upcoming version 2.9 release (or even name it 3.0 to communicate this potentially breaking change?).</div> <div class=3D""><br class=3D""> </div> <div class=3D"">We are planning to also come up with some = comparative performance measurements during this week, as people have asked for that before [1].</div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">Best regards,<br class=3D""> </div> <div class=3D"">Johannes and Peter</div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">[1] <a moz-do-not-send=3D"true" = href=3D"http://www.orocos.org/forum/rtt/rtt-dev/limitations-v2-port-implem= entation" = class=3D"">http://www.orocos.org/forum/rtt/rtt-dev/limitations-v2-port-imp= lementation</a> </div> <div class=3D"">[2] <a moz-do-not-send=3D"true" = href=3D"http://www.orocos.org/wiki/rtt/rtt-2.0/dataflow" = class=3D"">http://www.orocos.org/wiki/rtt/rtt-2.0/dataflow</a><br = class=3D""> </div> <div class=3D"">[3] <a moz-do-not-send=3D"true" = href=3D"https://docs.google.com/document/d/1zDnPPz4SiCVvfEFxYFUZBcVbXKj33o= 4KaotpCtXM4E0/pub" = class=3D"">https://docs.google.com/document/d/1zDnPPz4SiCVvfEFxYFUZBcVbXKj= 33o4KaotpCtXM4E0/pub</a></div> <div class=3D""><br class=3D""> </div> </div> <br class=3D""> <fieldset class=3D"mimeAttachmentHeader"></fieldset> <br class=3D""> </blockquote> <br class=3D""> <br class=3D""> <pre class=3D"moz-signature" cols=3D"72">--=20 Dipl. Inf. Janosch Machowinski SAR- & Sicherheitsrobotik Universit=E4t Bremen FB 3 - Mathematik und Informatik AG Robotik Robert-Hooke-Stra=DFe 1 28359 Bremen, Germany =20 Zentrale: +49 421 178 45-6611 =20 Besuchsadresse der Nebengesch=E4ftstelle:=20 Robert-Hooke-Stra=DFe 5 28359 Bremen, Germany =20 Tel.: +49 421 178 45-6614 Empfang: +49 421 178 45-6600 Fax: +49 421 178 45-4150 E-Mail: <a class=3D"moz-txt-link-abbreviated" = href=3D"mailto:[email protected]">jmachowinski@informa= tik.uni-bremen.de</a> Weitere Informationen: <a class=3D"moz-txt-link-freetext" = href=3D"http://www.informatik.uni-bremen.de/robotik">http://www.informatik= .uni-bremen.de/robotik</a> </pre> </div> -- <br class=3D"">Orocos-Dev mailing list<br class=3D""><a = href=3D"mailto:[email protected]" = class=3D"">[email protected]</a><br = class=3D"">http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev<br = class=3D""></div></blockquote></div><br class=3D""></body></html>= --Apple-Mail=_4A0C3AA2-EBA0-4533-9FDE-56EAC227572B-- --===============1953219044== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline -- Orocos-Dev mailing list [email protected] http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev --===============1953219044==--