Re: Updated dataflow semantics for RTT
Janosch Machowinski <[email protected]> Tue, 22 Sep 2015 19:01:07 +0200
| Newsgroups | gmane.science.robotics.orocos.devel |
|---|---|
| Message-ID | <[email protected]> |
This is a multi-part message in MIME format. --===============0178181149== Content-Type: multipart/alternative; boundary="------------080100000302090708040807" This is a multi-part message in MIME format. --------------080100000302090708040807 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Am 22.09.2015 um 18:15 schrieb Johannes Meyer: > Hi, > > On Tue, Sep 22, 2015 at 3:32 PM, Janosch Machowinski=20 > <[email protected] <mailto:[email protected]>> wrot= e: > > Hi, > here are my remarks: > First of all, I like the idea of the input buffer (push case). But > I also see some > problems with it. The restriction on the same buffer size and type > in the > connection policy will be annoying (In my mind I can see the > runtime errors pop up). > It will be hard to track this in bigger systems. Do I get it > correctly, that you only > want to preserve arrival order and runtime here ? > > > No, we think the current implementation with one buffer per connection=20 > could really be considered as a bug, at least for certain use cases.=20 > Consider two output ports W1 and W2 which are both connected to the=20 > same input port R with a data connection. What I would expect is: > > W1.write("foo") > W2.write("bar") > assert(R.read(x) =3D=3D NewData && x =3D=3D "bar") > assert(R.read(x) =3D=3D OldData && x =3D=3D "bar") > > so each writer overwrites the data sample from previous writes (think=20 > of a robot command). But what happens in RTT v2 is: > > W1.write("foo") > W2.write("bar") > assert(R.read(x) =3D=3D NewData && x =3D=3D "foo") > assert(R.read(x) =3D=3D NewData && x =3D=3D "bar") > > and I cannot even be sure if I first read "foo" or "bar" because it=20 > depends on what connection was read last. The first result would be=20 > the new default case (push), while the second can still be achieved by=20 > setting pull to true, which had no effect until now for local connectio= ns. > > Certainly there is a significant chance that the new restrictions=20 > break existing deployments and that's why we would only release it=20 > with a new minor, if not even major version number. But to fix this=20 > serious drawback of the current dataflow implementation it is=20 > certainly worth the effort. What you actually want here is a paradigm change. The buffer of the=20 input port should not be defined any more by the connection, but by the input port. Using this thought,=20 one could go further and infer the connection policy (buffer size / type) automatically. Ahh, I just realized, what is odd about your example. As a design=20 principle, a component does not know anything about the connections that are attached to the port. So, if you are a=20 robot controller and want to always use the latest sample, you MUST use R.readNewest(x). You just may not rely=20 on the fact that the outside world used the correct connection. Actually in this case you get the expected=20 behaviour, if the data order is not broken. > > I agree with Sylvain here: It is the transport's responsibility to not=20 > block on write or read operations and if the underlying implementation=20 > cannot guarantee that, to use single worker thread, thread pool or one=20 > thread per connection. If this would be enforced by the RTT connection=20 > factory already, it would be impossible to use real-time capable=20 > transports like Xenomai message queues. > > We indeed also have an updated CORBA transport ready which introduces=20 > oneway calls wherever possible and might solve some of the problems=20 > you observed. It could be considered as stable and I could prepare=20 > another pull request, but of course there would be plenty of conflicts=20 > with the updated-dataflow-semantics branch, so I would prefer to=20 > postpone this until this one is merged. Hm, this one has been bugging us for a long time... If it is stable, we=20 should perhaps merge it before the big data flow change. It is basically a bugfix to the last version with the=20 old dataflow... > > > For the creation of shared connections, I would recommend an new > API. The create call > should contain all involved members. This spares us the headache > of tracking buffer sizes > and types around in the code. Also this will simplify the code > logic, as we don't need the lookup. > e.g. > out.connectShared(b.in <http://b.in>, c.in <http://c.in>, d.in > <http://d.in>, policy) ; > > > A new API instead of adding the shared flag to the ConnPolicy? Yes > Or only as a thin wrapper which implies that the shared flag is set? > It should be noted that it is not the new concept of shared=20 > connections that adds constraints, but especially the per-input port=20 > buffers implied by the default pull =3D false setting, even for private= =20 > connections. What you probably mean is to introduce a separate API for=20 > all new dataflow models and make sure that the existing API calls=20 > behave the same as before. If I got I right, one could still create a shared output connection=20 together with a (second) normal buffered connection. The new API would just make sure that you don't get annoying runtime=20 errors, like non matching policies etc in the shared case. Using the old api for the shared case is just a bit to=20 implicit and error prone for my taste. Janosch --=20 Dipl. Inf. Janosch Machowinski SAR- & Sicherheitsrobotik Universit=C3=A4t Bremen FB 3 - Mathematik und Informatik AG Robotik Robert-Hooke-Stra=C3=9Fe 1 28359 Bremen, Germany =20 Zentrale: +49 421 178 45-6611 =20 Besuchsadresse der Nebengesch=C3=A4ftstelle: Robert-Hooke-Stra=C3=9Fe 5 28359 Bremen, Germany =20 Tel.: +49 421 178 45-6614 Empfang: +49 421 178 45-6600 Fax: +49 421 178 45-4150 E-Mail: [email protected] Weitere Informationen: http://www.informatik.uni-bremen.de/robotik --------------080100000302090708040807 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html> <head> <meta content=3D"text/html; charset=3Dutf-8" http-equiv=3D"Content-Ty= pe"> </head> <body text=3D"#000000" bgcolor=3D"#FFFFFF"> <div class=3D"moz-cite-prefix">Am 22.09.2015 um 18:15 schrieb Johanne= s Meyer:<br> </div> <blockquote cite=3D"mid:[email protected]= l.com" type=3D"cite"> <div dir=3D"ltr"> <div class=3D"gmail_extra">Hi,</div> <div class=3D"gmail_extra"><br> <div class=3D"gmail_quote">On Tue, Sep 22, 2015 at 3:32 PM, Janosch Machowinski <span dir=3D"ltr"><<a moz-do-not-send=3D"true" href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>></sp= an> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-lef= t-style:solid;padding-left:1ex"> <div text=3D"#000000" bgcolor=3D"#FFFFFF"> <div>Hi,<br> here are my remarks:<br> First of all, I like the idea of the input buffer (push case). But I also see some<br> problems with it. The restriction on the same buffer size and type in the <br> connection policy will be annoying (In my mind I can see the runtime errors pop up). <br> It will be hard to track this in bigger systems. Do I get it correctly, that you only<br> want to preserve arrival order and runtime here ?<br> </div> </div> </blockquote> <div><br> </div> <div>No, we think the current implementation with one buffer per connection could really be considered as a bug, at least for certain use cases. Consider two output ports W1 and W2 which are both connected to the same input port R with a data connection. What I would expect is:</div> <div><br> </div> <div>W1.write("foo")</div> <div>W2.write("bar")</div> <div>assert(R.read(x) =3D=3D NewData && x =3D=3D "bar= ")</div> <div>assert(R.read(x) =3D=3D OldData=C2=A0&& x =3D=3D= "bar")</div> <div><br> </div> <div>so each writer overwrites the data sample from previous writes (think of a robot command). But what happens in RTT v2 is:</div> <div><br> </div> <div> <div>W1.write("foo")</div> <div>W2.write("bar")</div> <div>assert(R.read(x) =3D=3D NewData && x =3D=3D "f= oo")</div> <div>assert(R.read(x) =3D=3D NewData && x =3D=3D "b= ar")</div> <div><br> </div> <div>and I cannot even be sure if I first read "foo" or "bar" because it depends on what connection was read last. The first result would be the new default case (push), while the second can still be achieved by setting pull to true, which had no effect until now for local connections.</div> </div> <div><br> </div> <div>Certainly there is a significant chance that the new restrictions break existing deployments and that's why we would only release it with a new minor, if not even major version number. But to fix this serious drawback of the current dataflow implementation it is certainly worth the effort.</div> </div> </div> </div> </blockquote> <br> What you actually want here is a paradigm change. The buffer of the input port should not be defined<br> any more by the connection, but by the input port. Using this thought, one could go further and <br> infer the connection policy (buffer size / type) automatically. <br> <br> Ahh, I just realized, what is odd about your example. As a design principle, a component does not know anything<br> about the connections that are attached to the port. So, if you are a robot controller and want to always use <br> the latest sample, you MUST use R.readNewest(x). You just may not rely on the fact that the outside world<br> used the correct connection. Actually in this case you get the expected behaviour, if the data order is not broken. <blockquote cite=3D"mid:[email protected]= l.com" type=3D"cite"> <div dir=3D"ltr"> <div class=3D"gmail_extra"> <div class=3D"gmail_quote"> <div><br> </div> <div>I agree with Sylvain here: It is the transport's responsibility to not block on write or read operations and if the underlying implementation cannot guarantee that, to use single worker thread, thread pool or one thread per connection. If this would be enforced by the RTT connection factory already, it would be impossible to use real-time capable transports like Xenomai message queues.</div> <div><br> </div> <div>We indeed also have an updated CORBA transport ready which introduces oneway calls wherever possible and might solve some of the problems you observed. It could be considered as stable and I could prepare another pull request, but of course there would be plenty of conflicts with the updated-dataflow-semantics branch, so I would prefer to postpone this until this one is merged.</div> </div> </div> </div> </blockquote> Hm, this one has been bugging us for a long time... If it is stable, we should perhaps merge it before the big<br> data flow change. It is basically a bugfix to the last version with the old dataflow...<br> <blockquote cite=3D"mid:[email protected]= l.com" type=3D"cite"> <div dir=3D"ltr"> <div class=3D"gmail_extra"> <div class=3D"gmail_quote"> <div>=C2=A0</div> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-lef= t-style:solid;padding-left:1ex"> <div text=3D"#000000" bgcolor=3D"#FFFFFF"> <div> <br> For the creation of shared connections, I would recommend an new API. The create call<br> should contain all involved members. This spares us the headache of tracking buffer sizes<br> and types around in the code. Also this will simplify the code logic, as we don't need the lookup.<br> e.g. <br> out.connectShared(<a moz-do-not-send=3D"true" href=3D"http://b.in" target=3D"_blank">b.in</a>, <a moz-do-not-send=3D"true" href=3D"http://c.in" target=3D"_blank">c.in</a>, <a moz-do-not-send=3D"tru= e" href=3D"http://d.in" target=3D"_blank">d.in</a>, poli= cy) ;<br> </div> </div> </blockquote> <div><br> </div> <div>A new API instead of adding the shared flag to the ConnPolicy?</div> </div> </div> </div> </blockquote> Yes<br> <blockquote cite=3D"mid:[email protected]= l.com" type=3D"cite"> <div dir=3D"ltr"> <div class=3D"gmail_extra"> <div class=3D"gmail_quote"> <div> Or only as a thin wrapper which implies that the shared flag is set?</div> <div>It should be noted that it is not the new concept of shared connections that adds constraints, but especially the per-input port buffers implied by the default pull =3D false setting, even for private connections. What you probably mean is to introduce a separate API for all new dataflow models and make sure that the existing API calls behave the same as before.</div> </div> </div> </div> </blockquote> If I got I right, one could still create a shared output connection together with a (second) normal buffered connection.<br> The new API would just make sure that you don't get annoying runtime errors, like non matching policies etc in the <br> shared case. Using the old api for the shared case is just a bit to implicit and error prone for my taste.<br> =C2=A0=C2=A0=C2=A0 Janosch<br> <pre class=3D"moz-signature" cols=3D"72">--=20 Dipl. Inf. Janosch Machowinski SAR- & Sicherheitsrobotik Universit=C3=A4t Bremen FB 3 - Mathematik und Informatik AG Robotik Robert-Hooke-Stra=C3=9Fe 1 28359 Bremen, Germany =20 Zentrale: +49 421 178 45-6611 =20 Besuchsadresse der Nebengesch=C3=A4ftstelle:=20 Robert-Hooke-Stra=C3=9Fe 5 28359 Bremen, Germany =20 Tel.: +49 421 178 45-6614 Empfang: +49 421 178 45-6600 Fax: +49 421 178 45-4150 E-Mail: <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:jmachowins= [email protected]">[email protected]</a> Weitere Informationen: <a class=3D"moz-txt-link-freetext" href=3D"http:/= /www.informatik.uni-bremen.de/robotik">http://www.informatik.uni-bremen.d= e/robotik</a> </pre> </body> </html> --------------080100000302090708040807-- --===============0178181149== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline -- Orocos-Dev mailing list [email protected] http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev --===============0178181149==--