Re: Updated dataflow semantics for RTT
S Roderick <[email protected]> Tue, 22 Sep 2015 18:20:48 -0400
| Newsgroups | gmane.science.robotics.orocos.devel |
|---|---|
| Message-ID | <[email protected]> |
--===============0849785390== Content-type: multipart/alternative; boundary="Apple-Mail=_A4F0EF9D-CE55-4595-AD90-BB3582B192EB" --Apple-Mail=_A4F0EF9D-CE55-4595-AD90-BB3582B192EB Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Sep 22, 2015, at 13:01, Janosch Machowinski = <[email protected]> wrote: >=20 > Am 22.09.2015 um 18:15 schrieb Johannes Meyer: >> Hi, >>=20 >> On Tue, Sep 22, 2015 at 3:32 PM, Janosch Machowinski = <[email protected] <mailto:[email protected]>> = wrote: >> Hi, >> here are my remarks: >> First of all, I like the idea of the input buffer (push case). But I = also see some >> problems with it. The restriction on the same buffer size and type in = the=20 >> connection policy will be annoying (In my mind I can see the runtime = errors pop up).=20 >> It will be hard to track this in bigger systems. Do I get it = correctly, that you only >> want to preserve arrival order and runtime here ? >>=20 >> No, we think the current implementation with one buffer per = connection could really be considered as a bug, at least for certain use = cases. Consider two output ports W1 and W2 which are both connected to = the same input port R with a data connection. What I would expect is: >>=20 >> W1.write("foo") >> W2.write("bar") >> assert(R.read(x) =3D=3D NewData && x =3D=3D "bar") >> assert(R.read(x) =3D=3D OldData && x =3D=3D "bar") >>=20 >> so each writer overwrites the data sample from previous writes (think = of a robot command). But what happens in RTT v2 is: >>=20 >> W1.write("foo") >> W2.write("bar") >> assert(R.read(x) =3D=3D NewData && x =3D=3D "foo") >> assert(R.read(x) =3D=3D NewData && x =3D=3D "bar") >>=20 >> and I cannot even be sure if I first read "foo" or "bar" because it = depends on what connection was read last. The first result would be the = new default case (push), while the second can still be achieved by = setting pull to true, which had no effect until now for local = connections. >>=20 >> Certainly there is a significant chance that the new restrictions = break existing deployments and that's why we would only release it with = a new minor, if not even major version number. But to fix this serious = drawback of the current dataflow implementation it is certainly worth = the effort. >=20 > What you actually want here is a paradigm change. The buffer of the = input port should not be defined > any more by the connection, but by the input port. Using this thought, = one could go further and=20 > infer the connection policy (buffer size / type) automatically.=20 I will have to disagree here. Sylvain=E2=80=99s got it right - it is a = system design issue to deal with connections. I do not want any thing = inferring what type of buffer policy/size/type someone else thinks my = application needs. No one but the system design of any given system has = enough knowledge to make those decisions. > Ahh, I just realized, what is odd about your example. As a design = principle, a component does not know anything > about the connections that are attached to the port. So, if you are a = robot controller and want to always use=20 > the latest sample, you MUST use R.readNewest(x). You just may not rely = on the fact that the outside world > used the correct connection. Actually in this case you get the = expected behaviour, if the data order is not broken. Actually, you don=E2=80=99t need to use readNewest(). The problem is = that systems designed with v1 port semantics in mind that use v2 ports = end up coupling the component with the type of connection, as they have = to work around the problem that Johannes so aptly described above (and = that is similar to a problem posted on the ML some time ago). >> I agree with Sylvain here: It is the transport's responsibility to = not block on write or read operations and if the underlying = implementation cannot guarantee that, to use single worker thread, = thread pool or one thread per connection. If this would be enforced by = the RTT connection factory already, it would be impossible to use = real-time capable transports like Xenomai message queues. >>=20 >> We indeed also have an updated CORBA transport ready which introduces = oneway calls wherever possible and might solve some of the problems you = observed. It could be considered as stable and I could prepare another = pull request, but of course there would be plenty of conflicts with the = updated-dataflow-semantics branch, so I would prefer to postpone this = until this one is merged. > Hm, this one has been bugging us for a long time... If it is stable, = we should perhaps merge it before the big > data flow change. It is basically a bugfix to the last version with = the old dataflow... >> =20 >>=20 >> For the creation of shared connections, I would recommend an new API. = The create call >> should contain all involved members. This spares us the headache of = tracking buffer sizes >> and types around in the code. Also this will simplify the code logic, = as we don't need the lookup. >> e.g.=20 >> out.connectShared(b.in <http://b.in/>, c.in <http://c.in/>, d.in = <http://d.in/>, policy) ; >>=20 >> A new API instead of adding the shared flag to the ConnPolicy? > Yes Could you state the goal of the existing API, and this new API, in a way = that they are orthogonal and not overlapping? >> Or only as a thin wrapper which implies that the shared flag is set? >> It should be noted that it is not the new concept of shared = connections that adds constraints, but especially the per-input port = buffers implied by the default pull =3D false setting, even for private = connections. What you probably mean is to introduce a separate API for = all new dataflow models and make sure that the existing API calls behave = the same as before. > If I got I right, one could still create a shared output connection = together with a (second) normal buffered connection. > The new API would just make sure that you don't get annoying runtime = errors, like non matching policies etc in the=20 > shared case. Using the old api for the shared case is just a bit to = implicit and error prone for my taste. I think that Johannes is trying to graft a fix for some of the existing = implementation issues on top of the existing implementation. Using a = second API would cause two APIs to exist, which would simply confuse = users. There=E2=80=99d also be the question of what is the future for = either API. I think that having one API that covers both cases, and = coping gracefully with backwards compatibility for a period of time, is = a better long term solution. Jumping from one to the other (ala v2.0) or = having two parallel APIs exist along side each other, is likely to lead = to a lot of confusion and trouble. YMMV S --Apple-Mail=_A4F0EF9D-CE55-4595-AD90-BB3582B192EB Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On Sep 22, 2015, at 13:01, Janosch Machowinski <<a = href=3D"mailto:[email protected]" = class=3D"">[email protected]</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""> =20 <meta content=3D"text/html; charset=3Dutf-8" = http-equiv=3D"Content-Type" class=3D""> =20 <div text=3D"#000000" bgcolor=3D"#FFFFFF" class=3D""> <div class=3D"moz-cite-prefix">Am 22.09.2015 um 18:15 schrieb = Johannes Meyer:<br class=3D""> </div> <blockquote = cite=3D"mid:[email protected]= .com" type=3D"cite" class=3D""> <div dir=3D"ltr" class=3D""> <div class=3D"gmail_extra">Hi,</div> <div class=3D"gmail_extra"><br class=3D""> <div class=3D"gmail_quote">On Tue, Sep 22, 2015 at 3:32 PM, Janosch Machowinski <span dir=3D"ltr" class=3D""><<a = moz-do-not-send=3D"true" href=3D"mailto:[email protected]" = target=3D"_blank" class=3D"">[email protected]</a>></span> wrote:<br class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px = 0px = 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left= -style:solid;padding-left:1ex"> <div text=3D"#000000" bgcolor=3D"#FFFFFF" class=3D""> <div class=3D"">Hi,<br class=3D""> here are my remarks:<br class=3D""> First of all, I like the idea of the input buffer (push case). But I also see some<br class=3D""> problems with it. The restriction on the same buffer size and type in the <br class=3D""> connection policy will be annoying (In my mind I can see the runtime errors pop up). <br class=3D""> It will be hard to track this in bigger systems. Do I get it correctly, that you only<br class=3D""> want to preserve arrival order and runtime here ?<br = class=3D""> </div> </div> </blockquote> <div class=3D""><br class=3D""> </div> <div class=3D"">No, we think the current implementation with = one buffer per connection could really be considered as a bug, at least for certain use cases. Consider two output ports W1 and W2 which are both connected to the same input port R with a data connection. What I would expect is:</div> <div class=3D""><br class=3D""> </div> <div class=3D"">W1.write("foo")</div> <div class=3D"">W2.write("bar")</div> <div class=3D"">assert(R.read(x) =3D=3D NewData && x = =3D=3D "bar")</div> <div class=3D"">assert(R.read(x) =3D=3D = OldData && x =3D=3D "bar")</div> <div class=3D""><br class=3D""> </div> <div class=3D"">so each writer overwrites the data sample = from previous writes (think of a robot command). But what happens in RTT v2 is:</div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div class=3D"">W1.write("foo")</div> <div class=3D"">W2.write("bar")</div> <div class=3D"">assert(R.read(x) =3D=3D NewData && = x =3D=3D "foo")</div> <div class=3D"">assert(R.read(x) =3D=3D NewData && = x =3D=3D "bar")</div> <div class=3D""><br class=3D""> </div> <div class=3D"">and I cannot even be sure if I first read = "foo" or "bar" because it depends on what connection was read last. The first result would be the new default case (push), while the second can still be achieved by setting pull to true, which had no effect until now for local connections.</div> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">Certainly there is a significant chance that = the new restrictions break existing deployments and that's why we would only release it with a new minor, if not even major version number. But to fix this serious drawback of the current dataflow implementation it is certainly worth the effort.</div> </div> </div> </div> </blockquote> <br class=3D""> What you actually want here is a paradigm change. The buffer of the input port should not be defined<br class=3D""> any more by the connection, but by the input port. Using this thought, one could go further and <br class=3D""> infer the connection policy (buffer size / type) automatically. <br = class=3D""></div></div></blockquote><div><br class=3D""></div>I will = have to disagree here. Sylvain=E2=80=99s got it right - it is a system = design issue to deal with connections. I do not want any thing inferring = what type of buffer policy/size/type someone else thinks my application = needs. No one but the system design of any given system has enough = knowledge to make those decisions.</div><div><br class=3D""><blockquote = type=3D"cite" class=3D""><div class=3D""><div text=3D"#000000" = bgcolor=3D"#FFFFFF" class=3D""> =20 Ahh, I just realized, what is odd about your example. As a design principle, a component does not know anything<br class=3D""> about the connections that are attached to the port. So, if you are a robot controller and want to always use <br class=3D""> the latest sample, you MUST use R.readNewest(x). You just may not rely on the fact that the outside world<br class=3D""> used the correct connection. Actually in this case you get the expected behaviour, if the data order is not broken. </div></div></blockquote><div><br class=3D""></div>Actually, you = don=E2=80=99t need to use readNewest(). The problem is that systems = designed with v1 port semantics in mind that use v2 ports end up = coupling the component with the type of connection, as they have to work = around the problem that Johannes so aptly described above (and that is = similar to a problem posted on the ML some time ago).</div><div><br = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div = text=3D"#000000" bgcolor=3D"#FFFFFF" class=3D""><blockquote = cite=3D"mid:[email protected]= .com" type=3D"cite" class=3D""> <div dir=3D"ltr" class=3D""> <div class=3D"gmail_extra"> <div class=3D"gmail_quote"> =20 <div class=3D"">I agree with Sylvain here: It is the = transport's responsibility to not block on write or read operations and if the underlying implementation cannot guarantee that, to use single worker thread, thread pool or one thread per connection. If this would be enforced by the RTT connection factory already, it would be impossible to use real-time capable transports like Xenomai message queues.</div> <div class=3D""><br class=3D""> </div> <div class=3D"">We indeed also have an updated CORBA = transport ready which introduces oneway calls wherever possible and might solve some of the problems you observed. It could be considered as stable and I could prepare another pull request, but of course there would be plenty of conflicts with the updated-dataflow-semantics branch, so I would prefer to postpone this until this one is merged.</div> </div> </div> </div> </blockquote> Hm, this one has been bugging us for a long time... If it is stable, we should perhaps merge it before the big<br class=3D""> data flow change. It is basically a bugfix to the last version with the old dataflow...<br class=3D""> <blockquote = cite=3D"mid:[email protected]= .com" type=3D"cite" class=3D""> <div dir=3D"ltr" class=3D""> <div class=3D"gmail_extra"> <div class=3D"gmail_quote"> <div class=3D""> </div> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px = 0px = 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left= -style:solid;padding-left:1ex"> <div text=3D"#000000" bgcolor=3D"#FFFFFF" class=3D""> <div class=3D""> <br class=3D""> For the creation of shared connections, I would recommend an new API. The create call<br class=3D""> should contain all involved members. This spares us the headache of tracking buffer sizes<br class=3D""> and types around in the code. Also this will simplify the code logic, as we don't need the lookup.<br = class=3D""> e.g. <br class=3D""> out.connectShared(<a moz-do-not-send=3D"true" = href=3D"http://b.in/" target=3D"_blank" class=3D"">b.in</a>, <a = moz-do-not-send=3D"true" href=3D"http://c.in/" target=3D"_blank" = class=3D"">c.in</a>, <a moz-do-not-send=3D"true" href=3D"http://d.in/" = target=3D"_blank" class=3D"">d.in</a>, policy) ;<br class=3D""> </div> </div> </blockquote> <div class=3D""><br class=3D""> </div> <div class=3D"">A new API instead of adding the shared flag = to the ConnPolicy?</div> </div> </div> </div> </blockquote> Yes<br class=3D""></div></div></blockquote><div><br = class=3D""></div>Could you state the goal of the existing API, and this = new API, in a way that they are orthogonal and not = overlapping?</div><div><br class=3D""><blockquote type=3D"cite" = class=3D""><div class=3D""><div text=3D"#000000" bgcolor=3D"#FFFFFF" = class=3D""> <blockquote = cite=3D"mid:[email protected]= .com" type=3D"cite" class=3D""> <div dir=3D"ltr" class=3D""> <div class=3D"gmail_extra"> <div class=3D"gmail_quote"> <div class=3D""> Or only as a thin wrapper which implies = that the shared flag is set?</div> <div class=3D"">It should be noted that it is not the new = concept of shared connections that adds constraints, but especially the per-input port buffers implied by the default pull =3D false setting, even for private connections. What you probably mean is to introduce a separate API for all new dataflow models and make sure that the existing API calls behave the same as before.</div> </div> </div> </div> </blockquote> If I got I right, one could still create a shared output connection together with a (second) normal buffered connection.<br class=3D""> The new API would just make sure that you don't get annoying runtime errors, like non matching policies etc in the <br class=3D""> shared case. Using the old api for the shared case is just a bit to implicit and error prone for my taste.<br = class=3D""></div></div></blockquote><div><br class=3D""></div>I think = that Johannes is trying to graft a fix for some of the existing = implementation issues on top of the existing implementation. Using a = second API would cause two APIs to exist, which would simply confuse = users. There=E2=80=99d also be the question of what is the future for = either API. I think that having one API that covers both cases, and = coping gracefully with backwards compatibility for a period of time, is = a better long term solution. Jumping from one to the other (ala v2.0) or = having two parallel APIs exist along side each other, is likely to lead = to a lot of confusion and trouble.</div><div><br = class=3D""></div><div>YMMV</div><div>S</div><div><br = class=3D""></div></body></html>= --Apple-Mail=_A4F0EF9D-CE55-4595-AD90-BB3582B192EB-- --===============0849785390== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline -- Orocos-Dev mailing list [email protected] http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev --===============0849785390==--