Re: Updated dataflow semantics for RTT

--===============0451313715==
Content-type: multipart/alternative;
	boundary="Boundary_(ID_DAUzl2H65HRHVNbCMLgcoA)"


--Boundary_(ID_DAUzl2H65HRHVNbCMLgcoA)
Content-type: text/plain; charset=utf-8; format=flowed
Content-transfer-encoding: quoted-printable

On Sep 28, 2015, at 08:31 AM, Sylvain Joyeux <[email protected]> wrot=
e:=0A=0A(@Stephen: re-sending, mistakenly sent to you only instead of all =
the=0Ainvolved parties)=0A=0AEnforcing in order arrive on the reading side=
 is one way of doing it. You=0Acan also do it at a system level if you cho=
se. They're equally valid use=0Acases.=0A=0ACan you describe a use case wh=
ere "order of arrival" is good enough=0Afor you ? That would help me under=
stand your use-case.=0A=0A(For some clearer thoughts on this,=0Ahttp://roc=
k-robotics.org/master/documentation/data_processing/data.html=0A=C2=A0=0AI=
 agree with much at the above, and we have similar problems that we've sol=
ved in similar ways. But there are also huge parts of our system that, whi=
le "data driven", aren't at all interested in time. The system itself is s=
ynchronized and time is (largely) irrelevant - everything operates on a pe=
r-cycle basis. In the long run, we time stamp only a tiny portion of the d=
ata in our system due to this.=C2=A0Equally valid, but different.=C2=A0=0A=
=0Ahttp://rock-robotics.org/master/documentation/data_processing/stream_al=
igner.html)=0A=0AExcept for small parts of our system/application, we don'=
t need to worry about this.=C2=A0=0A=0Aorogen only deals with single compo=
nents. Syskit is the one inmpacted by=0Athis change, and it is for me out =
of the picture to compute all possible=0Aconnections in all possible netwo=
rks. Pull connections are not practical as=0Asoon as you have a inter-host=
 connection (i.e. remote connection) as they=0Abreak the reading part.=0A=0A=
Can you explain this last part a little more Sylvain?=0A=0AReading on remo=
te+pull CORBA connections requires the reader to do=0Anetwork access (it's=
 actually the whole use-case of pull for us, that=0Athe writer is not at a=
ll affected by the connection)=0A=C2=A0=0ASometimes we want to control whe=
ther the network access (and hence delay/non-real-time part) is on the wri=
ter or reader side. That should be an application-level choice.=0A=0AIsn't=
 the current policy per connection, and not per port? If you have 2=0Aoutp=
ut ports and 3 input ports, and you connect them all together, then you=0A=
have up to 2*3=3D6 possible connections/policies. Right?=0A=0AIf you put t=
he buffer per-port, you end up with one policy per port.=0APort-driven tri=
ggering (event ports) are governed by the data object=0A(i.e. per-port), d=
ata object size and type becomes per-port as well.=0AWhat's left in the co=
nnection policy is the info necessary to reach=0Athe other side (i.e. the =
topic name in ROS), which means that it is=0Anot a policy anymore but a "r=
emote endpoint"=0A=C2=A0=0AWhat if you don't use event ports?=C2=A0=0A=0AT=
his issue for us boils down to the ability to selectively remove the conne=
ction-object-per-input-output-port-pair issue. As a system designer I want=
 to be able to=0A- specify one connection object between multiple input/ou=
tput ports (1-N or N-1 or N-N)=0A- specify where the connection object liv=
es (e.g. to cope with time delay at the system level)=0A=0ANote the "selec=
tively remove" above - we still want to be able to use v2-flow-status type=
 setups within our system.=0A=0AInterestingly, you'll have a hard time fig=
uring out how to configure=0Athe CORBA dispatcher (which currently uses th=
e same policy than the=0Awhole connection).=0A=C2=A0=0AWe have several hun=
dred local connections in our deployers. We have maybe less than 10 remote=
 connections.=0A=0AWhat do you mean by "dynamic" networks?=0A=0AThat are n=
ot established once, but are modified at runtime (i.e.=0Aconnections are c=
reated/removed at runtime). With syskit, that happens=0Aoften and computin=
g the network necessary to run them all is in=0Aprinciple feasible, but in=
 practice useless -- at the least in the=0Adevelopment phases.=0A=C2=A0=0A=
Got it. Thanks.=0A=0A> Buffer connection: read() returns all samples recei=
ved by the input port=0A> (apart from buffer size limitations).=0A=0AWell,=
 I'd phrase this differently. A buffer allows more than one sample to=0Aex=
ist, in an ordered form, between a writer and a reader. "All" samples can=0A=
only be "guaranteed" with very particular buffer sizing or additional=0Aap=
plication semantics.=0A=0A+1, closer to the current semantic.=0A=C2=A0=0AB=
uffers vs data connections definitely change the flavor of things. The vas=
t majority of our connections are data connections, not buffers. We typica=
lly only use buffers in the traditional producer/consumer type situation -=
 usually to cope with processing (e.g. real-time-ness) differences between=
 producer and consumer (e.g. sample something on one side and write it to =
disk on the other).=0A=0A=0A=0AI think that we're getting off topic as a w=
hole. Johannes proposed some modifications to cope with some known problem=
s with the existing implementation that directly affect some of the commun=
ity. There doesn't seem to be any dissent that something needs to be done =
about certain issues with using the current implementation. Does anyone ha=
ve a concrete alternative to Johannes current approach? Certainly we can d=
iscuss how to best phase in (i.e. transition) the proposed changes, or som=
ething similar to them.=0A=0AJohannes' proposed changes are focussed on=0A=
a) push vs pull, or specifying whether the data/buffer object is on the re=
ader or writer side. This is very useful for system designers dealing with=
 significant time delay.=C2=A0=0Ab) private vs shared, whether a connectio=
n is per input/output port pair or whether the connection is shared betwee=
n multiple input/output ports.=0Ac) mandatory, primarily intended to suppo=
rt knowing whether writes to a buffer succeeded (so that traditional produ=
cer/consumer relationships have some knowledge about the buffer state)=0A=0A=
It appears to me that most people are worried about changing the defaults,=
 more than they're worried about the actual proposed changes. Actual techn=
ical concerns with the proposed changes appear to be=0A=0A1) whether to en=
code the result of a write() in existing enums or not=0A2) how to add the =
"shared" concept within the current API, or to use a "new" API=0A3) changi=
ng the RTT defaults, which then affect an application's semantics.=0A=0ADi=
d I miss any technical issues?=0A=0A=0AMy huge personal caveat to all of t=
his is that I do *not* think we should change the current system semantics=
 in introducing this change. Too many Orocos changes have broken backwards=
 compatibility and forced system designers and implementors to change thei=
r system in response. I don't think we should force that on users without =
an incredibly good reason. Otherwise it breeds frustration in the communit=
y.=0A=0AIf there is some way to introduce the proposed changes to the syst=
em, but to not change the behavior of current systems, then I think that i=
s a win for all. There does seem to be some general agreement that there a=
re valid use cases that the current port implementation handles poorly, an=
d that this proposed approach does fix some of those. The proposed approac=
h also improves the situation for systems that don't want or need to use t=
he flow status approach. Both are equally valid approaches - IMHO it would=
 be beneficial to the community as a whole if RTT supported both approache=
s.=0A=0ACheers=0AStephen=0A=0A=

--Boundary_(ID_DAUzl2H65HRHVNbCMLgcoA)
Content-type: multipart/related;
	boundary="Boundary_(ID_6+JZUwdzFzaMsEa2rSFMrw)"; type="text/html"


--Boundary_(ID_6+JZUwdzFzaMsEa2rSFMrw)
Content-type: text/html; CHARSET=US-ASCII
Content-transfer-encoding: quoted-printable

<html><body><div><span style=3D"line-height: 1.5;">On Sep 28, 2015, at 08:31 AM, Sylva=
in Joyeux &lt;[email protected]&gt; wrote:</span></div><div><br></div=
><div><blockquote type=3D"cite"><div class=3D"msg-quote"><div class=3D"_st=
retch"><span class=3D"body-text-content"><span class=3D"body-text-content"=
>(@Stephen: re-sending, mistakenly sent to you only instead of all the<br>=
involved parties)<br><br></span></span><blockquote class=3D"quoted-plain-t=
ext" type=3D"cite">Enforcing in order arrive on the reading side is one wa=
y of doing it. You</blockquote><blockquote class=3D"quoted-plain-text" typ=
e=3D"cite">can also do it at a system level if you chose. They're equally =
valid use</blockquote><blockquote class=3D"quoted-plain-text" type=3D"cite=
">cases.</blockquote><span class=3D"body-text-content"><span class=3D"body=
-text-content"><br>Can you describe a use case where "order of arrival" is=
 good enough<br>for you ? That would help me understand your use-case.<br>=
<br>(For some clearer thoughts on this,<br> <a href=3D"http://rock-robotic=
s.org/master/documentation/data_processing/data.html">http://rock-robotics=
.org/master/documentation/data_processing/data.html</a></span></span></div=
></div></blockquote><span>&nbsp;</span></div><div>I agree with much at the=
 above, and we have similar problems that we've solved in similar ways. Bu=
t there are also huge parts of our system that, while "data driven", aren'=
t at all interested in time. The system itself is synchronized and time is=
 (largely) irrelevant - everything operates on a per-cycle basis. In the l=
ong run, we time stamp only a tiny portion of the data in our system due t=
o this.&nbsp;<span style=3D"line-height: 1.5;">Equally valid, but differen=
t.</span><span style=3D"line-height: 1.5;">&nbsp;</span></div><div><br></d=
iv><div><blockquote type=3D"cite"><div class=3D"msg-quote"><div class=3D"_=
stretch"><span class=3D"body-text-content"><span class=3D"body-text-conten=
t"><a href=3D"http://rock-robotics.org/master/documentation/data_processin=
g/data.html"></a><a href=3D"http://rock-robotics.org/master/documentation/=
data_processing/data.html" data-mce-href=3D"http://rock-robotics.org/maste=
r/documentation/data_processing/data.html"></a> <a href=3D"http://rock-rob=
otics.org/master/documentation/data_processing/stream_aligner.html" data-m=
ce-href=3D"http://rock-robotics.org/master/documentation/data_processing/s=
tream_aligner.html">http://rock-robotics.org/master/documentation/data_pro=
cessing/stream_aligner.html</a>)</span></span></div></div></blockquote><br=
></div><div>Except for small parts of our system/application, we don't nee=
d to worry about this.&nbsp;</div><div><br></div><div><blockquote type=3D"=
cite"><div class=3D"msg-quote"><div class=3D"_stretch"><blockquote class=3D=
"quoted-plain-text" type=3D"cite"><blockquote class=3D"quoted-plain-text" =
type=3D"cite">orogen only deals with single components. Syskit is the one =
inmpacted by</blockquote></blockquote><blockquote class=3D"quoted-plain-te=
xt" type=3D"cite"><blockquote class=3D"quoted-plain-text" type=3D"cite">th=
is change, and it is for me out of the picture to compute all possible</bl=
ockquote></blockquote><blockquote class=3D"quoted-plain-text" type=3D"cite=
"><blockquote class=3D"quoted-plain-text" type=3D"cite">connections in all=
 possible networks. Pull connections are not practical as</blockquote></bl=
ockquote><blockquote class=3D"quoted-plain-text" type=3D"cite"><blockquote=
 class=3D"quoted-plain-text" type=3D"cite">soon as you have a inter-host c=
onnection (i.e. remote connection) as they</blockquote></blockquote><block=
quote class=3D"quoted-plain-text" type=3D"cite"><blockquote class=3D"quote=
d-plain-text" type=3D"cite">break the reading part.</blockquote></blockquo=
te><blockquote class=3D"quoted-plain-text" type=3D"cite"><br></blockquote>=
<blockquote class=3D"quoted-plain-text" type=3D"cite">Can you explain this=
 last part a little more Sylvain?</blockquote><span class=3D"body-text-con=
tent"><span class=3D"body-text-content"><br>Reading on remote+pull CORBA c=
onnections requires the reader to do<br>network access (it's actually the =
whole use-case of pull for us, that<br>the writer is not at all affected b=
y the connection)</span></span></div></div></blockquote><span>&nbsp;</span=
></div><div>Sometimes we want to control whether the network access (and h=
ence delay/non-real-time part) is on the writer or reader side. That shoul=
d be an application-level choice.</div><div><br></div><div><blockquote typ=
e=3D"cite"><div class=3D"msg-quote"><div class=3D"_stretch"><blockquote cl=
ass=3D"quoted-plain-text" type=3D"cite">Isn't the current policy per conne=
ction, and not per port? If you have 2</blockquote><blockquote class=3D"qu=
oted-plain-text" type=3D"cite">output ports and 3 input ports, and you con=
nect them all together, then you</blockquote><blockquote class=3D"quoted-p=
lain-text" type=3D"cite">have up to 2*3=3D6 possible connections/policies.=
 Right?</blockquote><span class=3D"body-text-content"><span class=3D"body-=
text-content"><br>If you put the buffer per-port, you end up with one poli=
cy per port.<br>Port-driven triggering (event ports) are governed by the d=
ata object<br>(i.e. per-port), data object size and type becomes per-port =
as well.<br>What's left in the connection policy is the info necessary to =
reach<br>the other side (i.e. the topic name in ROS), which means that it =
is<br>not a policy anymore but a "remote endpoint"</span></span></div></di=
v></blockquote><span>&nbsp;</span></div><div>What if you don't use event p=
orts?&nbsp;</div><div><br></div><div>This issue for us boils down to the a=
bility to selectively remove the connection-object-per-input-output-port-p=
air issue. As a system designer I want to be able to</div><div>- specify o=
ne connection object between multiple input/output ports (1-N or N-1 or N-=
N)</div><div>- specify where the connection object lives (e.g. to cope wit=
h time delay at the system level)</div><div><br></div><div>Note the "selec=
tively remove" above - we still want to be able to use v2-flow-status type=
 setups within our system.</div><div><br><blockquote type=3D"cite"><div cl=
ass=3D"msg-quote"><div class=3D"_stretch"><span class=3D"body-text-content=
"><span class=3D"body-text-content">Interestingly, you'll have a hard time=
 figuring out how to configure<br>the CORBA dispatcher (which currently us=
es the same policy than the<br>whole connection).</span></span></div></div=
></blockquote><span>&nbsp;</span></div><div>We have several hundred local =
connections in our deployers. We have maybe less than 10 remote connection=
s.</div><div><br><blockquote type=3D"cite"><div class=3D"msg-quote"><div c=
lass=3D"_stretch"><blockquote class=3D"quoted-plain-text" type=3D"cite">Wh=
at do you mean by "dynamic" networks?</blockquote><span class=3D"body-text=
-content"><span class=3D"body-text-content"><br>That are not established o=
nce, but are modified at runtime (i.e.<br>connections are created/removed =
at runtime). With syskit, that happens<br>often and computing the network =
necessary to run them all is in<br>principle feasible, but in practice use=
less -- at the least in the<br>development phases.</span></span></div></di=
v></blockquote><span>&nbsp;</span></div><div><span style=3D"line-height: 1=
.5;">Got it. Thanks.</span></div><div><br><blockquote type=3D"cite"><div c=
lass=3D"msg-quote"><div class=3D"_stretch"><blockquote class=3D"quoted-pla=
in-text" type=3D"cite">&gt; Buffer connection: read() returns all samples =
received by the input port</blockquote><blockquote class=3D"quoted-plain-t=
ext" type=3D"cite">&gt; (apart from buffer size limitations).</blockquote>=
<blockquote class=3D"quoted-plain-text" type=3D"cite"><br></blockquote><bl=
ockquote class=3D"quoted-plain-text" type=3D"cite">Well, I'd phrase this d=
ifferently. A buffer allows more than one sample to</blockquote><blockquot=
e class=3D"quoted-plain-text" type=3D"cite">exist, in an ordered form, bet=
ween a writer and a reader. "All" samples can</blockquote><blockquote clas=
s=3D"quoted-plain-text" type=3D"cite">only be "guaranteed" with very parti=
cular buffer sizing or additional</blockquote><blockquote class=3D"quoted-=
plain-text" type=3D"cite">application semantics.</blockquote><span class=3D=
"body-text-content"><span class=3D"body-text-content"><br>+1, closer to th=
e current semantic.</span></span></div></div></blockquote><span>&nbsp;</sp=
an></div><div>Buffers vs data connections definitely change the flavor of =
things. The vast majority of our connections are data connections, not buf=
fers. We typically only use buffers in the traditional producer/consumer t=
ype situation - usually to cope with processing (e.g. real-time-ness) diff=
erences between producer and consumer (e.g. sample something on one side a=
nd write it to disk on the other).</div><div><br></div><div><br></div><div=
><br></div><div>I think that we're getting off topic as a whole. Johannes =
proposed some modifications to cope with some known problems with the exis=
ting implementation that directly affect some of the community. There does=
n't seem to be any dissent that something needs to be done about certain i=
ssues with using the current implementation. Does anyone have a concrete a=
lternative to Johannes current approach? Certainly we can discuss how to b=
est phase in (i.e. transition) the proposed changes, or something similar =
to them.</div><div><br></div><div>Johannes' proposed changes are focussed =
on</div><div>a) push vs pull, or specifying whether the data/buffer object=
 is on the reader or writer side. This is very useful for system designers=
 dealing with significant time delay.&nbsp;</div><div>b) private vs shared=
, whether a connection is per input/output port pair or whether the connec=
tion is shared between multiple input/output ports.</div><div>c) mandatory=
, primarily intended to support knowing whether writes to a buffer succeed=
ed (so that traditional producer/consumer relationships have some knowledg=
e about the buffer state)</div><div><br></div><div>It appears to me that m=
ost people are worried about changing the defaults, more than they're worr=
ied about the actual proposed changes. Actual technical concerns with the =
proposed changes appear to be</div><div><br></div><div>1) whether to encod=
e the result of a write() in existing enums or not</div><div>2) how to add=
 the "shared" concept within the current API, or to use a "new" API</div><=
div>3) changing the RTT defaults, which then affect an application's seman=
tics.</div><div><br></div><div>Did I miss any technical issues?</div><div>=
<br></div><div><br></div><div><div>My huge personal caveat to all of this =
is that I do *not* think we should change the current system semantics in =
introducing this change. Too many Orocos changes have broken backwards com=
patibility and forced system designers and implementors to change their sy=
stem in response. I don't think we should force that on users without an i=
ncredibly good reason. Otherwise it breeds frustration in the community.</=
div><div><br></div><div>If there is some way to introduce the proposed cha=
nges to the system, but to not change the behavior of current systems, the=
n I think that is a win for all. There does seem to be some general agreem=
ent that there are valid use cases that the current port implementation ha=
ndles poorly, and that this proposed approach does fix some of those. The =
proposed approach also improves the situation for systems that don't want =
or need to use the flow status approach. Both are equally valid approaches=
 - IMHO it would be beneficial to the community as a whole if RTT supporte=
d both approaches.</div></div><div><br></div><div>Cheers</div><div>Stephen=
</div><div><blockquote type=3D"cite"><div class=3D"msg-quote"><div class=3D=
"_stretch"><blockquote class=3D"quoted-plain-text" type=3D"cite"><br></blo=
ckquote></div></div></blockquote></div></body></html>=

--Boundary_(ID_6+JZUwdzFzaMsEa2rSFMrw)--

--Boundary_(ID_DAUzl2H65HRHVNbCMLgcoA)--

--===============0451313715==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

-- 
Orocos-Dev mailing list
[email protected]
http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev

--===============0451313715==--