Re: Updated dataflow semantics for RTT

--===============1598272345==
Content-type: multipart/alternative;
	boundary="Boundary_(ID_SOf2wMVgvBEoxCMhwvojyw)"


--Boundary_(ID_SOf2wMVgvBEoxCMhwvojyw)
Content-type: text/plain; charset=utf-8; format=flowed
Content-transfer-encoding: quoted-printable

On Sep 28, 2015, at 10:28 AM, Sylvain Joyeux <[email protected]> wrot=
e:=0A=0AI agree with much at the above, and we have similar problems that =
we've=0Asolved in similar ways. But there are also huge parts of our syste=
m that,=0Awhile "data driven", aren't at all interested in time. The syste=
m itself is=0Asynchronized and time is (largely) irrelevant - everything o=
perates on a=0Aper-cycle basis. In the long run, we time stamp only a tiny=
 portion of the=0Adata in our system due to this. Equally valid, but diffe=
rent.=0A=0AI don't dispute that you probably have a very good point, you u=
sually=0Ado, but could you provide me with a concrete example of such a se=
tup=0Athat uses buffers ? It would really help me grasp the situation.=0A=C2=
=A0=0ASome classic producer/consumer examples=0A=C2=A0- a camera captures =
images and sends them on to a processing pipeline (that is all data object=
 connections). A separate connection stores the camera images in a buffer,=
 which are dumped to disk.=0A- data is read (non-real-time) from a file an=
d pushed into a buffer for processing (real-time)=0A- data samples coming =
from a remote source (non-real-time) is buffered on arrival into the deplo=
yer for later processing (real-time).=0A=0ASometimes we want to control wh=
ether the network access (and hence=0Adelay/non-real-time part) is on the =
writer or reader side. That should be an=0Aapplication-level choice.=0A=0A=
Agreed, but this choice is currently bundled with having a buffer per=0Ach=
annel or one per input port.=0A=0AI don't follow, Sylvain. Bundled in the =
existing implementation or the proposed changes?=0A=0AIf you put the buffe=
r per-port, you end up with one policy per port.=0APort-driven triggering =
(event ports) are governed by the data object=0A(i.e. per-port), data obje=
ct size and type becomes per-port as well.=0AWhat's left in the connection=
 policy is the info necessary to reach=0Athe other side (i.e. the topic na=
me in ROS), which means that it is=0Anot a policy anymore but a "remote en=
dpoint"=0A=0A=0AWhat if you don't use event ports?=0A=0AHow is that releva=
nt ? Are we misunderstanding each other here ?=0A=C2=A0=0AFrom your commen=
t above "Port-driven triggering (event ports) =E2=80=A6"=0A=0AAs I underst=
and the proposal so far, we have: either you pull (hit=0Anetwork on read, =
one buffer per channel) or push (not hit network on=0Aread, have a one buf=
fer per port). The middle ground of=0Aone-buffer-per-channel and not hitti=
ng network is not covered, and=0Athat's what *greatly* bothers me.=0A=C2=A0=
=0AOur original intent was that pull/push would be independent of shared/p=
rivate (if you mean that by "one buffer per channel" and "one buffer per p=
ort").=0A=0AI know you guys probably have a good case here, I am trying ve=
ry hard=0Afor you to not break mine=0A=0AI'm with you on this, Sylvain!=0A=
=0ANote the "selectively remove" above - we still want to be able to use=0A=
v2-flow-status type setups within our system.=0A=0AMy understanding of the=
 proposal is that the v2 semantic are not=0Aavailable anymore. I'm not pus=
hing against what you need, I want you=0Ato not break what I need.=0A=C2=A0=
=0AI'm personally against changing the default, even though it would help =
me. This kind of pervasive breakage isn't good for the community ...=0A=0A=
=C2=A0- what are going to be the options of network/no-network and=0Aplace=
ment of buffer ?=0A- how can these options be or not be used with a single=
 output /=0Asingle input ? Are they mutually exclusive ? Can they be used =
together=0Aon the same input port ? On the same output port ?=0A=0ALooking=
 at Section 5.1 of=C2=A0Johannes' google doc, something looks wrong or I'm=
 remembering previous discussions incorrectly. I was expecting push/pull v=
s shared/private to be truly orthogonal, but=C2=A0the=C2=A0push/private fi=
gure says differently. Johannes? Peter?=0A=0AIn particular, from Section 5=
.2:=0A=0AOther than currently in v2, in the=C2=A0push=C2=A0case (pull=3Dfa=
lse) there will be only one buffer per input port, that is shared among al=
l connections. The connected output ports =E2=80=9Cpush=E2=80=9D their dat=
a through the channel and fill the input buffer (or overwrite the data obj=
ect), but every input port has its own buffer or data object and can read =
independently (Fig. 4)=0A=0AWhy doesn't every input port have a connection=
 object per associated output port? Why isn't the push/private figure the =
same as the pull/private figure, except that the connection object is on t=
he reader side (and for each input port, you'd have one connection object =
per output port)? Or is this to try and fix the non-determinstic behavior =
when an InputPort reads from multiple connection objects?=0A=0AWe have sev=
eral hundred local connections in our deployers. We have maybe=0Aless than=
 10 remote connections.=0A=0AAgain, how is this relevant ? If the policy i=
s specified per-port, you=0Awon't have any "kind-of-sane default" to confi=
gure the CORBA=0Adispatchers with.=C2=A0=0ABuffers vs data connections def=
initely change the flavor of things. The vast=0Amajority of our connection=
s are data connections, not buffers.=0A=0AI *know* which leads to this lon=
g discussion, because I am on the=0Acompletely other end of things: we mos=
tly use buffered connections.=0A=C2=A0=0AWhich is good IMHO, because it me=
ans that you and I effectively span the spectrum of system approaches. So =
if we can find a way to satisfy both our system approaches, then likely we=
'll cater for everyone else as well.=0A=0AI think that we're getting off t=
opic as a whole.=0A=0AI don't think so. So far, my concerns with the propo=
sal are still=0Athere: by getting closer to v1 semantics, the proposal bre=
aks v2=0Asemantics in a way that is fundamental to me. Peter's post actual=
ly=0Aleads me to think that this breakage and known and desired, which is=0A=
why I am spending so much time trying to get my point across, while I=0Aal=
ready have way too much on my plate.=0A=C2=A0=0AAgain, my intent was to ad=
d more v1-like semantics to the system, as well as have more control on wh=
ere latency occurred. Not to replace any aspect of the v2 semantics.=0A=0A=
a) push vs pull, or specifying whether the data/buffer object is on the=0A=
reader or writer side. This is very useful for system designers dealing wi=
th=0Asignificant time delay.=0Ab) private vs shared, whether a connection =
is per input/output port pair or=0Awhether the connection is shared betwee=
n multiple input/output ports.=0A=0ASo far, "private" is defined one buffe=
r *per input port* (change in=0Asemantic) not *per input/output port pair*=
 (current v2 semantic)=0A=C2=A0=0AI think we should postpone the "changing=
 the defaults" discussion. For a v2.9 type release that could include the =
proposed changes, having the default not change would be of most benefit t=
o the community I think. In the future, if/when we decide that a different=
 set of defaults is=C2=A0more appropriate, we could phase that into future=
 releases in a way that minimizes disruption to the community.=0A=0ACheers=
=0AStephen=

--Boundary_(ID_SOf2wMVgvBEoxCMhwvojyw)
Content-type: multipart/related;
	boundary="Boundary_(ID_OnQ0BUCm0GD50YR/hqDPWQ)"; type="text/html"


--Boundary_(ID_OnQ0BUCm0GD50YR/hqDPWQ)
Content-type: text/html; charset=utf-8
Content-transfer-encoding: quoted-printable

<html><body><div><span style=3D"line-height: 1.5;">On Sep 28, 2015, at 10:28 AM, Sylva=
in Joyeux &lt;[email protected]&gt; wrote:</span></div><div><br></div=
><div><blockquote type=3D"cite"><div class=3D"msg-quote"><div class=3D"_st=
retch"><blockquote class=3D"quoted-plain-text" type=3D"cite">I agree with =
much at the above, and we have similar problems that we've</blockquote><bl=
ockquote class=3D"quoted-plain-text" type=3D"cite">solved in similar ways.=
 But there are also huge parts of our system that,</blockquote><blockquote=
 class=3D"quoted-plain-text" type=3D"cite">while "data driven", aren't at =
all interested in time. The system itself is</blockquote><blockquote class=
=3D"quoted-plain-text" type=3D"cite">synchronized and time is (largely) ir=
relevant - everything operates on a</blockquote><blockquote class=3D"quote=
d-plain-text" type=3D"cite">per-cycle basis. In the long run, we time stam=
p only a tiny portion of the</blockquote><blockquote class=3D"quoted-plain=
-text" type=3D"cite">data in our system due to this. Equally valid, but di=
fferent.</blockquote><span class=3D"body-text-content"><span class=3D"body=
-text-content"><br>I don't dispute that you probably have a very good poin=
t, you usually<br>do, but could you provide me with a concrete example of =
such a setup<br>that uses buffers ? It would really help me grasp the situ=
ation.</span></span></div></div></blockquote><span>&nbsp;</span></div><div=
>Some classic producer/consumer examples</div><div>&nbsp;- a camera captur=
es images and sends them on to a processing pipeline (that is all data obj=
ect connections). A separate connection stores the camera images in a buff=
er, which are dumped to disk.</div><div>- data is read (non-real-time) fro=
m a file and pushed into a buffer for processing (real-time)</div><div>- d=
ata samples coming from a remote source (non-real-time) is buffered on arr=
ival into the deployer for later processing (real-time).</div><div><br><bl=
ockquote type=3D"cite"><div class=3D"msg-quote"><div class=3D"_stretch"><b=
lockquote class=3D"quoted-plain-text" type=3D"cite">Sometimes we want to c=
ontrol whether the network access (and hence</blockquote><blockquote class=
=3D"quoted-plain-text" type=3D"cite">delay/non-real-time part) is on the w=
riter or reader side. That should be an</blockquote><blockquote class=3D"q=
uoted-plain-text" type=3D"cite">application-level choice.</blockquote><spa=
n class=3D"body-text-content"><span class=3D"body-text-content"><br>Agreed=
, but this choice is currently bundled with having a buffer per<br>channel=
 or one per input port.</span></span></div></div></blockquote></div><div><=
span><br></span></div><div>I don't follow, Sylvain. Bundled in the existin=
g implementation or the proposed changes?</div><div><br><blockquote type=3D=
"cite"><div class=3D"msg-quote"><div class=3D"_stretch"><blockquote class=3D=
"quoted-plain-text" type=3D"cite">If you put the buffer per-port, you end =
up with one policy per port.</blockquote><blockquote class=3D"quoted-plain=
-text" type=3D"cite">Port-driven triggering (event ports) are governed by =
the data object</blockquote><blockquote class=3D"quoted-plain-text" type=3D=
"cite">(i.e. per-port), data object size and type becomes per-port as well=
.</blockquote><blockquote class=3D"quoted-plain-text" type=3D"cite">What's=
 left in the connection policy is the info necessary to reach</blockquote>=
<blockquote class=3D"quoted-plain-text" type=3D"cite">the other side (i.e.=
 the topic name in ROS), which means that it is</blockquote><blockquote cl=
ass=3D"quoted-plain-text" type=3D"cite">not a policy anymore but a "remote=
 endpoint"</blockquote><blockquote class=3D"quoted-plain-text" type=3D"cit=
e"><br></blockquote><blockquote class=3D"quoted-plain-text" type=3D"cite">=
<br></blockquote><blockquote class=3D"quoted-plain-text" type=3D"cite">Wha=
t if you don't use event ports?</blockquote><span class=3D"body-text-conte=
nt"><span class=3D"body-text-content"><br>How is that relevant ? Are we mi=
sunderstanding each other here ?</span></span></div></div></blockquote><sp=
an>&nbsp;</span></div><div>From your comment above "Port-driven triggering=
 (event ports) =E2=80=A6"</div><div><br><blockquote type=3D"cite"><div cla=
ss=3D"msg-quote"><div class=3D"_stretch"><span class=3D"body-text-content"=
><span class=3D"body-text-content">As I understand the proposal so far, we=
 have: either you pull (hit<br>network on read, one buffer per channel) or=
 push (not hit network on<br>read, have a one buffer per port). The middle=
 ground of<br>one-buffer-per-channel and not hitting network is not covere=
d, and<br>that's what *greatly* bothers me.</span></span></div></div></blo=
ckquote><span>&nbsp;</span></div><div>Our original intent was that pull/pu=
sh would be independent of shared/private (if you mean that by "one buffer=
 per channel" and "one buffer per port").</div><div><br><blockquote type=3D=
"cite"><div class=3D"msg-quote"><div class=3D"_stretch"><span class=3D"bod=
y-text-content"><span class=3D"body-text-content">I know you guys probably=
 have a good case here, I am trying very hard<br>for you to not break mine=
</span></span></div></div></blockquote></div><div><br></div><div>I'm with =
you on this, Sylvain!</div><div><br><blockquote type=3D"cite"><div class=3D=
"msg-quote"><div class=3D"_stretch"><blockquote class=3D"quoted-plain-text=
" type=3D"cite">Note the "selectively remove" above - we still want to be =
able to use</blockquote><blockquote class=3D"quoted-plain-text" type=3D"ci=
te">v2-flow-status type setups within our system.</blockquote><span class=3D=
"body-text-content"><span class=3D"body-text-content"><br>My understanding=
 of the proposal is that the v2 semantic are not<br>available anymore. I'm=
 not pushing against what you need, I want you<br>to not break what I need=
.</span></span></div></div></blockquote><span>&nbsp;</span></div><div>I'm =
personally against changing the default, even though it would help me. Thi=
s kind of pervasive breakage isn't good for the community ...</div><div><b=
r></div><div><blockquote type=3D"cite"><div class=3D"msg-quote"></div></bl=
ockquote><blockquote type=3D"cite"><div class=3D"msg-quote"><div class=3D"=
_stretch"><span class=3D"body-text-content"><span class=3D"body-text-conte=
nt">&nbsp;- what are going to be the options of network/no-network and<br>=
placement of buffer ?</span></span></div></div></blockquote></div><div><bl=
ockquote type=3D"cite"><div class=3D"msg-quote"><div class=3D"_stretch"><s=
pan class=3D"body-text-content"><span class=3D"body-text-content">- how ca=
n these options be or not be used with a single output /<br>single input ?=
 Are they mutually exclusive ? Can they be used together<br>on the same in=
put port ? On the same output port ?</span></span></div></div></blockquote=
><span><div><span><br></span></div><div><span>Looking at Section 5.1 of&nb=
sp;</span><span style=3D"line-height: 1.5;">Johannes' google doc, somethin=
g looks wrong or I'm remembering previous discussions incorrectly. I was e=
xpecting push/pull vs shared/private to be truly orthogonal, but&nbsp;</sp=
an>the<span style=3D"line-height: 1.5;">&nbsp;push/private figure says dif=
ferently. Johannes? Peter?</span></div><div><span style=3D"line-height: 1.=
5;"><br></span></div><div><span style=3D"line-height: 1.5;">In particular,=
 from Section 5.2:</span></div><div><span style=3D"line-height: 1.5;"><br>=
</span></div><div><em><span style=3D"font-family: Arial; line-height: norm=
al; orphans: 2; widows: 2;" data-mce-style=3D"font-family: Arial; line-hei=
ght: normal; orphans: 2; widows: 2;">Other than currently in v2, in the&nb=
sp;</span><span class=3D"c2" style=3D"font-weight: bold; font-family: Aria=
l; line-height: normal; orphans: 2; widows: 2;" data-mce-style=3D"font-wei=
ght: bold; font-family: Arial; line-height: normal; orphans: 2; widows: 2;=
">push&nbsp;</span><span style=3D"font-family: Arial; line-height: normal;=
 orphans: 2; widows: 2;" data-mce-style=3D"font-family: Arial; line-height=
: normal; orphans: 2; widows: 2;">case (</span><span class=3D"c0" style=3D=
"font-family: 'Courier New'; line-height: normal; orphans: 2; widows: 2;" =
data-mce-style=3D"font-family: 'Courier New'; line-height: normal; orphans=
: 2; widows: 2;">pull=3Dfalse</span><span style=3D"font-family: Arial; lin=
e-height: normal; orphans: 2; widows: 2;" data-mce-style=3D"font-family: A=
rial; line-height: normal; orphans: 2; widows: 2;">) there will be only on=
e buffer per input port, that is shared among all connections. The connect=
ed output ports =E2=80=9Cpush=E2=80=9D their data through the channel and =
fill the input buffer (or overwrite the data object), but every input port=
 has its own buffer or data object and can read independently (Fig. 4)</sp=
an></em></div><div><br></div><div>Why doesn't every input port have a conn=
ection object per associated output port? Why isn't the push/private figur=
e the same as the pull/private figure, except that the connection object i=
s on the reader side (and for each input port, you'd have one connection o=
bject per output port)? Or is this to try and fix the non-determinstic beh=
avior when an InputPort reads from multiple connection objects?</div></spa=
n></div><div><br></div><div><blockquote type=3D"cite"><div class=3D"msg-qu=
ote"><div class=3D"_stretch"><blockquote class=3D"quoted-plain-text" type=3D=
"cite">We have several hundred local connections in our deployers. We have=
 maybe</blockquote><blockquote class=3D"quoted-plain-text" type=3D"cite">l=
ess than 10 remote connections.</blockquote><span class=3D"body-text-conte=
nt"><span class=3D"body-text-content"><br>Again, how is this relevant ? If=
 the policy is specified per-port, you<br>won't have any "kind-of-sane def=
ault" to configure the CORBA<br>dispatchers with.</span></span><span style=
=3D"line-height: 1.5; font-family: 'Helvetica Neue', Helvetica, Arial, san=
s-serif;">&nbsp;</span></div></div></blockquote></div><div><blockquote typ=
e=3D"cite"><div class=3D"msg-quote"><div class=3D"_stretch"><blockquote cl=
ass=3D"quoted-plain-text" type=3D"cite">Buffers vs data connections defini=
tely change the flavor of things. The vast</blockquote><blockquote class=3D=
"quoted-plain-text" type=3D"cite">majority of our connections are data con=
nections, not buffers.</blockquote><span class=3D"body-text-content"><span=
 class=3D"body-text-content"><br>I *know* which leads to this long discuss=
ion, because I am on the<br>completely other end of things: we mostly use =
buffered connections.</span></span></div></div></blockquote><span>&nbsp;</=
span></div><div>Which is good IMHO, because it means that you and I effect=
ively span the spectrum of system approaches. So if we can find a way to s=
atisfy both our system approaches, then likely we'll cater for everyone el=
se as well.</div><div><br><blockquote type=3D"cite"><div class=3D"msg-quot=
e"><div class=3D"_stretch"><blockquote class=3D"quoted-plain-text" type=3D=
"cite">I think that we're getting off topic as a whole.</blockquote><span =
class=3D"body-text-content"><span class=3D"body-text-content"><br>I don't =
think so. So far, my concerns with the proposal are still<br>there: by get=
ting closer to v1 semantics, the proposal breaks v2<br>semantics in a way =
that is fundamental to me. Peter's post actually<br>leads me to think that=
 this breakage and known and desired, which is<br>why I am spending so muc=
h time trying to get my point across, while I<br>already have way too much=
 on my plate.</span></span></div></div></blockquote><span>&nbsp;</span></d=
iv><div>Again, my intent was to add more v1-like semantics to the system, =
as well as have more control on where latency occurred. Not to replace any=
 aspect of the v2 semantics.</div><div><br></div><div><blockquote type=3D"=
cite"><div class=3D"msg-quote"><div class=3D"_stretch"><blockquote class=3D=
"quoted-plain-text" type=3D"cite">a) push vs pull, or specifying whether t=
he data/buffer object is on the</blockquote><blockquote class=3D"quoted-pl=
ain-text" type=3D"cite">reader or writer side. This is very useful for sys=
tem designers dealing with</blockquote><blockquote class=3D"quoted-plain-t=
ext" type=3D"cite">significant time delay.</blockquote><blockquote class=3D=
"quoted-plain-text" type=3D"cite">b) private vs shared, whether a connecti=
on is per input/output port pair or</blockquote><blockquote class=3D"quote=
d-plain-text" type=3D"cite">whether the connection is shared between multi=
ple input/output ports.</blockquote><span class=3D"body-text-content"><br>=
So far, "private" is defined one buffer *per input port* (change in<br>sem=
antic) not *per input/output port pair* (current v2 semantic)</span></div>=
</div></blockquote><span>&nbsp;</span></div><div>I think we should postpon=
e the "changing the defaults" discussion. For a v2.9 type release that cou=
ld include the proposed changes, having the default not change would be of=
 most benefit to the community I think. In the future, if/when we decide t=
hat a different set of defaults is&nbsp;<span style=3D"line-height: 1.5;">=
more appropriate, we could phase that into future releases in a way that m=
inimizes disruption to the community.</span></div><div><span style=3D"line=
-height: 1.5;"><br></span></div><div><span style=3D"line-height: 1.5;">Che=
ers</span></div><div><span style=3D"line-height: 1.5;">Stephen</span></div=
></body></html>=

--Boundary_(ID_OnQ0BUCm0GD50YR/hqDPWQ)--

--Boundary_(ID_SOf2wMVgvBEoxCMhwvojyw)--

--===============1598272345==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

-- 
Orocos-Dev mailing list
[email protected]
http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev

--===============1598272345==--