Re: Updated dataflow semantics for RTT
Janosch Machowinski <[email protected]> Tue, 22 Sep 2015 15:32:35 +0200
| Newsgroups | gmane.science.robotics.orocos.devel |
|---|---|
| Message-ID | <[email protected]> |
This is a multi-part message in MIME format.
--===============1791124515==
Content-Type: multipart/alternative;
boundary="------------090606070703060800080708"
This is a multi-part message in MIME format.
--------------090606070703060800080708
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: quoted-printable
Hi,
here are my remarks:
First of all, I like the idea of the input buffer (push case). But I=20
also see some
problems with it. The restriction on the same buffer size and type in the
connection policy will be annoying (In my mind I can see the runtime=20
errors pop up).
It will be hard to track this in bigger systems. Do I get it correctly,=20
that you only
want to preserve arrival order and runtime here ?
If you are already changing the Connection implementation, I would=20
recommend, to
put a new thread in every connection in remote case. We experienced the=20
issue, that
bad Wifi connections, slowed our systems down, as it would hang on the=20
write call.
For the Feedback on the write I would recommend to create a second enum,=20
and not
to use FlowStatus. Sometimes I use switch/case to react to the read=20
FlowStatus. If you
Introduce new members, this will result in compile warnings, because of=20
unhandled
entrie.
For the creation of shared connections, I would recommend an new API.=20
The create call
should contain all involved members. This spares us the headache of=20
tracking buffer sizes
and types around in the code. Also this will simplify the code logic, as=20
we don't need the lookup.
e.g.
out.connectShared(b.in, c.in, d.in, policy) ;
Greetings
Janosch
Am 21.09.2015 um 15:15 schrieb Johannes Meyer:
> Dear Orocos community,
>
> during the last few weeks we have been working on the concept and=20
> implementation of a major redesign of the Orocos RTT dataflow, that we=20
> would like to share and ask for your valuable feedback. There was=20
> already a previous discussion on the mailing list about the topic=20
> earlier this year [1], and the following proposal tries to address all=20
> the issues mentioned there.
>
> With the version 2 of the Orocos toolchain some years ago, the=20
> previous data flow architecture, which was based on a single data=20
> object and buffer shared between all ports participating at the same=20
> connection, was replaced by a connection architecture based on data=20
> flow, where each pair of output and input ports (or streams) is=20
> connected through an individual channel, each having its own data=20
> object or buffer. While there was a well substantiated motivation for=20
> this update and it solved some of the shortcomings in RTT version 1=20
> [2], it also introduced some new, sometimes hidden, problems and broke=20
> other use cases. One example is an input port with multiple=20
> connections, where due to the ambiguity in which channel to read from=20
> the result is sometimes unexpected to the user, who expects that each=20
> write overwrites previous samples in case of simple data connections.
>
> We assembled a document (see attachment, or [3]) which introduces the=20
> version 1 and version 2 data flow semantics for those who are not=20
> aware of the details, three example use cases and presents the new=20
> dataflow architecture we have in mind, which ideally works for a=20
> superset of use cases that are supported by RTT version 1 and 2.
>
> We do not want to go into details here, so only a short summary for=20
> those who do not want to read the full document:
> We propose to add two more boolean flags to the ConnPolicy struct that=20
> describes the type of connection between two ports: "shared" and=20
> "mandatory". Furthermore the existing "pull" flag, which was only=20
> meaningful for remote connections, now has a big influence for local=20
> connections.
>
> *"push" vs. "pull"*
> *
> *
> Like it is already the case for remote connections via CORBA, the=20
> "pull" flag will decide whether the connection's data object or buffer=20
> is installed at the writer's side (pull) or the reader's side (push)=20
> of the connection. The first case is equivalent to the current version=20
> 2 connection model and readers have to select the connection/output=20
> port to read from, while for push connections there will be only one=20
> single buffer per input port. We think that the push connection should=20
> be the default connection type for local ports, but this might break=20
> existing applications which rely on having one buffer per connection no=
w.
>
> *"private" vs. "shared"*
>
> The shared flag reintroduces the shared data connection model that was=20
> standard in RTT version 1. For shared connections multiple input ports=20
> read from the same data object or consume samples from the same buffer=20
> instance, so their read operations can influence each other. Shared=20
> connections are especially useful for producer/consumer applications=20
> with multiple consumers.
>
> *"mandatory"*
>
> The introduction of the "mandatory" flag was necessary to decide on=20
> the return value of the write() operation on an output port. In RTT=20
> version 2 it was not possible anymore to get a feedback of whether a=20
> write operation was successful or failed, e.g. because of a full=20
> buffer or broken remote connection. Only connections with the=20
> mandatory flag set would be considered for the overall result of=20
> write() operations, while failures of non-mandatory connections will=20
> be ignored.
>
>
> For those who want to take a look at the current state of the=20
> implementation, this is the link to the working branch on GitHub:
> Tree:=20
> https://github.com/orocos-toolchain/rtt/tree/updated-dataflow-semantics
> Diff to master:=20
> https://github.com/orocos-toolchain/rtt/compare/master...updated-datafl=
ow-semantics
>
> The implementation is not fully working yet for non-standard=20
> transports (like Corba and mqueue), but especially the updated test=20
> ports_test.cpp already shows what would change from a user perspective.
>
> Obviously, such a major refactoring cannot be done without changes in=20
> the public API. However, we tried to keep the required changes in user=20
> code as minimal as possible.
> The new dataflow semantics are targeted for an upcoming version 2.9=20
> release (or even name it 3.0 to communicate this potentially breaking=20
> change?).
>
> We are planning to also come up with some comparative performance=20
> measurements during this week, as people have asked for that before [1]=
.
>
>
> Best regards,
> Johannes and Peter
>
>
>
> [1]=20
> http://www.orocos.org/forum/rtt/rtt-dev/limitations-v2-port-implementat=
ion=20
>
> [2] http://www.orocos.org/wiki/rtt/rtt-2.0/dataflow
> [3]=20
> https://docs.google.com/document/d/1zDnPPz4SiCVvfEFxYFUZBcVbXKj33o4Kaot=
pCtXM4E0/pub
>
>
>
--=20
Dipl. Inf. Janosch Machowinski
SAR- & Sicherheitsrobotik
Universit=E4t Bremen
FB 3 - Mathematik und Informatik
AG Robotik
Robert-Hooke-Stra=DFe 1
28359 Bremen, Germany
=20
Zentrale: +49 421 178 45-6611
=20
Besuchsadresse der Nebengesch=E4ftstelle:
Robert-Hooke-Stra=DFe 5
28359 Bremen, Germany
=20
Tel.: +49 421 178 45-6614
Empfang: +49 421 178 45-6600
Fax: +49 421 178 45-4150
E-Mail: [email protected]
Weitere Informationen: http://www.informatik.uni-bremen.de/robotik
--------------090606070703060800080708
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
<html>
<head>
<meta content=3D"text/html; charset=3Dwindows-1252"
http-equiv=3D"Content-Type">
</head>
<body text=3D"#000000" bgcolor=3D"#FFFFFF">
<div class=3D"moz-cite-prefix">Hi,<br>
here are my remarks:<br>
First of all, I like the idea of the input buffer (push case). But
I also see some<br>
problems with it. The restriction on the same buffer size and type
in the <br>
connection policy will be annoying (In my mind I can see the
runtime errors pop up). <br>
It will be hard to track this in bigger systems. Do I get it
correctly, that you only<br>
want to preserve arrival order and runtime here ?<br>
<br>
If you are already changing the Connection implementation, I would
recommend, to<br>
put a new thread in every connection in remote case. We
experienced the issue, that<br>
bad Wifi connections, slowed our systems down, as it would hang on
the write call.<br>
<br>
For the Feedback on the write I would recommend to create a second
enum, and not<br>
to use FlowStatus. Sometimes I use switch/case to react to the
read FlowStatus. If you<br>
Introduce new members, this will result in compile warnings,
because of unhandled<br>
entrie.<br>
<br>
For the creation of shared connections, I would recommend an new
API. The create call<br>
should contain all involved members. This spares us the headache
of tracking buffer sizes<br>
and types around in the code. Also this will simplify the code
logic, as we don't need the lookup.<br>
e.g. <br>
out.connectShared(b.in, c.in, d.in, policy) ;<br>
Greetings<br>
=A0=A0=A0 Janosch<br>
<br>
<br>
<br>
Am 21.09.2015 um 15:15 schrieb Johannes Meyer:<br>
</div>
<blockquote
cite=3D"mid:[email protected]=
ail.com"
type=3D"cite">
<div dir=3D"ltr">Dear Orocos community,
<div><br>
</div>
<div>during the last few weeks we have been working on the
concept and implementation of a major redesign of the Orocos
RTT dataflow, that we would like to share and ask for your
valuable feedback. There was already a previous discussion on
the mailing list about the topic earlier this year [1], and
the following proposal tries to address all the issues
mentioned there.</div>
<div><br>
</div>
<div>With the version 2 of the Orocos toolchain some years ago,
the previous data flow architecture, which was based on a
single data object and buffer shared between all ports
participating at the same connection, was replaced by a
connection architecture based on data flow, where each pair of
output and input ports (or streams) is connected through an
individual channel, each having its own data object or buffer.
While there was a well substantiated motivation for this
update and it solved some of the shortcomings in RTT version 1
[2], it also introduced some new, sometimes hidden, problems
and broke other use cases. One example is an input port with
multiple connections, where due to the ambiguity in which
channel to read from the result is sometimes unexpected to the
user, who expects that each write overwrites previous samples
in case of simple data connections.</div>
<div><br>
</div>
<div>We assembled a document (see attachment, or [3]) which
introduces the version 1 and version 2 data flow semantics for
those who are not aware of the details, three example use
cases and presents the new dataflow architecture we have in
mind, which ideally works for a superset of use cases that are
supported by RTT version 1 and 2.=A0</div>
<div><br>
</div>
<div>We do not want to go into details here, so only a short
summary for those who do not want to read the full document:<br=
>
</div>
<div>We propose to add two more boolean flags to the ConnPolicy
struct that describes the type of connection between two
ports: "shared" and "mandatory". Furthermore the existing
"pull" flag, which was only meaningful for remote connections,
now has a big influence for local connections.</div>
<div><br>
</div>
<div><b>"push" vs. "pull"</b></div>
<div><b><br>
</b></div>
<div>Like it is already the case for remote connections via
CORBA, the "pull" flag will decide whether the connection's
data object or buffer is installed at the writer's side (pull)
or the reader's side (push) of the connection. The first case
is equivalent to the current version 2 connection model and
readers have to select the connection/output port to read
from, while for push connections there will be only one single
buffer per input port. We think that the push connection
should be the default connection type for local ports, but
this might break existing applications which rely on having
one buffer per connection now.</div>
<div><br>
</div>
<div><b>"private" vs. "shared"</b></div>
<div><br>
</div>
<div>The shared flag reintroduces the shared data connection
model that was standard in RTT version 1. For shared
connections multiple input ports read from the same data
object or consume samples from the same buffer instance, so
their read operations can influence each other. Shared
connections are especially useful for producer/consumer
applications with multiple consumers.</div>
<div><br>
</div>
<div><b>"mandatory"</b></div>
<div><br>
</div>
<div>The introduction of the "mandatory" flag was necessary to
decide on the return value of the write() operation on an
output port. In RTT version 2 it was not possible anymore to
get a feedback of whether a write operation was successful or
failed, e.g. because of a full buffer or broken remote
connection. Only connections with the mandatory flag set would
be considered for the overall result of write() operations,
while failures of non-mandatory connections will be ignored.</d=
iv>
<div><br>
</div>
<div><br>
</div>
<div>For those who want to take a look at the current state of
the implementation, this is the link to the working branch on
GitHub:</div>
<div>Tree: <a moz-do-not-send=3D"true"
href=3D"https://github.com/orocos-toolchain/rtt/tree/updated-dataflow-sem=
antics">https://github.com/orocos-toolchain/rtt/tree/updated-dataflow-sem=
antics</a><br>
</div>
<div>Diff to master:=A0<a moz-do-not-send=3D"true"
href=3D"https://github.com/orocos-toolchain/rtt/compare/master...updated-=
dataflow-semantics">https://github.com/orocos-toolchain/rtt/compare/maste=
r...updated-dataflow-semantics</a></div>
<div><br>
</div>
<div>The implementation is not fully working yet for
non-standard transports (like Corba and mqueue), but
especially the updated test ports_test.cpp already shows what
would change from a user perspective.</div>
<div><br>
</div>
<div>Obviously, such a major refactoring cannot be done without
changes in the public API. However, we tried to keep the
required changes in user code as minimal as possible.</div>
<div>The new dataflow semantics are targeted for an upcoming
version 2.9 release (or even name it 3.0 to communicate this
potentially breaking change?).</div>
<div><br>
</div>
<div>We are planning to also come up with some comparative
performance measurements during this week, as people have
asked for that before [1].</div>
<div><br>
</div>
<div><br>
</div>
<div>Best regards,<br>
</div>
<div>Johannes and Peter</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>[1] <a moz-do-not-send=3D"true"
href=3D"http://www.orocos.org/forum/rtt/rtt-dev/limitations-v2-port-imple=
mentation">http://www.orocos.org/forum/rtt/rtt-dev/limitations-v2-port-im=
plementation</a>
</div>
<div>[2]=A0<a moz-do-not-send=3D"true"
href=3D"http://www.orocos.org/wiki/rtt/rtt-2.0/dataflow">http=
://www.orocos.org/wiki/rtt/rtt-2.0/dataflow</a><br>
</div>
<div>[3] <a moz-do-not-send=3D"true"
href=3D"https://docs.google.com/document/d/1zDnPPz4SiCVvfEFxYFUZBcVbXKj33=
o4KaotpCtXM4E0/pub">https://docs.google.com/document/d/1zDnPPz4SiCVvfEFxY=
FUZBcVbXKj33o4KaotpCtXM4E0/pub</a></div>
<div><br>
</div>
</div>
<br>
<fieldset class=3D"mimeAttachmentHeader"></fieldset>
<br>
</blockquote>
<br>
<br>
<pre class=3D"moz-signature" cols=3D"72">--=20
Dipl. Inf. Janosch Machowinski
SAR- & Sicherheitsrobotik
Universit=E4t Bremen
FB 3 - Mathematik und Informatik
AG Robotik
Robert-Hooke-Stra=DFe 1
28359 Bremen, Germany
=20
Zentrale: +49 421 178 45-6611
=20
Besuchsadresse der Nebengesch=E4ftstelle:=20
Robert-Hooke-Stra=DFe 5
28359 Bremen, Germany
=20
Tel.: +49 421 178 45-6614
Empfang: +49 421 178 45-6600
Fax: +49 421 178 45-4150
E-Mail: <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:jmachowins=
[email protected]">[email protected]</a>
Weitere Informationen: <a class=3D"moz-txt-link-freetext" href=3D"http:/=
/www.informatik.uni-bremen.de/robotik">http://www.informatik.uni-bremen.d=
e/robotik</a>
</pre>
</body>
</html>
--------------090606070703060800080708--
--===============1791124515==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
--
Orocos-Dev mailing list
[email protected]
http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev
--===============1791124515==--