Re: Updated dataflow semantics for RTT

(@Stephen: re-sending, mistakenly sent to you only instead of all the
involved parties)

> Enforcing in order arrive on the reading side is one way of doing it. You
> can also do it at a system level if you chose. They're equally valid use
> cases.

Can you describe a use case where "order of arrival" is good enough
for you ? That would help me understand your use-case.

(For some clearer thoughts on this,
   http://rock-robotics.org/master/documentation/data_processing/data.html
   http://rock-robotics.org/master/documentation/data_processing/stream_aligner.html)

>> orogen only deals with single components. Syskit is the one inmpacted by
>> this change, and it is for me out of the picture to compute all possible
>> connections in all possible networks. Pull connections are not practical as
>> soon as you have a inter-host connection (i.e. remote connection) as they
>> break the reading part.
>
> Can you explain this last part a little more Sylvain?

Reading on remote+pull CORBA connections requires the reader to do
network access (it's actually the whole use-case of pull for us, that
the writer is not at all affected by the connection)

> Isn't the current policy per connection, and not per port? If you have 2
> output ports and 3 input ports, and you connect them all together, then you
> have up to 2*3=6 possible connections/policies. Right?

If you put the buffer per-port, you end up with one policy per port.
Port-driven triggering (event ports) are governed by the data object
(i.e. per-port), data object size and type becomes per-port as well.
What's left in the connection policy is the info necessary to reach
the other side (i.e. the topic name in ROS), which means that it is
not a policy anymore but a "remote endpoint"

Interestingly, you'll have a hard time figuring out how to configure
the CORBA dispatcher (which currently uses the same policy than the
whole connection).

> What do you mean by "dynamic" networks?

That are not established once, but are modified at runtime (i.e.
connections are created/removed at runtime). With syskit, that happens
often and computing the network necessary to run them all is in
principle feasible, but in practice useless -- at the least in the
development phases.

> > Buffer connection: read() returns all samples received by the input port
> > (apart from buffer size limitations).
>
> Well, I'd phrase this differently. A buffer allows more than one sample to
> exist, in an ordered form, between a writer and a reader. "All" samples can
> only be "guaranteed" with very particular buffer sizing or additional
> application semantics.

+1, closer to the current semantic.

Sylvain

2015-09-28 8:32 GMT-03:00 Stephen Roderick <[email protected]>:
> On Sep 25, 2015, at 09:11 PM, Sylvain Joyeux <[email protected]> wrote:
>
>> It doesn't fix the out-of-order bug for the buffer-per-channel case.
>> You can still starve a certain channel and get old data much later.
>>
>> That's why we propose to change the default. This starving already
>>
>> happens in very simple/clean setups, unless you start to round-robin
>>
>> all the channels. But again, this does not prevent out-of-order arrivals.
>
>
> This is one thing where we have a completely different point of view. If you
> need in-order arrival, you have to enforce it on the reading side with
> something like Rock's stream aligner. The only case where the proposed
> change will allow you some in-order guarantees is the local connection case
> anyways, in all other cases you have unbounded latencies that do reordering.
> As for stale data: if you expect buffers to be used on your inputs, the
> receiving component must always read all data available (hence, no "very old
> stale data"). Otherwise, you will always have some form of buildup in the
> buffers, in a form you can't control. Finally, sharing a single buffer
> allows one high-frequency connection to starve all others by filling the
> buffer.
>
> Enforcing in order arrive on the reading side is one way of doing it. You
> can also do it at a system level if you chose. They're equally valid use
> cases.
>
> And agreed on local connections, which is primarily what we're hoping that
> this functionality fixes.
>
>>
>> I prefer the proposed solution since the default policy leads to a
>> much more predictable data flow. For appropriately sizing buffers
>>
>> for push channels in multi-rate component systems, we thought
>>
>> about it to specify on the input port level the buffer size
>> (there is already a default conn policy for each input port). A code
>>
>> generator like orogen could easily calculate the largest buffer size
>>
>> required between components and inform the input port, or stick to
>> pull connections.
>
>
> orogen only deals with single components. Syskit is the one inmpacted by
> this change, and it is for me out of the picture to compute all possible
> connections in all possible networks. Pull connections are not practical as
> soon as you have a inter-host connection (i.e. remote connection) as they
> break the reading part.
>
>
> Can you explain this last part a little more Sylvain?
>
>
> The bottom line is: the proposed change is really broken in the use case of
> networks that are dynamic, which is my very use case. Making the new (or the
> old) behaviour optional would be also hard because they really demand
> different APIs (you can't keep a policy in the connection API when truly the
> policy is per-port).
>
> Isn't the current policy per connection, and not per port? If you have 2
> output ports and 3 input ports, and you connect them all together, then you
> have up to 2*3=6 possible connections/policies. Right?
>
> What do you mean by "dynamic" networks?
>
>
>>
>> I would definately not 'imply' different behaviours depending on
>> buffer/data. After all, a data object is just a circular buffer of size 1.
>
>
> I thought the same, but start to disagree. Data and buffer have (and should
> have) different semantics. The fact that the data object is the same don't
> change that.
>
> Data connection: read() only returns the last sample received by the input
> port
>
>
> +1
>
> Buffer connection: read() returns all samples received by the input port
> (apart from buffer size limitations).
>
>
> Well, I'd phrase this differently. A buffer allows more than one sample to
> exist, in an ordered form, between a writer and a reader. "All" samples can
> only be "guaranteed" with very particular buffer sizing or additional
> application semantics.
>
> Cheers
> S
>
>
-- 
Orocos-Dev mailing list
[email protected]
http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev