Re: Updated dataflow semantics for RTT

On 28/9/15, 07:28, "Herman Bruyninckx"
<[email protected]> wrote:

>On Mon, 28 Sep 2015, BiggsGeoffrey wrote:
>
>> On 27/9/15, 14:10, "[email protected] on behalf
>>of
>> Herman Bruyninckx" <[email protected] on behalf
>>of
>> [email protected]> wrote:
>>>
>>> Thanks for this nice effort. I think it is really time to consider a
>>>next
>>> generation of Orocos RTT...
>>>
>>> If you allow me to bring in my two cents... It's possible that you
>>>won't
>>> like the
>>> message, or find it off-topic, but I think I should share it with you
>>> anyway...
>>>
>>> The best document I have read about communication that connects
>>> high-performance with flexibility and configurability is this one:
>>>
>>> 
>>><https://engineering.linkedin.com/distributed-systems/log-what-every-sof
>>>tw
>>> are-engineer-should-know-about-real-time-datas-unifying>
>>>
>>> It cleanly separates the communication of "events" from (i) what they
>>> mean,
>>> (ii) who is interested in them, and (iii) with what policy they are
>>> consumed.
>>>
>>> In our current research at KU Leuven, this is going to be _the_ basis
>>>for
>>> all interaction between asynchronous activities.
>>> The other basis will be the separation between (i) functional
>>>requirements
>>> of the application, (ii) "roles and responsibilities" of "components",
>>> (iii) software components to implement both, and (iv) deployment of
>>>these
>>> components on your hardware and OS platforms.
>>>
>>> All of the above are not present in Orocos RTT, which I consider design
>>> flaws.
>>> So solving these problems is definitely worthwhile. But, and this is a
>>>big
>>> but, my view on solving them (which is currently being transformed into
>>> a first set of models and code and tools) implies that we do not need
>>> Orocos RTT at all anymore. The reasons why we started RTT more than
>>> fifteen
>>> years ago have not disappeared, but have all been solved in more
>>> "mainstream"
>>> software projects than Orocos. Fifteen years ago, what we where after
>>>was
>>> a
>>> "node-port-connector" modelling-and-coding interface, where nodes
>>>could be
>>> deployed in realtime activities on an operating system. It was the
>>>time of
>>> single core CPUs, with only proprietary solutions to fast networking.
>>> Those
>>> days have gone, together with the need to use RTT as a software
>>>solution.
>>>
>>> More in particular, the "communication" aspect (which is the core of
>>>the
>>> document you distributed) is a lot better understood and supported in
>>> mainstream than 15 years ago. Taking the model of the event log and the
>>> corresponding log processing around it means that all the policies that
>>> your document wants to introduce as "API" methods can (should...)
>>>better
>>> be
>>> realised by configurations of the event log mechanism in your
>>>_application
>>> architecture_; there is no need to provide APIs in "middleware",
>>> "frameworks" or "libraries".
>>>
>>> _The_ core functionalities of RTT that still survive after 15 years
>>>are:
>>> - its introspection capabilities;
>>> - the lock-free data buffer.
>>> The latter is just a special case ("configuration") of the event log
>>> mechanism, that provides all other policies that you mention (and
>>>more!)
>>> just through other configurations.
>>> The introspection is a form of "model based" implementation, which
>>> considers models as first-class citizens in the software; this is
>>>great,
>>> and should be extended, starting with modelling all the relevant event
>>>log
>>> policies explicity. (KU Leuven is working on that; it takes a lot of
>>> efforts...) Lock-free data buffers are the core of event log
>>>processing,
>>> and are currently better known under the new buzzword "immutable data".
>>>
>>> The other "good-intuition-that-turned-out-to-be-badly-implemented" of
>>> fifteen years ago is that of "stored procedures" (known under various
>>> names, such as "microservices", proxy computations, etc.). RTT
>>>provides a
>>> very complex and ambiguous API to let one component execute code for
>>> another component. Again, that functionality should not be provided by
>>>an
>>> "RPC API", but by models and tools of _deployment_.
>>>
>>> The added value of Orocos as a project could shift from providing code
>>>to
>>> providing professional quality documentation and tutorials for "best
>>> practices" and "patterns", some specific realtime-optimized
>>> implementations, or patches to the large amount of existing libraries
>>>that
>>> support alul of the above.
>>>
>>> It is up to the Orocos community to decide whether to go that
>>>direction or
>>> not. I would be glad to cooperate in that directoion, because
>>>otherwise we
>>> will be forced to start a "competing" project.
>>>
>>> Note that this message does not contain threats to "fork" or whatever,
>>>but
>>> just my insights after working with Orocos RTT for about a decade, and
>>> having Orocos RTT work against the architectural requirements in our
>>> robotics projects for the subsequent five years...
>>>
>>> I frankly think it is time to reinvent the project. Big time.
>>>
>>> The easiest way is to _deprecate_ things, to make the _core_ things
>>> simpler
>>> for users, throwing out more and more code and start using other
>>>already
>>> existing software (ZeroMQ, libevent, the lock-free buffer library,
>>> RethinkDB,...) instead of trying to squeeze in their functionalities
>>> within
>>> NIH APIs and implementations.  In the end, RTT will be empty, and the
>>> development efforts can then finally shift to give Sylvain a much
>>>bigger
>>> hand with ROCK :-)
>>> (Just make sure to refactor ROCK in order to decouple the _models_ from
>>> the
>>>  Ruby and RTT _implementations_ :-))
>>>
>>> Best regards,
>>>
>>> Herman Bruyninckx
>>
>> There are some potentially interesting ideas in there, Herman. But
>>either
>> your email is uncharacteristically incoherent or I’m really jet lagged.
>> Could you explain more clearly what you currently think is the ideal?
>
>1. Complete separation of _any_ communication policy from the activity
>    execution policies: there are just way too many use cases that must be
>    supported to bring them all behind one API. (Which is also what ROS2.0
>    tries to do.)
>2. "Event processing" is the most flexible _mechanism_ to build other
> 	communication policies upon. The most important lesson that _I_ learned
> 	in this context is that it allows "clients" and "servers" to choose
>    whatever policy they like _locally_, since all data is immutable.
>(Except
>    for some "long term" garbage collection.)

Does this imply distributing all provided data that a component asks for
to that component and having it all buffered to be used according to the
local policies?

>3. Executing activities in realtime currently does not require a separate
>    library/framework anymore, but is realised by control groups (on
>Linux), or
>    similar functionalities on other OSs.
>4. Similarly, systemd has (or is in the process of) taken over all
>    configuration issues.

And caused many a flame war in the process. :)

>5. The expertise that the RTT community excels in is that of _designing_
>    realtime systems, with or without the RTT code. And that is more than
>    enough to warrant the continuation of this community, even when none
>of
>    the code is kept, since "best practices" are useful everywhere.

So reading between the lines, I think that what you’re saying is that we
should use a component-based approach at the model level, and the code
generated from that doesn’t need to have a component-based framework, it
can just use all these existing technologies to provide the functionality.
Is this correct?

Geoff


>
>> For clarity, we can skip the comments Orocos, since we’ve already
>> established that you think it needs to change.
>
>My thoughts go further: there is no need for it anymore.
>
>> Geoff
>
>Herman

-- 
Orocos-Dev mailing list
[email protected]
http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev