Re: Updated dataflow semantics for RTT

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

--8323329-1212148483-1443444449=:30972
Content-Type: TEXT/PLAIN; charset=ISO-8859-7; format=flowed
Content-ID: <alpine.DEB.2.11.1509281447351.30972@pma-15-011>
Content-Transfer-Encoding: quoted-printable

On Mon, 28 Sep 2015, Sylvain Joyeux wrote:

> Some thoughts on that executive summary (I'm like geoffrey, I had a
> hard time following the first email :P)
>
> One thing I definitely want to warn you about: don't think, ever, that
> code-generating everything is a replacement for a well-designed
> library that solve most of the problems, along with a code generator
> for the one percent.

I am very well aware of this trap, but it cannot be repeated enough.

> With this premise: I actually that your points are not making RTT
> obsolete. It is actually making it even more relevant than ever.
>
> RTT *is* the best starting point towards something where
> process/thread policies is delegated to the tooling (either accessing
> "raw" OS-specific functionality or exposing them under an
> easier-to-use interface). It is also *already* provides an
> encapsulation of computation, which allows you to "inverse" the
> current tendency and actually make your middleware central to your
> communication setup -- instead of hiding it behind a common API.

I agree with your analysis in the context of "the current tendency" to
information hiding behind APIs :-)

But I do not agree with the analysis that RTT is the best thing to hide
computations: I have seen way too many control loops implemented with one
single Orocos component (and activity) per block in a Simulink-like data
flow... Way too many.

> Sylvain

Herman

>
> 2015-09-28 2:28 GMT-03:00 Herman Bruyninckx
> <[email protected]>:
>> On Mon, 28 Sep 2015, BiggsGeoffrey wrote:
>>
>>> On 27/9/15, 14:10, "[email protected] on beha=
lf of
>>> Herman Bruyninckx" <[email protected] on beha=
lf of
>>> [email protected]> wrote:
>>>>
>>>>
>>>> Thanks for this nice effort. I think it is really time to consider a=
 next
>>>> generation of Orocos RTT...
>>>>
>>>> If you allow me to bring in my two cents... It's possible that you w=
on't
>>>> like the
>>>> message, or find it off-topic, but I think I should share it with yo=
u
>>>> anyway...
>>>>
>>>> The best document I have read about communication that connects
>>>> high-performance with flexibility and configurability is this one:
>>>>
>>>>
>>>> <https://engineering.linkedin.com/distributed-systems/log-what-every=
-softw
>>>> are-engineer-should-know-about-real-time-datas-unifying>
>>>>
>>>> It cleanly separates the communication of "events" from (i) what the=
y
>>>> mean,
>>>> (ii) who is interested in them, and (iii) with what policy they are
>>>> consumed.
>>>>
>>>> In our current research at KU Leuven, this is going to be _the_ basi=
s for
>>>> all interaction between asynchronous activities.
>>>> The other basis will be the separation between (i) functional
>>>> requirements
>>>> of the application, (ii) "roles and responsibilities" of "components=
",
>>>> (iii) software components to implement both, and (iv) deployment of =
these
>>>> components on your hardware and OS platforms.
>>>>
>>>> All of the above are not present in Orocos RTT, which I consider des=
ign
>>>> flaws.
>>>> So solving these problems is definitely worthwhile. But, and this is=
 a
>>>> big
>>>> but, my view on solving them (which is currently being transformed i=
nto
>>>> a first set of models and code and tools) implies that we do not nee=
d
>>>> Orocos RTT at all anymore. The reasons why we started RTT more than
>>>> fifteen
>>>> years ago have not disappeared, but have all been solved in more
>>>> "mainstream"
>>>> software projects than Orocos. Fifteen years ago, what we where afte=
r was
>>>> a
>>>> "node-port-connector" modelling-and-coding interface, where nodes co=
uld
>>>> be
>>>> deployed in realtime activities on an operating system. It was the t=
ime
>>>> of
>>>> single core CPUs, with only proprietary solutions to fast networking=
.
>>>> Those
>>>> days have gone, together with the need to use RTT as a software solu=
tion.
>>>>
>>>> More in particular, the "communication" aspect (which is the core of=
 the
>>>> document you distributed) is a lot better understood and supported i=
n
>>>> mainstream than 15 years ago. Taking the model of the event log and =
the
>>>> corresponding log processing around it means that all the policies t=
hat
>>>> your document wants to introduce as "API" methods can (should...) be=
tter
>>>> be
>>>> realised by configurations of the event log mechanism in your
>>>> _application
>>>> architecture_; there is no need to provide APIs in "middleware",
>>>> "frameworks" or "libraries".
>>>>
>>>> _The_ core functionalities of RTT that still survive after 15 years =
are:
>>>> - its introspection capabilities;
>>>> - the lock-free data buffer.
>>>> The latter is just a special case ("configuration") of the event log
>>>> mechanism, that provides all other policies that you mention (and mo=
re!)
>>>> just through other configurations.
>>>> The introspection is a form of "model based" implementation, which
>>>> considers models as first-class citizens in the software; this is gr=
eat,
>>>> and should be extended, starting with modelling all the relevant eve=
nt
>>>> log
>>>> policies explicity. (KU Leuven is working on that; it takes a lot of
>>>> efforts...) Lock-free data buffers are the core of event log process=
ing,
>>>> and are currently better known under the new buzzword "immutable dat=
a".
>>>>
>>>> The other "good-intuition-that-turned-out-to-be-badly-implemented" o=
f
>>>> fifteen years ago is that of "stored procedures" (known under variou=
s
>>>> names, such as "microservices", proxy computations, etc.). RTT provi=
des a
>>>> very complex and ambiguous API to let one component execute code for
>>>> another component. Again, that functionality should not be provided =
by an
>>>> "RPC API", but by models and tools of _deployment_.
>>>>
>>>> The added value of Orocos as a project could shift from providing co=
de to
>>>> providing professional quality documentation and tutorials for "best
>>>> practices" and "patterns", some specific realtime-optimized
>>>> implementations, or patches to the large amount of existing librarie=
s
>>>> that
>>>> support alul of the above.
>>>>
>>>> It is up to the Orocos community to decide whether to go that direct=
ion
>>>> or
>>>> not. I would be glad to cooperate in that directoion, because otherw=
ise
>>>> we
>>>> will be forced to start a "competing" project.
>>>>
>>>> Note that this message does not contain threats to "fork" or whateve=
r,
>>>> but
>>>> just my insights after working with Orocos RTT for about a decade, a=
nd
>>>> having Orocos RTT work against the architectural requirements in our
>>>> robotics projects for the subsequent five years...
>>>>
>>>> I frankly think it is time to reinvent the project. Big time.
>>>>
>>>> The easiest way is to _deprecate_ things, to make the _core_ things
>>>> simpler
>>>> for users, throwing out more and more code and start using other alr=
eady
>>>> existing software (ZeroMQ, libevent, the lock-free buffer library,
>>>> RethinkDB,...) instead of trying to squeeze in their functionalities
>>>> within
>>>> NIH APIs and implementations.  In the end, RTT will be empty, and th=
e
>>>> development efforts can then finally shift to give Sylvain a much bi=
gger
>>>> hand with ROCK :-)
>>>> (Just make sure to refactor ROCK in order to decouple the _models_ f=
rom
>>>> the
>>>>  Ruby and RTT _implementations_ :-))
>>>>
>>>> Best regards,
>>>>
>>>> Herman Bruyninckx
>>>
>>>
>>> There are some potentially interesting ideas in there, Herman. But ei=
ther
>>> your email is uncharacteristically incoherent or I=A2m really jet lag=
ged.
>>> Could you explain more clearly what you currently think is the ideal?
>>
>>
>> 1. Complete separation of _any_ communication policy from the activity
>>    execution policies: there are just way too many use cases that must=
 be
>>    supported to bring them all behind one API. (Which is also what ROS=
2.0
>>    tries to do.)
>> 2. "Event processing" is the most flexible _mechanism_ to build other
>>         communication policies upon. The most important lesson that _I=
_
>> learned
>>         in this context is that it allows "clients" and "servers" to c=
hoose
>>    whatever policy they like _locally_, since all data is immutable. (=
Except
>>    for some "long term" garbage collection.)
>> 3. Executing activities in realtime currently does not require a separ=
ate
>>    library/framework anymore, but is realised by control groups (on Li=
nux),
>> or
>>    similar functionalities on other OSs.
>> 4. Similarly, systemd has (or is in the process of) taken over all
>>    configuration issues.
>> 5. The expertise that the RTT community excels in is that of _designin=
g_
>>    realtime systems, with or without the RTT code. And that is more th=
an
>>    enough to warrant the continuation of this community, even when non=
e of
>>    the code is kept, since "best practices" are useful everywhere.
>>
>>> For clarity, we can skip the comments Orocos, since we=A2ve already
>>> established that you think it needs to change.
>>
>>
>> My thoughts go further: there is no need for it anymore.
>>
>>> Geoff
>>
>>
>> Herman
>> --
>> Orocos-Dev mailing list
>> [email protected]
>> http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev
>>
>
--8323329-1212148483-1443444449=:30972
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

-- 
Orocos-Dev mailing list
[email protected]
http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev

--8323329-1212148483-1443444449=:30972--