Pipes and filters (was Re: [stack] A Joy VM in OCaml

On Tue, 22 Sep 2009, chris glur wrote:

> This forum prompted me to investigate the power of
> linux 'pipe programming',
> which is astoundingly decetive.
> It's magically easy, for small jobs, but quickly becomes unmanageable.

Hmm. It's just another tool that sort of vanishes into the background
of stuff getting done. Go digging (hard) around a major unixy project
like gcc and you'll find pipes are quietly and unobtrusively used in
many places.

It's no panacea... but it it's a nifty solution to a largish pool of
commonly occurring problems.

I once wrote, before SQL became commonly and freely available, a
largish datamining project heavily using pipes and filters.

It was very similar to "NoSql", so rather than dig out my very old
programs to show you... let me just point you at NoSql.

http://www.strozzi.it/cgi-bin/CSA/tw7/I/en_US/nosql/Home%20Page

Most database projects get hung up on the need to do updates. In the
datamining world your data is usually dumped from one or more upstream
databases and then you just digest that static dump.

Programming with pipes and filter database like nosql is a breeze. You
build up your pipeline one element at a time inspecting the output.

You can then tie it all together by dumping intermediate results in
files and use "make" to track dependencies and update things as is
needed.

However, I'm not sure how we got on to this....

I'm not sure "pipes and filters" are quite equivalent to
"concatenative" although the similarities and differences might be
interesting to explore.

A way to explore it may be looking at NoSql filter elements and some
of the nosql example pipelines.

John Carter                             Phone : (64)(3) 358 6639
Tait Electronics                        Fax   : (64)(3) 359 4632
PO Box 1645 Christchurch                Email : [email protected]
New Zealand