Re: Using ENBD for neat abstraction and network data transpor

"Also sprach Frans Grotepass:"
> We have a system with a number of clients running an embbedded linux variant. 
> The clients must reliably transport their data to the server. Network failure 
> must be taken care of in an efficient way (not like NFS does). The data 
> transport is unidirectional. 

What's wrong with NFS?  As I recall, either NFS blocks until reconnect
(option "hard") or it times out (option "soft") and/or you may "time it
out" yourself at will (option "intr").  What other options exist in the
universe?

Well, maybe  there are problems if you time out once, and then revisit,
this time without timeout.  That may present coherency problems. But
that doesn't seem to me to be directly related to network failure,
rather a reaction to unspecified failures of any kind, resulting from
the inherently stateless nature of nfs transactions.

> The idea is to share space in the server for the client(s) to access via ENBD. 
> The clients write in the space and the server reads from this space. 

The clients will not share an equal vision of this space to write
TO, unless you remove all clientside caching at all levels (which means
essentially O_DIRECT and no file system). Well - there is another ptin:
use a network aware filesystem like gfs. That should maintain
coherency.

> The discussions that I found on the internet mentions kernel cache and file 
> system cache as the two main probs in this architecture. Is there a way to 
> use ENBD for this? 

One can't get rid of the kernel block device cache except via O_DIRECT,
and then one has the problem that the FS cache still remains.  If you
use a FS, you must use an apprpriate one - either no FS cache or shared
network FS cache).  Come to think of it, "no FS cache" is not enough
either, since you need to avoid races, and the management of free space
must be coordinated too.

I did once experiment with adding a mount option "direct" to sysmount that
converted every open into an open with O_DIRECT. That treats half of
the problem, the half that can be treated with "no data FS cache".
Still leaves other problems.

> The embedded system is too small for one of the heavyweight filesystems as GFS 
> or their equivalents.

If you use NO filesystem, then you are perfectly alright. You can read
and write raw data using ENBD and O_DIRECT fine. Then you merely have a
writing races problem, which you can solve as you think fit via
shared semaphores.

Peter