Re: seekable pipes

All,

I set this project aside when I realized I had bit off
more than I could chew. My original idea was simply to
be able to feed a pipe to any program that expected a
file, and have it just work. In particular, I wanted
to be able to feed a Postgres large-file to a
multimedia player.

I took this off the burner primarily because of the
necessity of handling multiple readers (or clients).
You suggest this as desirable, but in fact it's a
requirement, because the client can fork, and you have
to be able to handle that on both sides. Which makes
things a lot messier.

Given all this, I agree with Bryan that this makes
more sense when done with IOCTLs on the server side.
But the introduction of multiple readers brings in
concurrency, which is not easy at all. In particular,
my original idea allowed the kernel to buffer some
amount of data, just like a regular pipe. But if there
are multiple clients with read/write capability,
you're basically forced to engage in substantial
quantities of trickery, or else pass every client
system call over to the server, and wait for the
response. Not good.

All that said, I'm happy to dedicate my meager
intellectual resources to this project, but I think
the amount of hackery required to do this right is
probably beyond me.

All the best,

--Ian Turner

--- Frederik Eaton <[email protected]> wrote:

> Wow, what happened to Ian Turner's "seekable pipes"
> proposal? 
> 
>
http://marc.theaimsgroup.com/?l=linux-kernel&m=110153524529112&w=2
> 
> I have wanted to propose something very similar. I
> think it would be a
> great way of implementing a lot of functionality
> that we think of as
> belonging to more complicated userspace filesystems.
> For instance, if
> the kernel had seekable pipes then I could write a
> command that just
> serialized the protocol and allowed me to read and
> write files over an
> 'ssh' session, with very simple syntax:
> 
> some-editor <(ssh-file host:path)
> some-editor <(cmd-file ssh host open-file path)
> 
> or inside a tar file (maybe reading only), or on an
> FTP site, etc.
> 
> some-editor <(url-file ftp://host:path)
> 
> It might even make more sense than the AVFS way of
> doing things,
> because commands can take multiple file arguments:
> 
> some-editor <(patched-file file.c file.patch) #
> writes changes to file.patch
> 
> - difficult to do with AVFS file system modules
> which are "mounted" on
> a single base file.
> 
> Kernel support could mean support for mmap and exec,
> which you
> couldn't easily get from a library implementation.
> 
> Something like this could be the basis and core of a
> more complete
> userspace file system protocol. But since the
> seekable pipe concept
> also has independent applications I think it would
> be a good thing to
> factor out and focus on getting right independently
> of
> namespace-dependent functionality. Plus even without
> kernel support
> for allowing processes to export virtual directories
> to each other,
> seekable pipes would still give shared-library based
> userspace
> filesystems such as AVFS-preload a way to present
> host processes with
> a much more complete file system interface, by
> solving their mmap and
> exec problems.
> 
> Ian admits that his idea of a protocol was too
> simple, since - if you
> will allow me to use the terms 'master' and 'slave'
> - it didn't allow
> multiple slaves for one master. And it didn't allow
> the slave half of
> the pipe to write data, which I think would be an
> important feature;
> also, for good random-access performance I think the
> master should be
> able to provide a block of data with an expiration
> time which is
> cached independently of slave seek position.
> 
> But aside from that, the protocol need not be
> complex. Except for the
> problem of creating the special file descriptors,
> the rest could be
> done on the master side with read/write (read a
> request for data,
> write a block of data with a description header).
> Creating the
> descriptor pair could be done with an ioctl
> "promoting" the pipe
> descriptors as in Ian's proposal, although since
> seekable pipes
> wouldn't share much in common with regular pipes,
> the kernel pipe data
> structures would probably be created and then
> destroyed unnecessarily
> in this case - ugly if not a performance issue. I
> guess there are
> several other possibilities. Being able to name the
> slave as with a
> devpts-like solution could have advantages but
> perhaps not enough to
> justify the complexity.
> 
> I feel that this kind of feature, in some form,
> could be the shortest
> path to a lot of neat applications. What do other
> people think?
> 
> Frederik
> 
> -- 
> http://ofb.net/~frederik/
> 



		
__________________________________ 
Discover Yahoo! 
Stay in touch with email, IM, photo sharing and more. Check it out! 
http://discover.yahoo.com/stayintouch.html
-
To unsubscribe from this list: send the line "unsubscribe linux-userfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html