SUMMARY: Multiprocess programming and enterprise indexing

Hey there, everyone.

As always, LinuxManagers was very helpful.  bishop and Evan Cofsky 
replied, suggesting that pipes are not the way to go for interprocess 
communication.  Since we plan to develop most of our software in Python 
(except perhaps for the Windows search client), it made sense to delve 
into sockets.  Turns out sockets are a very UNIX-(and windows)portable 
way to do what we want to do: have multiple clients post questions and 
get replies from one server.  Domain sockets have the same API as INET 
sockets and are, in fact, what PostgreSQL and MySQL seem to use 
(/var/lib/pgsql/*.sock).  Turns out that there are already a couple of 
concurrent socket server implementations for Python, and the best one 
was XML-RPC lib for python.  Three lines of code and you have a server 
running.  This fits within our model of open interprocess 
communication.  There is ample documentation in the subject, as well.

Good.  That was summary for question two.  Question one was a feeble 
attempt at gathering interest around this project.  That one didn't get 
much positive input.  When it's ready, be assured that the OSS community 
and Linux in general will see itself two cents closer to world domination. 

As always, thanks a lot.  Good luck!

      Manuel

---------------------
Quoting my full questionnaire:

Hi there, everyone!  I'm a frequent reader of LinuxManagers and I'd like 
to come up with another question for LinuxManagers to solve =)

This is the deal.  I'm developing an application which will serve as an 
enterprise real-time full-text-index for Linux.  We plan to release a 
fully functional version as open-source, and release an Enterprise 
edition with the ability to index networked NFS partitions, reply to 
queries posed by Windows SAMBA clients, and the ability to index 
proprietary file formats.  Our release date would be 6 months from now.  
The end-user model of the product is a search tool, akin to the Windows, 
KDE and GNOME search tools (we sure hope that at least the GNOME and KDE 
people piggyback on our services).  This search tool would let the user 
find any file based on properties/metadata (author, album, date, title, 
subject, etcetera) and data (full-text-indexing) in under ten seconds 
for ten thousand files.

Two questions.  On the business side of it, what would it be like for 
your company/corporation/workplace to have such a tool?  For most large 
corporations (or any with a large file server), there will exist the 
need for the enterprise version, because of the networked index 
capabilities, and because of the proprietary indexing thingie.

The second question is that we're planning to serve search requests 
through a search service (a daemon).  This daemon would listen through a 
UNIX named pipe (FIFO).  Now, if two different searches come through the 
same pipe, how can the search service distinguish between one search and 
the other?  What I mean is, how can I multiplex a single pipe's multiple 
inputs?  Are pipes two-way in Linux/Solaris?  This is an easy task if 
it's done with TCP/IP, but it's much trickier if using pipes.  Now, I 
can't use TCP/IP because of security and functionality constraints in 
our requirements list, but pipes aren't ruled out.

To try to make it more clear: applications wishing to pose a search 
query write to a pipe, and on the other side of the pipe the search 
service gets the query, queries the index and replies with a set of 
results.  Now, I've seen that PostgreSQL and MySQL both communicate with 
their clients through a single pipe (and TCP/IP).  How do they 
distinguish between multiple simultaneous requests?

Tricky question?  I understand how to make a program into a daemon, and 
how to read from/write to pipes.
good luck, and thanks for your assistance =)

       Manuel Amador
_______________________________________________
LinuxManagers mailing list - http://www.linuxmanagers.org
submissions: LinuxManagers-35TzE1X9F6582KRnZfj+bdi2O/[email protected]
subscribe/unsubscribe: http://www.linuxmanagers.org/mailman/listinfo/linuxmanagers