Re: ifile internals

[email protected] said:
> I've been taking a long look today at trying to daemonize ifile.  The
> fact of the matter is, the internals are a bit tangled up for that.
> Before I get deep into a code overhaul larger than the
> memory-leak-hunt, I figured I'd check in with you with some questions.
> (I'm deliberately asking off the mailing list, but you're welcome to
> forward this there if you'd prefer.)

> - Does this seem like something actually worth doing?  Is anyone
>   besides me complaining about performance issues?

Yes and yes.  Besides the fact that it would be cool to see <grin>,
daemonizing ifile would open up a range of possibilities.  Besides the
fact that it would make things much faster, it would also provide much
better infrastructure for remote applications.  Imagine an AI Lab-wide
spam filter daemon.  Individual clients could connect and get judgements
on incoming e-mails.  This could alleviate the individual burden
(requiring each person to set up their own spam-filtering system) while
also making it easier to detect junk e-mails since the filter could run on
a much larger corpus and it could do smart things like looking for
identical messages.

> - Do you have any insight on the best place to split client/server
>   functionality?  (I'm currently leaning toward letting the client
>   parse a message, and then shoving a serialized version of the 
>   hashtable over to the server for comparison vs. the database.)

I agree with your leaning.  I'd suggest having the client send features
(words) and counts.  The message you sent me might be trasmitted as: the 
word "long" occurred twice, the word "today" occurred once, the word 
"ifile" occurred twice, etc. (albeit in a slightly more compressed format 
:)  The server would take this "bag of words" and produce a best folder 
or list of top folders.

> - Has ifile been evolving long enough that it might be time for a 
>   fresh rewrite/simplification pass?

Certainly.  The last major re-vamping was done late '97-early '98.  
Removing all of the complex parsing/lexing from the core engine would be 
a plus, I think.

> - Will I be stepping on your toes if I take a shot at any of this? 

Not at all.  Also, considering how drastic of changes you want to make, 
it might be best to give the code a new name, maybe jfile? :)

Jason D. M. Rennie
MIT AI Lab
[email protected]
(617) 253-5339
http://www.ai.mit.edu/~jrennie/