Re: Efficiency issues/thoughts
Jason Rennie <[email protected]>
| Newsgroups | gmane.mail.ifile.general |
|---|---|
| Message-ID | <[email protected]> |
[email protected] said: > I've trained ifile on my entire backlog of mail; as a result, .idata > is nearly 840 meg. 840k I hope? I don't think an .idata file could hit 840 meg :) My 70M of mail yields a 508k .idata file. Also, that may shrink some as you use ifile. Fewer words expire when e-mails are processed by category (like the knowledge_base.mh script does). [email protected] said: > One way of speeding up the invocation would be to have ifile run as a > daemon, and have command-line invocations just speak to the running > daemon. This has been discussed in the past, but hasn't been implemented. It would be some work and would add complexity to ifile. Certainly worth pursuing if we determine that it is necessary, but there's at least one thing to try that's much easier that might satisfy you. If you are currently using the --query-insert option (-Q) to filter new messages, try using the --query option (-q) and build .idata (e.g. run knowledge_base.mh via cron) once a day to pick up new messages. This will also speed up refiles---there's no need to hook them into ifile since they'll be picked up with the daily .idata build. The --query-insert option reads and writes the .idata file. The read is quite fast, but the write slows things down a bit. The --query option just reads the parts of .idata that it needs. Jason D. M. Rennie MIT AI Lab [email protected] (617) 253-5339 http://www.ai.mit.edu/~jrennie/