Re: [binc] program HOOKS - feature request?

>I still don't see why you want to put this type of "space compressing"
>in the IMAP layer rather then to put it in the maildrop layer where it
>would do the most good.

The maildrop is the one doing the symlink and managing email ID's.
Binc just acts like a normal IMAPD reading a  NFS mounted  Maildir..
BUT   I just wanted Binc to have a text filter when it opens each email from 
the Maildir.
-- This is just a feature request... for the filter.
%%full_name%%  %%company%%  tags are unlikely to change, so to the end 
client they
won't see a message that appears different everytime they download it.


>Having the MDA sorting out if the recent 100 droped mails could be the
>same and then moving it to a shared location and linking it into the
>users Maildir seems to me to be the effect you want.

Yes it is. And the MDA does it.

>Having the space compressed right away insteed of the MDA duplicating the 
>mails and
>waiting for binc to HOOKS it right and _then_ compressing.
>That would mean that the 10MB mail gets duplicated 100 times until the
>users checks its INBOX.

Binc has nothing to do with the Symlink system other than CAT (reading) the 
symlink file.
>Again, this should be done directly by the MDA.
It is.


> If performance and reliability is important, then you must remove the NFS
> mount and store the emails physically on disk. NFS is unreliable in its
> very nature, as there is no guarantee that emails are committed to disk.

The NFS is 99% ONE-WAY [read], the MDA uses sockets to store the messages on 
the NFS server.
Binc would only WRITE to the NFS when a person checks their email (to 
uncheck it) or moves a message.
http://www.time-travellers.org/shane/papers/NFS_considered_harmful.html

I don't think caching would be a big issue, as the client is reading their 
email from
which-ever IMAPD server they connected to. [caching could be turned off 
also, to ensure no latency]

> If you store emails on an NFS mount and your machines crash, then email
> data will certainly be lost
Depends when it crashed and what it was doing. If it was unchecking an email 
then
the user would see it as unread still (minor), if it was moving a message 
then as long as the headers were
written to the NFS then the symlink system can recover. [Clients create a 
new file and copy the
contents of the EMAIL in the new folder when a message is moved.] A cron job 
on the NFS scans for
files that are NOT symlinks... reads the ID in the headers, deletes the 
email, and makes a symlink.
The client would notice that an email was deleted and that a new email
 (marked as read or unread) now exists and would download it.

I've moved  files back and forth in on a shell to controll spam and never 
had trouble with deleting or moving maildir files.
The imap client just updates it's  email list and downloads/deletes messages 
as they exist in the FS.



>. Also performancewise, NFS is a killer. The load on your system will be
>much higher than with a physical volume.
Thats expected. Though the Gigabit network should have enough bandwidth.
There are OTHER alternatives to NFS, like SHFS and

Unless we put 3 TB of space on EVERY imap server and use Rsync, I don't see 
an alternative.
-- NFS is pretty widely used in the hosting world. I can have a dozen 
servers load-balance a single site.
When the user FTP's they all get updated.


> If saving space is important, then of course, symlinking emails will do
> some good for you. But it's much better to do what the huge email stores
> do, and use a file system that eliminates duplicates for you. If you want
> to use the accessor, or IMAP server, to work with symlinks and so on, then
> you will be breaking important features in the mailbox specification that
> directly affect reliability.

Binc IS NOT the program doing the Symlinks! It just has to read them.
I'm just asking if a Text Filter could be added to Binc when it reads a 
files in the Maildir
so that the 'single email'  could be customized based on the users login ID.
(personalize the email if tags were embedded)

> If I was to do something like what you are describing, then I'd define a
> specialized mailbox format that stores IDs only, with message bodies
> stored in a shared collection. Then I'd write a backend for the mailstore
> accessors that allowed them to deal with this format..

Already done. It's now part of the MTA. We noticed that our Tagged Emails 
now break as
there is nothing to filter them. The MTA can't filter them as a dozen or 
more people read the same one.

> In any case, if you start messing with message contents after they have
> been delivered to a Maildir mailbox (or any other mailbox), then IMAP
> clients are at risk to go haywire... :-)

The client should never notice the contents changing. The filter (added to 
Binc)
would make the same replacements, time and again when the user RE-downloads 
a message.
-- Clients usually cache known/read emails and don't redownload unless told 
to. As the filename
is the ID, I'd doubt Binc would have to refilter the same message more than 
once a month.

Notes that the Filter is actually a 3rd party perl script (yes, there is a 
performance hit)
I just wanted to know or ask if I could get a HOOK in the   fileopen section 
of binc to call the script, if it exists.