Re: need a sane approach to MIME attachment organization

Mike's unattended mail spake on Tuesday 23-Oct-2012@11:11:17
> On 2012-10-21, LuKreme <[email protected]> wrote:
>>> * you lose the ability to automatically process a file attachment
>>>   and carry out realtime actions on that file the spot (unless you
>>>   keep your MUA running 24/7 and code it to do the job of procmail).
>> 
>> Actually, you don’t. Processing the file does not necessitate
>> stripping the file.
> 
> Sure, but solving one problem causes another.  Either you lose the
> ability to automate the management of your files, or you lose the
> ability to avoid redundant copies.  An inherently flawed structure
> forces you to choose a problem.  Theoretically, there is no reason we
> can't have it all.
> 
>> To my mind, the original message stays intact, period. Anything you
>> do after that should never change the messages that was received.
> 
> Why?  You seem to lack confidence that automated tools could handle
> this, no?  Is a dedicated special purpose tool more likely to corrupt
> a file than a fully featured MUA?

There are many issues here. The primary one is that the original email should always be recoverable. The original email should always be archivable. The original message should also be forward-able.

For example, let's say I have an IMAP account address of [email protected] and I change my ISP and have a new IMAP account address of [email protected].

If the messages are not destroyed by the MTA/Procmail then to transfer all my mail archives from domain1 to domain2 I simply drag the folders from one iMAP store to the other. With your method, it is impossible, because the original message has been destroyed.

> Duplication is not just a waste of space.  There are update anomalies.

No there aren't because the original email always contains the original document.

> It's generally poor management of data to have multiple copies running
> around.

Nonsense. That's exactly what a backup is.

>  When you need to destroy something, you potentially have to
> find two copies to delete.. or just one, because you won't necessarily
> know where the file came from or if it's replicated.  And when you
> update the metadata on an external file, the mailed copy doesn't get
> the update.

That's right, because the mail is untouched.

> Now suppose I accept your idea that there should be two copies, one in
> the email and a proper file that exists independantly on a filesystem.

But you're already showing a prejudice; there are not "two copies" there is the original email, and there is the document independent of the email.

> What links or associates the proper file to the email containing the
> duplicate attachment?  ATM, I've found no tools that make it trivial
> to jump between the two.

Like I said, finding an attachment in my MUA is trivial. They are not linked, nor should they be linked, any more than a backup from last month should reflect the changes you made to the document last week.

> I would be all in favor of dumping mbox for a format that maintains a
> relationship between the message and the file.  But I would not be
> keen to replace mbox with something else that also does not adequately
> establish links and metadata between the message and external content.
> 
> Suggestions?

No, because in my opinion you are looking at it completely wrong.

If the original message with the original content is not useful, then the user can delete it.

> I'll give another example.  Having the file encoded in base64 ASCII,
> and embedded in a message, is a lousy way to store a file.

But it's an excellent way to store a MESSAGE, and that's what I see; not a file, but an email.

> Consider having a need to search for text or some object embedded within your files. Your search tool must then be aware of your email archive format, and it must extract each attachment, decode it, and search it. It's CPU intensive, and human intensive as well if files are embedded in encrypted messages.

So you might need to go back to the original message to get the original attachment? Or not? Because you appear to be saying both things here.

>>> * the file must be extracted and potentially decrypted every single
>>>   time it is opened from the message (if not duplicated).
>> 
>> This is a good thing.
> 
> Not if you've forgotten the password that decrypts the private key
> needed to decrypt the file.  This has actually happened to me, btw.
> And keys expire.

If you extracted the file originally, why would *you* ever need to go back to the message to re-extract the file? You are specifically asking for a way to prevent this automatically, aren't you?

If the message is not valuable as it is, delete it.

> Hold on.  A /search/ is one way to find things, and
> browsing/navigation is another.  Having a good search capability is
> only half of it.  If you optimize your organization for browsing
> attachments, you compromise on browsing for messages, and vice-versa.

No, that doesn't follow at all.

>> procmail was written long before MIME email became common.
> 
> Does this mean the language (and thus tool) does not evolve?

procmail has not received significant updates since the late 90's. The current version, featuring a few tweaks and bug fixes, is from 2001. I would say that in computer terms procmail is definitely not evolved, and practically Jurassic.

> If I wanted to join the project to expand the language and add some basic
> features, would this be welcome?

Welcome? Yes. Problematic? Probably also yes. I do not know who 'owns' the repo at this point. You would probably need to fork the current source and create your own repo. So far, no one has taken up the task of updating procmail, despite various threats. For those of us using it, it's good enough for what we need to do. If it falls down too much for what we need, then we look at something like maildrop.

Maildrop is especially good when using single-file-per-message stores like Maildir.

That said, MIME and attachment handling is problematic, period.

-- 
When treading water in a circle of sharks, a wizard will always consider
other wizards to be the most immediate danger. --The Last Continent