Fidelity versus conformance

Maybe this is one digression too far from the header folding issues, but as 
I'm using Mail::Box for spam filtering and tagging, understanding 
Mail::Box's intended effects is important to me.

I was looking through RFC2822 recently -- chasing down a problem my fiancee 
had on hotmail with an email address like "O'[email protected]" (turns out 
this is legal, but if not escaped properly, screws up hotmail's javascript 
which uses single-quotes...)

I noticed that the 78 character recommendation is a SHOULD -- that's a lot 
stronger than an OPTIONAL.  Now, usually, it's been my experience that 
Mail::Box takes broken messages (particularly headers) and tries to put 
them into some appropriate form.  For example, Mail-Box will add a 
Message-ID header to a message that doesn't have one.  This actually broke 
a test case of mine when I was using a minimal message (From, Date) as a 
test case and Mail::Box kept spitting it back with a Message-ID I wasn't 
expecting.  (N.B. The RFC specifies that messages SHOULD have a message-id.)

In the case of the 78 character line, Mark made the call to not implement 
the "SHOULD" -- for performance reasons and to keep the in-memory and 
in-file versions the same.  In the case of the Message-ID, Mark follows the 
"SHOULD", despite a processing cost to generate the unique ID and the 
result that the in-memory version is different than the file input.  In 
both cases, I think he probably made the right call -- certainly, Mail::Box 
needs Message-ID for its own use anyway.)

However, this raises the broader question of whether Mail::Box should be 
trying to be as faithful as possible to the original message (from input or 
stored in a file) or whether Mail::Box should be trying to bring messages 
as far into complience with the RFC standards as possible.  Or, perhaps, 
whether this should be a user-selectable configuration option or a 
user-callable function as different users may wish to trade off conformance 
and performance/fidelity.  I.e.:

   Mail::Message->read($input, conform => 1);

or

   $msg->conformToRFC();

A related question is whether Mail::Box should instead always try to 
output/save in conformance with the RFC (including all "SHOULD" 
recommendations), i.e. when printing a message or saving/editing a message 
in a mailbox, even if it doesn't conform the message at the time it reads 
it in?  This would allow tools like SpamAssassin to retreive the message 
verbatim, but would ensure that any messages modified or generated by 
Mail::Box itself are output in the proper form.

This is clearly a philosophical debate and probably varies by user 
depending on intended use, but I think it might be worth some consideration 
and likely documentation.

Mark and others -- I'm curious to hear your views on the competing goals of 
"fidelity of original" versus "conformance to RFC".

Personally, I lean a little more towards conformance.  I'm willing to use a 
fairly heavy module like Mail::Box even with its performance costs (and 
memory footprint) because it's bulletproof or nearly so.  I want something 
that I can throw the most horribly deformed messages at (from the poorest 
implemented or most intentionally malicious spam mailers) and get something 
back that makes sense of it within the standards.  If I wanted/needed 
fidelity of source, I think I would just read the input raw and use some 
regexes for simple parsing (or else use the new Email::Simple module).

Regards, and apologies for the long-windedness,

David Golden