Re: [Imap-protocol] Parsing, part numbering and BODYSTRUCTURE

--===============3854513059260697794==
Content-Type: multipart/alternative; boundary=089e01294004f8c4c00519a3f99e

--089e01294004f8c4c00519a3f99e
Content-Type: text/plain; charset=UTF-8

On 29 June 2015 at 07:11, David Harris <[email protected]> wrote:

> For reasons that aren't relevant here, I'm in the process of rewriting my
> MIME
> parser for about the fifth time in twenty-five years. Each time I do this
> I find I spend
> a lot of time trying to reconcile the way I do my parsing with the demands
> of IMAP. I
> should probably keep notes each time, but I never do. *sigh*.
>
> A lot of the trouble I have comes from the paucity of detail in RFC3501
> over two key
> issues - part numbering, and BODYSTRUCTURE. This is not helped by what
> appears to me to be an erratum - the sample numbering scheme shown on page
> 56, which appears to suggest that the bare part number for any part of a
> message
> references the first byte of the part INCLUDING any MIME headers it might
> have (if
> you look at 4.1, it is *followed* by 4.1.MIME, which appears to suggest
> that the
> MIME headers are a subset of 4.1).
>
> So here's my first question: could someone confirm for me that a bare part
> number
> (such as "4.1") refers to the part starting at the first byte *following*
> the CRLF at the
> end of its MIME headers?
>
>
Yes. Well, for leaf parts, anyway.


> Next, in a BODYSTRUCTURE, do the line and octet counts for such a part
> include
> the MIME headers, or not? I believe the correct answer is "not", but would
> like to
> know for sure.
>
>
I'd agree, octet counts there are expected to be those for the part itself
and not the headers.


> This leads to my next question, which is "is BODYSTRUCTURE reversible"?
> That
> is, if you parse a message, build a BODYSTRUCTURE from the parsed data,
> then
> re-parse the BODYSTRUCTURE, will the two parses be the same? I have to
> clarify
> here, because this question depends on context: if you're parsing for an
> IMAP
> server, it's quite reasonable to assume that your parser will build two
> entries for
> each part, the first tracking the offset of the  MIME headers for the
> part, the second
> tracking the offset of the part itself: this allows you to do a simple
> lookup to satisfy
> fetches for both <partnumber> and <partnumber>.MIME... Yet it seems to me
> that
> you cannot reconstruct this information from a BODYSTRUCTURE - you would
> lose the offset to the MIME headers. Why am I asking this? I'm trying to
> work out if
> it's possible to use BODYSTRUCTURE as a way of storing a parse between
> invocations, since it's always going to be far quicker to parse a
> BODYSTRUCTURE
> than it is to read the entire message again.
>
>
So by "is BODYSTRUCTURE reversible", I thought you meant something else
entirely.

But no, BODYSTRUCTURE itself doesn't contain the offsets into the message,
and may have normalized other parts of the data. A server would need more
data, and as I recall it's not quite a superset either - there are items
you need for the BODYSTRUCTURE which aren't otherwise useful for a server.

But - also as I recall - Cyrus IMAP does a single parse which extracts both
a server-side structure and the BODYSTRUCTURE.


> Finally, is there a detailed discussion of part numbering and BODYSTRUCTURE
> anywhere? I had a look through the RFC index and couldn't see any other
> documents that might expand on these subjects, and google didn't yield
> anything
> helpful either. And in a similar vein, is there a repository anywhere of
> sample
> messages with matching canonical part number listings and bodystructures?
> This
> would be extremely helpful in testing parsers and bodystructure generators.
>
>
I vaguely recall a lengthy discussion on a mailing list (either this one or
imapext) a few years back, but I can't find it immediately either.


> I'm sure this has all been asked a billion times before, and I apologize
> for that, but
> any guidance would be gratefully received.
>
> Cheers!
>
> -- David --
>
> ------------------ David Harris -+- Pegasus Mail ----------------------
> Box 5451, Dunedin, New Zealand | e-mail: [email protected]
>            Phone: +64 3 453-6880 | Fax: +64 3 453-6612
>
> Thought for the day:
>    A diplomat is a man who can convince his wife she'd look
>    stout in a fur coat.
>
>
>
> _______________________________________________
> Imap-protocol mailing list
> [email protected]
> http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol
>

--089e01294004f8c4c00519a3f99e
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><br><div class=3D"gmail_extra"><br><div class=3D"gmail_quo=
te">On 29 June 2015 at 07:11, David Harris <span dir=3D"ltr">&lt;<a href=3D=
"mailto:[email protected]" target=3D"_blank">[email protected]=
.nz</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"mar=
gin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">For reasons tha=
t aren&#39;t relevant here, I&#39;m in the process of rewriting my MIME<br>
parser for about the fifth time in twenty-five years. Each time I do this I=
 find I spend<br>
a lot of time trying to reconcile the way I do my parsing with the demands =
of IMAP. I<br>
should probably keep notes each time, but I never do. *sigh*.<br>
<br>
A lot of the trouble I have comes from the paucity of detail in RFC3501 ove=
r two key<br>
issues - part numbering, and BODYSTRUCTURE. This is not helped by what<br>
appears to me to be an erratum - the sample numbering scheme shown on page<=
br>
56, which appears to suggest that the bare part number for any part of a me=
ssage<br>
references the first byte of the part INCLUDING any MIME headers it might h=
ave (if<br>
you look at 4.1, it is *followed* by 4.1.MIME, which appears to suggest tha=
t the<br>
MIME headers are a subset of 4.1).<br>
<br>
So here&#39;s my first question: could someone confirm for me that a bare p=
art number<br>
(such as &quot;4.1&quot;) refers to the part starting at the first byte *fo=
llowing* the CRLF at the<br>
end of its MIME headers?<br>
<br></blockquote><div><br></div><div>Yes. Well, for leaf parts, anyway.</di=
v><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Next, in a BODYSTRUCTURE, do the line and octet counts for such a part incl=
ude<br>
the MIME headers, or not? I believe the correct answer is &quot;not&quot;, =
but would like to<br>
know for sure.<br>
<br></blockquote><div><br></div><div>I&#39;d agree, octet counts there are =
expected to be those for the part itself and not the headers.</div><div>=C2=
=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;borde=
r-left:1px #ccc solid;padding-left:1ex">
This leads to my next question, which is &quot;is BODYSTRUCTURE reversible&=
quot;? That<br>
is, if you parse a message, build a BODYSTRUCTURE from the parsed data, the=
n<br>
re-parse the BODYSTRUCTURE, will the two parses be the same? I have to clar=
ify<br>
here, because this question depends on context: if you&#39;re parsing for a=
n IMAP<br>
server, it&#39;s quite reasonable to assume that your parser will build two=
 entries for<br>
each part, the first tracking the offset of the=C2=A0 MIME headers for the =
part, the second<br>
tracking the offset of the part itself: this allows you to do a simple look=
up to satisfy<br>
fetches for both &lt;partnumber&gt; and &lt;partnumber&gt;.MIME... Yet it s=
eems to me that<br>
you cannot reconstruct this information from a BODYSTRUCTURE - you would<br=
>
lose the offset to the MIME headers. Why am I asking this? I&#39;m trying t=
o work out if<br>
it&#39;s possible to use BODYSTRUCTURE as a way of storing a parse between<=
br>
invocations, since it&#39;s always going to be far quicker to parse a BODYS=
TRUCTURE<br>
than it is to read the entire message again.<br>
<br></blockquote><div><br></div><div>So by &quot;is BODYSTRUCTURE reversibl=
e&quot;, I thought you meant something else entirely.</div><div><br></div><=
div>But no, BODYSTRUCTURE itself doesn&#39;t contain the offsets into the m=
essage, and may have normalized other parts of the data. A server would nee=
d more data, and as I recall it&#39;s not quite a superset either - there a=
re items you need for the BODYSTRUCTURE which aren&#39;t otherwise useful f=
or a server.</div><div><br></div><div>But - also as I recall - Cyrus IMAP d=
oes a single parse which extracts both a server-side structure and the BODY=
STRUCTURE.</div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D=
"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Finally, is there a detailed discussion of part numbering and BODYSTRUCTURE=
<br>
anywhere? I had a look through the RFC index and couldn&#39;t see any other=
<br>
documents that might expand on these subjects, and google didn&#39;t yield =
anything<br>
helpful either. And in a similar vein, is there a repository anywhere of sa=
mple<br>
messages with matching canonical part number listings and bodystructures? T=
his<br>
would be extremely helpful in testing parsers and bodystructure generators.=
<br>
<br></blockquote><div><br></div><div>I vaguely recall a lengthy discussion =
on a mailing list (either this one or imapext) a few years back, but I can&=
#39;t find it immediately either.</div><div>=C2=A0</div><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex">
I&#39;m sure this has all been asked a billion times before, and I apologiz=
e for that, but<br>
any guidance would be gratefully received.<br>
<br>
Cheers!<br>
<br>
-- David --<br>
<br>
------------------ David Harris -+- Pegasus Mail ----------------------<br>
Box 5451, Dunedin, New Zealand | e-mail: <a href=3D"mailto:David.Harris@pma=
il.gen.nz">[email protected]</a><br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Phone: <a href=3D"tel:%2B64%203%20=
453-6880" value=3D"+6434536880">+64 3 453-6880</a> | Fax: <a href=3D"tel:%2=
B64%203%20453-6612" value=3D"+6434536612">+64 3 453-6612</a><br>
<br>
Thought for the day:<br>
=C2=A0 =C2=A0A diplomat is a man who can convince his wife she&#39;d look<b=
r>
=C2=A0 =C2=A0stout in a fur coat.<br>
<br>
<br>
<br>
_______________________________________________<br>
Imap-protocol mailing list<br>
<a href=3D"mailto:[email protected]">[email protected]=
n.edu</a><br>
<a href=3D"http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol=
" rel=3D"noreferrer" target=3D"_blank">http://mailman13.u.washington.edu/ma=
ilman/listinfo/imap-protocol</a><br>
</blockquote></div><br></div></div>

--089e01294004f8c4c00519a3f99e--

--===============3854513059260697794==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Imap-protocol mailing list
[email protected]
http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol
--===============3854513059260697794==--