Re: Anticipatory header download

Hi John,

Thanks for your considered reply - and for not CCing me directly ;)

I agree essentially with everything you said. I never claimed TB to be
perfect, and there are a lot of things that I would love to see
fixed/added/changed.

This discussion has been valuable to me because it has point out some
shortcomings I hadn't even considered.

I'm off to the dovecot list to ask some general IMAP questions in
anticipation of adding another enhancement request or two dealing with
this issue of caching.

Because a lot of our users have literally 10's of thousands (one is over
100,000) of messages in certain folders, I would love to see an option
to limit the initial number of message headers that are downloaded on an
initial/new install to something sane (ie, your referenced 'anticipatory
mode') - say maybe 50 or 100, then continue downloading the additional
headers in a much lower priority background thread. In my enhancement
request I will even include requesting an option to limit the total
number (there is already an option for this in the newsgroups settings)
unless/until the user does something to indicate they want to see more -
ie, the user scrolls down to where they are approaching the bottom of
the list of currently downloaded headers, or maybe even clicks a button.

I don't really see a value today for anything like the 'kiosk mode' you
described, so won't go there. I vaguely see the argument, but don't
think it would be worth the trouble of implementing - although, if a
caching mechanism is smart enough, I guess it would be easy enough to
implement.

The bottom line though is, the best thing that could be done for
performance for a new install/account setup is to download the bare
minimum of headers in the main thread, then pass the rest to a low
priority background thread.

I don't believe TB does this now, but I think there are already
enhancement requests there for breaking things out into threads,
although it will likely require a lot of work.

Anyway, thanks again for your comments, and my apologies to anyone
offended by the continued discussion of TB here...

On 5/12/2015 7:39 AM, John C Klensin <[email protected]> wrote:
> Hi.  I think we are developing some confusion among "how
> Thunderbird is implemented", "how Mulberry is implemented", and
> "what are the basic capabilities required to have a competent
> IMAP client implementation".  This note is written largely from
> the perspective of those distinctions rather than to attack one
> or the other.
> 
> 
> --On Tuesday, 06:31:34 -0400 Tanstaafl
> <[email protected]> wrote:
> 
>> ...
>> It is physically impossible for any local mail client to
>> display to you the list of messages available in a folder
>> without downloading some of the headers, as these contain the
>> metadata necessary to display enough about the message so that
>> you can decide if you want to read it or not.
> 
> Yes.  And IMAP was quite carefully designed to allow a "list of
> messages" (aka "table of contents" (TOC)) download for selected
> messages or message ranges operation.  It had that capability in
> the original, online-only, mode and has always been set up so
> that there was no requirement to retain that information across
> sessions.  IIR, the earliest versions of Mulberry supported this
> via an extreme form of Online mode called "Kiosk mode" which
> like the original Stanford IMAP, was intended to be usable on
> shared client devices without causing lots of security and
> privacy problems.  That is, btw, the reason IMSP is so
> prominently supported.  It, or substitutes for it, are important
> if one wants to share preference and configuration information
> among machines, but it is essential if one wants to keep
> configuration/preference information without any permanent
> machines of one's own, especially if one uses machines that also
> do not allow access to portable storage (diskettes long ago,
> presumably USB sticks now).
> 
>> I do believe that even using a web browser to read from an
>> IMAP server will store *some* meta data of the messages
>> locally, although not in the same form as a full local mail
>> client does.
> 
> If it is implementing IMAP and not being really stuid, it is
> going to get that TOC information and it is going to get it in
> the same form everything else does.  How it organizes it is, of
> course, another matter.
> 
>> ...
> 
> The way to deal with TOC information in a large folder ("large"
> = "TOC requires more than a few screen's worth" has been
> well-understood since at least the first versions of PCMAIL and,
> I believe, early Pine.  That is to support an anticipatory mode
> in which only sufficient headers are retrieved to show a window
> into the TOC (typically a screen full), that screen is
> displayed, and more information is retrieved while the user is
> busy reading.  How important that is in practice depends on
> bandwidth, available processing power and retrieval/storage
> speed on both client and server machines, etc.  As usual, enough
> available power and speed, rewards, or at least hides, sloppy
> implementations.
> 
> Note too that more recent versions of IMAP allow retrieving and
> displaying TOC-like information using criteria other than
> explicit and continuous message ranges.
> 
> On an absolute scale of 10 and considering environments in which
> it matters, I'd give Mulberry about a 6, maybe a 7, on how it
> handles these things.  The last time I looked closely at
> Thunderbird, I would have given it about a 4.  But both are well
> above "didn't support the basic IMAP capability" or "completely
> screwed up the implementation".
> 
>    john
>