Re: moving mail between folders is intermittently failing

Mark Crispin wrote:

Comments on your message:

[5] It sounds like you are using traditional UNIX mailbox
format, a.k.a.

"mbox format"; a flat-file containing all the messages, one after
another,

with each message preceded with a line starting with "From ". If this
is

the case: traditional UNIX mailbox format was designed in a time when a

"very large mailbox" was between 100 KB and 800 KB in size. It does
not

scale well to sizes of 100 MB to 800 MB.

I don't know what other formats are available, moreover the "mbox"
format

I thought was actually the canonical format of a file containing
multiple email

messages and is used as well by POP. After reading about the Cyrus IMAP

server I decided not to allow "it" to determine to me what I could and
couldn't

do without it; uw-imap "plays with others".

These developers simply can not grasp the concept of a
stateful protocol

like IMAP [...]

Unfortunately, years of repeated attempts to educate these developers

proved to be utterly futile. When you talk about "state", you get
vacant

stares.

If they can understand the difference between TCP and UDP, they can
understand

state.

Thus, there is no hope of a working version of crapware
such as Outlook or

Thunderbird. The only thing that you can do is see what you can do on
the

server to not unduly stress the limitations of the crapware.

Well, with Thunderbird at least the avenue is (still?) open to muck
with the source.

Perhaps when I've nothing else to do, try to rewrite their IMAP support
to behave

properly and send them the code, or (ha ha) offer my IMAP client as a
"plugin" for

Thunderbird (ha ha.)

One way to do accomplish this is to switch mailbox formats
to mix format.

I can't imagine how anyone can tolerate traditional UNIX mailbox format

with a 100 MB mailbox, much less larger. mix format was designed to

accomodate such large mailboxes; and does in a matter of a second or
two

what can take minutes with traditional UNIX mailbox format.

[6] What you describe is typical of scaling problems. It seems to
"work"

until you reach a critical point. Very likely, the mailboxes got bit

enough that operations now tend to take long enough for Thunderbird to

decide to give up.

I've counseled splitting mail folders up and also patience to allow
operations to

complete, but evidently the volume for the users in question puts the
splitting

behind the congestion. I'm averse to storing user mail in a
noncompatible format

(meaning the POP server can't play) but may have to, for these users at
least

(I'll put them in their own sandbox.) If POP supports the mix format,
then that

objection vanishes. The fact that "mbox" is also easily human-readable
is a big

plus, but by the time the mail folder becomes 800 MB in size, being
tortured by

trying to use an editor on the file is no fun either. Then one plays
games with

"head" and "tail" and "wc". Yuk.

[7] Both Thunderbird and Outlook have a way to record protocol

negotiations. The c-client library also has a mechanism to record

protocol negotiations, at the client (not server) level.

You want to do client-based telemetry, not server-based, due to the
size

of the protocol logs. IMAP protocol logs are HUGE, and on anything
beyond

a small test system you will quickly be buried under the weight of the

logs. On a small test system, you can easily set up a script for
server

logs using "tee", at the cost of giving up SSL capability.

You quickly find out that the server logs are much less useful than you

thought. I went through this exercise on my new IMAP server.

The effects you describe explain what I see, and you've provided an
"out" (mix format),

so I will not worry about the logs. Thanks much. Thanks also to Linda
for the reference

to the other server software. I'll do fixes one way (uw+mix) or the
other (suite to test it.)

Argh, obviously the question becomes how does one convert existing mail
folders to mix

format ... or is there a magic cookie such that the "mix handler" knows
to bow out,

and or is there any automatic (blind) way that upon first use the
program can do a

conversion in the background? All I'd need is a filter to convert
"mbox" to "mix", and

stop the users getting in until that is done.

Thanks again.

-ecsd (Eric Dynamic)

Oakland, Californa

Remember to eat GILROY (California) garlic, not the imported tasteless
Chinese stuff, thanks. :)

==

On Fri, 30 Apr 2010, support wrote:

We have users whose folders are between 100
MB and 800 MB in size.

Most of those users are using Outlook but some are using Thunderbird.

Lately (and seemingly suddenly) the users are encountering trouble in
that

they will move one or more items from the inbox to another folder and
get

inconsistent results:

* sometimes the move is honored, with the item appearing in the target

folder and disappearing from the source folder.

* sometimes the move is partly honored in that the item appears in the

target folder, but is NOT deleted from the source folder.

* and sometimes the move is not honored at all, despite the client
email

program thinking for a moment that it was (before seeing in moments

after that it wasn't.)

The server is FreeBSD 5.4 using imap2006e, I think. I'll upgrade to

imap2007, whatever's current in the FreeBSD ports tree, to see if it

helps.

(1) I insured that /var/mail and /usr/tmp were permission 1777 as

suggested, but just now so I don't know if that will prove to help.

Things used to work just fine without these changes; the only things

that may have changed are (a) the users' mail folders have gotten

larger, (b) the versions of Outlook they are using are new. Thunderbird

(the latest in general) also suffers the failed transfers, though, and

only "suddenly" now for no apparent reason.

(2) I see no reference to debugging on the website. I thought I could

perhaps turn on a flag for imap in inetd to be able to track requests
to

try to see what the server was being told to do, but I see no mention
of

any such facility for either imap or c-client.

-- Mark --

http://panda.com/mrc

Democracy is two wolves and a sheep deciding what to eat for lunch.

Liberty is a well-armed sheep contesting the vote.

_______________________________________________
Imap-uw mailing list
[email protected]
http://mailman2.u.washington.edu/mailman/listinfo/imap-uw