Re: Frequent timeouts

Bob Atkins <[email protected]> Fri, 30 Jul 2010 10:43:26 -0700
Newsgroups gmane.mail.imap.uw.c-client
Organization DigiLink, Inc.
Message-ID <[email protected]>
Moray,

You are running into the limitations of mbx formatted boxes - especially 
when they are >2GB. Everytime a client requests a listing of the 
contents of the mail box - the server must perform a complete scan of an 
mbx file. In addition, when a message is appended to an mbx file another 
scan is performed. There simply is no way to get around the fact that 
mbx is basically a modified serial text file, no random access/insert is 
possible and when the size runs into the multi gigabyte level it just 
takes a /_*lot*_/ of I/O to manage it. I suspect that you will find your 
system is struggling with high levels of I/O wait.

 From personal experience I can say that the solution to your problem is 
to convert to mix mailbox format. I realize that the conversion of all 
users' mail stores on an established mail server is not trivial, 
especially on a live system so, in an effort to help, I have attached a 
script that I wrote and used to perform the conversion on our mail 
severs that are Sun Solaris systems. The script should be easily adapted 
to a linux system. It does all of the necessary checks, descends the 
user's mail folder tree (correctly determining whether it is mbox, mbx 
or a combination of mbx and mix folders) and does it's best to do the 
'right' thing, avoiding re-converting any already existing mix format 
mail folders. It also preserves the original mail folder in case 
something goes wrong. You can convert a single user or all users in the 
passwd file. It won't touch uids <500 or >60000. It also won't create 
mail stores for a user if none already exist. I offer this script 
without any support or warranty, express or implied. I suggest running 
it against a test user account to make sure that you are satisfied with 
the results. I hope that it will be helpful to anyone struggling with 
the type of problem that you are having with overloaded mbx files.

In addition to the conversion you will need to change the default 
operation of your uw-imap to use mix format folders and you /_*should*_/ 
also convert to using tmail as the local delivery agent. We did this 
when we compiled uw-imap. You don't have to use tmail, but I highly 
recommend it.

Good luck.
-- 
*Bob Atkins * 
/President/CEO/

*DigiLink, Inc. <http://www.digilink.net>*
Business Inter-net-working
*/The Cure for the Common ISP!/*

	

Phone: (310) 577-9450
Fax: (310) 577-3360
eMail: [email protected]



Guy Dawson wrote:
> On 30/07/2010 11:34, Moray Henderson wrote:
>> Can anyone give me some pointers on tracking down timeout problems?  I
>> have users saying Thunderbird is frequently giving "Connection to IMAP
>> server timed out" on their local network.  They are using uw-imap-2007e
>> with MBX mail folders on CentOS 5.4 server, Thunderbird 3.1.1 on Windows
>> client (with or without SSL).  Apparently it used to happen a few times
>> with uw-imap-2002d on CentOS 4, but is much worse now that they have
>> upgraded.
>
> When we virtualised our email server we started with a test system
> running 2007e and mbx and did experience some client timeout issues. The
> old physical server was running and older UW-IMAP server and mbx OK.
> I do wonder if 2007e is slower when accessing mbx than earler versions.
>
>> Does MIX store everything in one file like MBX, or is it all in separate
>> files?
>
> Neither. It stores multiple messages per file and has a maximum file
> site which can be configured at compile time. Lots of little messages
> will be stored in the same file while large messages will get a file
> of their own. MIX also has separate index files.
>
> I have one user with a 3.5GB inbox, 6954 .mix data files, a 5MB .mixindex
> file, a 2.7MB .mixstatus file and approximately 120,000 messages.
>
> Touch wood, mix is working OK!
>
> Regards,
> Guy
> -- --------------------------------------------------------------------
> Guy Dawson                 I.T. Systems Manager         Crossflight Ltd
> [email protected]         07973  797819                01753 776104
>
>
> *******************************************************************
> This message contains the views and opinions of a Crossflight
> Limited employee and at this stage are in no way a direct
> representation of Crossflight Limited.
>
> Crossflight Limited is an international express courier, mailing
> and logistics service provider. This communication and any
> attachments are confidential and may be protected from disclosure.
> We endorse no advice or opinion contained in this communication
> that is not the subject of a contract between the recipient and
> ourselves.
>
> If you have received it in error please notify us immediately and
> note that any storage, use or disclosure is strictly prohibited and
> may be unlawful. Those communicating with us by electronic mail
> will be deemed to have accepted the risks associated with
> interception, amendment, loss and late or incomplete delivery.
> They will also be deemed to have consented to our intercepting and
> monitoring such communications.
>
> This footnote also confirms that this message has been checked for
> the presence of computer viruses. We strongly recommend that you
> check this email with your own virus software as Crossflight
> Limited will not be held responsible for any damage caused by
> viruses as a result of opening this email.
> *******************************************************************
> _______________________________________________
> Imap-uw mailing list
> [email protected]
> http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
>

_______________________________________________
Imap-uw mailing list
[email protected]
http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
mail2mix.sh (text/plain, 6.2 KB)
#!/bin/sh
#
# -- Convert mailboxes to mix format for every user
#
# -- Author: Bob Atkins, DigiLink Network Services
#
# -- First Release : 2/16/08
#
# -- Revised:
#	8/25/09 - Added options to convert a single user. Added checks for already
#		converted mix format boxes so that they are not re-converted.
#	9/9/09 -  Create user INBOX after the conversion if it doesn't already exist.
#
#	Copyright (C) 2008, Bob Atkins
#
#    This program is free software: you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation, either version 3 of the License, or
#    (at your option) any later version.

#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.

#    You should have received a copy of the GNU General Public License
#    along with this program.  If not, see <http://www.gnu.org/licenses/>
#
PROGNAME=`basename $0`
USAGE="Usage: $PROGNAME [-u username] [-x]"
SIGNAL=-TERM
MIXCVT=mixcvt
MAILUTIL=mailutil
OSVER=`uname -r`
SHUTDOWN=y
umask 0077

usage ()
{	echo $USAGE

cat << "-EOF-"

Converts existing mail stores of all users in the system passwd file to
UW-Imap mix format.

Where:

	-u	Specifiy a single user to convert
	-x	Do NOT shutdown imap/pop mail services.
		Use with CAUTION!

    mail2mix.sh  Copyright (C) 2008, Bob Atkins
    This program comes with ABSOLUTELY NO WARRANTY.
    This is free software, and you are welcome to redistribute it
    under certain conditions.
-EOF-

}

# -- Process any args

if [ $# -ne 0 ]; then
    while getopts h\?u:x flags; do
	case $flags in
	u)	USERNAME="${OPTARG}";;	# Pickup arg to append to the package name
	x)	SHUTDOWN=n;;
	? | h)	usage; exit 4;;
    esac

    done

    if [ -z "$USERNAME" ]; then
	echo "$PROGNAME -- Invalid arg/no username specified"
	usage; exit 4
    fi

else
    echo "\n!!!!  WARNING  !!!!\n"
    echo "This script will convert existing mail stores of all users in the system"
    echo "passwd file to UW-Imap mix format.\n"

    echo "Are you sure that you want to proceed [n] ? \c"
    read ans
    if [ "$ans" != "y" ]; then
	exit 1
    fi
fi

# -- Shutdown all imap and pop mail services

if [ "$SHUTDOWN" = "y" -a -z "$USERNAME" ]; then

    echo "$PROGNAME -- Shutting down system wide client mail services...\c"

    if [ "$OSVER" = "5.10" ]; then

	svcadm disable pop2 pop3 pop3s imap imaps

    elif [ "$OSVER" = "5.8" ]; then

	rm -f /tmp/inetd.conf
	sed -e 's/^pop/#pop/g
		s/^imap/#imap/g' /home/bob/tmp/inetd.conf > /tmp/inetd.conf

	mv /etc/inetd.conf /etc/inetd.conf.sav
	mv /tmp/inetd.conf /etc/inetd.conf

	pkill -HUP inetd

	pkill imapd
	pkill pop

    fi

# - Make sure that all imapd and pop services have stopped before continuing

    nimapd=`ps -ef | grep imap | wc -l`
    npop=`ps -ef | grep pop | wc -l`
    nproc=`expr $nimapd + $npop`
    while [ nproc -gt 0 ]
    do
	sleep 1
	echo ".\c"
	nimapd=`ps -ef | grep imap | wc -l`
	npop=`ps -ef | grep pop | wc -l`
	nproc=`expr $nimapd + $npop`
    done
fi

echo "Done!"

if [ ! -z "$USERNAME" ]; then
    grep "$USERNAME:" /etc/passwd
else
    cat /etc/passwd
fi | while read line
do
    username=`echo $line|cut -d: -f1`
    uid=`echo $line|cut -d: -f3`
    gid=`echo $line|cut -d: -f4`
    homedir=`echo $line|cut -d: -f6`
    maildir=$homedir/mail
    premix=".premix"
    mixmaildir=$homedir/mail.mix

    #echo $line
    if [ $uid -ge 100 -a $uid -lt 500 -o $username = "guest" ]; then

	echo "\n\n$PROGNAME -- Processing User '$username' mail files..."

	#echo "User = '$username'"
	#echo "Maildir = '$maildir'"
	if [ -d "$maildir" -a ! -d "${maildir}${premix}" ]; then

	    if [ "$SHUTDOWN" = "y" -a ! -z "$USERNAME" ]; then

		echo "$PROGNAME -- Shutting down user client mail services...\c"
		pkill -u $username imap
		pkill -u $username pop

		nimapd=`ps -u $username | grep imap | wc -l`
		npop=`ps -u $username | grep pop | wc -l`
		nproc=`expr $nimapd + $npop`
		while [ nproc -gt 0 ]
		do
		    sleep 1
		    echo ".\c"
		    nimapd=`ps -u $username | grep imap | wc -l`
		    npop=`ps -u $username | grep pop | wc -l`
		    nproc=`expr $nimapd + $npop`
		done

		echo "Done!"

	    fi
		
	    find "$maildir" -name ".*" -prune -o -print | grep -v ".lock" | while read mailfile
	    do
		if [ "$maildir" != "$mailfile" ]; then
		    newmailfile=`echo $mailfile|sed -e "s^$maildir^$mixmaildir^g"`
		    smailfile=`echo $mailfile|sed -e "s^$maildir^^g"`
		    echo "'$smailfile'...\c"

# -- Is this already a mix format directory box?

		    if [ ! -f "$mailfile/.mixindex" ]; then

# -- Create the destination directory if it doesn't exist

			newmailfiled="`dirname "$newmailfile"`"
			if [ ! -d "$newmailfiled" ]; then
			    mkdir -p "$newmailfiled"
			fi

# -- Convert to mix format

			#message=`$MIXCVT "$mailfile" "$newmailfile" 2>&1`
			$MIXCVT "$mailfile" "$newmailfile"

			if [ $? -eq 0 ]; then
			    echo "Done!"
			else
			    message=`echo $message|cut -d: -f2`
			    echo "$message"
			fi
		    else
			echo "Skipped - already mix format!"
			mv "$mailfile" "`dirname "$newmailfile"`"
		    fi
		fi
	    done

	    if [ -d "${maildir}${premix}" -a ! -d "$mixmaildir" ]; then

# -- There were no folders converted (empty maildir) so no mix dir was created

		mkdir "$mixmaildir"
	    fi

# -- If there is no INBOX then create it

	    if [ ! -d "$mixmaildir/INBOX" ]; then

		echo "$PROGNAME -- Creating '$mixmaildir/INBOX'"
		$MAILUTIL create "#driver.mix/$mixmaildir/INBOX"

	    fi

# -- Make sure that everything in destination is properly owned by the user

	    chown -R $uid:$gid "$mixmaildir"
	    mv "$maildir" "${maildir}${premix}"
	    mv "$mixmaildir" "$maildir"

	elif [ -d "${maildir}${premix}" ]; then
	    echo "User '$username' - already converted - skipped!"
	else
	    echo "No '$username' maildir found - skipped!"
	fi
    fi
done

# - Restore the client mail services

if [ "$SHUTDOWN" = "y" -a -z "$USERNAME" ]; then

    echo "\n$PROGNAME -- Re-enabling system wide client mail services...\c"

    if [ "$OSVER" = "5.10" ]; then

	svcadm enable pop2 pop3 pop3s imap imaps

    elif [ "$OSVER" = "5.8" ]; then

	mv /etc/inetd.conf.sav /etc/inetd.conf

	pkill -HUP inetd
    fi
fi

echo "Done!"

exit