Re: Frequent timeouts
Bob Atkins <[email protected]> Fri, 30 Jul 2010 10:43:26 -0700
| Newsgroups | gmane.mail.imap.uw.c-client |
|---|---|
| Organization | DigiLink, Inc. |
| Message-ID | <[email protected]> |
Moray, You are running into the limitations of mbx formatted boxes - especially when they are >2GB. Everytime a client requests a listing of the contents of the mail box - the server must perform a complete scan of an mbx file. In addition, when a message is appended to an mbx file another scan is performed. There simply is no way to get around the fact that mbx is basically a modified serial text file, no random access/insert is possible and when the size runs into the multi gigabyte level it just takes a /_*lot*_/ of I/O to manage it. I suspect that you will find your system is struggling with high levels of I/O wait. From personal experience I can say that the solution to your problem is to convert to mix mailbox format. I realize that the conversion of all users' mail stores on an established mail server is not trivial, especially on a live system so, in an effort to help, I have attached a script that I wrote and used to perform the conversion on our mail severs that are Sun Solaris systems. The script should be easily adapted to a linux system. It does all of the necessary checks, descends the user's mail folder tree (correctly determining whether it is mbox, mbx or a combination of mbx and mix folders) and does it's best to do the 'right' thing, avoiding re-converting any already existing mix format mail folders. It also preserves the original mail folder in case something goes wrong. You can convert a single user or all users in the passwd file. It won't touch uids <500 or >60000. It also won't create mail stores for a user if none already exist. I offer this script without any support or warranty, express or implied. I suggest running it against a test user account to make sure that you are satisfied with the results. I hope that it will be helpful to anyone struggling with the type of problem that you are having with overloaded mbx files. In addition to the conversion you will need to change the default operation of your uw-imap to use mix format folders and you /_*should*_/ also convert to using tmail as the local delivery agent. We did this when we compiled uw-imap. You don't have to use tmail, but I highly recommend it. Good luck. -- *Bob Atkins * /President/CEO/ *DigiLink, Inc. <http://www.digilink.net>* Business Inter-net-working */The Cure for the Common ISP!/* Phone: (310) 577-9450 Fax: (310) 577-3360 eMail: [email protected] Guy Dawson wrote: > On 30/07/2010 11:34, Moray Henderson wrote: >> Can anyone give me some pointers on tracking down timeout problems? I >> have users saying Thunderbird is frequently giving "Connection to IMAP >> server timed out" on their local network. They are using uw-imap-2007e >> with MBX mail folders on CentOS 5.4 server, Thunderbird 3.1.1 on Windows >> client (with or without SSL). Apparently it used to happen a few times >> with uw-imap-2002d on CentOS 4, but is much worse now that they have >> upgraded. > > When we virtualised our email server we started with a test system > running 2007e and mbx and did experience some client timeout issues. The > old physical server was running and older UW-IMAP server and mbx OK. > I do wonder if 2007e is slower when accessing mbx than earler versions. > >> Does MIX store everything in one file like MBX, or is it all in separate >> files? > > Neither. It stores multiple messages per file and has a maximum file > site which can be configured at compile time. Lots of little messages > will be stored in the same file while large messages will get a file > of their own. MIX also has separate index files. > > I have one user with a 3.5GB inbox, 6954 .mix data files, a 5MB .mixindex > file, a 2.7MB .mixstatus file and approximately 120,000 messages. > > Touch wood, mix is working OK! > > Regards, > Guy > -- -------------------------------------------------------------------- > Guy Dawson I.T. Systems Manager Crossflight Ltd > [email protected] 07973 797819 01753 776104 > > > ******************************************************************* > This message contains the views and opinions of a Crossflight > Limited employee and at this stage are in no way a direct > representation of Crossflight Limited. > > Crossflight Limited is an international express courier, mailing > and logistics service provider. This communication and any > attachments are confidential and may be protected from disclosure. > We endorse no advice or opinion contained in this communication > that is not the subject of a contract between the recipient and > ourselves. > > If you have received it in error please notify us immediately and > note that any storage, use or disclosure is strictly prohibited and > may be unlawful. Those communicating with us by electronic mail > will be deemed to have accepted the risks associated with > interception, amendment, loss and late or incomplete delivery. > They will also be deemed to have consented to our intercepting and > monitoring such communications. > > This footnote also confirms that this message has been checked for > the presence of computer viruses. We strongly recommend that you > check this email with your own virus software as Crossflight > Limited will not be held responsible for any damage caused by > viruses as a result of opening this email. > ******************************************************************* > _______________________________________________ > Imap-uw mailing list > [email protected] > http://mailman2.u.washington.edu/mailman/listinfo/imap-uw > _______________________________________________ Imap-uw mailing list [email protected] http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
mail2mix.sh
(text/plain, 6.2 KB)
#!/bin/sh
#
# -- Convert mailboxes to mix format for every user
#
# -- Author: Bob Atkins, DigiLink Network Services
#
# -- First Release : 2/16/08
#
# -- Revised:
# 8/25/09 - Added options to convert a single user. Added checks for already
# converted mix format boxes so that they are not re-converted.
# 9/9/09 - Create user INBOX after the conversion if it doesn't already exist.
#
# Copyright (C) 2008, Bob Atkins
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>
#
PROGNAME=`basename $0`
USAGE="Usage: $PROGNAME [-u username] [-x]"
SIGNAL=-TERM
MIXCVT=mixcvt
MAILUTIL=mailutil
OSVER=`uname -r`
SHUTDOWN=y
umask 0077
usage ()
{ echo $USAGE
cat << "-EOF-"
Converts existing mail stores of all users in the system passwd file to
UW-Imap mix format.
Where:
-u Specifiy a single user to convert
-x Do NOT shutdown imap/pop mail services.
Use with CAUTION!
mail2mix.sh Copyright (C) 2008, Bob Atkins
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions.
-EOF-
}
# -- Process any args
if [ $# -ne 0 ]; then
while getopts h\?u:x flags; do
case $flags in
u) USERNAME="${OPTARG}";; # Pickup arg to append to the package name
x) SHUTDOWN=n;;
? | h) usage; exit 4;;
esac
done
if [ -z "$USERNAME" ]; then
echo "$PROGNAME -- Invalid arg/no username specified"
usage; exit 4
fi
else
echo "\n!!!! WARNING !!!!\n"
echo "This script will convert existing mail stores of all users in the system"
echo "passwd file to UW-Imap mix format.\n"
echo "Are you sure that you want to proceed [n] ? \c"
read ans
if [ "$ans" != "y" ]; then
exit 1
fi
fi
# -- Shutdown all imap and pop mail services
if [ "$SHUTDOWN" = "y" -a -z "$USERNAME" ]; then
echo "$PROGNAME -- Shutting down system wide client mail services...\c"
if [ "$OSVER" = "5.10" ]; then
svcadm disable pop2 pop3 pop3s imap imaps
elif [ "$OSVER" = "5.8" ]; then
rm -f /tmp/inetd.conf
sed -e 's/^pop/#pop/g
s/^imap/#imap/g' /home/bob/tmp/inetd.conf > /tmp/inetd.conf
mv /etc/inetd.conf /etc/inetd.conf.sav
mv /tmp/inetd.conf /etc/inetd.conf
pkill -HUP inetd
pkill imapd
pkill pop
fi
# - Make sure that all imapd and pop services have stopped before continuing
nimapd=`ps -ef | grep imap | wc -l`
npop=`ps -ef | grep pop | wc -l`
nproc=`expr $nimapd + $npop`
while [ nproc -gt 0 ]
do
sleep 1
echo ".\c"
nimapd=`ps -ef | grep imap | wc -l`
npop=`ps -ef | grep pop | wc -l`
nproc=`expr $nimapd + $npop`
done
fi
echo "Done!"
if [ ! -z "$USERNAME" ]; then
grep "$USERNAME:" /etc/passwd
else
cat /etc/passwd
fi | while read line
do
username=`echo $line|cut -d: -f1`
uid=`echo $line|cut -d: -f3`
gid=`echo $line|cut -d: -f4`
homedir=`echo $line|cut -d: -f6`
maildir=$homedir/mail
premix=".premix"
mixmaildir=$homedir/mail.mix
#echo $line
if [ $uid -ge 100 -a $uid -lt 500 -o $username = "guest" ]; then
echo "\n\n$PROGNAME -- Processing User '$username' mail files..."
#echo "User = '$username'"
#echo "Maildir = '$maildir'"
if [ -d "$maildir" -a ! -d "${maildir}${premix}" ]; then
if [ "$SHUTDOWN" = "y" -a ! -z "$USERNAME" ]; then
echo "$PROGNAME -- Shutting down user client mail services...\c"
pkill -u $username imap
pkill -u $username pop
nimapd=`ps -u $username | grep imap | wc -l`
npop=`ps -u $username | grep pop | wc -l`
nproc=`expr $nimapd + $npop`
while [ nproc -gt 0 ]
do
sleep 1
echo ".\c"
nimapd=`ps -u $username | grep imap | wc -l`
npop=`ps -u $username | grep pop | wc -l`
nproc=`expr $nimapd + $npop`
done
echo "Done!"
fi
find "$maildir" -name ".*" -prune -o -print | grep -v ".lock" | while read mailfile
do
if [ "$maildir" != "$mailfile" ]; then
newmailfile=`echo $mailfile|sed -e "s^$maildir^$mixmaildir^g"`
smailfile=`echo $mailfile|sed -e "s^$maildir^^g"`
echo "'$smailfile'...\c"
# -- Is this already a mix format directory box?
if [ ! -f "$mailfile/.mixindex" ]; then
# -- Create the destination directory if it doesn't exist
newmailfiled="`dirname "$newmailfile"`"
if [ ! -d "$newmailfiled" ]; then
mkdir -p "$newmailfiled"
fi
# -- Convert to mix format
#message=`$MIXCVT "$mailfile" "$newmailfile" 2>&1`
$MIXCVT "$mailfile" "$newmailfile"
if [ $? -eq 0 ]; then
echo "Done!"
else
message=`echo $message|cut -d: -f2`
echo "$message"
fi
else
echo "Skipped - already mix format!"
mv "$mailfile" "`dirname "$newmailfile"`"
fi
fi
done
if [ -d "${maildir}${premix}" -a ! -d "$mixmaildir" ]; then
# -- There were no folders converted (empty maildir) so no mix dir was created
mkdir "$mixmaildir"
fi
# -- If there is no INBOX then create it
if [ ! -d "$mixmaildir/INBOX" ]; then
echo "$PROGNAME -- Creating '$mixmaildir/INBOX'"
$MAILUTIL create "#driver.mix/$mixmaildir/INBOX"
fi
# -- Make sure that everything in destination is properly owned by the user
chown -R $uid:$gid "$mixmaildir"
mv "$maildir" "${maildir}${premix}"
mv "$mixmaildir" "$maildir"
elif [ -d "${maildir}${premix}" ]; then
echo "User '$username' - already converted - skipped!"
else
echo "No '$username' maildir found - skipped!"
fi
fi
done
# - Restore the client mail services
if [ "$SHUTDOWN" = "y" -a -z "$USERNAME" ]; then
echo "\n$PROGNAME -- Re-enabling system wide client mail services...\c"
if [ "$OSVER" = "5.10" ]; then
svcadm enable pop2 pop3 pop3s imap imaps
elif [ "$OSVER" = "5.8" ]; then
mv /etc/inetd.conf.sav /etc/inetd.conf
pkill -HUP inetd
fi
fi
echo "Done!"
exit