Re: Xapian index size

"J. Roeleveld via Info" <[email protected]> Wed, 21 May 2025 09:43:02 +0200

Newsgroups	gmane.mail.imap.cyrus
Message-ID	<13783671.uLZWGnKmhe@persephone>

On Wednesday, 11 September 2024 16:08:05 CEST Jean Charles Delépine via Info 
wrote:
> Hello,
> 
> Quoting [email protected]:
> > An obvious addition.
> > Yes, index size decreases to expected 50% of source if I remove old
> > index (rm *) and create completely new instead. Is it the indended
> > maintenance method?
> 
> You can read this mail from Bron Gondwana at Fastmail on how they use
> the Xapian index, it's the better doc I find and it helps me
> configuring it.
> 
> https://lists.tartarus.org/pipermail/xapian-discuss/2014-October/009112.html
> 
> Sincerly,
>     Jean Charles Delépine
> 

As addition to that mail, which I used as a basis as well, here is how I 
configured mine. Hope this helps others.

- settings in imapd.conf:

tempsearchpartition-main: /var/spool/index/temp
activesearchpartition-main: /var/spool/index/active
archivesearchpartition-main: /var/spool/index/archive

- In cyrus.conf (added the 'reindex' section):

START {
  # do not delete this entry!
  recover       cmd="ctl_cyrusdb -r"
  statscleanup  cmd="promstatsd -c"
  reindex       cmd="squatter -v -i"
}

- The filesystems used (from fstab):

tmpfs           /var/spool/index/temp           tmpfs 
defaults,uid=cyrus,gid=mail,mode=0755,noatime,size=12G

/dev/xvdd1      /var/spool/index/active         ext4    defaults,noatime

/dev/xvdd2      /var/spool/index/archive        ext4    defaults,noatime

- My regularly running script to handle the indexes:

see file "xapian_index_management.txt"

My script can be adjusted to limit sizes and usage-levels

--
Joost
------------------------------------------
Cyrus: Info
Permalink: https://cyrus.topicbox.com/groups/info/Tdcc91f096c0df53c-M40317592fe04e62b73b4e5bb
Delivery options: https://cyrus.topicbox.com/groups/info/subscription

xapian_index_management.txt (text/plain, 3 KB)

#!/bin/bash
#
maxtempperc=10
maxactiveperc=30

# 100MB = 100*1024*1024 = 104857600
maxtemp=104857600

# 500MB = 500 * 1024 * 1024 = 524288000
maxactive=524288000

tempperc=`/bin/df /var/spool/index/temp | /bin/grep temp | /usr/bin/awk '{ USED=($3*1000); SIZE=($2*1000); USAGE=(USED/SIZE)*100;  print USAGE}'`
activeperc=`/bin/df /var/spool/index/active | /bin/grep active | /usr/bin/awk '{ USED=($3*1000); SIZE=($2*1000); USAGE=(USED/SIZE)*100;  print USAGE}'`

/bin/echo "tempperc is [ ${tempperc} ]"
/bin/echo "activeperc is [ ${activeperc} ]"

if /usr/bin/awk -v n1="${tempperc}" -v n2="${maxtempperc}" 'BEGIN { exit ( n1 > n2 ) }' /dev/null; then
  /bin/echo "Temp usage [ ${tempperc} ] is smaller than max temp usage [ ${maxtempperc} ]"
else
  /bin/echo "Temp usage [ ${tempperc} ] is larger than max temp usage [ ${maxtempperc} ]"
  /bin/echo "REDUCING maxtemp"
  maxtemp=1048576
fi

if /usr/bin/awk -v n1="${activeperc}" -v n2="${maxactiveperc}" 'BEGIN { exit ( n1 > n2 ) }' /dev/null; then
  /bin/echo "Active usage [ ${activeperc} ] is smaller than max active usage [ ${maxactiveperc} ]"
else
  /bin/echo "Active usage [ ${activeperc} ] is larger than max active usage [ ${maxactiveperc} ]"
  /bin/echo "REDUCING maxarchive"
  maxarchive=1048576
fi
/bin/echo ""
/bin/echo "Checking individual temp usage"
if /bin/ls /var/spool/index/temp/*/user/*/xapian* 1> /dev/null 2>&1; then
  /usr/bin/du -bs /var/spool/index/temp/*/user/*/xapian* |
      while read size name
      do
        username=`/bin/echo ${name} | /bin/sed 's/\(.*\)\/user\/\(.*\)\/xapia\(.*\)/\2/'`
        if [ "${size}" -le "${maxtemp}" ]; then
          /bin/echo "username [ ${username} ] with size [ ${size} ] is smaller than [ ${maxtemp} ]"
        else
          /bin/echo "username [ ${username} ] with size [ ${size} ] is greater than [ ${maxtemp} ]"
          /bin/echo "EXEC : squatter -v -o -z active -t temp -u ${username}"
          /usr/sbin/squatter -v -o -z active -t temp -u "${username}"
        fi
      done
else
  /bin/echo "No active users found in temp"
fi
echo ""
echo "Checking individual active usage"
if /bin/ls /var/spool/index/active/*/user/*/xapian* 1> /dev/null 2>&1; then
  find /var/spool/index/active -type d -name "xapian*" 2>/dev/null | /bin/sed 's/\(.*\)\/xapian\(.*\)/\1/' | sort -u |
    while read longpath
    do
      path=`/bin/echo ${longpath} | /bin/sed 's/\(.*\)\/xapian\(.*\)/\1/' | sort -u`
      echo ${path}
      /usr/bin/du -bs ${path} |
        while read size name
        do
          username=`/bin/echo ${name} | /bin/sed 's/\(.*\)\/user\/\(.*\)/\2/'`
          if [ "${size}" -le "${maxactive}" ]; then
            /bin/echo "username [ ${username} ] with size [ ${size} ] is smaller than [ ${maxactive} ]"
          else
            /bin/echo "username [ ${username} ] with size [ ${size} ] is greater than [ ${maxactive} ]"
            /bin/echo "EXEC : squatter -v -z archive -t active,archive -u ${username}"
            /usr/sbin/squatter -v -z archive -t active,archive -u "${username}"
          fi
        done
      done
else
  /bin/echo "No active users found in active"
fi