Xapian initial index really that slow? - 3.6.beta3-2+

Hi,

I cloned a production server (zfs + lxc) to test the xapian search and
started to squatter.

What I am experiencing is a very slow progress, especially comparing to
the search_engine:squat.

Besides of the very slow indexing, the whole cyrus seems slowed down
for the user, which is under indexing (squattering).

When I set the batch to 8192, I got longlock for around 120 seconds,
with 1024 now I got about 18 seconds (numbers are not constants of
course).

The full indexing supposed to slow with xapian engine?

Details:
Config file:
# Sync log - rolling squat
sync_log: on
sync_log_channels: indexer
search_engine: xapian
search_index_headers: no
search_batchsize: 1024
#search_batchsize: 8192
#search_indexed_db: twoskip
defaultsearchtier: t1
t1searchpartition-default: /var/spool/cyrus/search


As I checked the size of the search partition, I got 300GB, even
squatter only started with 2 users (never finished any of them).
I have about 1.1 TByte email messages in total with about 50 users, so
about 300 GB for 2 users seems very strange to me.

I guess, I really missed something really important here.

The steps were:
- cyrus-imapd stopped
- changing config files to meet the xapian requirements (config like
above, rolling invocation of squatter in cyrus.conf:  cmd="squatter -R"
)
  - (the squatter in daemon section was squatter -R -n indexer, but as
I read in one email, -R and -n are not really friends
- start cyrus-imapd
- start an initial squatter for everybody:  /usr/lib/cyrus/bin/squatter
-v -p 
  - stopping this squatter
- starting squatter for specific users
  - stoppping after about 24 hours
- starting again with another user
- realizing, something really not ok

I read the emails and manual before I wrote this email, because I still
in doubt, how can I put this into production.
At this moment it is not possible to start to convert search engine to
xapian, due to that, the initial indexing (which I guess necessary for
old emails) takes a lot of time and technically block the email server
to serve at least the client, whos emails are under squattering.

Could you provide me a link to a recent documentation and/or give me
some hints?

Thanks,
István

------------------------------------------
Cyrus: Info
Permalink: https://cyrus.topicbox.com/groups/info/T108841899413a60e-M817b98689f7aad1c96cfc67e
Delivery options: https://cyrus.topicbox.com/groups/info/subscription