Re: How to disable mail indexing with Akonadi search

Am Donnerstag, 11. Februar 2016, 12:31:56 CET schrieb Martin Steigerwald:
> Am Donnerstag, 11. Februar 2016, 12:12:11 CET schrieb Martin Steigerwald:
> > Am Montag, 8. Februar 2016, 10:15:55 CET schrieb Daniel Vrátil:
> > > On Saturday, February 6, 2016 7:35:17 PM CET Martin Steigerwald wrote:
> > > > Back in Baloo times it was easy to disable mail indexing. But with
> > > > Akonadi
> > > > Search which as I understand is basically a renamed Baloo for mail
> > > > stuff
> > > > I
> > > > didn´t find a way yet. Can one disable it?
> > > > 
> > > > I searched for an option in akonadi_indexing_agent settings, but
> > > > didn´t
> > > > find one.
> > > > 
> > > > I excluded the biggest folders yet it still crawls Akonadi to a halt
> > > > with
> > > > at least 90-100% CPU load on mysqld for tenth of minutes or more up to
> > > > the point where something happens that doesn´t happen with Akonadi +
> > > > KDEPIM from master otherwise since all the performance improvements:
> > > > KMail looses connection to Akonadi and is unusable unless I restart
> > > > it.
> > > > 
> > > > Also currently the mail index would grow to big that BTRFS gets into
> > > > trouble with free space, up to kernel 4.3 even freezing up the whole
> > > > machine with a kworker thread searching for free space (known issue in
> > > > BTRFS, that may or may not be fixed in 4.4 or later).
> > > > 
> > > > For now at least for the storage thing I did:
> > > > 
> > > > touch ~/.local/share/akonadi/search_db
> > > > 
> > > > then akonadi indexing agent does not create a directory there and no
> > > > space
> > > > is used up.
> > > > 
> > > > As you know my mail setup is a bit crazy, so I know the index is big,
> > > > but
> > > > still after 5 GiB I keep wondering whether it needs to be that big,
> > > > especially as I excluded at least the folders with LKML mails.
> > > > 
> > > > For now until I have more free SSD space I just want to disable it.
> > > > 
> > > > I help myself with repeating
> > > > 
> > > > killall /home/kde/install/bin/akonadi_indexing_agent
> > > > 
> > > > until Akonadi gives up starting the indexing agent after each time I
> > > > start
> > > > Akonadi by logging into a Plasma session or restarting it after a new
> > > > compile from master.
> > > 
> > > You can prevent the Indexer from starting by modifying
> > > /usr/share/akonadi/
> > > agents/akonadindexingagent.desktop, unsetting "AutoStart" capability.
> > > 
> > > 
> > > You use maildir, so the main suspect why the IO goes crazy is that when
> > > indexing is triggered, we are forced to upload all the mails to Akonadi
> > > and
> > > index it, then Akonadi expires the data again and they are removed from
> > > the
> > > database - so you get maildir READ + DB write, then DB read, then Xapian
> > > write (super-IO-intensive) and again DB write (delete) - I suppose this
> > > together adds up to some insane IO numbers.
> > 
> > I thought about this again after having had the massive reduction of db
> > size on akonadi maintenance with akonadictl fsck ; akonadictl vacuum.
> > 
> > I missed one start of the indexing agent and it did some work (despite it
> > could save the Xapian database cause of my touch ~/.local/share/akonadi/
> > search_db and I think its better off with not doing any work at all if it
> > can not save it).
> > 
> > And after that db was about 1,4 GiB larger again.
> > 
> > Then I wondered I still have SizeThreshold=16768 in my akonadiserverrc
> > back
> > from time where Akonadi massively leaked files in file_db_data and then it
> > didn´t have the multi leveled cache. But when I understand you correctly
> > on
> > indexing it would stuff all mails below 16 KiB into the database and as
> > MySQL AFAIK does not auto-vacuum they stay there.
> > 
> > So I will attempt now to let the indexer run, but without my special
> > SizeThreshold setting, so it takes the default of 4 KiB again. Maybe this
> > way it will complete a full indexing run without blocking up Akonadi
> > completely.
> 
> Jup, I think this is it. I removed the setting and tada:
> 
> martin@merkaba:~/.local/share/akonadi> date ; find file_db_data | wc -l
> Do 11. Feb 12:29:12 CET 2016
> 8180
> martin@merkaba:~/.local/share/akonadi> date ; find file_db_data | wc -l
> Do 11. Feb 12:29:19 CET 2016
> 8864
> martin@merkaba:~/.local/share/akonadi> date ; find file_db_data | wc -l
> Do 11. Feb 12:29:29 CET 2016
> 9826
> 
> So still extremely inefficient, yet according to atop much less load on
> MySQL database. Around 20-50% cpu load from some quick glances instead of
> around 100%.

Wait, that actually means that Akonadi is actually *duplicating* my complete 
local maildir on initial indexing – and with SizeThreshold I specify what part 
of the duplication it puts into the database and which part it puts into 
file_db_data?

martin@merkaba:~/.local/share#130> du -sh local-mail 
20G     local-mail
martin@merkaba:~/.local/share> find local-mail -type f | wc -l
1738725

Wow! BTRFS RAID 1 there is quite some (actually unnecessary) work to do.

Hopefully from file_db_data it expires the stuff after the indexing.

I think I now fully got why the change you propose would create a *massive* 
performance improvement for local maildir users.

-- 
Martin
_______________________________________________
KDE PIM users mailing list
Subscription management: https://mail.kde.org/mailman/listinfo/kdepim-users