Work-around to issues with Akonadi file based caching (was: Re: rant)

I changed the topic so that I can easily reference it in the future. As it 
takes me quite some time to write mails like this. Also others may find it 
more easily.


Am Donnerstag, 12. Februar 2015, 20:06:50 schrieb E. Hakan Duran:
> On Thursday, February 12, 2015 08:20:35 AM Martin Steigerwald wrote:
> > ...
> > I do think the caching in Akonadi is broke.
> > 
> > It caches to much for too long.
> > 
> > For offline IMAP, sure, download everything. I´d prefer a maildir for
> > that, cause its introspectable and not all in one directory, but
> > well.
> > 
> > I wrote enough about this already. Reference my other posts about it,
> > might have been on kdepim-ml as well.
> 
> Thank you Martin. I just wanted to report one more thing I found. I left
> akonadi running overnight. About 24 hours later (now) the size of the
> file_db_data was about 11.5 GBs. I ran akonadi fsck again to find that
> the ~7.5 GB addition to this folder was all unlinked files. akonadi
> fsck put all that data into the lost+find folder, reducing the size of
> the file_db_data folder size back to about 4.2 GB.
> 
> Maybe I should setup a cronjob to run akonadi fsck daily, what do you
> think?

I recommend you to try the SizeTreshold thing instead. Cause even after at 
least a week of using it I still have:

martin@merkaba:~/.local/share/akonadi> du -sh file_db_data 
3,1M    file_db_data
martin@merkaba:~/.local/share/akonadi> find file_db_data | wc -l
26

martin@merkaba:~/.local/share/akonadi> du -sch db_data/akonadi/* | sort -
rh | head -5
2,0G    insgesamt
1,5G    db_data/akonadi/parttable.ibd
516M    db_data/akonadi/pimitemtable.ibd
23M     db_data/akonadi/pimitemflagrelation.ibd
240K    db_data/akonadi/collectiontable.ibd


Yes, you may get a bit larger database, but it appears to me that in there 
Akonadi at least cleans up old cached items as it doesn´t seem that it got 
much larger since I setup this.

What I use basically is:

martin@merkaba:~> cat .config/akonadi/akonadiserverrc 
[%General]
Driver=QMYSQL
SizeThreshold=32768
[…]

Then I did a akonadictl fsck and a akonadictl vacuum *once*. And it still 
seems to be good. 

KMail doesn´t usually loose the connection with Akonadi anymore as well.

I do think the file based caching, while I would prefer it, is just broke. 
And for another reason as well:

1) It caches too much.

2) It caches too many files in *one* directory (it broke a storage 
appliance standard setup at work)

3) It caches too long.

And now in addition to that:

4) It doesn´t clean up properly after itself as akonadictl fsck clearly 
shows. The fsck action seems to clean up files Akonadi server doesn´t know 
about anymore, i.e. that have no reference in database. But then, my 
expectation would be that if Akonadi deletes a cache entry it deletes the 
reference in the *database* and the *file*. Yet, it seems that Akonadi 
looses track of what its doing. I think thats even another bug.


So in my eyes the Akonadi file based code has *at least* four severe bugs. 
In addition to the performance problems it seems to cause.

So while I´d prefer that Akonadi uses am introspectable directory 
structure with files to cache things and only keeps references in the 
database that just doesn´t seem to work.

Above is now with my POP3 maildir based setup...


Additional to that see:

Bug 338402 - File system cache is inneficient : too many file per directory
https://bugs.kde.org/show_bug.cgi?id=338402#c9

Re: Possible akonadi problem?
https://lists.debian.org/debian-kde/2015/01/msg00055.html

> I already commented on one of the bugs you mentioned earlier Martin. Do
> you think I should report this there or as a separate bug?

I think a bug report that clearly shows that akonadictl fsck again and 
again produces files in lost + found that Akonadi didn´t know about it 
anynmore and just didn´t delete it self, i.e. that Akonadi file based cache 
doesn´t seem to clean up after itself properly might be useful.

But I think, spending time to develop Akonadi or maybe better Akonadi Next 
is even more helpful. I currently have other priorities, so while I think 
current Akonadi still has quite some serious issues, I also I am also 
prepared to deal this them for now.

The SizeThreshold thing helped me with that. But have caution: This is no 
official recommendation of Akonadi developers. Still, it gave me a good 
improvement with performance and latency *and* seemed to have solved the 
insane amount of file and data in file_db_data issue for me. Or at least it 
worked around it successfully.

I have switched two IMAP and one POP3 setup to this and upto now have no 
issues that I could attribute to that configuration change.

Thanks,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

_______________________________________________
KDE PIM users mailing list
Subscription management: https://mail.kde.org/mailman/listinfo/kdepim-users

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEABECAAYFAlTdxSkACgkQmRvqrKWZhMfqKACfSDPn9kZ7e/fIuQY5YuPPBf48
J44AoJ68kqemV4PSLETzxE1OrinPJaDK
=TYfG
-----END PGP SIGNATURE-----