Re: rant

Hi Martin, and everybody that responded. I will try to respond below to the 
issues brought up:

On Wednesday, January 28, 2015 09:26:29 AM Martin Steigerwald wrote:

>...
> Wow, you hold a record. 75 GiB in ~/.local/share/akonadi/file_db_data – how
> large is your mail account? That is more than *ten* times larger than the
> maximum size I have seen on one of my systems. Are you sure it was 75 GiB?

I am positive. I realized it when there was almost no space left in my /home 
directory. I compressed it to a 7z file and moved it to another HDD. The 
compression took somewhere between 12 and 24 hours (no kidding), and the 
compressed file size is 9.6 GiB. Right now I am attempting to open it with ark 
just to count the number of files in it, as you pointed. Looks like this will 
take some time too, lol. I even submitted a feature request for ark to have an 
implementation for multi-threaded compression (still no kidding).

> And yeah, I think you found a real issue with the caching mechanism of
> Akonadi.

I am not sure if I should feel proud, lol.

> Have you checked how many files have been in there?

I used the CLI command 7z instead of ark, which reported 422,359 files in the 
archive. Of note, I have executed akonadictl fsck before archiving these files, 
which reported so many duplicates, but didn't actually change the size of the 
folder significantly. Since I needed space in my /home directory, I didn't have 
much choice rather than emptying this folder, even though I was aware this 
wasn't the recommended action.

> Also see:
> 
> [Akonadi] [Bug 338402] File system cache is inneficient : too many file per
> directory
> 
> Bug 332013 - NFS with NetApp FAS: please split payload files in file_db_data
> into several directories to avoid reaching maxdirsize limit on Ontap / WAFL
> filesystem
> 
> Bug 341884 - dozens of duplicate mails in
> ~/.local/share/akonadi/file_db_data
> 
> And please add your feedback to there, I suggest using 338402.

Thank you so much for your valuable input here. I will definitely add feedback 
to 338402.


> ...
> One question regarding this, do you receive *very large* mails? In
> addition to the bug reports above, I think there may be another issue at
> play: If you receive very large mails with huge attachments and if Akonadi
> caches them, no wonder you get a large file_db_data. So I would really be
> interested in that. How many files did you have in there? Was it "just" a
> few very big files for very big mails, or was it hundreds of thousands (I
> am *not* kidding, maximum I saw here with a huge IMAP account is >850000
> files) of files?

I don't receive large emails. The account you see here is my largest gmail 
account and it is used for email lists and therefore filled with a lot of small 
sized emails. Even my other accounts rarely receive attachments that are >= 5 
MB.

> As you cleaned the directory already, you cannot check how many duplicated
> mails you have in there, if you do have backup, please go check and also
> add it to the duplicate mail bug report about.

Please see my comment above.

> Also see my thread [Kde-pim] Akonadi(Next): Thoughts on caching on kde-pim
> mailing list.
> 
> 
> And:
> 
> I tested a setting (SizeTreshold=32768) that seems to make Akonadi create
> *a lot* less files in there:
> 
> https://bugs.kde.org/show_bug.cgi?id=338402#c9
> 
> At the expense of a larger database.
> 
> I am only testing it since a few days, but so far it looks good.
> 

I will definitely take a look at this and probably try myself.

> 
> 
> Also, it seems that at least during some time after moving mails, Akonadi
> may *write* cache them into ~/.local/share/akonadi either in database (<4
> KiB with standars SizeTreshold setting) or as files (>4 KiB) and does not
> write the mail to its final location immediately, so if you are using local
> maildirs for storing mails either via POP3 or by moving mails from IMAP
> account to local maildir, then be cautious about cleaning file_db_data, as
> you may loose mails in that case. So better clean things with
> 
> akonadictl fsck
> 
> instead of manually removing files from there. You may add a higher
> SizeTreshold setting as well.
> 
> [kmail2] [Bug 338238] copied or moved mails are often not processed
> correctly (POP3)
> 
> [kmail2] [Bug 339181] moving mails from cached imap to local folder leaves
> mails without RID

I don't move mails to local maildirs, but I was aware of this before 
compressing that folder. Please see my comment above. I truly don't think I 
lost any email in the process, perhaps due to being lucky.

> > I used to love and admire Kontact/Kmail, now it is a love and hate
> > relationship, just like a seasoned marriage. Unfortunately I lack the
> > expertise to be able to contribute to the improvement of this now very
> > complicated piece of software. I just hope something miraculous happens
> > soon and the usability of KDEPIM improves significantly; both on a
> > day-to-day basis and in the long term (see the rant above about
> > significant accumulation of junk data in folders in time). As a
> > consumer, I am yet to be convinced that the switch to akonadi backbone
> > was a good decision.
> > 
> > I am sorry to take your time, and thanks to those who read up to this
> > point.
> 
> I understand your mode.
> 
> I had my own share of issues with Akonadi.
> 
> Developers are aware of some of its shortcomings and there are plans for
> Akonadi Next which likely would do things quite a bit differently. It may
> even go without a RDMS like MySQL for caching, use mmap for efficient access
> and what not, there is a presentation about it, I think Aaron Seigo put it
> up in his blog.

Thank you Martin for your empathy and kind words.

> There also have been database performance improvements with Akonadi git,
> also in 1.13 branch thats easily compilable.

I don't feel competent enough to compile Akonadi git 1.13 branch, since I 
don't have technical/software background. I am more likely to break things 
further in the process, lol.

> However I agree to the principle of Akonadi in itself. Having the mail and
> PIM item handling separate from the mail program in a framework with some
> kind of database or caching is a very good idea in my oppinion.
> 
> Its just, that, IMHO, the implementation is lacking.
> 
> In addition to that, I think KDEPIM team needs more developers. It is just
> a few people working regularily on KMail, KDEPIM and Akonadi.

+100 on that. I have enormous respect for the developers, who dedicate their 
own time voluntarily for such a noble cause. Is there a way to donate 
specifically to the KDEPIM subproject?

> So it may take a while, and I think as to the current caching behaviour,
> please make your feedback heard in the bug report, best, based with some
> facts, so if you can check how many files you had in there from a backup,
> all the better.

Will do, and the number again is 422.359 ;). Almost half a million it is!

> I know it can be painful. I still think KMail is the best GUI based mail
> client I have ever seen. But issues with Akonadi make it difficult for me to
> enjoy it at times. And sometimes I want to throw it out of the window.
> 

You spoke my mind here. That is the reason I cannot abandon KDEPIM, but seeing 
the potential and the way it is not fully realized is very frustrating at 
times.

Thank you,

Hakan

_______________________________________________
KDE PIM users mailing list
Subscription management: https://mail.kde.org/mailman/listinfo/kdepim-users

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEABECAAYFAlTJqBEACgkQu5WqKQPUC0uN4wCePG1Zs1c95CVKzU0OjV8Vw6oi
K7cAn3AHIncF3AkrlwmMxKt5SMbCBYMy
=606S
-----END PGP SIGNATURE-----