KMail does no longer filter or process emails on huge maildirs. w shows allways load >3 and top kontact 100% cpu load

Hello,

I got currently, since a long lasting period, performance issues on kmail/kontact, which renders the application useless.

My kmail is stuck due to a huge amount of MLs I joined and a big set of filter rules I guess.

e.g. on startup I get this message: grouped 94610 Threads of 170558, synchronizing email folder of user 0%

I'm using kmail2 5.1.3, Ubuntu 16.04.2 LTS, Qt: 5.5.1, KDE: 5.18.0,

ii akonadi-backend-mysql 4:15.12.3-0ubuntu6 all MySQL storage backend for Akonadi

ii kontact 4:15.12.3-0ubuntu1 amd64,

ii mysql-server 5.7.17-0ubuntu0.16.04.1 all

15:48:27 up 7:47, 3 users, load average: 24,99, 11,38, 6,54

USER TTY VON ANMELD@ UNTÄ JCPU PCPU WAS

gerhard pts/1 :0 10:00 5:47 m 0.00 s 5.34 s kded5 [kdeinit5]

gerhard pts/2 :0 10:01 5.00 s 30:50 0.12 s /bin/bash

gerhard pts/3 :0 14:38 55:47 0.20 s 0.20 s /bin/bash

gerhard@ubuntu:~$ top

top - 15:53:17 up 7:52, 3 users, load average: 7,27, 8,62, 6,64

Tasks: 387 gesamt, 4 laufend, 382 schlafend, 0 gestoppt, 1 Zombie

%CPU(s): 50,5 be, 5,5 sy, 0,0 ni, 4,8 un, 39,1 wa, 0,0 hi, 0,1 si, 0,0 st

KiB Spch : 7920528 gesamt, 131540 frei, 7148308 belegt, 640680 Puff/Cache

KiB Swap: 8388604 gesamt, 1796328 frei, 6592276 belegt. 208612 verfü Spch

PID USER PR NI VIRT RES SHR S %CPU %MEM ZEIT+ BEFEHL

13143 gerhard 20 0 4149576 509324 2720 R 100,0 6,4 5:00.17 kontact

13137 gerhard 20 0 5302392 1,333g 9896 R 99,7 17,6 32:12.18 kmail

26523 gerhard 20 0 2503420 203740 0 S 5,6 2,6 31:38.37 mysqld

25652 gerhard 20 0 3168536 26600 18064 S 5,3 0,3 3:49.29 kwin_x11

3553 root 20 0 719000 55416 48904 R 4,3 0,7 11:51.37 Xorg

11404 gerhard 20 0 1542396 474108 47704 S 3,7 6,0 24:19.58 Web Content

26752 gerhard 20 0 664192 31812 21536 S 2,0 0,4 0:16.69 konsole

11300 gerhard 20 0 3742384 84340 27276 S 1,7 1,1 0:31.60 plasmashell

11327 gerhard 20 0 1395996 192104 56612 S 1,0 2,4 6:50.34 firefox

7 root 20 0 0 0 0 S 0,3 0,0 0:36.05 rcu_sched

1722 root 20 0 281396 2172 2044 S 0,3 0,0 0:00.57 accounts-daemon

3014 root 20 0 7572 28 0 S 0,3 0,0 0:16.09 gpm

3023 debian-+ 20 0 82756 14040 2648 S 0,3 0,2 0:11.37 tor

3558 root 20 0 185844 144 0 S 0,3 0,0 2:29.29 teamviewerd

14132 gerhard 20 0 47508 2488 1480 R 0,3 0,0 0:00.27 top

26160 gerhard 20 0 792120 3400 2688 S 0,3 0,0 0:12.64 krunner

26213 gerhard 20 0 538076 7532 4924 S 0,3 0,1 0:24.53 python

26486 gerhard 20 0 356328 2392 1560 S 0,3 0,0 0:01.85 akonadi_control

1 root 20 0 120796 2168 1616 S 0,0 0,0 0:03.15 systemd

If try ctrl + J messages seems no longer be filtered. Load spikes from time to time to 24! (w)

I thought by upgrading from lts 12.04 to 14.04 and further to 16.04 would help with the issue, but unfortunately that wasn't the case.

Now I'm thinking on upgrading to the latest version by using a backport ppa: https://www.ubuntuupdates.org/packagn/base/kontact

I also think about migrating to thunderbird instead. Is tehre a fine HOWTO on how to migrate filter-rules and all structure (tree of folders, where the filter sortes the mails in) and config to thunderbird, at least the most of the current configuration? I got some experience with scripting.

Maybe this debian bug is related, though I'm not sure:

~/.local/share/akonadi/file_db_data/ is completly inneficient and render the whole system slow.

Directory shoud not store more than a thousand of entry. With my multi gigabyte mail box I have more than 900000 entry in this directory.

See http://etutorials.org/Server+Adminisdir+Directive/ paragraph 7.1.4 L1 and L2 for how to do.

thumbnail cache, git and squid use this technique.

I am tented to raise to RC severity because some filesystem does not support this insane number of file per directory.

What locate do you use? mlocate? slocate?

By memory mlocate but I use ext3.I think the main inifficiency is ext3 I am goind to switch to ext4.

/Sune

https://bugs.debian.org/cgi-bin/bugrcgi?bug=757844

I got a similar huge inbox, with 230289 emails. tree ~/Mail, shows 100 directories, 49907 files (12 GiB laut du -sh), but the mentioned path in the bug above got, according to tree, 101 directories und 227 files.

Alone the content of debian-de ML is according to KMail 1,6GiB with 195663 emails.

~/Mail (12GiB) ist ein Symlink auf eine SSD Partition (SanDisk SDSSDX240GG25)

Gerät Boot Start Ende Sektoren Größe Id Typ

/dev/sda6 394270720 468862127 74591408 35,6G 83 Linux

Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf

/dev/sda6 35G 18G 17G 52% /mnt/sda6

/dev/sda6: Linux rev 1.0 ext4 filesystem data

Maybe the issue comes froma failed Migration? https://userbase.kde.org/KMail/FAQs__Kmail_is_slow

kaddressbook and notes are also need a lot of time to show up.

This way kontact isn't useable at all for me.

Thank you in advance.