Re: kde-pim dev needed, for a sponsored open source effort, to remove akonadi

Dear Paul.

This may still contain spelling mistakes and so on, but I read through it 
about 2 times, so sending it as is now:

First off: I acknowledge that there are performance *and* stability related 
issues with KMail on Akonadi. There is no point in denying that. They are in 
part due to how Akonadi currently handles some things. I was busy with other 
things and I bet Dan was too, but I really look forward to provide the in-
depth technical information that Dan shared with Sandro, Pablo and me with the 
public, even if initially just by posting some mails to KDEPIM developer 
mailing list.

Second: I easily spend about one hour for writing this detailed response. On 
hour that I did not use to do other things to help moving beyond the current 
situations with KDEPIM + Akonadi. I am certainly not interested in engaging a 
lot in a discussion on this list that may just end up being fruitless. I am 
interested in helping to move things along, i.e. really doing something that 
is effective. As I am not doing any of this during my day job as well. So 
after this long and elaborate mail I may not respond much more to this thread.



Paul Vixie - 16.03.18, 22:18:
> so there are two things i am unclear on, wrt akonadi's problems.
> 
> looked at: https://marc.info/?l=kde-pim&m=151777761106658&w=2
> 
> question 1: how did it ever get this bad? clearly the design and
> implementation was not stress tested or stringently reviewed.

I am not a KDEPIM dev, I just spent and spend some effort to help moving 
things in a positive direction. Take my answer as my personal impression:

I think the answer to your question is two-fold:

1) It is challenging to get right from the beginning. Doing such an 
asynchronous personal information management system that works independently 
from the applications, like Akonadi, is a challenging task. Protocols like 
IMAP are complex already, how to handle errors during connection, like for 
example not being able to store a mail onto the IMAP server, is even more so. 
Akonadi currently misses replaying payloads (that is mails, appointments) to 
their backend storage (be it local maildir or IMAP server) in case the initial 
attempt to store them there failed. Server-side change as proposed by Dan can 
fix that.

2) There are only a few developers working on KDEPIM. Often in their spare 
time beside their daily job. This is where funding might be able to help 
depending on how flexible their current employers are and whether those 
developers would be willing to accept money from sponsors. The Krita project 
people successfully raised quite some money several times and has a developer 
(or even two at the end?) work on specific topics.

Paul, your approach to pay to basically rewrite quite some amount of KDEPIM to 
work with something other than Akonadi might even make sense… but I take the 
word of KDEPIM developers like Dan, Sandro, and others that it would be a 
really, really huge effort. I think that Sink which is used for Kube is not 
there yet as it focuses on just IMAP based setups, but I am happily told 
otherwise. And also Sink is quite different in a lot of ways, so porting 
things over to it, may also be a quite huge effort.

So I find it highly unlikely that one of them would agree to work on a fork of 
KDEPIM. Why? 

1. I think it is safe for me to say that current KDEPIM developers plan to use 
Akonadi. They don´t think its feasible to replace it with something else at 
the current point in time. 

2. It would split the developer, tester and user community even more.

3. And how I understood Dan who really looked deep into the internals of 
Akonadi with the four major changes I mentioned in the other mail, it would be 
possible to remove the reason for the majority of all current bug reports.

So I believe that going about fixing Akonadi may easily still be way less 
effort than basically rewriting lots of KDEPIM from scratch.

However it appears to still be quite some effort. And there are only a few 
developers available at the moment who are capable to do this work. And these 
are developing on KDEPIM in their spare time. That is why I thought that 
publishing what Dan wrote to us during a very constructive private mail 
exchange about the inner workings of Akonadi, the current technical 
shortcomings and ways on how to solve them for good, can be helpful to attract 
other developers at least for some parts of the work.

However, putting a condition on the sponsoring that developers have to remove 
Akonadi… would be requiring the current KDEPIM developers to work against what 
they think is most suitable. I think it is unlikely for you this way to 
attract one of the current KDEPIM developers as you are basically opposing 
they way they want to fix things in KDEPIM.

I do think you may be able to go a longer way if you offer money so that 
KDEPIM developers can spend more time to work on the four major changes I 
mentioned in the other mail. However I fully appreciate that you want to read 
more about it before committing to spend money on it and… it would still be 
the decision of the developer whether to agree to step back from his current 
job a bit in order to free up more time for a temporary development task.

I plan to spend some of my time to help making the information Dan shared 
privately with us available on KDEPIM development mailing list. I handed it to 
Dan himself to post the information, but it seems he did not have the time, so 
I offered my help to him just today.

> question 2: why would we stress test or stringently review a fix, given
> that akonadi + kde-pim as it is, wasn't tested or reviewed, and is
> unusably bad?

I don´t get what you mean by that?

As far as I got it from Dan and other developers Akonadi has been tested and 
reviewed. One issue in all of that is that Akonadi to KDEPIM developers quite 
some issues don´t seem to happen on developer boxes. One reason for that may 
be the sheer amount of different ways to configure and setup Akonadi and 
KDEPIM. And well some shortcomings in Akonadi that can lead to bugs that only 
happen some of the time.

> background:
> 
> i have to shut down kmail and akonadi whenever i'm not reading e-mail,
> to keep it from eating my 12-hour battery in 45 minutes, and to protect
> my network bandwidth. when i start it back up i have to do it in a very
> particular order since only the first imap connection to come up will be
> able to do IDLE, and the kolabsys imap server _only_ works if you do
> IDLE. and then akonadi has to check _every single folder_ to find out if
> anything has changed. and then if anything has changed it has to
> re-download _every single message_ in that folder. including trash,
> which usually has thousands of messages in it. so i'm in it for a
> minimum of a gigabyte of bandwith and 15% of my battery just to start
> kmail back up, which i have to do whenever i receive encrypted e-mail.

First off, do you use disconnected IMAP?

Second: AFAIK it does not redownload every message, but with disconnected IMAP 
it synchronizes what is has locally with the contents of the folders on the 
IMAP server. I do think that it may be doing that *way* too often and it would 
be beneficial to be able to specify how often it does that. I have seen it 
doing it after pressing the delete key to delete a mail about 5-10 times in a 
folder. I have reported several bug reports related to that.

> conclusion:
> 
> my view is, don't ask for second set of eyes on code or testing, just
> commit patches to make akonadi better, because nothing you can break
> that way would be worse than the current situation.

While I certainly acknowledge that you are facing real issues, let me tell you 
this: Not everyone does.

I still have KDEPIM + Akonadi 17.08 here and I not see most of the issues you 
mentioned. I do see performance related issues with my large POP3 + maildir 
private setup and I reduced this recently by tarring together old mails from 
mailing lists and then removing them from the maildir. But I am talking about 
way more than a 1 million of mails here. The work related account with uses 
IMAP works quite fine. There are some issues with losing the IMAP connection 
rarely, but as Akonadi talkes to an Exchange based IMAP account, I am not 
really all that surprised it. For me it helped setting it not to cache all 
mails locally. So it just caches the most recent ones.

Thanks,
-- 
Martin