Re: [Tiki-devel] doc.tiki.org is down?!

Thanks Aris and Marc (and people doing their best to improve current situation with room-for-improvement)
One small addition to something to keep in mind for t.o (and other sites) as registered users clicking here and there. Any action that triggers some email sending to users watching pages (through user/group-watches system, and/or the notification-system in Tiki, etc) might add an extra important waiting time for the user after posting a comment/editing a wiki page/changing some tracker item, etc), if Tiki is not setup to use some imap queue to send emails.
Tiki allows that configuration (and it was proven to work years ago, I haven't tested for years on heavy tiki sites), but my experience was that it was a configuration that was not set in most *.t.o places.

HTH

Best

Xavi


On Wed, 21 Jul 2021 19:00:04 +0300 Tiki developers [email protected] said

> Please see below
> 
> 
> 
> On 2021-07-20 02:06, aris002 via TikiWiki-devel wrote:
> > Hi again,
> > 
> > Sorry @Marc, but
> > 
> > 1. The situation of the tiki.org websites/subdomains
> > slowliness/deadliness and the unwillingness to improve it just have
> > been wearing me down so much and for so long that I have just run out
> > of helpful comments. :)
> > 
> 
> Thank you Aris,
> 
> This message is helpful.
> 
> No one has an "unwillingness to improve it". It can be a lack of 
> information, time, skill or server access.
> 
> 
> 
> > 2. Any good comments how and when might the servers and the websites
> > be improved with regards to my ideas below?
> > 
> 
> I will reply inline
> 
> 
> > 3. Are tiki.org and wikisuite websites using load balancer and/or any
> > emergincy measures to switch to an emergency website or webpage to
> > show something and white screen of death if the server or one of the
> > domains is down?
> >  And I guess they are on the same server?
> > 
> 
> There are no load balancers. Some could be added of course.
> 
> And various sites are on various servers managed by various people, 
> which makes the whole thing more complicated.
> 
> 
> 
> > 4. Is tiki.org lacking computer power then? Because it can't be that
> > on my 25 Eu/month VPS server I have 0.01s->0.6s page generation time,
> > when hosting some other websites even without load balancer.
> > And tiki.org has between 2.3->37s and that sometimes I lose patience
> > to even record it.
> > Some web theory says that unacceptible speed makes redundant any other
> > website and/or navigation improvements...
> > 
> 
> 
> Different sites are on different servers. So the problem on each can be 
> very different...
> 
> tiki.org is on one server. I don't know if it lacks computing power.
> 
> The server that houses dev.tiki.org. doc.tiki.org, themes.tiki.org and 
> many others is severely lacking CPU. See image.
> 
> Some of the *.tiki.org sites have a quantity and complexity of data 
> rarely seen in other Tiki instances (15+ years of data)
> 
> 
> > 5. If tiki offer a product as a dog food it must not even smell that
> > slow...
> > Google suggests normal response time 0.2 sec to the First Byte. And
> > most open source projects I am using somehow don't have such slow
> > websites.
> > How do they manage that? Do we need to lauch a campaign to get
> > donations for the server?
> > 
> 
> Donations would be nice because the current setup is costing too much.
> https://tiki.org/Contribute-to-Tiki
> 
> I have rented a dedicated server to which we can moved to, so more CPU 
> is coming.
> 
> But throwing money at the problem with more CPUs is not the full 
> solution. We should improve the code/config as well.
> 
> 
> >  6. I have got couple more guesses that quite often the server gets
> > overloaded with some other things...
> > 
> 
> Indeed. I suspect specific triggers.
> 
> 
> > 7. And also it might be something wrong with the drop down menu
> > scripts when switching domains and/or with tiki multi domain code
> >  because sometimes there is more than 10 times difference or more in
> > response times comparing to when opened in a browser directly or from
> > browser bookmarks.
> > 
> 
> Yes, multi-domain code and perspectives are likely a source of issues. 
> And they are difficult to test on another server.
> 
> FYI, I tried to delete some links but hit a bug:
> https://avan.tech/item65090-I-can-t-modify-content-of-Modules-on-tiki-org-via-https-tiki-org-tiki-admin_modules-php-I-suspect-an-issue-with-perspectives
> 
> 
> > 8. But then again, willingness to improve, accept problems, tell the
> > real problems, accept help can prevent not only unhelpful comments but
> > also loss of willing actively help and participate community members.
> > 
> 
> So you like the blunt truth:
> 
> 1- Some people promised they would help and they were unreliable so it 
> adds more work to the reliable ones.
> 
> 2- In the past, we have trusted people to take initiatives and make 
> changes, and some messed things up, and disappeared. And the reliable 
> ones had to clean up the mess.
> 
> 3- Some people would like to help but do not have the necessary 
> competence to solve these issues.
> 
> 4- Server access requires trust level, way different than making merge 
> requests on the code, or updating docs.
> 
> 5- While clients fund a decent portion of new stuff:
> https://doc.tiki.org/Tiki23
> https://doc.tiki.org/Tiki22
> https://doc.tiki.org/Tiki21
> the biggest clients that pay for the development of Tiki barely use 
> *.tiki.org sites and even when they do, they will not allocate budgets 
> to it. They pay firms like EvoluData.com to manage their Tiki and train 
> them.
> 
> 6- The people that have the access, skills and trust are usually very 
> busy helping clients.
> 
> 7- Some of the *.tiki.org site are 15+ years old and have a huge amount 
> of data. And can use features in an unusual way. They can hit edge case 
> bugs and different challenges than the rest of the community. Here are 
> examples:
> 
> "Limit number of expired tokens (and temp users) to delete to 10,000 - 
> there are about 2 million on doc.t.o"
> https://gitlab.com/tikiwiki/tiki/-/commit/5292fbdf9d59832f8015e1bd455e333d7b4f7d14
> 
> "FIX] tokens: Remove token created for PDF generation after use, and 
> reduce the number of dead tokens to remove to avoid timeout errors (2000 
> should take about 10 seconds max) plus some other optimisations"
> https://gitlab.com/tikiwiki/tiki/-/commit/0672f897f8babe2598f0de500cbfd9aaea86c416
> 
> This is good dogfood of course! But still painful.
> 
> 
> 8- We are lacking an active https://tiki.org/Analytics-Team that would 
> track the traffic on our sites and warn us of issues. If anyone reading 
> this has interest/skills in this area, please reach out to me. Maybe 
> sites are drowning from bots trying to create thousands of PDFs. If we 
> knew that was the source, we could turn off PDF generation, block 
> robots, or pre-generate files.
> 
> 9- This change has eaten up a lot of time:
> https://wikisuite.org/blogpost16-WikiSuite-will-now-support-all-major-Linux-distros
> 
> 10- We are busy releasing Tiki 23 which involves upgrading a bunch of 
> sites
> 
> 11- There were a lot of infrastructure / back-end changes (PSR-12, 
> Composer 2, etc.) which took time in recent months. (From the same 
> people that can help with servers)
> 
> 12- We are working on a new server infrastructure (more info in next 
> message: "Show3 and performance / reliability of community servers"
> 
> 
> But all those 12 points above are not the main issue. The main issue is 
> that we do not have a proper monitoring system. We are somewhat blind to 
> what are the root causes of issues.
> 
> And we can waste a lot of time because we don't have the right tools.
> 
> As an example: We struggled for over a year for random server crashes, 
> and in the end, it was this: https://bugs.php.net/bug.php?id=71135
> But we thought it was a Tiki bug, and we wasted a lot of time.
> 
> 
> > 9. Btw, if someone need helping hand I am also still available most of
> > the time.
> > 
> 
> Great!!
> 
> Please help clean us the monitoring code that was started and never 
> finished:
> https://dev.tiki.org/Monitoring-revamp#Improve_Tiki_code_to_provide_Tiki-specific_alert_to_Zabbix_and_or_NetData_help_needed_
> 
> And please help up test and improve https://dev.tiki.org/GlitchTip   
> Specifically, GlitchTip should be triggers on slow pages.
> 
> Then, we can set up NetData, Zabbix and GlitchTip to give the right info 
> to the right person at the right time to solve the root causes of the 
> performance/reliability problems. And we can run this on pre-dogfood 
> servers so we can quickly catch any regressions in the future.
> 
> Thank you and best regards,
> 
> Marc
> 
> 
> > Best regards,
> > Aris
> > 
> >  On Monday, 19 July 2021, 23:36:25 EEST, <[email protected]> wrote:
> > 
> > 
> > @Aris: Such comments are not helpful.
> > 
> > On 2021-07-19 13:16, aris002 via TikiWiki-devel wrote:
> >> I guess, tiki.org is not down. This is just a normal performance
> > with
> >> several minutes response time to some pages and/or subdomains. I
> > talk
> >> about this every week on navigation improvement meetings, monthly
> >> meetings, and tiki fests.
> >> But it seems too many people at tiki.org are just addicted to this
> >> kind of performance.
> >> 
> >> sorry.
> >> 
> >>  On Monday, 19 July 2021, 12:34:43 EEST, Karsten Wemheuer
> >> <[email protected]> wrote:
> >> 
> >> Hi *,
> >> 
> >> It looks to me like the site https://doc.tiki.org/ [1]is [1]down.
> > Who is
> >> 
> >> responsible or must be informed to fix this?
> >> 
> >> Best regards,
> >> 
> >> Karsten
> >> 
> >> _______________________________________________
> >> 
> >> TikiWiki-devel mailing list
> >> 
> >> [email protected]
> >> 
> >> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
> >> 
> >> 
> >> Links:
> > 
> >> ------
> >> [1] https://doc.tiki.org/ is
> >> _______________________________________________
> >> TikiWiki-devel mailing list
> >> [email protected]
> >> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
> > 
> > 
> > Links:
> > ------
> > [1] https://doc.tiki.org/
> > _______________________________________________
> > TikiWiki-devel mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel