Re: [Tiki-devel] doc.tiki.org is down?!

Please see below



On 2021-07-20 02:06, aris002 via TikiWiki-devel wrote:
> Hi again,
> 
> Sorry @Marc, but
> 
> 1. The situation of the tiki.org websites/subdomains
> slowliness/deadliness and the unwillingness to improve it just have
> been wearing me down so much and for so long that I have just run out
> of helpful comments. :)
> 

Thank you Aris,

This message is helpful.

No one has an "unwillingness to improve it". It can be a lack of 
information, time, skill or server access.



> 2. Any good comments how and when might the servers and the websites
> be improved with regards to my ideas below?
> 

I will reply inline


> 3. Are tiki.org and wikisuite websites using load balancer and/or any
> emergincy measures to switch to an emergency website or webpage to
> show something and white screen of death if the server or one of the
> domains is down?
>  And I guess they are on the same server?
> 

There are no load balancers. Some could be added of course.

And various sites are on various servers managed by various people, 
which makes the whole thing more complicated.



> 4. Is tiki.org lacking computer power then? Because it can't be that
> on my 25 Eu/month VPS server I have 0.01s->0.6s page generation time,
> when hosting some other websites even without load balancer.
> And tiki.org has between 2.3->37s and that sometimes I lose patience
> to even record it.
> Some web theory says that unacceptible speed makes redundant any other
> website and/or navigation improvements...
> 


Different sites are on different servers. So the problem on each can be 
very different...

tiki.org is on one server. I don't know if it lacks computing power.

The server that houses dev.tiki.org. doc.tiki.org, themes.tiki.org and 
many others is severely lacking CPU. See image.

Some of the *.tiki.org sites have a quantity and complexity of data 
rarely seen in other Tiki instances (15+ years of data)


> 5. If tiki offer a product as a dog food it must not even smell that
> slow...
> Google suggests normal response time 0.2 sec to the First Byte. And
> most open source projects I am using somehow don't have such slow
> websites.
> How do they manage that? Do we need to lauch a campaign to get
> donations for the server?
> 

Donations would be nice because the current setup is costing too much.
https://tiki.org/Contribute-to-Tiki

I have rented a dedicated server to which we can moved to, so more CPU 
is coming.

But throwing money at the problem with more CPUs is not the full 
solution. We should improve the code/config as well.


>  6. I have got couple more guesses that quite often the server gets
> overloaded with some other things...
> 

Indeed. I suspect specific triggers.


> 7. And also it might be something wrong with the drop down menu
> scripts when switching domains and/or with tiki multi domain code
>  because sometimes there is more than 10 times difference or more in
> response times comparing to when opened in a browser directly or from
> browser bookmarks.
> 

Yes, multi-domain code and perspectives are likely a source of issues. 
And they are difficult to test on another server.

FYI, I tried to delete some links but hit a bug:
https://avan.tech/item65090-I-can-t-modify-content-of-Modules-on-tiki-org-via-https-tiki-org-tiki-admin_modules-php-I-suspect-an-issue-with-perspectives


> 8. But then again, willingness to improve, accept problems, tell the
> real problems, accept help can prevent not only unhelpful comments but
> also loss of willing actively help and participate community members.
> 

So you like the blunt truth:

1- Some people promised they would help and they were unreliable so it 
adds more work to the reliable ones.

2- In the past, we have trusted people to take initiatives and make 
changes, and some messed things up, and disappeared. And the reliable 
ones had to clean up the mess.

3- Some people would like to help but do not have the necessary 
competence to solve these issues.

4- Server access requires trust level, way different than making merge 
requests on the code, or updating docs.

5- While clients fund a decent portion of new stuff:
https://doc.tiki.org/Tiki23
https://doc.tiki.org/Tiki22
https://doc.tiki.org/Tiki21
the biggest clients that pay for the development of Tiki barely use 
*.tiki.org sites and even when they do, they will not allocate budgets 
to it. They pay firms like EvoluData.com to manage their Tiki and train 
them.

6- The people that have the access, skills and trust are usually very 
busy helping clients.

7- Some of the *.tiki.org site are 15+ years old and have a huge amount 
of data. And can use features in an unusual way. They can hit edge case 
bugs and different challenges than the rest of the community. Here are 
examples:

"Limit number of expired tokens (and temp users) to delete to 10,000 - 
there are about 2 million on doc.t.o"
https://gitlab.com/tikiwiki/tiki/-/commit/5292fbdf9d59832f8015e1bd455e333d7b4f7d14

"FIX] tokens: Remove token created for PDF generation after use, and 
reduce the number of dead tokens to remove to avoid timeout errors (2000 
should take about 10 seconds max) plus some other optimisations"
https://gitlab.com/tikiwiki/tiki/-/commit/0672f897f8babe2598f0de500cbfd9aaea86c416

This is good dogfood of course! But still painful.


8- We are lacking an active https://tiki.org/Analytics-Team that would 
track the traffic on our sites and warn us of issues. If anyone reading 
this has interest/skills in this area, please reach out to me. Maybe 
sites are drowning from bots trying to create thousands of PDFs. If we 
knew that was the source, we could turn off PDF generation, block 
robots, or pre-generate files.

9- This change has eaten up a lot of time:
https://wikisuite.org/blogpost16-WikiSuite-will-now-support-all-major-Linux-distros

10- We are busy releasing Tiki 23 which involves upgrading a bunch of 
sites

11- There were a lot of infrastructure / back-end changes (PSR-12, 
Composer 2, etc.) which took time in recent months. (From the same 
people that can help with servers)

12- We are working on a new server infrastructure (more info in next 
message: "Show3 and performance / reliability of community servers"


But all those 12 points above are not the main issue. The main issue is 
that we do not have a proper monitoring system. We are somewhat blind to 
what are the root causes of issues.

And we can waste a lot of time because we don't have the right tools.

As an example: We struggled for over a year for random server crashes, 
and in the end, it was this: https://bugs.php.net/bug.php?id=71135
But we thought it was a Tiki bug, and we wasted a lot of time.


> 9. Btw, if someone need helping hand I am also still available most of
> the time.
> 

Great!!

Please help clean us the monitoring code that was started and never 
finished:
https://dev.tiki.org/Monitoring-revamp#Improve_Tiki_code_to_provide_Tiki-specific_alert_to_Zabbix_and_or_NetData_help_needed_

And please help up test and improve https://dev.tiki.org/GlitchTip   
Specifically, GlitchTip should be triggers on slow pages.

Then, we can set up NetData, Zabbix and GlitchTip to give the right info 
to the right person at the right time to solve the root causes of the 
performance/reliability problems. And we can run this on pre-dogfood 
servers so we can quickly catch any regressions in the future.

Thank you and best regards,

Marc


> Best regards,
> Aris
> 
>  On Monday, 19 July 2021, 23:36:25 EEST, <[email protected]> wrote:
> 
> 
> @Aris: Such comments are not helpful.
> 
> On 2021-07-19 13:16, aris002 via TikiWiki-devel wrote:
>> I guess, tiki.org is not down. This is just a normal performance
> with
>> several minutes response time to some pages and/or subdomains. I
> talk
>> about this every week on navigation improvement meetings, monthly
>> meetings, and tiki fests.
>> But it seems too many people at tiki.org are just addicted to this
>> kind of performance.
>> 
>> sorry.
>> 
>>  On Monday, 19 July 2021, 12:34:43 EEST, Karsten Wemheuer
>> <[email protected]> wrote:
>> 
>> Hi *,
>> 
>> It looks to me like the site https://doc.tiki.org/ [1]is [1]down.
> Who is
>> 
>> responsible or must be informed to fix this?
>> 
>> Best regards,
>> 
>> Karsten
>> 
>> _______________________________________________
>> 
>> TikiWiki-devel mailing list
>> 
>> [email protected]
>> 
>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>> 
>> 
>> Links:
> 
>> ------
>> [1] https://doc.tiki.org/ is
>> _______________________________________________
>> TikiWiki-devel mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
> 
> 
> Links:
> ------
> [1] https://doc.tiki.org/
> _______________________________________________
> TikiWiki-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel

_______________________________________________
TikiWiki-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel