Re: Update on CPANTesters.org stability

On 4/29/25 22:53, Scott Baker wrote:
> CPT Community:
> 
> Here is a quick update on some changes we've made to stabilize 
> *cpantesters.org*. Preaction was able to get a handful of us SSH access 
> to the CPT servers. The "outages" and latency we've all noticed lately 
> have been related to Apache resource exhaustion on the CPT4 web server 
> that handles: www, api, reports, etc. We're still not 100% what is 
> causing Apache to go unresponsive but we wanted to let everyone know 
> we've made some improvements in the last week:
> 
>   * Increased Apache listeners to facilitate higher levels of traffic
>   * Preaction reconfigured the Fastly CDN to remove a failed node
>   * 8GB swapfile was added to give the box some extra resources
>   * API.pm cleanup to fix a Mojolicious configuration artifact
>   * Robust monitoring tools installed to watch health in real time
> 
> Next we will need to investigate the health of the DB server as well. If 
> you have MySQL skills we could use your help diagnosing high load we're 
> seeing. We have a rather large MySQL DB (~1.5TB) that contains about 20 
> years of CPT history. The DB server gets ~37 queries per second and has 
> a sustained throughput of about 600Mb/s of read from the disks. From 
> what I can tell the DB server is healthy and functional, but it's 
> working /*hard*/.
> 
> If you have expertise in this field please reach out, or speak out at 
> PTC so we can not only stabilize CPT but improve it and take it to the 
> next level.
> 
> -- Scottchiefbaker
> 

Thank you very much for your investigation.  As I reported on 
#cpantesters-discuss yesterday, I am able to (i) upload CPANtesters 
reports from two different machines; (ii) view them 5 minutes later in 
https://metabase.cpantesters.org/tail/log.txt; (iii) then see them via 
fast-matrix 
(http://fast-matrix.cpantesters.org/?dist=File-Fetch;perl=5.41.11;reports=1). 
  So the situation currently DWIMs for me.