Re: wiki web failling

Ever since the Mysql errors we got about a month or so ago, both the 
wiki and the forum have been incredibly slow compared to before those 
Mysql errors were popping up. I just figured something was changed to 
fix the Mysql database. If this is something else, it would be wonderful 
if we could get it fixed.

Here in the midwest it has been hit or miss, mostly miss. However, right 
this instant it appears to be about normal, as in relatively fast, both 
wiki and forum.


I did not read through this entire thread so forgive me if the answer is 
already in the messages.


------------------------------------------------------------------------------------------------

On 5/23/2026 12:19 PM, Josh Davidson wrote:
> Hi
> Seems the wiki is inordinately slow/unusable again :(
>
> --
> Josh Davidson
> ------------------------------------------------------------------------
> *From:* Zebediah C. McClure <[email protected]>
> *Sent:* Sunday, March 15, 2026 15:09
> *To:* [email protected] 
> <[email protected]>
> *Subject:* Re: [Flightgear-devel] wiki web failling
> I'd like to second that.  I regularly use the web with 'non-normal' user
> agents.  Rate limiting, firewalling and other solutions are superior in
> my opinion to giving all our traffic to cloudflare.
>
> zmc
>
> On 2026-03-14 12:18, [email protected] wrote:
> > Also,
> > about non "usual browser/user agents"
> >
> > I would like to warn us against endorsing any and all "normalization"
> > (excluding diversity).
> >
> > "Curl" is frequently used by Flightgear and, I think it should not
> > become a reason to filter out users.
> > I personally avoid "mainstream" browser/user-agents. I prefer browsers
> > and tools in which I can "put my hand" on the code.
> >
> > AI do not "invent" solutions "that was not existing before".
> > That is quite a frequent misconception.
> >
> > They provide an help (a very big help) crawling in the ocean of
> > already given data (the e-data ocean :), in order to find those
> > answers who most match plebiscites.
> >
> > Because if an answer if "good for many/everyone" it must be the right
> > answer...
> > I do not feel very happy with that rule, but it is what it is.
> >
> > I believe the best creation lives outside the skirt of the existing
> > normality.
> > I thus prefer to endorse diversity instead of normalization.
> >
> > Configuring Cloudflare to enforce browser normality is changing a
> > problem for another.
> > As I see it. ... no offence to any one preferences.
> >
> > Kind regards
> > Lawrence
> >
> >
> > On 2026-03-13 19:22, Patrizio Melis wrote:
> >> ------------------------------
> >> *Technical proposal to mitigate AI‑agent load on the FlightGear forum
> >> and
> >> wiki*
> >>
> >> Hi everyone,
> >> in recent months many open‑source projects have been experiencing a
> >> significant increase in traffic generated by AI agents (crawlers,
> >> scrapers,
> >> retrieval tools). This traffic is often unregulated, does not respect
> >> robots.txt, and can easily saturate community‑run servers.
> >>
> >> I would like to propose a set of *realistic, low‑cost, and quickly
> >> deployable measures* to protect the forum and wiki without reducing
> >> accessibility for human users.
> >> ------------------------------
> >> 1. *Enable Cloudflare (free plan)*
> >>
> >> This is the most effective and immediate measure.
> >> Benefits:
> >>
> >>    - aggressive caching of static wiki pages
> >>    - automatic rate‑limiting for suspicious traffic
> >>    - JS challenges for non‑compliant bots
> >>    - basic DDoS protection
> >>
> >> Implementation:
> >>
> >>    - point DNS to Cloudflare
> >>    - enable “Bot Fight Mode”
> >>    - set a simple rate‑limit rule:
> >>    *e.g., max 30 requests / 10 seconds per anonymous IP*
> >>
> >> No changes to the existing servers are required.
> >> ------------------------------
> >> 2. *Soft‑throttle anonymous access to the wiki*
> >>
> >> There is no need to lock the wiki; slowing down anonymous traffic is
> >> enough.
> >> Possible measures:
> >>
> >>    - reduce response speed for non‑logged‑in IPs
> >>    - limit anonymous access to heavier pages
> >>    - introduce a simple cookie‑check to distinguish bots from real
> >> browsers
> >>
> >> This drastically reduces large‑scale scraping.
> >> ------------------------------
> >> 3. *Block or degrade suspicious user‑agents*
> >>
> >> Many scrapers identify themselves with recognizable user‑agents.
> >> Actions:
> >>
> >>    - block known user‑agents (python-requests, aiohttp, curl,
> >> Go-http-client,
> >>    etc.)
> >>    - degrade responses (1–2 second delay) for non‑standard user‑agents
> >>
> >> Aggressive bots often fail when latency is introduced.
> >> ------------------------------
> >> 4. *Local caching + database optimization*
> >>
> >> For the forum (phpBB or similar):
> >>
> >>    - enable Redis or Memcached if not already active
> >>    - increase cache TTL
> >>    - optimize indexes on the most stressed tables (search, sessions)
> >>
> >> For the wiki (MediaWiki):
> >>
> >>    - enable $wgMainCacheType = CACHE_ACCEL
> >>    - use $wgParserCacheType = CACHE_ACCEL
> >>    - increase $wgCacheEpoch to reduce unnecessary regenerations
> >>
> >> These changes reduce CPU load by 30–50%.
> >> ------------------------------
> >> 5. *Publish an official “AI Access Policy”*
> >>
> >> Many AI agents respect policies when they exist.
> >> Recommended contents:
> >>
> >>    - scraping limits (e.g., max 1 req/sec)
> >>    - requirement for identifiable user‑agents
> >>    - prohibition of large‑scale forum scraping
> >>    - contact requirement for bulk dataset access
> >>
> >> Publish it in:
> >>
> >>    - robots.txt
> >>    - a dedicated wiki page
> >>    - the X-Robots-Tag HTTP header
> >>
> >> This won’t stop malicious bots, but it reduces traffic from
> >> “well‑behaved”
> >> ones.
> >> ------------------------------
> >> 6. *Traffic monitoring and identification of problematic IPs*
> >>
> >> Recommended free tools:
> >>
> >>    - goaccess for real‑time log analysis
> >>    - fail2ban to block IPs with abnormal patterns
> >>    - netdata for lightweight server monitoring
> >>
> >> This allows quick reaction to abnormal spikes.
> >> ------------------------------
> >> 7. *Extreme but effective option: login required for forum access*
> >>
> >> Not the first choice, but if traffic becomes unmanageable:
> >>
> >>    - read‑only access for anonymous users
> >>    - full access only for registered accounts
> >>
> >> Many bots cannot handle login and cookies.
> >> ------------------------------
> >> *Goal of this proposal*
> >>
> >> The goal is not to “block AI”, but to:
> >>
> >>    - protect community resources
> >>    - ensure stability of the forum and wiki
> >>    - prevent large‑scale scraping from degrading user experience
> >>    - keep FlightGear accessible without additional costs
> >>
> >> All these measures can be implemented with moderate skills and no
> >> financial
> >> investment.
> >> ------------------------------
> >>
> >>
> >> Patrizio <aka> cobe571
> >>
> >> _______________________________________________
> >> Flightgear-devel mailing list
> >> [email protected]
> >> 
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fflightgear-devel&data=05%7C02%7C%7C93e89e3353c84519a86708de834a2c2d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C639092551559647895%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=7AcjqWEbgB%2F3uNQIOLv0%2B0wENkvqR6siHC3FCDqBa4Y%3D&reserved=0 
> <https://lists.sourceforge.net/lists/listinfo/flightgear-devel>
> >
> >
> > _______________________________________________
> > Flightgear-devel mailing list
> > [email protected]
> > 
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fflightgear-devel&data=05%7C02%7C%7C93e89e3353c84519a86708de834a2c2d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C639092551559702073%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=bR3mJKU1B66ndxagUZ4PNfIe7Ao4gNLbP0IFGdYRNN8%3D&reserved=0 
> <https://lists.sourceforge.net/lists/listinfo/flightgear-devel>
>
>
> _______________________________________________
> Flightgear-devel mailing list
> [email protected]
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fflightgear-devel&data=05%7C02%7C%7C93e89e3353c84519a86708de834a2c2d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C639092551559751404%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2V6CLBQCqXsDqMwvtKoZfNnZeuZbrFSPvc19YC%2BPOyA%3D&reserved=0 
> <https://lists.sourceforge.net/lists/listinfo/flightgear-devel>
>
>
> _______________________________________________
> Flightgear-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/flightgear-devel

_______________________________________________
Flightgear-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/flightgear-devel