Re: wiki web failling

Hi
Seems the wiki is inordinately slow/unusable again :(

--
Josh Davidson
________________________________
From: Zebediah C. McClure <[email protected]>
Sent: Sunday, March 15, 2026 15:09
To: [email protected] <[email protected]>
Subject: Re: [Flightgear-devel] wiki web failling

I'd like to second that.  I regularly use the web with 'non-normal' user
agents.  Rate limiting, firewalling and other solutions are superior in
my opinion to giving all our traffic to cloudflare.

zmc

On 2026-03-14 12:18, [email protected] wrote:
> Also,
> about non "usual browser/user agents"
>
> I would like to warn us against endorsing any and all "normalization"
> (excluding diversity).
>
> "Curl" is frequently used by Flightgear and, I think it should not
> become a reason to filter out users.
> I personally avoid "mainstream" browser/user-agents. I prefer browsers
> and tools in which I can "put my hand" on the code.
>
> AI do not "invent" solutions "that was not existing before".
> That is quite a frequent misconception.
>
> They provide an help (a very big help) crawling in the ocean of
> already given data (the e-data ocean :), in order to find those
> answers who most match plebiscites.
>
> Because if an answer if "good for many/everyone" it must be the right
> answer...
> I do not feel very happy with that rule, but it is what it is.
>
> I believe the best creation lives outside the skirt of the existing
> normality.
> I thus prefer to endorse diversity instead of normalization.
>
> Configuring Cloudflare to enforce browser normality is changing a
> problem for another.
> As I see it. ... no offence to any one preferences.
>
> Kind regards
> Lawrence
>
>
> On 2026-03-13 19:22, Patrizio Melis wrote:
>> ------------------------------
>> *Technical proposal to mitigate AI‑agent load on the FlightGear forum
>> and
>> wiki*
>>
>> Hi everyone,
>> in recent months many open‑source projects have been experiencing a
>> significant increase in traffic generated by AI agents (crawlers,
>> scrapers,
>> retrieval tools). This traffic is often unregulated, does not respect
>> robots.txt, and can easily saturate community‑run servers.
>>
>> I would like to propose a set of *realistic, low‑cost, and quickly
>> deployable measures* to protect the forum and wiki without reducing
>> accessibility for human users.
>> ------------------------------
>> 1. *Enable Cloudflare (free plan)*
>>
>> This is the most effective and immediate measure.
>> Benefits:
>>
>>    - aggressive caching of static wiki pages
>>    - automatic rate‑limiting for suspicious traffic
>>    - JS challenges for non‑compliant bots
>>    - basic DDoS protection
>>
>> Implementation:
>>
>>    - point DNS to Cloudflare
>>    - enable “Bot Fight Mode”
>>    - set a simple rate‑limit rule:
>>    *e.g., max 30 requests / 10 seconds per anonymous IP*
>>
>> No changes to the existing servers are required.
>> ------------------------------
>> 2. *Soft‑throttle anonymous access to the wiki*
>>
>> There is no need to lock the wiki; slowing down anonymous traffic is
>> enough.
>> Possible measures:
>>
>>    - reduce response speed for non‑logged‑in IPs
>>    - limit anonymous access to heavier pages
>>    - introduce a simple cookie‑check to distinguish bots from real
>> browsers
>>
>> This drastically reduces large‑scale scraping.
>> ------------------------------
>> 3. *Block or degrade suspicious user‑agents*
>>
>> Many scrapers identify themselves with recognizable user‑agents.
>> Actions:
>>
>>    - block known user‑agents (python-requests, aiohttp, curl,
>> Go-http-client,
>>    etc.)
>>    - degrade responses (1–2 second delay) for non‑standard user‑agents
>>
>> Aggressive bots often fail when latency is introduced.
>> ------------------------------
>> 4. *Local caching + database optimization*
>>
>> For the forum (phpBB or similar):
>>
>>    - enable Redis or Memcached if not already active
>>    - increase cache TTL
>>    - optimize indexes on the most stressed tables (search, sessions)
>>
>> For the wiki (MediaWiki):
>>
>>    - enable $wgMainCacheType = CACHE_ACCEL
>>    - use $wgParserCacheType = CACHE_ACCEL
>>    - increase $wgCacheEpoch to reduce unnecessary regenerations
>>
>> These changes reduce CPU load by 30–50%.
>> ------------------------------
>> 5. *Publish an official “AI Access Policy”*
>>
>> Many AI agents respect policies when they exist.
>> Recommended contents:
>>
>>    - scraping limits (e.g., max 1 req/sec)
>>    - requirement for identifiable user‑agents
>>    - prohibition of large‑scale forum scraping
>>    - contact requirement for bulk dataset access
>>
>> Publish it in:
>>
>>    - robots.txt
>>    - a dedicated wiki page
>>    - the X-Robots-Tag HTTP header
>>
>> This won’t stop malicious bots, but it reduces traffic from
>> “well‑behaved”
>> ones.
>> ------------------------------
>> 6. *Traffic monitoring and identification of problematic IPs*
>>
>> Recommended free tools:
>>
>>    - goaccess for real‑time log analysis
>>    - fail2ban to block IPs with abnormal patterns
>>    - netdata for lightweight server monitoring
>>
>> This allows quick reaction to abnormal spikes.
>> ------------------------------
>> 7. *Extreme but effective option: login required for forum access*
>>
>> Not the first choice, but if traffic becomes unmanageable:
>>
>>    - read‑only access for anonymous users
>>    - full access only for registered accounts
>>
>> Many bots cannot handle login and cookies.
>> ------------------------------
>> *Goal of this proposal*
>>
>> The goal is not to “block AI”, but to:
>>
>>    - protect community resources
>>    - ensure stability of the forum and wiki
>>    - prevent large‑scale scraping from degrading user experience
>>    - keep FlightGear accessible without additional costs
>>
>> All these measures can be implemented with moderate skills and no
>> financial
>> investment.
>> ------------------------------
>>
>>
>> Patrizio <aka> cobe571
>>
>> _______________________________________________
>> Flightgear-devel mailing list
>> [email protected]
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fflightgear-devel&data=05%7C02%7C%7C93e89e3353c84519a86708de834a2c2d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C639092551559647895%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=7AcjqWEbgB%2F3uNQIOLv0%2B0wENkvqR6siHC3FCDqBa4Y%3D&reserved=0<https://lists.sourceforge.net/lists/listinfo/flightgear-devel>
>
>
> _______________________________________________
> Flightgear-devel mailing list
> [email protected]
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fflightgear-devel&data=05%7C02%7C%7C93e89e3353c84519a86708de834a2c2d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C639092551559702073%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=bR3mJKU1B66ndxagUZ4PNfIe7Ao4gNLbP0IFGdYRNN8%3D&reserved=0<https://lists.sourceforge.net/lists/listinfo/flightgear-devel>


_______________________________________________
Flightgear-devel mailing list
[email protected]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fflightgear-devel&data=05%7C02%7C%7C93e89e3353c84519a86708de834a2c2d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C639092551559751404%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2V6CLBQCqXsDqMwvtKoZfNnZeuZbrFSPvc19YC%2BPOyA%3D&reserved=0<https://lists.sourceforge.net/lists/listinfo/flightgear-devel>

_______________________________________________
Flightgear-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/flightgear-devel