[TikiWiki-commits] [Git][tikiwiki/tiki][master] [NEW] Implement SEO preference to prevent site crawling and update robots.txt accordingly

Jonny Bradley pushed to branch master at Tiki Wiki CMS Groupware / Tiki


Commits:
4b98516c by Elvis Ansima at 2026-07-16T14:15:14+00:00
[NEW] Implement SEO preference to prevent site crawling and update robots.txt accordingly
---
* [REF] remove dead code

* UPD: Set default preference to yes so by default sites will not be crawled

https://dev.tiki.org/item7919-Simple-way-to-tell-search-engine-robot-to-index-or-not-a-Tiki-site

* [NEW] Implement SEO preference to prevent site crawling and update robots.txt accordingly

See merge request tikiwiki/tiki!9520

- - - - -


4 changed files:

- + lib/prefs/seo.php
- + templates/admin/include_seo.tpl
- templates/wizard/admin_advanced.tpl
- tiki-robots.php


Changes:

=====================================
lib/prefs/seo.php
=====================================
@@ -0,0 +1,19 @@
+<?php
+
+// (c) Copyright by authors of the Tiki Wiki CMS Groupware Project
+//
+// All Rights Reserved. See copyright.txt for details and a complete list of authors.
+// Licensed under the GNU LESSER GENERAL PUBLIC LICENSE. See license.txt for details.
+function prefs_seo_list()
+{
+    return [
+
+        'seo_prevent_crawling' => [
+            'name' => tr('Prevent entire site from being crawled'),
+            'description' => tr('Blocks all search engine crawlers from indexing the site. Useful for development, staging, or private sites.'),
+            'type' => 'flag',
+            'default' => 'y',
+            'keywords' => 'seo robots crawl search engine noindex block',
+        ],
+    ];
+}


=====================================
templates/admin/include_seo.tpl
=====================================
@@ -0,0 +1,16 @@
+<form action="tiki-admin.php?page=seo" onreset="return(confirm('{tr}Cancel Edit{/tr}'))" class="admin" method="post">
+{ticket}
+<div class="row">
+    <div class="mb-3 col-lg-12 clearfix">
+        {include file='admin/include_apply_top.tpl'}
+    </div>
+</div>
+<div class="styling-element">
+    <fieldset>
+        <legend class="h3">{tr}Search Engine Crawling{/tr}{help url="SEO"}</legend>
+
+        {preference name=seo_prevent_crawling}
+    </fieldset>
+    {include file='admin/include_apply_bottom.tpl'}
+</div>
+</form>


=====================================
templates/wizard/admin_advanced.tpl
=====================================
@@ -14,7 +14,11 @@
             {preference name=workspace_ui}
             <em>{tr}See also{/tr} <a href="https://doc.tiki.org/Workspaces UI" target="_blank">{tr}Workspaces UI in doc.tiki.org{/tr}</a></em>
         </fieldset>
-        <fieldset>
+        <fieldset class="mt-3">
+            <legend>{tr}Search Engines Optimization{/tr}</legend>
+            <div class="admin clearfix featurelist">
+                {preference name=seo_prevent_crawling}
+            </div>
             <legend>{tr}Dependencies{/tr}</legend>
             <div class="admin clearfix featurelist">
                 {preference name=feature_categories}


=====================================
tiki-robots.php
=====================================
@@ -11,6 +11,30 @@ if ($prefs['sitemap_enable'] == 'y') {
     $dynamicOptions .= "Sitemap: {$base_url}storage/public/sitemap-index.xml";
 }
 
+if ($prefs['seo_prevent_crawling'] == 'y') {
+    $disallowRule = "# Prevent crawling of the entire site, you can change this via preferences" . PHP_EOL;
+    $disallowRule .= "Disallow: /" . PHP_EOL;
+    echo <<<EOF
+    # This is a robots.txt file to instruct all search bots that we don't want them to crawl in the paths beginning with the strings below.
+    # If you are viewing this via example.org/robots.txt, it is an alias of tiki-robots.php (likely via an instruction in .htaccess)
+    # tiki-robots.php permits to dynamically generate a robots.txt file according to preferences set in the admin panel
+    # 
+    # For an installation in a subdirectory, you have to copy this file in root of your domain and add /yoursubdirname on each line.
+    #
+    # (c) Copyright by authors of the Tiki Wiki CMS Groupware Project
+    #
+    # All Rights Reserved. See copyright.txt for details and a complete list of authors.
+    # Licensed under the GNU LESSER GENERAL PUBLIC LICENSE. See license.txt for details.
+
+    User-agent: *
+    {$disallowRule}
+    EOF;
+    exit;
+} else {
+    $disallowRule = "# Allow site crawling" . PHP_EOL;
+    $disallowRule .= "Allow: /" . PHP_EOL;
+}
+
 echo <<<EOF
 # This is a robots.txt file to instruct all search bots that we don't want them to crawl in the paths beginning with the strings below.
 # If you are viewing this via example.org/robots.txt, it is an alias of tiki-robots.php (likely via an instruction in .htaccess)
@@ -24,8 +48,7 @@ echo <<<EOF
 # Licensed under the GNU LESSER GENERAL PUBLIC LICENSE. See license.txt for details.
 
 User-agent: *
-# Uncomment the following line to indicate to robots __not__ to crawl your site.
-# Disallow: /
+{$disallowRule}
 
 {$dynamicOptions}
 



View it on GitLab: https://gitlab.com/tikiwiki/tiki/-/commit/4b98516cf64ce169b3a98ff9795fea256d78fe4e

-- 
View it on GitLab: https://gitlab.com/tikiwiki/tiki/-/commit/4b98516cf64ce169b3a98ff9795fea256d78fe4e
You're receiving this email because of your account on gitlab.com. Manage all notifications: https://gitlab.com/-/profile/notifications | Help: https://gitlab.com/help

_______________________________________________
TikiWiki-cvs mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-cvs