Re: [Tiki-devel] Reviewing and limiting crawling to pages we don't need people to see as search results

Hi Benard,

It is an ongoing battle. Sometimes it is handled very well, other times it could be handled better. It partly depends on what features are enabled/used.

Having an automated system of motoring would be perfect. Particularly if it could be integrated into unit tests, but we are not there yet.

We had an issue not so long ago where the canonical for PDFs of wiki pages was not being handled. So google decided what page to show. A bunch of times in the TIki documentation it would only show the PDF version of the page instead of the HTML version. This is where an HTTP canonical entry needs to be made on the PDF to tell google it's a duplicate of the wiki page and to link to the wiki page.

It is something that I have put a reasonable amount of effort into in the past. I think Tiki does handle it well much of the time, but there is still room for improvement.

One area where we could be better is in sending the correct HTTP response. We could send more 401, 403, 406, 418 & 201 codes. Google even guesses sometimes that our codes are wrong and re-classifies them sometimes. I am sure we could get slightly better rankings by giving them correct codes, to begin with. Some non-important pages don’t have noindex indicated. They are results one wouldn’t normally find because of the lack of content, but it's better than useless information is not indexed. Grated that type of thing is hard to see unless one is digging for it.

In any case, I agree that Tiki is largely on the right track :)

Brendan



> On Aug 21, 2021, at 12:52 PM, Bsfez Tiki via TikiWiki-devel <[email protected]> wrote:
> 
> Hi Brendan,
> 
> Happy we have you with us. ;-)
> 
> I don’t have your knowledge about this and even after reading I feel confused with what I can do in a Tiki(22) website.
> 
> https://developers.google.com/search/docs/advanced/crawling/consolidate-duplicate-urls <https://developers.google.com/search/docs/advanced/crawling/consolidate-duplicate-urls>
> https://www.semrush.com/blog/canonical-url-guide <https://www.semrush.com/blog/canonical-url-guide>
> 
> However I have to say that working little by little on my own websites (canonical, sitemap, using page description for metatag) I kind of feel like things are handled properly by Tiki.
> I only add to disallow /404 and /item* (I don’t want people to land on items, but on pages ;-) )
> 
> Bernard
> 
> 
>> On 20 Aug 2021, at 19:46 , Dr. Sassafras <[email protected] <mailto:[email protected]>> wrote:
>> 
>> Blocking these will not do what you want. Canonicalization is what you want. https://developers.google.com/search/docs/advanced/guidelines/duplicate-content <https://developers.google.com/search/docs/advanced/guidelines/duplicate-content>
>> 
>> Years ago I started blocking content of one of my sites to get better search results. Much of the existing blocking in robots.txt is from me. I knew it wasn't the best way then but it kinda works and was easy for me to handle changing a test file. It has major drawbacks.
>> 
>> Instead, I would suggest canonicalization for any duplicate content and in-page tag of noindex. It would be better if we could change that from an HTML tag to anHTTP header. Then we could apply that to any content in tiki, even files or pdfs, etc. After content has in-page control of these things, the corresponding robots.txt entry can be deleted.
>> 
>> I agree that we should be better at this. It perhaps should also be a unit test? To make sure every page is classified…
>> 
>> Brendan
>> 
>> 
>>> On Aug 20, 2021, at 4:50 AM, Bsfez Tiki via TikiWiki-devel <[email protected] <mailto:[email protected]>> wrote:
>>> 
>>> Hello,
>>> 
>>> I took on me to review t.o Google search results.
>>> There is many case of pages that should be crawled and suggested as results.
>>> 
>>> One way is to log them on the Google console and it will be temporary.
>>> 
>>> The right way is to block robot from indexing such pages.
>>> Also this is good for Tiki to learn from such results and improve the robot.txt we are releasing.
>>> 
>>> 
>>> To do so I need one of those:
>>> * A __safe and non time consuming__ method to be able to do it.
>>> * A volunteer to review the links and block them (or not) from being crawled
>>> 
>>> Exemples for this month:
>>> https://doc.tiki.org/tiki-slideshow.php?page=elFinder&pdf=1&landscape=1 <https://doc.tiki.org/tiki-slideshow.php?page=elFinder&pdf=1&landscape=1>
>>> 
>>> Thank you for helping making things better !
>>> 
>>> Bernard 
>>> 
>>> _______________________________________________
>>> TikiWiki-devel mailing list
>>> [email protected] <mailto:[email protected]>
>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel <https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel>
>> _______________________________________________
>> TikiWiki-devel mailing list
>> [email protected] <mailto:[email protected]>
>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel <https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel>
> 
> _______________________________________________
> TikiWiki-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel

_______________________________________________
TikiWiki-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel