Re: [Tiki-devel] Reviewing and limiting crawling to pages we don't need people to see as search results
Bsfez Tiki via TikiWiki-devel <[email protected]>
| Newsgroups | gmane.comp.cms.tiki.devel |
|---|---|
| Message-ID | <[email protected]> |
Hi Brendan, Happy we have you with us. ;-) I don’t have your knowledge about this and even after reading I feel confused with what I can do in a Tiki(22) website. https://developers.google.com/search/docs/advanced/crawling/consolidate-duplicate-urls <https://developers.google.com/search/docs/advanced/crawling/consolidate-duplicate-urls> https://www.semrush.com/blog/canonical-url-guide <https://www.semrush.com/blog/canonical-url-guide> However I have to say that working little by little on my own websites (canonical, sitemap, using page description for metatag) I kind of feel like things are handled properly by Tiki. I only add to disallow /404 and /item* (I don’t want people to land on items, but on pages ;-) ) Bernard > On 20 Aug 2021, at 19:46 , Dr. Sassafras <[email protected] <mailto:[email protected]>> wrote: > > Blocking these will not do what you want. Canonicalization is what you want. https://developers.google.com/search/docs/advanced/guidelines/duplicate-content <https://developers.google.com/search/docs/advanced/guidelines/duplicate-content> > > Years ago I started blocking content of one of my sites to get better search results. Much of the existing blocking in robots.txt is from me. I knew it wasn't the best way then but it kinda works and was easy for me to handle changing a test file. It has major drawbacks. > > Instead, I would suggest canonicalization for any duplicate content and in-page tag of noindex. It would be better if we could change that from an HTML tag to anHTTP header. Then we could apply that to any content in tiki, even files or pdfs, etc. After content has in-page control of these things, the corresponding robots.txt entry can be deleted. > > I agree that we should be better at this. It perhaps should also be a unit test? To make sure every page is classified… > > Brendan > > >> On Aug 20, 2021, at 4:50 AM, Bsfez Tiki via TikiWiki-devel <[email protected] <mailto:[email protected]>> wrote: >> >> Hello, >> >> I took on me to review t.o Google search results. >> There is many case of pages that should be crawled and suggested as results. >> >> One way is to log them on the Google console and it will be temporary. >> >> The right way is to block robot from indexing such pages. >> Also this is good for Tiki to learn from such results and improve the robot.txt we are releasing. >> >> >> To do so I need one of those: >> * A __safe and non time consuming__ method to be able to do it. >> * A volunteer to review the links and block them (or not) from being crawled >> >> Exemples for this month: >> https://doc.tiki.org/tiki-slideshow.php?page=elFinder&pdf=1&landscape=1 <https://doc.tiki.org/tiki-slideshow.php?page=elFinder&pdf=1&landscape=1> >> >> Thank you for helping making things better ! >> >> Bernard >> >> _______________________________________________ >> TikiWiki-devel mailing list >> [email protected] <mailto:[email protected]> >> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel <https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel> > _______________________________________________ > TikiWiki-devel mailing list > [email protected] <mailto:[email protected]> > https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel _______________________________________________ TikiWiki-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel