Re: [Tiki-devel] Reviewing and limiting crawling to pages we don't need people to see as search results
"Dr. Sassafras" <[email protected]>
| Newsgroups | gmane.comp.cms.tiki.devel |
|---|---|
| Message-ID | <[email protected]> |
Blocking these will not do what you want. Canonicalization is what you want. https://developers.google.com/search/docs/advanced/guidelines/duplicate-content Years ago I started blocking content of one of my sites to get better search results. Much of the existing blocking in robots.txt is from me. I knew it wasn't the best way then but it kinda works and was easy for me to handle changing a test file. It has major drawbacks. Instead, I would suggest canonicalization for any duplicate content and in-page tag of noindex. It would be better if we could change that from an HTML tag to anHTTP header. Then we could apply that to any content in tiki, even files or pdfs, etc. After content has in-page control of these things, the corresponding robots.txt entry can be deleted. I agree that we should be better at this. It perhaps should also be a unit test? To make sure every page is classified… Brendan > On Aug 20, 2021, at 4:50 AM, Bsfez Tiki via TikiWiki-devel <[email protected]> wrote: > > Hello, > > I took on me to review t.o Google search results. > There is many case of pages that should be crawled and suggested as results. > > One way is to log them on the Google console and it will be temporary. > > The right way is to block robot from indexing such pages. > Also this is good for Tiki to learn from such results and improve the robot.txt we are releasing. > > > To do so I need one of those: > * A __safe and non time consuming__ method to be able to do it. > * A volunteer to review the links and block them (or not) from being crawled > > Exemples for this month: > https://doc.tiki.org/tiki-slideshow.php?page=elFinder&pdf=1&landscape=1 > > Thank you for helping making things better ! > > Bernard > > _______________________________________________ > TikiWiki-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel _______________________________________________ TikiWiki-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel