following redirects
BOCZ Peter <[email protected]> Thu, 15 Jan 2004 13:13:23 +0100
| Newsgroups | gmane.comp.web.perlfect-search |
|---|---|
| Message-ID | <[email protected]> |
Hi Is it a feature that web crawler follows and indexes redirects to external URLs? My conf.pl allows the startup site to be indexed only, but due to redirections to external sites, they get indexed (while indexing external content correclty stops here). I wish it could be configured by the administrator and search page shows the source URL, not the destination. Peter _______________________________________________ perlfect-search mailing list [email protected] To unsubscribe, set other personal options or view the list archives please visit: http://perlfect.com/mailman/listinfo/perlfect-search