bagder: curl-www robots.txt,1.3,1.4

Update of /cvsroot/curl/curl-www
In directory giant.haxx.se:/var/tmp/cvs-serv19811

Modified Files:
	robots.txt 
Log Message:
robots should not index the CVS cgi


Index: robots.txt
===================================================================
RCS file: /cvsroot/curl/curl-www/robots.txt,v
retrieving revision 1.3
retrieving revision 1.4
diff -u -d -r1.3 -r1.4
--- robots.txt	14 Jan 2004 08:48:28 -0000	1.3
+++ robots.txt	26 Mar 2010 07:01:48 -0000	1.4
@@ -1,7 +1,4 @@
 # robots.txt for curl.haxx.se and mirrors
 
 User-agent: *
-Disallow: /archive_pre_oct_99/ # this is just sooo old
-Disallow: /archive/
-Disallow: /stats/
-Disallow: /lxr/              # shows too much "raw" stuff!
+Disallow: /cvs.cgi/           # shows too much "raw" stuff!