Solr web crawler with recursive option

--_000_MWHPR2201MB15821892A409544E58AD3A45C62F0MWHPR2201MB1582_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hello Team,


                I am working on solr for the first time and got the setup d=
one. Now I have created a core using command line and want to perform webcr=
awl of a third party site.
If I try it with individual links, I am able to do the crawl and index it t=
o the core.This was done using >
java -Ddata=3Dweb -Durl=3Dhttps://solr:8985/solr/solrhelp/update -jar post.=
jar http://www.example.com

Now what I intend to do is to give a url and using the recursive option (-D=
recursive) and let it crawl the entire site.
Note that I am pointing to a website that has around 125 pages and I am usi=
ng the below command >
java -Ddata=3Dweb -Durl=3Dhttps://solr:8985/solr/solrhelp/update -Drecursiv=
e=3Dyes -jar post.jar http://www.example.com  and
java -Ddata=3Dweb -Durl=3Dhttps://solr:8985/solr/solrhelp/update -Drecursiv=
e=3D2 -jar post.jar http://www.example.com

and I am getting the below error message.
Error:


POSTed web resource http://www.example.com (depth: 0)
[Fatal Error] :1:1: Content is not allowed in prolog.
Exception in thread "main" java.lang.RuntimeException: org.xml.sax.SAXParse=
Exception; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog=
.
        at org.apache.solr.util.SimplePostTool$PageFetcher.getLinksFromWebP=
age(SimplePostTool.java:1252)
        at org.apache.solr.util.SimplePostTool.webCrawl(SimplePostTool.java=
:616)
        at org.apache.solr.util.SimplePostTool.postWebPages(SimplePostTool.=
java:563)
        at org.apache.solr.util.SimplePostTool.doWebMode(SimplePostTool.jav=
a:365)
        at org.apache.solr.util.SimplePostTool.execute(SimplePostTool.java:=
187)
        at org.apache.solr.util.SimplePostTool.main(SimplePostTool.java:172=
)
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; C=
ontent is not allowed in prolog.
        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unkno=
wn Source)
        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.pars=
e(Unknown Source)
        at javax.xml.parsers.DocumentBuilder.parse(Unknown Source)
        at org.apache.solr.util.SimplePostTool.makeDom(SimplePostTool.java:=
1061)
        at org.apache.solr.util.SimplePostTool$PageFetcher.getLinksFromWebP=
age(SimplePostTool.java:1232)
        ... 5 more



I would be very grateful if anyone could get me to solve this issue I have =
been trying to fix for a couple of days.


Regards,
ShivprasadS


Confidentiality Notice: This e-mail message, including any attachments, is =
for the sole use of the intended recipient(s) and may contain confidential =
and privileged information. Any unauthorized review, use, disclosure or dis=
tribution is prohibited. If you are not the intended recipient, please cont=
act the sender by reply e-mail, delete and then destroy all copies of the o=
riginal message.

--_000_MWHPR2201MB15821892A409544E58AD3A45C62F0MWHPR2201MB1582_--