Problems with indexing
"Roger Growe" <[email protected]> Mon, 6 Dec 2004 16:18:50 -0500
| Newsgroups | gmane.comp.web.perlfect-search |
|---|---|
| Message-ID | <001901c4dbd9$2eb8b6a0$0f02a8c0@URSA> |
This is a multi-part message in MIME format.
--===============0439556738==
Content-Type: multipart/alternative;
boundary="----=_NextPart_000_0016_01C4DBAF.45625B10"
This is a multi-part message in MIME format.
------=_NextPart_000_0016_01C4DBAF.45625B10
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Problems with indexing
=20
I 'd really appreciate a hand with this script.
After working though the instructions, all the correspondence in the =
mail list archives and other sources I am still getting this when =
starting the index:
Using DB_File...
Checking for old temp files...
Building string of special characters...
Loading 'no index' regular expressions:
- frontpage2.html
- frontpage.html
[etc.]
=20
Loading stopwords...371 stopwords loaded.
Starting crawler...
Note: I will not visit more than $HTTP_MAX_PAGES=3D150 pages.
Loading http://www.quinacrine.com/robots.txt...
Error: Couldn't get 'http://www.quinacrine.com/robots.txt': response =
code 500
Not using any robots.txt.
Error: Couldn't get 'http://www.quinacrine.com/index.html': response =
code 500
=20
Crawler finished: indexed 0 files, 0 terms (0 different terms).
Ignored 0 files because of conf/no_index.txt
Ignored 0 files because of robots.txt
=20
I thought it might be the structure of the site that was the problem. =
The pages are not in the root but in the 'web' directory, like so:
=20
root
.config
.sessions
cgi-bin
logs
web
=20
In cgi-bin I have these:
=20
searchsite
conf
data
Perlfect
temp
templates
=20
I installed manually by necessity and need to index though http for the =
same reason. All syntax, permissions, and other rules that I can find =
check out. Unix server.
=20
The main sections of config.pl look like this now:
=20
=20
$DOCUMENT_ROOT =3D 'http://www.quinacrine.com/';
=20
# The base url of your site (normally that's the URL which
# corresponds to $DOCUMENT_ROOT).
$BASE_URL =3D 'http://www.quinacrine.com';
=20
# The url in which Perlfect Search is located (usually somewhere in =
cgi-bin/).
$CGIBIN =3D "/cgi-bin/searchsite/";
=20
# The full-path of the directory where Perlfect Search is installed.
$INSTALL_DIR =3D '/nfs/cust/5/80/46/564085/cgi-bin/searchsite/';
=20
# Only files with these extensions should be indexed (case-sensitive).=20
# This is only relevant for file system indexing, when you index files =
via
# http you need to set @HTTP_CONTENT_TYPES instead. [re-index]
@EXT =3D ("html", "htm", "shtml", "txt");
=20
=20
[Password section]
=20
#########################################################################=
##
### http configuration
### You only need this if you want to index your pages via http
=20
# Where you want the indexer to start via http. Leave empty if
# you want to index the files in the filesystem ($DOCUMENT_ROOT).
# ** WARNING **: Do not use for foreign servers! It might use too many
# resources on other people's servers. [re-index]
# example: $HTTP_START_URL =3D 'http://localhost/';
$HTTP_START_URL =3D 'http://www.quinacrine.com/index.html';
=20
Thinking that the file structure could be the issue, I put a copy of =
robots.txt in the root, still the 500 response. I've left =
$HTTP_START_URL =3D blank, used 'http://www.quinacrine.com/' as well as =
other things I could think of to break this jam.
=20
Thanks for your help in advance,
=20
Roger Growe
------=_NextPart_000_0016_01C4DBAF.45625B10
Content-Type: text/html;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=3DContent-Type content=3D"text/html; =
charset=3Diso-8859-1">
<META content=3D"MSHTML 6.00.2800.1400" name=3DGENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
<DIV><FONT face=3DArial size=3D2>
<P class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><FONT face=3D"Times =
New Roman"=20
size=3D3>Problems with indexing</FONT></P>
<P class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><?xml:namespace =
prefix =3D o ns =3D=20
"urn:schemas-microsoft-com:office:office" /><o:p><FONT face=3D"Times New =
Roman"=20
size=3D3> </FONT></o:p></P>
<P class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><FONT face=3D"Times =
New Roman"=20
size=3D3>I 'd really appreciate a hand with this script.</FONT></P>
<P class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><FONT face=3D"Times =
New Roman"=20
size=3D3>After working though the instructions, all the correspondence =
in the mail=20
list archives and other sources I am still getting this when starting =
the=20
index:</FONT></P>
<P class=3DMsoNormal style=3D"MARGIN: 0in 0in =
0pt"> </P></DIV></FONT>
<DIV class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><FONT =
size=3D2><FONT=20
face=3D"Courier New"><o:p> </o:p>Using =
DB_File...</FONT></FONT></DIV>
<DIV class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><FONT =
face=3D"Courier New"=20
size=3D2>Checking for old temp files...</FONT></DIV>
<DIV class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><FONT =
face=3D"Courier New"=20
size=3D2>Building string of special characters...</FONT></DIV>
<DIV class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><FONT =
face=3D"Courier New"=20
size=3D2>Loading 'no index' regular expressions:</FONT></DIV>
<DIV><FONT size=3D2><FONT face=3D"Courier New"><SPAN=20
style=3D"mso-spacerun: yes"> </SPAN>-=20
frontpage2.html</FONT></FONT></DIV>
<DIV><FONT size=3D2><FONT face=3D"Courier New"><SPAN=20
style=3D"mso-spacerun: yes"> </SPAN>-=20
frontpage.html</FONT></FONT></DIV>
<DIV><FONT face=3D"Courier New" size=3D2>[etc.]</FONT></DIV>
<DIV><o:p><FONT face=3D"Courier New" size=3D2> </FONT></o:p></DIV>
<DIV><FONT face=3D"Courier New" size=3D2>Loading stopwords...371 =
stopwords=20
loaded.</FONT></DIV>
<DIV><FONT face=3D"Courier New" size=3D2>Starting =
crawler...</FONT></DIV>
<DIV><FONT face=3D"Courier New" size=3D2>Note: I will not visit more =
than=20
$HTTP_MAX_PAGES=3D150 pages.</FONT></DIV>
<DIV><FONT face=3D"Courier New" size=3D2>Loading=20
http://www.quinacrine.com/robots.txt...</FONT></DIV>
<DIV><FONT face=3D"Courier New" size=3D2>Error: Couldn't get=20
'http://www.quinacrine.com/robots.txt': response code 500</FONT></DIV>
<DIV><FONT face=3D"Courier New" size=3D2>Not using any =
robots.txt.</FONT></DIV>
<DIV><FONT face=3D"Courier New" size=3D2>Error: Couldn't get=20
'http://www.quinacrine.com/index.html': response code 500</FONT></DIV>
<DIV><o:p><FONT face=3D"Courier New" size=3D2> </FONT></o:p></DIV>
<DIV><FONT face=3D"Courier New" size=3D2>Crawler finished: indexed 0 =
files, 0 terms=20
(0 different terms).</FONT></DIV>
<DIV><FONT face=3D"Courier New" size=3D2>Ignored 0 files because of=20
conf/no_index.txt</FONT></DIV>
<DIV><FONT face=3D"Courier New" size=3D2>Ignored 0 files because of=20
robots.txt</FONT></DIV>
<DIV class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><o:p><FONT =
face=3D"Courier New"=20
size=3D2> </FONT></o:p></DIV><FONT face=3DArial size=3D2>
<P class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><FONT face=3D"Times =
New Roman"=20
size=3D3>I thought it might be the structure of the site that was the=20
problem.<SPAN style=3D"mso-spacerun: yes"> </SPAN>The pages are =
not in the=20
root but in the 'web' directory, like so:</FONT></P>
<P class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><o:p><FONT =
face=3D"Times New Roman"=20
size=3D3> </FONT></o:p></P>
<DIV><FONT face=3D"Times New Roman" size=3D3>root</FONT></DIV>
<DIV><FONT face=3D"Times New Roman"><FONT size=3D3><SPAN=20
style=3D"mso-tab-count: 1"> =20
</SPAN>.config</FONT></FONT></DIV>
<DIV><FONT face=3D"Times New Roman"><FONT size=3D3><SPAN=20
style=3D"mso-tab-count: 1"> =20
</SPAN>.sessions</FONT></FONT></DIV>
<DIV><FONT face=3D"Times New Roman"><FONT size=3D3><SPAN=20
style=3D"mso-tab-count: 1"> =20
</SPAN>cgi-bin</FONT></FONT></DIV>
<DIV><FONT face=3D"Times New Roman"><FONT size=3D3><SPAN=20
style=3D"mso-tab-count: 1"> =20
</SPAN>logs</FONT></FONT></DIV>
<DIV><FONT face=3D"Times New Roman"><FONT size=3D3><SPAN=20
style=3D"mso-tab-count: 1"> =20
</SPAN>web</FONT></FONT></DIV>
<DIV><o:p><FONT face=3D"Times New Roman" =
size=3D3> </FONT></o:p></DIV>
<DIV><FONT face=3D"Times New Roman" size=3D3>In cgi-bin I have =
these:</FONT></DIV>
<DIV><o:p><FONT face=3D"Times New Roman" =
size=3D3> </FONT></o:p></DIV>
<DIV><FONT face=3D"Times New Roman" size=3D3>searchsite</FONT></DIV>
<DIV><FONT face=3D"Times New Roman"><FONT size=3D3><SPAN=20
style=3D"mso-tab-count: 1"> =20
</SPAN>conf</FONT></FONT></DIV>
<DIV><FONT face=3D"Times New Roman"><FONT size=3D3><SPAN=20
style=3D"mso-tab-count: 1"> =20
</SPAN>data</FONT></FONT></DIV>
<DIV><FONT face=3D"Times New Roman"><FONT size=3D3><SPAN=20
style=3D"mso-tab-count: 1"> =20
</SPAN>Perlfect</FONT></FONT></DIV>
<DIV><FONT face=3D"Times New Roman"><FONT size=3D3><SPAN=20
style=3D"mso-tab-count: 1"> =20
t</SPAN>emp</FONT></FONT></DIV>
<DIV><FONT face=3D"Times New Roman"><FONT size=3D3><SPAN=20
style=3D"mso-tab-count: 1"> =20
t</SPAN>emplates</FONT></FONT></DIV>
<DIV>
<P class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><o:p><FONT =
face=3D"Times New Roman"=20
size=3D3> </FONT></o:p></P>
<P class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><FONT face=3D"Times =
New Roman"=20
size=3D3>I installed manually by necessity and need to index though http =
for the=20
same reason.<SPAN style=3D"mso-spacerun: yes"> </SPAN>All syntax,=20
permissions, and other rules that I can find check out. Unix =
server.</FONT></P>
<P class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><o:p><FONT =
face=3D"Times New Roman"=20
size=3D3> </FONT></o:p></P>
<P class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><FONT face=3D"Times =
New Roman"=20
size=3D3>The main sections of config.pl look like this now:</FONT></P>
<P class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><o:p><FONT =
face=3D"Times New Roman"=20
size=3D3> </FONT></o:p></P>
<P class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><o:p><FONT =
face=3D"Times New Roman"=20
size=3D3> </FONT></o:p></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><SPAN lang=3DFR=20
style=3D"mso-ansi-language: FR"><FONT face=3D"Courier =
New">$DOCUMENT_ROOT =3D=20
'http://www.quinacrine.com/';<o:p></o:p></FONT></SPAN></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><SPAN lang=3DFR=20
style=3D"mso-ansi-language: FR"><o:p><FONT=20
face=3D"Courier New"> </FONT></o:p></SPAN></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><FONT =
face=3D"Courier New"># The=20
base url of your site (normally that's the URL =
which<o:p></o:p></FONT></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><FONT =
face=3D"Courier New">#=20
corresponds to $DOCUMENT_ROOT).<o:p></o:p></FONT></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><FONT=20
face=3D"Courier New">$BASE_URL =3D=20
'http://www.quinacrine.com';<o:p></o:p></FONT></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><o:p><FONT=20
face=3D"Courier New"> </FONT></o:p></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><FONT =
face=3D"Courier New"># The=20
url in which Perlfect Search is located (usually somewhere in=20
cgi-bin/).<o:p></o:p></FONT></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><FONT=20
face=3D"Courier New">$CGIBIN =3D =
"/cgi-bin/searchsite/";<o:p></o:p></FONT></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><o:p><FONT=20
face=3D"Courier New"> </FONT></o:p></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><FONT =
face=3D"Courier New"># The=20
full-path of the directory where Perlfect Search is=20
installed.<o:p></o:p></FONT></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><FONT=20
face=3D"Courier New">$INSTALL_DIR =3D=20
'/nfs/cust/5/80/46/564085/cgi-bin/searchsite/';<o:p></o:p></FONT></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><o:p><FONT=20
face=3D"Courier New"> </FONT></o:p></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><FONT =
face=3D"Courier New">#=20
Only files with these extensions should be indexed (case-sensitive).=20
<o:p></o:p></FONT></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><FONT =
face=3D"Courier New">#=20
This is only relevant for file system indexing, when you index files=20
via<o:p></o:p></FONT></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><FONT =
face=3D"Courier New">#=20
http you need to set @HTTP_CONTENT_TYPES instead.=20
[re-index]<o:p></o:p></FONT></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><FONT =
face=3D"Courier New">@EXT=20
=3D ("html", "htm", "shtml", "txt");<o:p></o:p></FONT></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><o:p><FONT=20
face=3D"Courier New"> </FONT></o:p></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><o:p><FONT=20
face=3D"Courier New"> </FONT></o:p></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><FONT=20
face=3D"Courier New">[Password section]</FONT></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><o:p><FONT=20
face=3D"Courier New"> </FONT></o:p></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><FONT=20
face=3D"Courier =
New">####################################################################=
#######<o:p></o:p></FONT></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><FONT =
face=3D"Courier New">###=20
http configuration<o:p></o:p></FONT></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><FONT =
face=3D"Courier New">###=20
You only need this if you want to index your pages via=20
http<o:p></o:p></FONT></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><o:p><FONT=20
face=3D"Courier New"> </FONT></o:p></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><FONT =
face=3D"Courier New">#=20
Where you want the indexer to start via http. Leave empty=20
if<o:p></o:p></FONT></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><FONT =
face=3D"Courier New"># you=20
want to index the files in the filesystem=20
($DOCUMENT_ROOT).<o:p></o:p></FONT></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><FONT =
face=3D"Courier New"># **=20
WARNING **: Do not use for foreign servers! It might use too=20
many<o:p></o:p></FONT></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><FONT =
face=3D"Courier New">#=20
resources on other people's servers. [re-index]<o:p></o:p></FONT></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><FONT =
face=3D"Courier New">#=20
example: $HTTP_START_URL =3D 'http://localhost/';<o:p></o:p></FONT></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><FONT=20
face=3D"Courier New">$HTTP_START_URL =3D=20
'http://www.quinacrine.com/index.html';<o:p></o:p></FONT></P>
<P class=3DMsoPlainText style=3D"MARGIN: 0in 0in 0pt"><o:p><FONT=20
face=3D"Courier New"> </FONT></o:p></P>
<P class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><FONT face=3D"Times =
New Roman"=20
size=3D3>Thinking that the file structure could be the issue, I put a =
copy of=20
robots.txt in the root, still the 500 response.<SPAN=20
style=3D"mso-spacerun: yes"> </SPAN>I've left $HTTP_START_URL =
=3D<SPAN=20
style=3D"mso-spacerun: yes"> </SPAN>blank, used =
'http://www.quinacrine.com/'=20
as well as other things I could think of to break this jam.</FONT></P>
<P class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><o:p><FONT =
face=3D"Times New Roman"=20
size=3D3> </FONT></o:p></P>
<P class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><FONT face=3D"Times =
New Roman"=20
size=3D3>Thanks for your help in advance,</FONT></P>
<P class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><o:p><FONT =
face=3D"Times New Roman"=20
size=3D3> </FONT></o:p></P>
<P class=3DMsoNormal style=3D"MARGIN: 0in 0in 0pt"><FONT face=3D"Times =
New Roman"=20
size=3D3>Roger Growe</FONT></P></FONT></DIV></BODY></HTML>
------=_NextPart_000_0016_01C4DBAF.45625B10--
--===============0439556738==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
perlfect-search mailing list
[email protected]
To unsubscribe, set other personal options or view the list archives please visit:
http://perlfect.com/mailman/listinfo/perlfect-search
--===============0439556738==--