Re: Htdig and MHT files
<[email protected]> Fri, 8 Feb 2008 11:58:12 -0000
| Newsgroups | gmane.comp.web.htdig.general |
|---|---|
| Message-ID | <498C92D39B24504CA6C753B477F19E6C04E1276E@E03MVZ4-UKDY.domain1.systemhost.net> |
This is a multi-part message in MIME format. --===============1077651792== Content-class: urn:content-classes:message Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C86A49.E18FB89C" This is a multi-part message in MIME format. ------_=_NextPart_001_01C86A49.E18FB89C Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Ainhoa, Can I ask you to check whether _new_ PDF's are getting indexed correctly? =20 I notice that the syntax used in the very first, commented, line of the external_parsers section looks different to the rest: application/pdf->text/html /usr/local/bin/conv_doc.pl Note the 'arrow' and mime-type bit after application/pdf. All of the external_parsers declarations in my config have this same bit, which makes me suspect that none of your declarations will be working just now, though if you have not rebuilt your databases from scratch this may not be obvious. You probably want to be using at least -vv (two letter v's) to get verbose output from the dig process - this should tell you what is happening during the indexing. My other thought is to check whether the mht files are being served to you with that mime-type - this won't work correctly if not, and you may need more than one external_parsers declaration to cover all possibilities. Regards, Mike ________________________________ From: Ainhoa L [mailto:[email protected]]=20 Sent: Wednesday, February 06, 2008 5:29 PM To: Brockington,MJ,Michael,JPGA4X R Cc: [email protected] Subject: Re: [htdig] Htdig and MHT files =09 =09 Hi Mike, =20 You are talking about the version with the mht parser, right? I write here an extract of where I mention mht things and I attach the whole file and the parser (originally the parser would create files for the files appearing in the mht. I modified it so it will only output the code in the htm file). Maybe this parser I modified is sending some other garbage that can't be read by the indexer? =09 =20 bad_extensions: .wav .gz .z .sit .au .zip .tar .hqx .exe .com .gif \ .jpg .jpeg .aiff .class .map .ram .tgz .bin .rpm .mpg .mov .avi .css=20 =20 valid_extensions: .html .htm .shtml .php .uhtml .phtml .txt .pdf .mht=20 =20 external_parsers: application/postscript /usr/local/apache/htdocs/htdig-3.1.6/contrib/parsepdf.pl\ application/pdf /usr/local/apache/htdocs/htdig-3.1.6/contrib/parsepdf.pl \=20 application/mht /opt/vin/mht2html2.pl =20 Thanks a lot for your help! Regards, =20 Ainhoa =20 On Feb 5, 2008 9:58 PM, <[email protected]> wrote: =09 Can you show us at least an extract of your config file - as you describe it this should work. =09 Regards, Mike =09 -----Original Message----- From: [email protected] on behalf of Ainhoa L Sent: Tue 2/5/2008 4:09 PM To: [email protected] Subject: [htdig] Htdig and MHT files =09 Hi! Maybe this is a very stupid question but, is it possible to index mht files with htdig? I have tried with the mht in the valid_extensions list, etc. Obviously htdig doesn't take them as html and refuses to index them. I looked for a parser and found a mht2html parser, modified it so it just sends through output the html. I added it to the parsers in the htdig config file. This didn't work, although the parser returns valid html... I would like to know if there is any way to index mht files with htdig? Thanks a lot for your help. =09 =09 ------_=_NextPart_001_01C86A49.E18FB89C Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <HTML><HEAD> <META http-equiv=3DContent-Type content=3D"text/html; = charset=3Dus-ascii"> <META content=3D"MSHTML 6.00.2900.2802" name=3DGENERATOR></HEAD> <BODY> <DIV dir=3Dltr align=3Dleft><SPAN class=3D651195111-08022008><FONT = face=3DArial=20 color=3D#0000ff size=3D2>Ainhoa,</FONT></SPAN></DIV> <DIV dir=3Dltr align=3Dleft><SPAN class=3D651195111-08022008><FONT = face=3DArial=20 color=3D#0000ff size=3D2>Can I ask you to check whether _new_ PDF's are = getting=20 indexed correctly?</FONT></SPAN></DIV> <DIV dir=3Dltr align=3Dleft><SPAN class=3D651195111-08022008><FONT = face=3DArial=20 color=3D#0000ff size=3D2></FONT></SPAN> </DIV> <DIV dir=3Dltr align=3Dleft><SPAN class=3D651195111-08022008><FONT = face=3DArial=20 color=3D#0000ff size=3D2>I notice that the syntax used in the very = first, commented,=20 line of the external_parsers section looks different to the=20 rest:</FONT></SPAN></DIV> <DIV dir=3Dltr align=3Dleft><SPAN class=3D651195111-08022008> <P><FONT size=3D2>application/pdf->text/html=20 /usr/local/bin/conv_doc.pl</FONT></P> <P><SPAN class=3D651195111-08022008><FONT face=3DArial color=3D#0000ff = size=3D2>Note the=20 'arrow' and mime-type bit after application/pdf. All of the =20 external_parsers declarations in my config have this same bit, = which makes=20 me suspect that none of your declarations will be working just now, = though if=20 you have not rebuilt your databases from scratch this may not be = obvious. You=20 probably want to be using at least -vv (two letter v's) to = get=20 verbose output from the dig process - this should tell you what is = happening=20 during the indexing. My other thought is to check whether the mht files = are=20 being served to you with that mime-type - this won't work correctly if = not, and=20 you may need more than one external_parsers declaration to = cover all=20 possibilities.</FONT></SPAN></P> <P><SPAN class=3D651195111-08022008><FONT face=3DArial color=3D#0000ff=20 size=3D2>Regards,<BR>Mike</FONT></SPAN></P></SPAN></DIV><BR> <BLOCKQUOTE=20 style=3D"PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px = solid; MARGIN-RIGHT: 0px"> <DIV class=3DOutlookMessageHeader lang=3Den-us dir=3Dltr align=3Dleft> <HR tabIndex=3D-1> <FONT face=3DTahoma size=3D2><B>From:</B> Ainhoa L = [mailto:[email protected]]=20 <BR><B>Sent:</B> Wednesday, February 06, 2008 5:29 PM<BR><B>To:</B>=20 Brockington,MJ,Michael,JPGA4X R<BR><B>Cc:</B>=20 [email protected]<BR><B>Subject:</B> Re: [htdig] = Htdig and=20 MHT files<BR></FONT><BR></DIV> <DIV></DIV> <DIV>Hi Mike,</DIV> <DIV> </DIV> <DIV>You are talking about the version with the mht parser, = right?</DIV> <DIV>I write here an extract of where I mention mht things and I = attach the=20 whole file and the parser (originally the parser would create files = for the=20 files appearing in the mht. I modified it so it will only output the = code in=20 the htm file). Maybe this parser I modified is sending some other = garbage that=20 can't be read by the indexer?</DIV><FONT size=3D2> <DIV> </DIV> <DIV>bad_extensions: .wav .gz .z .sit .au .zip .tar .hqx .exe .com = .gif=20 \</DIV> <DIV>.jpg .jpeg .aiff .class .map .ram .tgz .bin .rpm .mpg .mov .avi = .css=20 </DIV> <DIV> </DIV> <DIV>valid_extensions: .html .htm .shtml .php .uhtml .phtml .txt .pdf = .mht=20 </DIV> <DIV> </DIV> <DIV></DIV></FONT><FONT size=3D2>external_parsers: = application/postscript=20 /usr/local/apache/htdocs/htdig-3.1.6/contrib/parsepdf.pl\ = application/pdf=20 /usr/local/apache/htdocs/htdig-3.1.6/contrib/parsepdf.pl \</FONT>=20 <DIV><FONT size=3D2>application/mht /opt/vin/mht2html2.pl</FONT></DIV> <DIV> </DIV> <DIV>Thanks a lot for your help!</DIV> <DIV>Regards,</DIV> <DIV> </DIV> <DIV>Ainhoa</DIV> <DIV><BR><BR> </DIV> <DIV class=3Dgmail_quote>On Feb 5, 2008 9:58 PM, <<A=20 = href=3D"mailto:[email protected]">[email protected]</A>= >=20 wrote:<BR> <BLOCKQUOTE class=3Dgmail_quote=20 style=3D"PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: = #ccc 1px solid">Can=20 you show us at least an extract of your config file - as you = describe it=20 this should work.<BR><BR>Regards,<BR>Mike<BR> <DIV> <DIV></DIV> <DIV class=3DWj3C7c><BR><BR>-----Original Message-----<BR>From: <A=20 = href=3D"mailto:[email protected]">htdig-general= [email protected]</A>=20 on behalf of Ainhoa L<BR>Sent: Tue 2/5/2008 4:09 PM<BR>To: <A=20 = href=3D"mailto:[email protected]">[email protected]= ourceforge.net</A><BR>Subject:=20 [htdig] Htdig and MHT files<BR><BR>Hi! Maybe this is a very stupid = question=20 but, is it possible to index mht<BR>files with htdig?<BR>I have = tried with=20 the mht in the valid_extensions list, etc. Obviously = htdig<BR>doesn't take=20 them as html and refuses to index them. I looked for a parser<BR>and = found a=20 mht2html parser, modified it so it just sends through output = the<BR>html. I=20 added it to the parsers in the htdig config file. This didn't=20 work,<BR>although the parser returns valid html...<BR>I would like = to know=20 if there is any way to index mht files with htdig?<BR>Thanks a lot = for your=20 = help.<BR><BR></DIV></DIV></BLOCKQUOTE></DIV><BR></BLOCKQUOTE></BODY></HTM= L> ------_=_NextPart_001_01C86A49.E18FB89C-- --===============1077651792== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ --===============1077651792== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ ht://Dig general mailing list: <[email protected]> ht://Dig FAQ: http://htdig.sourceforge.net/FAQ.html List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-general --===============1077651792==--