Re: Htdig and MHT files
<[email protected]> Mon, 11 Feb 2008 16:13:03 -0000
| Newsgroups | gmane.comp.web.htdig.general |
|---|---|
| Message-ID | <498C92D39B24504CA6C753B477F19E6C04E6F716@E03MVZ4-UKDY.domain1.systemhost.net> |
This is a multi-part message in MIME format. --===============1911991377== Content-class: urn:content-classes:message Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C86CC8.FAE20028" This is a multi-part message in MIME format. ------_=_NextPart_001_01C86CC8.FAE20028 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Ainhoa, I see what you mean - that certainly doesn't look quite right! =20 Before we go any further, can I ask if you have tried indexing this file as 'plain' HTML? I know that it doesn't look quite right, but it would appear to me that the content should be okay for htdig's own html parser, if things are set up correctly. Since we know that the config wasn't correct (at first) for the mime-type etc it would be worth checking that over - no point doing an MHT -> HTML translation if it is HTML to begin with! =20 I am afraid that I don't have a working example, but http://www.htdig.org/attrs.html#external_parsers describes how to target a file at the internal parser =20 Regards, Mike =20 PS None of my messages have been coming back to me via the list - have you been getting one copy or two? =20 ________________________________ From: [email protected] [mailto:[email protected]] On Behalf Of Ainhoa L Sent: Monday, February 11, 2008 3:19 PM To: Brockington,MJ,Michael,JPGA4X R Cc: [email protected] Subject: Re: [htdig] Htdig and MHT files =09 =09 Yeah you are right, I think it doesn't like the output at all. Instead of the words it is taking as words:=20 word: Read@0 word: 8192@33 word: from@67 word: document@100 =09 :S So I suppose htdig just doesn't really like the output of the parser. I'm attaching the output of parser (executed manually) and the output of dig just in case you have any more ideas :) =09 Thanks a lot! =09 Ainhoa =09 =09 =09 On Feb 11, 2008 12:16 PM, <[email protected]> wrote: =09 Ainhoa, My first instinct would now be to check the parser output - try adding another v to your config, (and possibly restricting your indexing to just this one file) and check the log output - it may be that htdig does not like the output from your PERL script. www.htdig.org explains what the output means. I seem to recall you saying that you had already tested that it ran on its own, but possibly there is something not right there, or a typo in the config that neither of us can see. =20 Regards, Mike =20 =20 ________________________________ =09 From: Ainhoa L [mailto:[email protected]]=20 =09 Sent: Monday, February 11, 2008 9:33 AM=20 To: Brockington,MJ,Michael,JPGA4X R Cc: [email protected] Subject: Re: [htdig] Htdig and MHT files =09 Hi Mike, Yes you were right, I was missing that part and I didn't even noticed! I changed the config file and wrote this: application/pdf->text/html /usr/local/apache/htdocs/htdig-3.1.6/contrib/parsepdf.pl \ application/vnd.wap.xhtml+xml->text/html /opt/vin/mht2html.pl vnd.wap.xhtml+xml was the MIME type for my mht documents.=20 So I run dig and everything seems to go fine, having at the end: 0/http://172.26.0.169/testdig/ =09 1/http://172.26.0.169/testdig/About_comments_eex3.mht 2/http://172.26.0.169/testdig/aster.pdf 3/http://172.26.0.169/testdig/beepmacro.mht 4/http://172.26.0.169/testdig/index.txt 5/http://172.26.0.169/testdig/test.html =20 (I am doing this in a test folder) =20 But when I go to the search page, it won't find words inside the mht files. It works for the pdf, txt and html ones, but can't find the words that are in the mht ones. =20 I suppose I am missing something here... do I need to setup any other settings for the search engine? =20 Thanks a lot for all your help, =20 Ainhoa ------_=_NextPart_001_01C86CC8.FAE20028 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <HTML><HEAD> <META http-equiv=3DContent-Type content=3D"text/html; = charset=3Dus-ascii"> <META content=3D"MSHTML 6.00.2900.2802" name=3DGENERATOR></HEAD> <BODY> <DIV dir=3Dltr align=3Dleft><SPAN class=3D249043715-11022008><FONT = face=3DArial=20 color=3D#0000ff size=3D2>Ainhoa,</FONT></SPAN></DIV> <DIV dir=3Dltr align=3Dleft><SPAN class=3D249043715-11022008><FONT = face=3DArial=20 color=3D#0000ff size=3D2>I see what you mean - that certainly doesn't = look quite=20 right!</FONT></SPAN></DIV> <DIV dir=3Dltr align=3Dleft><SPAN class=3D249043715-11022008><FONT = face=3DArial=20 color=3D#0000ff size=3D2></FONT></SPAN> </DIV> <DIV dir=3Dltr align=3Dleft><SPAN class=3D249043715-11022008><FONT = face=3DArial=20 color=3D#0000ff size=3D2>Before we go any further, can I ask if you have = tried=20 indexing this file as 'plain' HTML? I know that it doesn't look quite = right, but=20 it would appear to me that the content should be okay for htdig's own = html=20 parser, if things are set up correctly. Since we know that the config = wasn't=20 correct (at first) for the mime-type etc it would be worth checking that = over -=20 no point doing an MHT -> HTML translation if it is HTML to begin=20 with!</FONT></SPAN></DIV> <DIV dir=3Dltr align=3Dleft><SPAN class=3D249043715-11022008><FONT = face=3DArial=20 color=3D#0000ff size=3D2></FONT></SPAN> </DIV> <DIV dir=3Dltr align=3Dleft><SPAN class=3D249043715-11022008><FONT = face=3DArial=20 color=3D#0000ff size=3D2>I am afraid that I don't have a working = example, but <A=20 href=3D"http://www.htdig.org/attrs.html#external_parsers">http://www.htdi= g.org/attrs.html#external_parsers</A> =20 describes how to target a file at the internal = parser</FONT></SPAN></DIV> <DIV dir=3Dltr align=3Dleft><SPAN class=3D249043715-11022008><FONT = face=3DArial=20 color=3D#0000ff size=3D2></FONT></SPAN> </DIV> <DIV dir=3Dltr align=3Dleft><SPAN class=3D249043715-11022008><FONT = face=3DArial=20 color=3D#0000ff size=3D2>Regards,</FONT></SPAN></DIV> <DIV dir=3Dltr align=3Dleft><SPAN class=3D249043715-11022008><FONT = face=3DArial=20 color=3D#0000ff size=3D2>Mike</FONT></SPAN></DIV> <DIV dir=3Dltr align=3Dleft><SPAN class=3D249043715-11022008><FONT = face=3DArial=20 color=3D#0000ff size=3D2></FONT></SPAN> </DIV> <DIV dir=3Dltr align=3Dleft><SPAN class=3D249043715-11022008><FONT = face=3DArial=20 color=3D#0000ff size=3D2>PS None of my messages have been coming back to = me via the=20 list - have you been getting one copy or two?</FONT></SPAN></DIV> <DIV dir=3Dltr align=3Dleft><SPAN class=3D249043715-11022008><FONT = face=3DArial=20 color=3D#0000ff size=3D2></FONT></SPAN> </DIV><BR> <BLOCKQUOTE=20 style=3D"PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px = solid; MARGIN-RIGHT: 0px"> <DIV class=3DOutlookMessageHeader lang=3Den-us dir=3Dltr align=3Dleft> <HR tabIndex=3D-1> <FONT face=3DTahoma size=3D2><B>From:</B>=20 [email protected]=20 [mailto:[email protected]] <B>On Behalf Of=20 </B>Ainhoa L<BR><B>Sent:</B> Monday, February 11, 2008 3:19 = PM<BR><B>To:</B>=20 Brockington,MJ,Michael,JPGA4X R<BR><B>Cc:</B>=20 [email protected]<BR><B>Subject:</B> Re: [htdig] = Htdig and=20 MHT files<BR></FONT><BR></DIV> <DIV></DIV>Yeah you are right, I think it doesn't like the output at = all.=20 Instead of the words it is taking as words: <BR>word: Read@0<BR>word:=20 8192@33<BR>word: from@67<BR>word: document@100<BR><BR>:S<BR>So I = suppose htdig=20 just doesn't really like the output of the parser. I'm attaching the = output of=20 parser (executed manually) and the output of dig just in case you have = any=20 more ideas :)<BR><BR>Thanks a lot!<BR><BR>Ainhoa<BR><BR><BR> <DIV class=3Dgmail_quote>On Feb 11, 2008 12:16 PM, <<A=20 = href=3D"mailto:[email protected]">[email protected]</A>= >=20 wrote:<BR> <BLOCKQUOTE class=3Dgmail_quote=20 style=3D"PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: = rgb(204,204,204) 1px solid"> <DIV> <DIV dir=3Dltr align=3Dleft><SPAN><FONT face=3DArial color=3D#0000ff = size=3D2>Ainhoa,</FONT></SPAN></DIV> <DIV dir=3Dltr align=3Dleft><SPAN><FONT face=3DArial color=3D#0000ff = size=3D2>My first=20 instinct would now be to check the parser output - try adding = another =20 v to your config, (and possibly restricting your indexing to = just this=20 one file) and check the log output - it may be that htdig does not = like the=20 output from your PERL script. <A href=3D"http://www.htdig.org"=20 target=3D_blank>www.htdig.org</A> explains what the output = means. I seem=20 to recall you saying that you had already tested that it ran on its = own, but=20 possibly there is something not right there, or a typo in the config = that=20 neither of us can see.</FONT></SPAN></DIV> <DIV dir=3Dltr align=3Dleft><SPAN><FONT face=3DArial color=3D#0000ff = size=3D2></FONT></SPAN> </DIV> <DIV dir=3Dltr align=3Dleft><SPAN><FONT face=3DArial color=3D#0000ff = size=3D2>Regards,</FONT></SPAN></DIV> <DIV dir=3Dltr align=3Dleft><SPAN><FONT face=3DArial color=3D#0000ff = size=3D2>Mike</FONT></SPAN></DIV> <DIV dir=3Dltr align=3Dleft><SPAN><FONT face=3DArial color=3D#0000ff = size=3D2></FONT></SPAN> </DIV> <DIV dir=3Dltr align=3Dleft><SPAN><FONT face=3DArial color=3D#0000ff = size=3D2></FONT></SPAN> </DIV> <DIV dir=3Dltr align=3Dleft> <HR> </DIV> <DIV dir=3Dltr align=3Dleft><FONT face=3DTahoma size=3D2> <DIV class=3DIh2E3d><B>From:</B> Ainhoa L [mailto:<A=20 href=3D"mailto:[email protected]" = target=3D_blank>[email protected]</A>]=20 <BR></DIV><B>Sent:</B> Monday, February 11, 2008 9:33 AM <DIV class=3DIh2E3d><BR><B>To:</B> Brockington,MJ,Michael,JPGA4X=20 R<BR><B>Cc:</B> <A = href=3D"mailto:[email protected]"=20 = target=3D_blank>[email protected]</A><BR><B>Subject:</B= > Re:=20 [htdig] Htdig and MHT files<BR></DIV></FONT><BR></DIV> <BLOCKQUOTE=20 style=3D"PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: = rgb(0,0,255) 2px solid; MARGIN-RIGHT: 0px"> <DIV></DIV> <DIV>Hi Mike,</DIV> <DIV class=3DIh2E3d> <DIV>Yes you were right, I was missing that part and I didn't even = noticed!<BR>I changed the config file and wrote this:</DIV><FONT = size=3D2> <P>application/pdf->text/html=20 /usr/local/apache/htdocs/htdig-3.1.6/contrib/parsepdf.pl \</P> <P>application/vnd.wap.xhtml+xml->text/html = /opt/vin/mht2html.pl</P> <DIV></DIV></FONT>vnd.wap.xhtml+xml was the MIME type for my mht=20 documents.=20 <DIV>So I run dig and everything seems to go fine, having at = the=20 end:</DIV> = <DIV><BR><BR>0/http://172.26.0.169/testdig/<BR>1/http://172.26.0.169/test= dig/About_comments_eex3.mht<BR>2/http://172.26.0.169/testdig/aster.pdf<BR= >3/http://172.26.0.169/testdig/beepmacro.mht<BR>4/http://172.26.0.169/tes= tdig/index.txt<BR>5/http://172.26.0.169/testdig/test.html</DIV> <DIV> </DIV> <DIV>(I am doing this in a test folder)</DIV> <DIV> </DIV> <DIV>But when I go to the search page, it won't find words inside = the mht=20 files. It works for the pdf, txt and html ones, but can't find the = words=20 that are in the mht ones.</DIV> <DIV> </DIV> <DIV>I suppose I am missing something here... do I need to setup = any other=20 settings for the search engine?</DIV> <DIV> </DIV> <DIV>Thanks a lot for all your help,</DIV> <DIV> </DIV> = <DIV>Ainhoa</DIV></DIV></BLOCKQUOTE></DIV></BLOCKQUOTE></DIV><BR></BLOCKQ= UOTE></BODY></HTML> ------_=_NextPart_001_01C86CC8.FAE20028-- --===============1911991377== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ --===============1911991377== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ ht://Dig general mailing list: <[email protected]> ht://Dig FAQ: http://htdig.sourceforge.net/FAQ.html List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-general --===============1911991377==--