Re: Re: wiktionary parser
"George V. Reilly" <[email protected]> Sun, 3 Feb 2013 07:54:35 -0800
| Newsgroups | gmane.comp.windows.off-topic |
|---|---|
| Message-ID | <CAL3j_b_v1ys6di0p_ZqrjCufSOWL7-G=if+PyLZU5prcy3vbow@mail.gmail.com> |
--e89a8f3b9eb140284304d4d3fe2e Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable In the Python world, BeautifulSoup is a very powerful screen scraper. You could drive it with IronPython, probably. Googling for "BeautifulSoup IronPython" led me to http://htmlagilitypack.codeplex.com/, which may be more what you want. --=20 /George V. Reilly [email protected] Twitter: @georgevreilly http://www.georgevreilly.com/blog http://blogs.cozi.com/tech On Sun, Feb 3, 2013 at 7:34 AM, Thomas <[email protected]> wrote: > ** > > > Three years ago I needed to do some web page scraping for a project that = I > was working on. I looked for an open source .Net solution to do it, but > couldn't find one. I ended up using a an open source Java project called > HtmlUnit. HtmlUnit implements a "headless browser". It gives you a DOM vi= ew > of the HTML page, and lets you find elements using XPath syntax. > > But of course HtmlUnit is Java code, and you can't just call Java code > from a .Net application. That's where IKVM.NET comes in. IKVM.NET is a an > implementation of Java for the .Net Framework. It includes a Java Virtual > Machine that runs under the .Net CLR, a .Net implementation of the Java > libraries, and a program to convert Java JAR files to .Net assemblies. So > when you're done, you deploy .Net assemblies. You don't need to install > Java on your target machine. > > This blog post explains how to use HtmlUnit in a .Net program using > IKVM.NET: > > > http://blog.stevensanderson.com/2010/03/30/using-htmlunit-on-net-for-head= less-browser-automation/ > > One drawback of using HtmlUnit this way is that when you are writing code > that uses the HtmlUnit API, you need to live in two worlds at once. You'r= e > writing C# (or VB.Net) code, but HtmlUnit uses Java naming conventions. I= t > also uses Java listeners rather than .Net events. So you may need to writ= e > Java-style listeners in C#. Also, the documentation is in Javadoc format. > It helps if you have some Java experience. > > I know that this all seems like an ugly kludge. But it works well. > HtmlUnit and IKVM.NET are mature, well-supported projects. Both are > hosted on SourceForge. Using them together is a lot less work than writin= g > your own HTML parser. > > Tom > > > --- In [email protected], Adrian Vintu wrote: > > > > Thank you Frank, but I only have VS 2010, Win 7. I also need not WinRT > code to post on shared server. > > > > Best regards, > > Adrian > > > > > > ________________________________ > > From: Frank Lavigne > > To: Adrian Vintu ; "[email protected]" > > Sent: Friday, February 1, 2013 2:14 PM > > Subject: RE: [OT] wiktionary parser > > > > > > =C3=82 > > Hi, > > > > I wrote a screen scraping utility that might help. > > > > It's open source and a free download. > > > > Thanks, > > Frank > > > > > > > http://code.msdn.microsoft.com/windowsapps/Screen-Scraping-Utility-11e3d4= 57 > > > > Sent from my Windows Phone > > > > ________________________________ > > From: Adrian Vintu > > Sent: 2/1/2013 4:13 AM > > To: [email protected] > > Subject: [OT] wiktionary parser > > > > =C3=82 > > Hi guys, > > > > I have been looking for some C# Wiktionary parser but just cannot get m= y > hands on an easy to use dll. > > > > I have found only half solutions, or very complex ways of extracting > data. > > > > I could use a parser that either scraps the html pages, or looks into > the downloadable en/de/fr wiki pages. > > > > Any ideas? > > > > Thank you, > > > > Best regards, > > Adrian > > > >=20=20 > --e89a8f3b9eb140284304d4d3fe2e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/htm= l4/strict.dtd"> <html> <head> </head> <body style=3D"background-color: #fff;"> <span style=3D"display:none"> </span> <!--~-|**|PrettyHtmlStartT|**|-~--> <div id=3D"ygrp-mlmsg" style=3D"position:relative;"> <div id=3D"ygrp-msg" style=3D"z-index: 1;"> <!--~-|**|PrettyHtmlEndT|**|-~--> <div id=3D"ygrp-text" > =20=20=20=20=20=20 =20=20=20=20=20=20 <p><font face=3D"trebuchet ms,sans-serif">In the Python world, Beauti= fulSoup is a very powerful screen scraper. You could drive it with IronPyth= on, probably.</font><div><font face=3D"trebuchet ms,sans-serif"><br></font>= </div><div> <font face=3D"trebuchet ms,sans-serif">Googling for "BeautifulSoup Iro= nPython" led me to=C2=A0<a href=3D"http://htmlagilitypack.codeplex.com= /">http://htmlagilitypack.codeplex.com/</a>, which may be more what you wan= t.</font><div> <font face=3D"trebuchet ms, sans-serif"><div dir=3D"ltr"><font face=3D"treb= uchet ms,sans-serif"><font face=3D"trebuchet ms,sans-serif"><font face=3D"t= rebuchet ms,sans-serif"><div>--=C2=A0</div><div>/George V. Reilly =C2=A0<a = href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a> = =C2=A0Twitter: @georgevreilly</div> <div><a href=3D"http://www.georgevreilly.com/blog" target=3D"_blank">http:/= /www.georgevreilly.com/blog</a> =C2=A0<a href=3D"http://blogs.cozi.com/tech= " target=3D"_blank">http://blogs.cozi.com/tech</a>=C2=A0</div></font></font= ></font></div> <br></font><div><br><div class=3D"gmail_quote">On Sun, Feb 3, 2013 at 7:34 = AM, Thomas <span dir=3D"ltr"><<a href=3D"mailto:[email protected]" t= arget=3D"_blank">[email protected]</a>></span> wrote:<br><blockquote= class=3D"gmail_quote" style=3D"border-left:1px #ccc solid;"> <u></u> <div> <span>=C2=A0</span> <div> <div> <div> =20=20=20=20=20=20 =20=20=20=20=20=20 <p>Three years ago I needed to do some web page scraping for a projec= t that I was working on. I looked for an open source .Net solution to do it= , but couldn't find one. I ended up using a an open source Java project= called HtmlUnit. HtmlUnit implements a "headless browser". It gi= ves you a DOM view of the HTML page, and lets you find elements using XPath= syntax.<br> <br> But of course HtmlUnit is Java code, and you can't just call Java code = from a .Net application. That's where <a href=3D"http://IKVM.NET" targe= t=3D"_blank">IKVM.NET</a> comes in. <a href=3D"http://IKVM.NET" target=3D"_= blank">IKVM.NET</a> is a an implementation of Java for the .Net Framework. = It includes a Java Virtual Machine that runs under the .Net CLR, a .Net imp= lementation of the Java libraries, and a program to convert Java JAR files = to .Net assemblies. So when you're done, you deploy .Net assemblies. Yo= u don't need to install Java on your target machine. <br> <br> This blog post explains how to use HtmlUnit in a .Net program using <a href= =3D"http://IKVM.NET" target=3D"_blank">IKVM.NET</a>:<br> <br> <a href=3D"http://blog.stevensanderson.com/2010/03/30/using-htmlunit-on-net= -for-headless-browser-automation/" target=3D"_blank">http://blog.stevensand= erson.com/2010/03/30/using-htmlunit-on-net-for-headless-browser-automation/= </a><br> <br> One drawback of using HtmlUnit this way is that when you are writing code t= hat uses the HtmlUnit API, you need to live in two worlds at once. You'= re writing C# (or VB.Net) code, but HtmlUnit uses Java naming conventions. = It also uses Java listeners rather than .Net events. So you may need to wri= te Java-style listeners in C#. Also, the documentation is in Javadoc format= . It helps if you have some Java experience.<br> <br> I know that this all seems like an ugly kludge. But it works well. HtmlUnit= and <a href=3D"http://IKVM.NET" target=3D"_blank">IKVM.NET</a> are mature,= well-supported projects. Both are hosted on SourceForge. Using them togeth= er is a lot less work than writing your own HTML parser.<br> <br> Tom</p><div class=3D"im"><br> <br> --- In <a href=3D"mailto:win_tech_off_topic%40yahoogroups.com" target=3D"_b= lank">[email protected]</a>, Adrian Vintu wrote:<br> ><br> > Thank you Frank, but I only have VS 2010, Win 7. I also need not WinRT= code to post on shared server.<br> > <br> > Best regards,<br> > Adrian<br> > <br> > <br> > ________________________________<br> > From: Frank Lavigne <br></div> > To: Adrian Vintu ; "<a href=3D"mailto:win_tech_off_topic%40yahoog= roups.com" target=3D"_blank">[email protected]</a>" = <br><div class=3D"im"> > Sent: Friday, February 1, 2013 2:14 PM<br> > Subject: RE: [OT] wiktionary parser<br> > <br> > <br></div> > =C3=82=C2=A0 <br><div class=3D"im"> > Hi,<br> > <br> > I wrote a screen scraping utility that might help.<br> > <br> > It's open source and a free download.<br> > <br> > Thanks,<br> > Frank<br> > <br> > <br> > <a href=3D"http://code.msdn.microsoft.com/windowsapps/Screen-Scraping-= Utility-11e3d457" target=3D"_blank">http://code.msdn.microsoft.com/windowsa= pps/Screen-Scraping-Utility-11e3d457</a><br> > <br> > Sent from my Windows Phone<br> > <br> > ________________________________<br> > From: Adrian Vintu<br> > Sent: 2/1/2013 4:13 AM<br> > To: <a href=3D"mailto:win_tech_off_topic%40yahoogroups.com" target=3D"= _blank">[email protected]</a><br> > Subject: [OT] wiktionary parser<br> > <br></div> > =C3=82=C2=A0 <br><div class=3D"im"> > Hi guys,<br> > <br> > I have been looking for some C# Wiktionary parser but just cannot get = my hands on an easy to use dll.<br> > <br> > I have found only half solutions, or very complex ways of extracting d= ata.<br> > <br> > I could use a parser that either scraps the html pages, or looks into = the downloadable en/de/fr wiki pages.<br> > <br> > Any ideas?<br> > <br> > Thank you,<br> > <br> > Best regards,<br> > Adrian<br> ><br> <br> </div><p></p> </div> =20=20=20=20=20 =20=20=20=20 <div style=3D"color:#fff;"></div> </div> =20=20 </div></div></blockquote></div><br></div></div></div> </p> </div> =20=20=20=20=20 <!--~-|**|PrettyHtmlStart|**|-~--> <div style=3D"color: #fff; height: 0;">__._,_.___</div> =20=20=20=20=20=20=20=20 =20=20 =20=20 <table cellspacing=3D4px style=3D"margin-top: 20px; margin-bottom: 10px= ;"> <tbody> <tr> <td style=3D"font-size: 12px; font-family: arial; font-weight: bo= ld; padding: 7px 5px 5px; color: #FFF; background-color: #F2F2F2; border: 1= px solid #EAEAEA " > <a style=3D"text-decoration: none; color: #2D50FD= " href=3D"http://groups.yahoo.com/group/win_tech_off_topic/post;_ylc=3DX3oD= MTJxOXFrN2U2BF9TAzk3MzU5NzE0BGdycElkAzM3MDE1MjQEZ3Jwc3BJZAMxNzA1MDA2NjY5BG1= zZ0lkAzYwNDk3BHNlYwNmdHIEc2xrA3JwbHkEc3RpbWUDMTM1OTkwNjg3Nw--?act=3Dreply&m= essageNum=3D60497">Reply via web post</a> </td> <td style=3D"font-size: 12px; font-family: arial; padding: 7px 5p= x 5px; color: #FFF; background-color: #F2F2F2; border: 1px solid #EAEAEA; "= > <a href=3D"mailto:[email protected]?subject=3DRe%3A%20%5BOT%5D%= 20Re%3A%20wiktionary%20parser" style=3D"text-decoration: none; color: #2D50= FD;"> Reply to sender </a>=20 </td> <td style=3D"font-size: 12px; font-family: arial; padding: 7px 5p= x 5px; color: #FFF; background-color: #F2F2F2; border: 1px solid #EAEAEA; "= > <a href=3D"mailto:[email protected]?subject=3D= Re%3A%20%5BOT%5D%20Re%3A%20wiktionary%20parser" style=3D"text-decoration: n= one; color: #2D50FD"> Reply to group </a>=20 </td> <td style=3D"font-size: 12px; font-family: arial; padding: 7px 5p= x 5px; color: #FFF; background-color: #F2F2F2; border: 1px solid #EAEAEA; "= > <a href=3D"http://groups.yahoo.com/group/win_tech_off_topic/pos= t;_ylc=3DX3oDMTJlZnF0MmhvBF9TAzk3MzU5NzE0BGdycElkAzM3MDE1MjQEZ3Jwc3BJZAMxNz= A1MDA2NjY5BHNlYwNmdHIEc2xrA250cGMEc3RpbWUDMTM1OTkwNjg3Nw--" style=3D"text-d= ecoration: none; color: #2D50FD">Start a New Topic</a> </td> <td style=3D"font-size: 12px; font-family: arial; padding: 7px 5p= x 5px; color: #2D50FD; background-color: #F2F2F2; border: 1px solid #EAEAEA= ; " > <a href=3D"http://groups.yahoo.com/group/win_te= ch_off_topic/message/60483;_ylc=3DX3oDMTM2ZTNudW9vBF9TAzk3MzU5NzE0BGdycElkA= zM3MDE1MjQEZ3Jwc3BJZAMxNzA1MDA2NjY5BG1zZ0lkAzYwNDk3BHNlYwNmdHIEc2xrA3Z0cGME= c3RpbWUDMTM1OTkwNjg3NwR0cGNJZAM2MDQ4Mw--" style=3D"text-decoration: none; c= olor: #2D50FD;">Messages in this topic</a> (14) </td> </tr> </tbody> </table> =20=20=20=20=20=20=20=20 <!------- Start Nav Bar ------> <!-- |**|begin egp html banner|**| --> <div id=3D"ygrp-vital" style=3D"background-color: #f2f2f2; font-family: Ver= dana; font-size: 10px; margin-bottom: 10px; padding: 10px;"> <span id=3D"vithd" style=3D"font-weight: bold; color: #333; text-tran= sform: uppercase; ">Recent Activity:</span> <ul style=3D"list-style-type: none; margin: 0; padding: 0; display: inl= ine;"> </ul> =20=20=20=20 <div style=3D"clear: both; padding-top: 2px; color: #1e66ae;"> <a href=3D"http://groups.yahoo.com/group/win_tech_off_topic;_ylc=3DX3oD= MTJlNjJ2NTQ2BF9TAzk3MzU5NzE0BGdycElkAzM3MDE1MjQEZ3Jwc3BJZAMxNzA1MDA2NjY5BHN= lYwN2dGwEc2xrA3ZnaHAEc3RpbWUDMTM1OTkwNjg3Nw--" style=3D"text-decoration: no= ne;">Visit Your Group</a> </div> </div> =20=20 <div id=3D"ft" style=3D"font-family: Arial; font-size: 11px; margin-top: 5p= x; padding: 0 2px 0 0; clear: both;"> <a href=3D"http://groups.yahoo.com/;_ylc=3DX3oDMTJkbmtyY3FoBF9TAzk3NDc2NT= kwBGdycElkAzM3MDE1MjQEZ3Jwc3BJZAMxNzA1MDA2NjY5BHNlYwNmdHIEc2xrA2dmcARzdGltZ= QMxMzU5OTA2ODc3" style=3D"float: left;"><img src=3D"http://l.yimg.com/a/i/u= s/yg/logo/us.gif" height=3D"15" width=3D"137" alt=3D"Yahoo! Groups" style= =3D"border: 0;"/></a> <div style=3D"color: #747575; float: right;">Switch to: <a href=3D"mailto= :win_tech_off_topic-traditional-hHKSG33TihhbjbujkaE4pw@public.gmane.org?subject=3DChange Delivery F= ormat: Traditional" style=3D"text-decoration: none;">Text-Only</a>, <a href= =3D"mailto:win_tech_off_topic-digest-hHKSG33TihhbjbujkaE4pw@public.gmane.org?subject=3DEmail Delive= ry: Digest" class=3D"margin-rt" style=3D"text-decoration: none;">Daily Dige= st</a> • <a href=3D"mailto:win_tech_off_topic-unsubscribe@yahoogroups.= com?subject=3DUnsubscribe" style=3D"text-decoration: none;">Unsubscribe</a>= • <a href=3D"http://docs.yahoo.com/info/terms/" style=3D"text-decorat= ion: none;">Terms of Use</a> • <a href=3D"mailto:ygroupsnotifications@= yahoogroups.com?subject=3DFeedback on the redesigned individual mail v1" st= yle=3D"text-decoration: none;">Send us Feedback </a></div> </div> <!-- |**|end egp html banner|**| --> </div> <!-- ygrp-msg --> <!-- Sponsor --> <!-- |**|begin egp html banner|**| --> <div id=3D"ygrp-sponsor" style=3D"width:160px; float:right; clear:none; m= argin:0 0 25px 0; background: #fff;"> <!-- Start Recommendations --> <div id=3D"ygrp-reco"> </div> <!-- End Recommendations --> </div> <!-- |**|end egp html banner|**| --> <div style=3D"clear:both; color: #FFF; font-size:1px;">.</div> </div> <img src=3D"http://geo.yahoo.com/serv?s=3D97359714/grpId=3D3701524/grpspI= d=3D1705006669/msgId=3D60497/stime=3D1359906877/nc1=3D3848642/nc2=3D5191953= /nc3=3D5741392" width=3D"1" height=3D"1"> <br> <div style=3D"color: #fff; height: 0;">__,_._,___</div> <!--~-|**|PrettyHtmlEnd|**|-~--> </body> <!--~-|**|PrettyHtmlStart|**|-~--> <head> <style type=3D"text/css"> <!-- #ygrp-mkp { border: 1px solid #d8d8d8; font-family: Arial; margin: 10px 0; padding: 0 10px; } #ygrp-mkp hr { border: 1px solid #d8d8d8; } #ygrp-mkp #hd { color: #628c2a; font-size: 85%; font-weight: 700; line-height: 122%; margin: 10px 0; } #ygrp-mkp #ads { margin-bottom: 10px; } #ygrp-mkp .ad { padding: 0 0; } #ygrp-mkp .ad p { margin: 0; } #ygrp-mkp .ad a { color: #0000ff; text-decoration: none; } #ygrp-sponsor #ygrp-lc { font-family: Arial; } #ygrp-sponsor #ygrp-lc #hd { margin: 10px 0px; font-weight: 700; font-size: 78%; line-height: 122%; } #ygrp-sponsor #ygrp-lc .ad { margin-bottom: 10px; padding: 0 0; } #actions { font-family: Verdana; font-size: 11px; padding: 10px 0; } #activity { background-color: #e0ecee; float: left; font-family: Verdana; font-size: 10px; padding: 10px; } #activity span { font-weight: 700; } #activity span:first-child { text-transform: uppercase; } #activity span a { color: #5085b6; text-decoration: none; } #activity span span { color: #ff7900; } #activity span .underline { text-decoration: underline; } .attach { clear: both; display: table; font-family: Arial; font-size: 12px; padding: 10px 0; width: 400px; } .attach div a { text-decoration: none; } .attach img { border: none; padding-right: 5px; } .attach label { display: block; margin-bottom: 5px; } .attach label a { text-decoration: none; } =20=20 blockquote { margin: 0 0 0 4px; } .bold { font-family: Arial; font-size: 13px; font-weight: 700; } .bold a { text-decoration: none; } dd.last p a { font-family: Verdana; font-weight: 700; } dd.last p span { margin-right: 10px; font-family: Verdana; font-weight: 700; } dd.last p span.yshortcuts { margin-right: 0; } div.attach-table div div a { text-decoration: none; } div.attach-table { width: 400px; } div.file-title a, div.file-title a:active, div.file-title a:hover, div.fi= le-title a:visited { text-decoration: none; } div.photo-title a, div.photo-title a:active, div.photo-title a:hover, div= .photo-title a:visited { text-decoration: none; } div#ygrp-mlmsg #ygrp-msg p a span.yshortcuts { font-family: Verdana; font-size: 10px; font-weight: normal; } .green { color: #628c2a; } .MsoNormal { margin: 0 0 0 0; } o { font-size: 0; } #photos div { float: left; width: 72px; } #photos div div { border: 1px solid #666666; height: 62px; overflow: hidden; width: 62px; } #photos div label { color: #666666; font-size: 10px; overflow: hidden; text-align: center; white-space: nowrap; width: 64px; } #reco-category { font-size: 77%; } #reco-desc { font-size: 77%; } .replbq { margin: 4px; } #ygrp-actbar div a:first-child { /* border-right: 0px solid #000;*/ margin-right: 2px; padding-right: 5px; } #ygrp-mlmsg { font-size: 13px; font-family: Arial, helvetica,clean, sans-serif; *font-size: small; *font: x-small; } #ygrp-mlmsg table { font-size: inherit; font: 100%; } #ygrp-mlmsg select, input, textarea { font: 99% Arial, Helvetica, clean, sans-serif; } #ygrp-mlmsg pre, code { font:115% monospace; *font-size:100%; } #ygrp-mlmsg * { line-height: 1.22em; } #ygrp-mlmsg #logo { padding-bottom: 10px; } #ygrp-msg p a { font-family: Verdana; } #ygrp-msg p#attach-count span { color: #1E66AE; font-weight: 700; } #ygrp-reco #reco-head { color: #ff7900; font-weight: 700; } #ygrp-reco { margin-bottom: 20px; padding: 0px; } #ygrp-sponsor #ov li a { font-size: 130%; text-decoration: none; } #ygrp-sponsor #ov li { font-size: 77%; list-style-type: square; padding: 6px 0; }=20 #ygrp-sponsor #ov ul { margin: 0; padding: 0 0 0 8px; } #ygrp-text { font-family: Georgia; } #ygrp-text p { margin: 0 0 1em 0; } #ygrp-text tt { font-size: 120%; } #ygrp-vital ul li:last-child { border-right: none !important;=20 }=20 --> </style> </head> <!--~-|**|PrettyHtmlEnd|**|-~--> </html> <!-- end group email --> --e89a8f3b9eb140284304d4d3fe2e--