Re: Text-Extraction Libraries for Omindex
Bruno Baruffaldi <[email protected]> Sat, 15 Jun 2019 14:13:43 -0300
| Newsgroups | gmane.comp.search.xapian.devel |
|---|---|
| Message-ID | <CAND-c9OcLB=uUgABNsgt6NZzVshp4hx3M55XZ7Vp_3bq+OnHCg@mail.gmail.com> |
--0000000000005d9fc4058b5fe36c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello, I have been looking libarchive and it seems a great candidate! I think we can also add libstaroffice <https://github.com/fosnola/libstaroffice> and libmarkdown2-dev. I wasn't sure about adding libmarkdown2-dev to the list because I couldn't find much information about it. El s=C3=A1b., 15 de jun. de 2019 a la(s) 00:49, Olly Betts ([email protected]= ) escribi=C3=B3: > On Fri, Jun 14, 2019 at 08:52:51AM -0300, Bruno Baruffaldi wrote: > > This is a list with some libraries that I have been looking at. > > > > The idea is to discuss the advantages and disadvantages of adding some = of > > these libraries to Xapian. > > I think we should prioritise formats which are widely used (among > current and potential users of Omega particularly), and also formats > which we don't already support (or which we could support better by > using a library). > > > > > If anyone knows another library that could be add to the list it would = be > > great! > > > > Libfreexl: > > * For Excel (.xls) > > * Last release: 2018-02 > > * Info: gaia-gis.it/fossil/freexl/index > > * License: MPL tri-license > > I've not come across this before. It looks like it is currently only > used in GIS software which is probably more interested in numbers than > text, so before we commit a lot of effort to supporting it I'd suggest > we try it out and compare how it does with the command line tool we > currently use (xls2csv). > > > Libzip: > > * For zip archives(C library) > > * Last release 2018-04 > > * Info: libzip.org > > * License: 3-clause BSD > > > > Libzipios++: > > * For zip archives > > * Last release 2019-04 > > * Info: zipios.sourceforge.net > > * License: GNU Lesser General Public License (LGPL) > > > > I have been thinking about unzip. It is widely use in omindex an it mig= ht > > be an option to replace unzip with one of this libraries. I know that i= t > is > > not the best solution, but it could be something to consider for some > > formats. > > I'd suggest libarchive for zip files - it's widely used, and supports > reading other archive formats rather than just zip files (I actually > wrote a prototype patch for omindex a while back to support indexing > files in archive files which used libarchive, though it hasn't been > merged yet). > > I think this is probably one to prioritise since we use unzip for a > number of common formats. > > > Djvulibre: > > * For DjVu files > > * Last release: 2015-02 > > * Info: djvu.sourceforge.net > > * License: GNU General Public License version 2 > > While DjVu is an interesting format, it doesn't seem to be widely used > and we can already index these files using the command line djvutxt > tool. > > > Libe-book: > > * For ebooks formats > > * Last release 2018-01 > > * It shows little activity > > * Status: Beta > > * Info: sourceforge.net/projects/libebook/ > > * License: GNU Lesser GPL 2.1+ and MPL 2.0+ > > > > I have been reading the code of this library, but it seems a bit comple= x. > > It could be a good option, but it will take a while to figure it out ho= w > it > > works. > > This is used by libreoffice. > > There's a command line tool in the libe-book source to extract text > (though for some reason this tool isn't packaged for Debian it seems). > You can see the source here, which shows how to use the API to extract > text: > > > https://sources.debian.org/src/libe-book/0.1.3-1/src/conv/text/ebook2text= .cpp/ > > This would add support for several popular formats we don't currently > support at all, so seems another one to prioritise. > > > Libetonyek-dev: > > * For Apple iWork documents > > * Status: Beta > > * Info: wiki.documentfoundation.org/DLP/Libraries/libetonyek > > * License: MPL 2.0+ > > We use this via a command line tool currently. I'd guess it's popular > on Macs so this is probably a good candidate. > > > Libabw: > > * For AbiWord documents > > * Last release 2017-12 > > * Info: wiki.documentfoundation.org/DLP/Libraries/libabw > > * License: MPL 2.0 > > This is an XML-based format which we have a built-in parser for, so > there's probably not a lot to gain from using an external library. > It's also not a very widely used format in my experience. > > > Other Options: > > * libreoffice-dev(SDK) > > I guess this is "libreofficekit"? > > I actually maintain a command line tool which is a thin wrapper > around that: > > https://gitlab.com/ojwb/lloconv > > It works pretty well, but it's rather slow even reusing the > lok::Office() object (lloconv has a feature where it can fork a > daemon process to allow such reuse). > > Much of the import code libreoffice uses has now beep split out into > libraries (like libabw, libe-book and libetonyek from your list) and > I think we'd do better to use such libraries directly. > > You can find a list of these libraries here: > > https://www.documentliberation.org/projects/#import-libs > > Cheers, > Olly > --=20 Atte. Bruno Baruffaldi --0000000000005d9fc4058b5fe36c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-family:tahoma,s= ans-serif">Hello,</div><div class=3D"gmail_default" style=3D"font-family:ta= homa,sans-serif"><br></div><div class=3D"gmail_default" style=3D"font-famil= y:tahoma,sans-serif">I have been looking libarchive and it seems a great ca= ndidate!</div><div class=3D"gmail_default" style=3D"font-family:tahoma,sans= -serif"><br></div><div class=3D"gmail_default" style=3D"font-family:tahoma,= sans-serif">I think we can also add <a href=3D"https://github.com/fosnola/l= ibstaroffice" target=3D"_blank">libstaroffice</a> and libmarkdown2-dev. I w= asn't sure about adding libmarkdown2-dev to the list because I couldn&#= 39;t find much information about it.<br></div><div class=3D"gmail_default" = style=3D"font-family:tahoma,sans-serif"><br></div></div><br><div class=3D"g= mail_quote"><div dir=3D"ltr" class=3D"gmail_attr">El s=C3=A1b., 15 de jun. = de 2019 a la(s) 00:49, Olly Betts (<a href=3D"mailto:[email protected]">olly@= survex.com</a>) escribi=C3=B3:<br></div><blockquote class=3D"gmail_quote" s= tyle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);pad= ding-left:1ex">On Fri, Jun 14, 2019 at 08:52:51AM -0300, Bruno Baruffaldi w= rote:<br> > This is a list with some libraries that I have been looking at.<br> > <br> > The idea is to discuss the advantages and disadvantages of adding some= of<br> > these libraries to Xapian.<br> <br> I think we should prioritise formats which are widely used (among<br> current and potential users of Omega particularly), and also formats<br> which we don't already support (or which we could support better by<br> using a library).<br> <br> > <br> > If anyone knows another library that could be add to the list it would= be<br> > great!<br> > <br> > Libfreexl:<br> > * For Excel (.xls)<br> > * Last release: 2018-02<br> > * Info: <a href=3D"http://gaia-gis.it/fossil/freexl/index" rel=3D"nore= ferrer" target=3D"_blank">gaia-gis.it/fossil/freexl/index</a><br> > * License: MPL tri-license<br> <br> I've not come across this before.=C2=A0 It looks like it is currently o= nly<br> used in GIS software which is probably more interested in numbers than<br> text, so before we commit a lot of effort to supporting it I'd suggest<= br> we try it out and compare how it does with the command line tool we<br> currently use (xls2csv).<br> <br> > Libzip:<br> > * For zip archives(C library)<br> > * Last release 2018-04<br> > * Info: <a href=3D"http://libzip.org" rel=3D"noreferrer" target=3D"_bl= ank">libzip.org</a><br> > * License: 3-clause BSD<br> > <br> > Libzipios++:<br> > * For zip archives<br> > * Last release 2019-04<br> > * Info: <a href=3D"http://zipios.sourceforge.net" rel=3D"noreferrer" t= arget=3D"_blank">zipios.sourceforge.net</a><br> > * License: GNU Lesser General Public License (LGPL)<br> > <br> > I have been thinking about unzip. It is widely use in omindex an it mi= ght<br> > be an option to replace unzip with one of this libraries. I know that = it is<br> > not the best solution, but it could be something to consider for some<= br> > formats.<br> <br> I'd suggest libarchive for zip files - it's widely used, and suppor= ts<br> reading other archive formats rather than just zip files (I actually<br> wrote a prototype patch for omindex a while back to support indexing<br> files in archive files which used libarchive, though it hasn't been<br> merged yet).<br> <br> I think this is probably one to prioritise since we use unzip for a<br> number of common formats.<br> <br> > Djvulibre:<br> > * For DjVu files<br> > * Last release: 2015-02<br> > * Info: <a href=3D"http://djvu.sourceforge.net" rel=3D"noreferrer" tar= get=3D"_blank">djvu.sourceforge.net</a><br> > * License: GNU General Public License version 2<br> <br> While DjVu is an interesting format, it doesn't seem to be widely used<= br> and we can already index these files using the command line djvutxt<br> tool.<br> <br> > Libe-book:<br> > * For ebooks formats<br> > * Last release 2018-01<br> > * It shows little activity<br> > * Status: Beta<br> > * Info: <a href=3D"http://sourceforge.net/projects/libebook/" rel=3D"n= oreferrer" target=3D"_blank">sourceforge.net/projects/libebook/</a><br> > * License: GNU Lesser GPL 2.1+ and MPL 2.0+<br> > <br> > I have been reading the code of this library, but it seems a bit compl= ex.<br> > It could be a good option, but it will take a while to figure it out h= ow it<br> > works.<br> <br> This is used by libreoffice.<br> <br> There's a command line tool in the libe-book source to extract text<br> (though for some reason this tool isn't packaged for Debian it seems).<= br> You can see the source here, which shows how to use the API to extract<br> text:<br> <br> <a href=3D"https://sources.debian.org/src/libe-book/0.1.3-1/src/conv/text/e= book2text.cpp/" rel=3D"noreferrer" target=3D"_blank">https://sources.debian= .org/src/libe-book/0.1.3-1/src/conv/text/ebook2text.cpp/</a><br> <br> This would add support for several popular formats we don't currently<b= r> support at all, so seems another one to prioritise.<br> <br> > Libetonyek-dev:<br> > * For Apple iWork documents<br> > * Status: Beta<br> > * Info: <a href=3D"http://wiki.documentfoundation.org/DLP/Libraries/li= betonyek" rel=3D"noreferrer" target=3D"_blank">wiki.documentfoundation.org/= DLP/Libraries/libetonyek</a><br> > * License: MPL 2.0+<br> <br> We use this via a command line tool currently.=C2=A0 I'd guess it's= popular<br> on Macs so this is probably a good candidate.<br> <br> > Libabw:<br> > * For AbiWord documents<br> > * Last release 2017-12<br> > * Info: <a href=3D"http://wiki.documentfoundation.org/DLP/Libraries/li= babw" rel=3D"noreferrer" target=3D"_blank">wiki.documentfoundation.org/DLP/= Libraries/libabw</a><br> > * License: MPL 2.0<br> <br> This is an XML-based format which we have a built-in parser for, so<br> there's probably not a lot to gain from using an external library.<br> It's also not a very widely used format in my experience.<br> <br> > Other Options:<br> > * libreoffice-dev(SDK)<br> <br> I guess this is "libreofficekit"?<br> <br> I actually maintain a command line tool which is a thin wrapper<br> around that:<br> <br> <a href=3D"https://gitlab.com/ojwb/lloconv" rel=3D"noreferrer" target=3D"_b= lank">https://gitlab.com/ojwb/lloconv</a><br> <br> It works pretty well, but it's rather slow even reusing the<br> lok::Office() object (lloconv has a feature where it can fork a<br> daemon process to allow such reuse).<br> <br> Much of the import code libreoffice uses has now beep split out into<br> libraries (like libabw, libe-book and libetonyek from your list) and<br> I think we'd do better to use such libraries directly.<br> <br> You can find a list of these libraries here:<br> <br> <a href=3D"https://www.documentliberation.org/projects/#import-libs" rel=3D= "noreferrer" target=3D"_blank">https://www.documentliberation.org/projects/= #import-libs</a><br> <br> Cheers,<br> =C2=A0 =C2=A0 Olly<br> </blockquote></div><br clear=3D"all"><br>-- <br><div dir=3D"ltr" class=3D"g= mail_signature"><div dir=3D"ltr">Atte. Bruno Baruffaldi<br></div></div> --0000000000005d9fc4058b5fe36c--