Re: [GSoC] Questions about project Text-Extraction Libraries

--000000000000af57850585156cb4
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

I think you are right and I will try with another approach.

One last query, I was thinking if it would be worth trying to use an
external filter (when it is available) in case a particular library fails
on run time.

Have you considered it?

El mar., 26 de mar. de 2019 a la(s) 19:39, Olly Betts ([email protected])
escribi=C3=B3:

> On Sat, Mar 23, 2019 at 03:42:36PM -0300, Bruno Baruffaldi wrote:
> > Thanks!
> > That was really useful!
> >
> > I wanted to share my approach to this project with the hope that you ca=
n
> > give me some feedback.
> >
> > I am think that applying a design that foresees the incorporation of ne=
w
> > file formats is the most suitable way to solve the problem.
> >
> > In the attached sketch we can see:
> > * Bug_Box: It is responsible for encapsulating and handling errors.
> > * File_extrator: It presents an interface for the different formats.
> > * File_X: Encapsulates a particular library for the X file format.
> > * File_Hadle: It is responsible for directing the extraction. More
> > specifically, it determines the file format and which extractor to use.
> > * Ominex: It represents the rest of the project.
>
> I'm not entirely sure what these boxes are meant to actually be
> (classes? programs? something else?), but in general I'd tend to steer
> GSoC projects towards an evolutionary approach rather than trying to
> rewrite everything in sight, or even refactor everything into some
> entirely new structure.
>
> With an evolutionary approach you can get to something that basically
> works much sooner, and then fill in the missing pieces, fix bugs, etc.
> It lends itself much better to incremental cycles of implement, test,
> document, review, merge, which is easier to work through for both
> mentors and students, and if the work doesn't get fully completed, at
> least there's something to show for it.
>
> With a revolutionary approach, there's nothing you can show working
> for much longer, and you'll need to do a lot of extra testing for
> all the existing functionality to make sure your reimplementation
> works (unfortunately there's currently no testsuite for omindex you can
> lean on here).
>
> Review is painful because it involves wading through thousands of
> lines of code, so you're likely to need to wait longer for a review
> because it's harder for mentors to find enough time in one go for
> that.
>
> And if the work doesn't get fully completed, there's a big pile of
> non-functioning code, which it's unlikely anyone is going to have the
> time or enthusiasm to do anything further with.
>
> More specifically to this case we already have code which encapsulates
> extraction in a subprocess for an external filter program, and code
> which determines the file format and which extractor to use.  If you
> are proposing to replace those, you're going to need to convince us
> what you think is deficient about the existing code, how you can
> do better, and why that's a good use of the limited GSoC coding time
> (if you spend time doing X, then you can't do Y).
>
> > The idea of organizing the code in this way focuses on two fundamental
> > items:
> > * The possibility of changing a particular library for another that
> > fulfills the same purpose without affecting the project.
>
> That's achievable without a major restructure (e.g. wrap each library
> in a helper program).
>
> > * The possibility of extending Xapian's support in terms of file format=
.
>
> People have been adding new file formats for years within the current
> structure.
>
> > One of the major advantages is that if a particular programmer wishes t=
o
> > add support for a new file format or improve an existing one, they shou=
ld
> > only modify the objects that are in red. In this way, with a proper
> > documentation this kind of tasks should not be a complex task.
>
> We've had prospective GSoC students who were new to the codebase add
> support for new formats successful, which suggests it isn't all that
> complex currently.
>
> To show what's typically involved, here's the patch to add support for
> iWork documents (which is the most recent format added):
>
>
> https://git.xapian.org/?p=3Dxapian;a=3Dcommitdiff;h=3D10e2cf5e64c8acd0a13=
5e54007e1ba8eff2c53ee
>
> (The gen-mimemap tweak is only because this happened to mean the
> generated mimetype lookup table now needs 2 byte offsets).
>
> Cheers,
>     Olly
>


--=20
Atte. Bruno Baruffaldi

--000000000000af57850585156cb4
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><div class=3D"gmail_default" style=3D"fon=
t-family:tahoma,sans-serif">I think you are right and I will try with anoth=
er approach.</div><div class=3D"gmail_default" style=3D"font-family:tahoma,=
sans-serif"><br></div><div class=3D"gmail_default" style=3D"font-family:tah=
oma,sans-serif">One last query, I was thinking if it would be worth trying =
to use an external filter (when it is available) in case a particular libra=
ry fails on run time.</div><div class=3D"gmail_default" style=3D"font-famil=
y:tahoma,sans-serif"><br></div><div class=3D"gmail_default" style=3D"font-f=
amily:tahoma,sans-serif">Have you considered it?<br></div></div></div><br><=
div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">El mar., 26=
 de mar. de 2019 a la(s) 19:39, Olly Betts (<a href=3D"mailto:[email protected]=
om">[email protected]</a>) escribi=C3=B3:<br></div><blockquote class=3D"gmail=
_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204=
,204);padding-left:1ex">On Sat, Mar 23, 2019 at 03:42:36PM -0300, Bruno Bar=
uffaldi wrote:<br>
&gt; Thanks!<br>
&gt; That was really useful!<br>
&gt; <br>
&gt; I wanted to share my approach to this project with the hope that you c=
an<br>
&gt; give me some feedback.<br>
&gt; <br>
&gt; I am think that applying a design that foresees the incorporation of n=
ew<br>
&gt; file formats is the most suitable way to solve the problem.<br>
&gt; <br>
&gt; In the attached sketch we can see:<br>
&gt; * Bug_Box: It is responsible for encapsulating and handling errors.<br=
>
&gt; * File_extrator: It presents an interface for the different formats.<b=
r>
&gt; * File_X: Encapsulates a particular library for the X file format.<br>
&gt; * File_Hadle: It is responsible for directing the extraction. More<br>
&gt; specifically, it determines the file format and which extractor to use=
.<br>
&gt; * Ominex: It represents the rest of the project.<br>
<br>
I&#39;m not entirely sure what these boxes are meant to actually be<br>
(classes? programs? something else?), but in general I&#39;d tend to steer<=
br>
GSoC projects towards an evolutionary approach rather than trying to<br>
rewrite everything in sight, or even refactor everything into some<br>
entirely new structure.<br>
<br>
With an evolutionary approach you can get to something that basically<br>
works much sooner, and then fill in the missing pieces, fix bugs, etc.<br>
It lends itself much better to incremental cycles of implement, test,<br>
document, review, merge, which is easier to work through for both<br>
mentors and students, and if the work doesn&#39;t get fully completed, at<b=
r>
least there&#39;s something to show for it.<br>
<br>
With a revolutionary approach, there&#39;s nothing you can show working<br>
for much longer, and you&#39;ll need to do a lot of extra testing for<br>
all the existing functionality to make sure your reimplementation<br>
works (unfortunately there&#39;s currently no testsuite for omindex you can=
<br>
lean on here).<br>
<br>
Review is painful because it involves wading through thousands of<br>
lines of code, so you&#39;re likely to need to wait longer for a review<br>
because it&#39;s harder for mentors to find enough time in one go for<br>
that.<br>
<br>
And if the work doesn&#39;t get fully completed, there&#39;s a big pile of<=
br>
non-functioning code, which it&#39;s unlikely anyone is going to have the<b=
r>
time or enthusiasm to do anything further with.<br>
<br>
More specifically to this case we already have code which encapsulates<br>
extraction in a subprocess for an external filter program, and code<br>
which determines the file format and which extractor to use.=C2=A0 If you<b=
r>
are proposing to replace those, you&#39;re going to need to convince us<br>
what you think is deficient about the existing code, how you can<br>
do better, and why that&#39;s a good use of the limited GSoC coding time<br=
>
(if you spend time doing X, then you can&#39;t do Y).<br>
<br>
&gt; The idea of organizing the code in this way focuses on two fundamental=
<br>
&gt; items:<br>
&gt; * The possibility of changing a particular library for another that<br=
>
&gt; fulfills the same purpose without affecting the project.<br>
<br>
That&#39;s achievable without a major restructure (e.g. wrap each library<b=
r>
in a helper program).<br>
<br>
&gt; * The possibility of extending Xapian&#39;s support in terms of file f=
ormat.<br>
<br>
People have been adding new file formats for years within the current<br>
structure.<br>
<br>
&gt; One of the major advantages is that if a particular programmer wishes =
to<br>
&gt; add support for a new file format or improve an existing one, they sho=
uld<br>
&gt; only modify the objects that are in red. In this way, with a proper<br=
>
&gt; documentation this kind of tasks should not be a complex task.<br>
<br>
We&#39;ve had prospective GSoC students who were new to the codebase add<br=
>
support for new formats successful, which suggests it isn&#39;t all that<br=
>
complex currently.<br>
<br>
To show what&#39;s typically involved, here&#39;s the patch to add support =
for<br>
iWork documents (which is the most recent format added):<br>
<br>
<a href=3D"https://git.xapian.org/?p=3Dxapian;a=3Dcommitdiff;h=3D10e2cf5e64=
c8acd0a135e54007e1ba8eff2c53ee" rel=3D"noreferrer" target=3D"_blank">https:=
//git.xapian.org/?p=3Dxapian;a=3Dcommitdiff;h=3D10e2cf5e64c8acd0a135e54007e=
1ba8eff2c53ee</a><br>
<br>
(The gen-mimemap tweak is only because this happened to mean the<br>
generated mimetype lookup table now needs 2 byte offsets).<br>
<br>
Cheers,<br>
=C2=A0 =C2=A0 Olly<br>
</blockquote></div><br clear=3D"all"><br>-- <br><div dir=3D"ltr" class=3D"g=
mail_signature"><div dir=3D"ltr">Atte. Bruno Baruffaldi<br></div></div>

--000000000000af57850585156cb4--