Re: Ideas for Biopython 2.0

--===============7487400461886814071==
Content-Type: multipart/alternative; boundary="001a11411b8466aa0f05532a909c"

--001a11411b8466aa0f05532a909c
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hello everyone,

> So, in theory, you can end up with dozens of extensions that are out of
sync with the core in terms of 'core' dependencies and it's up to the
maintainers of those extensions to keep them up-to-date. That's my main
concern. We have a monolith that works. It is conservative and that's
exactly why it works. Otherwise, for example, I would install the core and
your Cython-based lightining fast PopGen code and then you grow bored of
it, a new version of Cython comes along that has some incompatibilities
with your code, and now it doesn't work. Worse even, I release a blinding
fast PDBParser using a new feature in Cython and now you have two
incompatible extensions.

Yes, that is one concern that the modular setup can introduce. It is a
trouble when you want to use two non-core extensions which depend on an
incompatible versions of the core module. This applies if your application
wants to make use of both non-core modules.

However, I would argue that in that case, it would be easier for you push
for a change in the PopGen module (taking on your example). You would have
less code to look at (since you only want to change PopGen) and less person
to convince that your changes are worth a new release (which in this case
is Tiago, the PopGen maintainer). It would also be easier for you to fork
PopGen and maintain further, if you wish to do so. All of this boils down
to PopGen being smaller and, to a larger degree than it is now, independent
from the rest of Biopython.

Contrast this with what happens in the monolith setup: PDBParser is stuck
using a suboptimal parser because another module is not updated. Yes, there
is consistency in a sense that everything works together. But this is
consistency by the lowest common denominator: all modules need to adhere to
the oldest (and probably least maintained) module. Not to mention the
increased burden to the core developers when a maintainer for a module
decides to spend less time maintaining.

If you are interested in using non-compatible core modules inside a
pipeline (so both modules are used at different steps in the pipeline),
there are various pipeline frameworks and/or containers today that provide
more granular level of isolation. In other words, you can run incompatible
biopython versions in the same pipeline.

> I agree that we should 'refresh' our dependencies to be able to do cool
things. And modularity is incredibly attractive. But the main advantage of
modularity - lowering the standards of biopython code - is also, in my
opinion, it's main disadvantage.

I would say it actually encourages people to contribute more :). And yes,
this may come at a cost of some biopython non-core modules having 'lower
quality' than other modules. But I think that is not necessarily bad.
Useful modules with good documentation and good code quality are more
likely to be used.

> Question: How is scikit handling their different modules?

They have different teams for the core package and the toolkits (modules),
as far as I know [https://www.scipy.org/scikits.html]. It seems that anyone
can register a module with the scikit namespace (at least I could not find
any specific mention of the toolkit development requiring permission from
the core team). Even the license of the toolkit can be different. (slightly
unrelated, but there is also a scikit-bio package: http://scikit-bio.org/).

(P.S. just so that everyone is on the same page, we actually do have a
GitHub ticket on this modularization proposal here:
https://github.com/biopython/biopython/issues/349 ~ there were already some
discussion as you can see).

Best regards,
Bow

On Wed, Jun 28, 2017 at 7:10 PM Jo=C3=A3o Rodrigues <
[email protected]> wrote:

> So, in theory, you can end up with dozens of extensions that are out of
> sync with the core in terms of 'core' dependencies and it's up to the
> maintainers of those extensions to keep them up-to-date. That's my main
> concern. We have a monolith that works. It is conservative and that's
> exactly why it works. Otherwise, for example, I would install the core an=
d
> your Cython-based lightining fast PopGen code and then you grow bored of
> it, a new version of Cython comes along that has some incompatibilities
> with your code, and now it doesn't work. Worse even, I release a blinding
> fast PDBParser using a new feature in Cython and now you have two
> incompatible extensions.
>
> Being very honest, there is no barrier to anyone developing anything new
> here. If you want to use Cython, go ahead and do it and then worst case
> scenario, we have a dependency warning and check that can simply skip
> compilation of that code. It's what happens with Numpy at the moment IIRC=
.
> You can also bundle the c code, instead of compiling the pyx on install,
> and compile it regularly using GCC, skipping Cython altogether.
>
> I agree that we should 'refresh' our dependencies to be able to do cool
> things. And modularity is incredibly attractive. But the main advantage o=
f
> modularity - lowering the standards of biopython code - is also, in my
> opinion, it's main disadvantage.
>
> Question: How is scikit handling their different modules?
>
> 2017-06-28 8:05 GMT-07:00 Tiago Ant=C3=A3o <[email protected]>:
>
>> It can see plenty of issues where it could help. In my specific case all
>> the PopGen code is stopped for 10 years because I would need to write ve=
ry
>> fast code (say in Cython). This would be an extension module, not a core
>> module because it would impart a very big dependency on the system.
>>
>> Modules would allow a core with very strict policies and dependencies
>> _but_ extensions that could be way more relaxed.
>>
>> It would also lower the barrier of entry for new content. Everyone could
>> publish an extension. If the extension would survive time (which most do
>> not - creating a maintenance burden in the core) then it could eventuall=
y
>> be made a core extension. Now the policy in practice is to add very litt=
le
>> innovation out of the fear that it will become stagnant and not-supporte=
d
>> by the main author (say after publication). An extension system would
>> accommodate both innovation whereas preserving the core quality.
>>
>> Currently we have a gigantic monolith that in practice imposes very
>> conservative technologies and changes. I suspect that is why we do not s=
ee
>> anything really exciting with Biopython for the better part of the last
>> decade,
>>
>> On 28 June 2017 at 04:25, Michiel de Hoon <[email protected]> wrote:
>>
>>> I agree with Joao here. I don't see an immediate and overriding problem
>>> that modularity would solve, and I can see many drawbacks.
>>>
>>> Best,
>>> -Michiel
>>>
>>>
>>> On Monday, June 26, 2017 11:03 AM, Jo=C3=A3o Rodrigues <
>>> [email protected]> wrote:
>>>
>>>
>>> Copied from the other thread where I mistakenly posted:
>>>
>>> I think we should focus on other topics such as modularity. What do the
>>> proponents of the said modularity say about it? What are its advantages=
? I
>>> personally think a big disadvantage is that with one package install yo=
u
>>> get a wide array of tools for a variety of subjects. With a constellati=
on
>>> of modules you might end up with an up-to-date core and an out-of-date =
lone
>>> module somewhere, which makes things much much harder not only to maint=
ain
>>> but also to debug in case of issues.
>>>
>>>
>>> _______________________________________________
>>> Biopython-dev mailing list
>>> [email protected]
>>> http://mailman.open-bio.org/mailman/listinfo/biopython-dev
>>>
>>>
>>>
>>
>>
>> --
>> Tiago Antao
>> Scientific and HPC programmer
>> http://tiago.org
>> https://github.com/tiagoantao/
>>
>
> _______________________________________________
> Biopython-dev mailing list
> [email protected]
> http://mailman.open-bio.org/mailman/listinfo/biopython-dev

--001a11411b8466aa0f05532a909c
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hello everyone,</div><div><br></div>&gt;=C2=A0<span s=
tyle=3D"color:rgb(33,33,33)">So, in theory, you can end up with dozens of e=
xtensions that are out of sync with the core in terms of &#39;core&#39; dep=
endencies and it&#39;s up to the maintainers of those extensions to keep th=
em up-to-date. That&#39;s my main concern. We have a monolith that works. I=
t is conservative and that&#39;s exactly why it works. Otherwise, for examp=
le, I would install the core and your Cython-based lightining fast PopGen c=
ode and then you grow bored of it, a new version of Cython comes along that=
 has some incompatibilities with your code, and now it doesn&#39;t work. Wo=
rse even, I release a blinding fast PDBParser using a new feature in Cython=
 and now you have two incompatible extensions.<br><br>Yes, that is one conc=
ern that the modular setup can introduce. It is a trouble when you want to =
use two non-core extensions which depend on an incompatible versions of the=
 core module. This applies if your application wants to make use of both no=
n-core modules.<br><br>However, I would argue that in that case, it would b=
e easier for you push for a change in the PopGen module (taking on your exa=
mple). You would have less code to look at (since you only want to change P=
opGen) and less person to convince that your changes are worth a new releas=
e (which in this case is Tiago, the PopGen maintainer). It would also be ea=
sier for you to fork PopGen and maintain further, if you wish to do so. All=
 of this boils down to PopGen being smaller and, to a larger degree than it=
 is now, independent from the rest of Biopython.<br><br>Contrast this with =
what happens in the monolith setup: PDBParser is stuck using a suboptimal p=
arser because another module is not updated. Yes, there is consistency in a=
 sense that everything works together. But this is consistency by the lowes=
t common denominator: all modules need to adhere to the oldest (and probabl=
y least maintained) module. Not to mention the increased burden to the core=
 developers when a maintainer for a module decides to spend less time maint=
aining.<br><br>If you are interested in using non-compatible core modules i=
nside a pipeline (so both modules are used at different steps in the pipeli=
ne), there are various pipeline frameworks and/or containers today that pro=
vide more granular level of isolation. In other words, you can run incompat=
ible biopython versions in the same pipeline.<br><br>&gt;=C2=A0</span><span=
 style=3D"color:rgb(33,33,33)">I agree that we should &#39;refresh&#39; our=
 dependencies to be able to do cool things. And modularity is incredibly at=
tractive. But the main advantage of modularity - lowering the standards of =
biopython code - is also, in my opinion, it&#39;s main disadvantage.<br><br=
>I would say it actually encourages people to contribute more :). And yes, =
this may come at a cost of some biopython non-core modules having &#39;lowe=
r quality&#39; than other modules. But I think that is not necessarily bad.=
 Useful modules with good documentation and good code quality are more like=
ly to be used.<br><br>&gt;=C2=A0</span><font color=3D"#212121">Question: Ho=
w is scikit handling their different modules?</font><br><br><font color=3D"=
#212121">They have different teams for the core package and the toolkits (m=
odules), as far as I know [<a href=3D"https://www.scipy.org/scikits.html">h=
ttps://www.scipy.org/scikits.html</a>]. It seems that anyone can register a=
 module with the scikit namespace (at least I could not find any specific m=
ention of the toolkit development requiring permission from the core team).=
 Even the license of the toolkit can be different. (slightly unrelated, but=
 there is also a scikit-bio package:=C2=A0<a href=3D"http://scikit-bio.org/=
">http://scikit-bio.org/</a>).<br><br>(P.S. just so that everyone is on the=
 same page, we actually do have a GitHub ticket on this modularization prop=
osal here: <a href=3D"https://github.com/biopython/biopython/issues/349">ht=
tps://github.com/biopython/biopython/issues/349</a>=C2=A0~ there were alrea=
dy some discussion as you can see).<br><br>Best regards,<br>Bow</font></div=
><br><div class=3D"gmail_quote"><div dir=3D"ltr">On Wed, Jun 28, 2017 at 7:=
10 PM Jo=C3=A3o Rodrigues &lt;<a href=3D"mailto:[email protected]=
om">[email protected]</a>&gt; wrote:<br></div><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex"><div dir=3D"ltr">So, in theory, you can end up with dozens of=
 extensions that are out of sync with the core in terms of &#39;core&#39; d=
ependencies and it&#39;s up to the maintainers of those extensions to keep =
them up-to-date. That&#39;s my main concern. We have a monolith that works.=
 It is conservative and that&#39;s exactly why it works. Otherwise, for exa=
mple, I would install the core and your Cython-based lightining fast PopGen=
 code and then you grow bored of it, a new version of Cython comes along th=
at has some incompatibilities with your code, and now it doesn&#39;t work. =
Worse even, I release a blinding fast PDBParser using a new feature in Cyth=
on and now you have two incompatible extensions.=C2=A0<div><br></div><div>B=
eing very honest, there is no barrier to anyone developing anything new her=
e. If you want to use Cython, go ahead and do it and then worst case scenar=
io, we have a dependency warning and check that can simply skip compilation=
 of that code. It&#39;s what happens with Numpy at the moment IIRC. You can=
 also bundle the c code, instead of compiling the pyx on install, and compi=
le it regularly using GCC, skipping Cython altogether.=C2=A0</div><div><br>=
</div><div>I agree that we should &#39;refresh&#39; our dependencies to be =
able to do cool things. And modularity is incredibly attractive. But the ma=
in advantage of modularity - lowering the standards of biopython code - is =
also, in my opinion, it&#39;s main disadvantage.=C2=A0</div><div><br></div>=
<div>Question: How is scikit handling their different modules?</div></div><=
div class=3D"gmail_extra"><br><div class=3D"gmail_quote">2017-06-28 8:05 GM=
T-07:00 Tiago Ant=C3=A3o <span dir=3D"ltr">&lt;<a href=3D"mailto:tiagoantao=
@gmail.com" target=3D"_blank">[email protected]</a>&gt;</span>:<br><bloc=
kquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #cc=
c solid;padding-left:1ex"><div dir=3D"ltr">It can see plenty of issues wher=
e it could help. In my specific case all the PopGen code is stopped for 10 =
years because I would need to write very fast code (say in Cython). This wo=
uld be an extension module, not a core module because it would impart a ver=
y big dependency on the system.<div><br></div><div>Modules would allow a co=
re with very strict policies and dependencies _but_ extensions that could b=
e way more relaxed.</div><div><br></div><div>It would also lower the barrie=
r of entry for new content. Everyone could publish an extension. If the ext=
ension would survive time (which most do not - creating a maintenance burde=
n in the core) then it could eventually be made a core extension. Now the p=
olicy in practice is to add very little innovation out of the fear that it =
will become stagnant and not-supported by the main author (say after public=
ation). An extension system would accommodate both innovation whereas prese=
rving the core quality.=C2=A0<br></div><div><br></div><div>Currently we hav=
e a gigantic monolith that in practice imposes very conservative technologi=
es and changes. I suspect that is why we do not see anything really excitin=
g with Biopython for the better part of the last decade,</div></div><div cl=
ass=3D"gmail_extra"><div><div class=3D"m_6630240820384689508h5"><br><div cl=
ass=3D"gmail_quote">On 28 June 2017 at 04:25, Michiel de Hoon <span dir=3D"=
ltr">&lt;<a href=3D"mailto:[email protected]" target=3D"_blank">mjldehoon=
@yahoo.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=
=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><di=
v style=3D"color:#000;background-color:#fff;font-family:Helvetica Neue,Helv=
etica,Arial,Lucida Grande,sans-serif;font-size:10px"><div id=3D"m_663024082=
0384689508m_-5291783102745619670m_4366980446051291029yui_3_16_0_1_149864422=
5839_45791" dir=3D"ltr"><span id=3D"m_6630240820384689508m_-529178310274561=
9670m_4366980446051291029yui_3_16_0_1_1498644225839_45863">I agree with Joa=
o here. I don&#39;t see an immediate and overriding problem that modularity=
 would solve, and I can see many drawbacks.</span></div><div dir=3D"ltr"><s=
pan id=3D"m_6630240820384689508m_-5291783102745619670m_4366980446051291029y=
ui_3_16_0_1_1498644225839_45863"><br></span></div><div dir=3D"ltr"><span id=
=3D"m_6630240820384689508m_-5291783102745619670m_4366980446051291029yui_3_1=
6_0_1_1498644225839_45863">Best,</span></div><div dir=3D"ltr"><span id=3D"m=
_6630240820384689508m_-5291783102745619670m_4366980446051291029yui_3_16_0_1=
_1498644225839_45863">-Michiel</span></div> <div class=3D"m_663024082038468=
9508m_-5291783102745619670m_4366980446051291029qtdSeparateBR"><br><br></div=
><div class=3D"m_6630240820384689508m_-5291783102745619670m_436698044605129=
1029yahoo_quoted" style=3D"display:block"> <div style=3D"font-family:Helvet=
ica Neue,Helvetica,Arial,Lucida Grande,sans-serif;font-size:10px"> <div sty=
le=3D"font-family:HelveticaNeue,Helvetica Neue,Helvetica,Arial,Lucida Grand=
e,sans-serif;font-size:16px"><span> <div dir=3D"ltr"><font size=3D"2" face=
=3D"Arial"> On Monday, June 26, 2017 11:03 AM, Jo=C3=A3o Rodrigues &lt;<a h=
ref=3D"mailto:[email protected]" target=3D"_blank">j.p.g.l.m.ro=
[email protected]</a>&gt; wrote:<br></font></div>  <br><br> </span><div cla=
ss=3D"m_6630240820384689508m_-5291783102745619670m_4366980446051291029y_msg=
_container"><span><div id=3D"m_6630240820384689508m_-5291783102745619670m_4=
366980446051291029yiv5820356184"><div><div dir=3D"ltr">Copied from the othe=
r thread where I mistakenly posted:<br clear=3D"none"><br clear=3D"none">I =
think we should focus on other topics such as=20
modularity. What do the proponents of the said modularity say about it?=20
What are its advantages? I personally think a big disadvantage is that=20
with one package install you get a wide array of tools for a variety of=20
subjects. With a constellation of modules you might end up with an=20
up-to-date core and an out-of-date lone module somewhere, which makes=20
things much much harder not only to maintain but also to debug in case=20
of issues. <div class=3D"m_6630240820384689508m_-5291783102745619670m_43669=
80446051291029yiv5820356184yqt2693074091" id=3D"m_6630240820384689508m_-529=
1783102745619670m_4366980446051291029yiv5820356184yqtfd23041"><br clear=3D"=
none"><br clear=3D"none"></div></div><div class=3D"m_6630240820384689508m_-=
5291783102745619670m_4366980446051291029yiv5820356184yqt2693074091" id=3D"m=
_6630240820384689508m_-5291783102745619670m_4366980446051291029yiv582035618=
4yqtfd49977">
</div></div></div></span><span><div class=3D"m_6630240820384689508m_-529178=
3102745619670m_4366980446051291029yqt2693074091" id=3D"m_663024082038468950=
8m_-5291783102745619670m_4366980446051291029yqtfd49057">___________________=
____________________________<br clear=3D"none">Biopython-dev mailing list<b=
r clear=3D"none"><a shape=3D"rect" href=3D"mailto:[email protected]=
n-bio.org" target=3D"_blank">[email protected]</a><br clea=
r=3D"none"><a shape=3D"rect" href=3D"http://mailman.open-bio.org/mailman/li=
stinfo/biopython-dev" target=3D"_blank">http://mailman.open-bio.org/mailman=
/listinfo/biopython-dev</a></div><br><br></span></div>  </div> </div>  </di=
v></div></div></blockquote></div><br><br clear=3D"all"><div><br></div></div=
></div><span>-- <br><div class=3D"m_6630240820384689508m_-52917831027456196=
70gmail_signature" data-smartmail=3D"gmail_signature"><div dir=3D"ltr">Tiag=
o Antao<div>Scientific and HPC programmer</div><div><a href=3D"http://tiago=
.org" target=3D"_blank">http://tiago.org</a></div><div><a href=3D"https://g=
ithub.com/tiagoantao/" target=3D"_blank">https://github.com/tiagoantao/</a>=
<br></div></div></div>
</span></div>
</blockquote></div><br></div>
_______________________________________________<br>
Biopython-dev mailing list<br>
<a href=3D"mailto:[email protected]" target=3D"_blank">Bio=
[email protected]</a><br>
<a href=3D"http://mailman.open-bio.org/mailman/listinfo/biopython-dev" rel=
=3D"noreferrer" target=3D"_blank">http://mailman.open-bio.org/mailman/listi=
nfo/biopython-dev</a></blockquote></div>

--001a11411b8466aa0f05532a909c--

--===============7487400461886814071==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Biopython-dev mailing list
[email protected]
http://mailman.open-bio.org/mailman/listinfo/biopython-dev
--===============7487400461886814071==--