Re: Ideas for Biopython 2.0

--===============3693099315252760816==
Content-Type: multipart/alternative; boundary="089e08e5bdcdfc705d0553081669"

--089e08e5bdcdfc705d0553081669
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

So, in theory, you can end up with dozens of extensions that are out of
sync with the core in terms of 'core' dependencies and it's up to the
maintainers of those extensions to keep them up-to-date. That's my main
concern. We have a monolith that works. It is conservative and that's
exactly why it works. Otherwise, for example, I would install the core and
your Cython-based lightining fast PopGen code and then you grow bored of
it, a new version of Cython comes along that has some incompatibilities
with your code, and now it doesn't work. Worse even, I release a blinding
fast PDBParser using a new feature in Cython and now you have two
incompatible extensions.

Being very honest, there is no barrier to anyone developing anything new
here. If you want to use Cython, go ahead and do it and then worst case
scenario, we have a dependency warning and check that can simply skip
compilation of that code. It's what happens with Numpy at the moment IIRC.
You can also bundle the c code, instead of compiling the pyx on install,
and compile it regularly using GCC, skipping Cython altogether.

I agree that we should 'refresh' our dependencies to be able to do cool
things. And modularity is incredibly attractive. But the main advantage of
modularity - lowering the standards of biopython code - is also, in my
opinion, it's main disadvantage.

Question: How is scikit handling their different modules?

2017-06-28 8:05 GMT-07:00 Tiago Ant=C3=A3o <[email protected]>:

> It can see plenty of issues where it could help. In my specific case all
> the PopGen code is stopped for 10 years because I would need to write ver=
y
> fast code (say in Cython). This would be an extension module, not a core
> module because it would impart a very big dependency on the system.
>
> Modules would allow a core with very strict policies and dependencies
> _but_ extensions that could be way more relaxed.
>
> It would also lower the barrier of entry for new content. Everyone could
> publish an extension. If the extension would survive time (which most do
> not - creating a maintenance burden in the core) then it could eventually
> be made a core extension. Now the policy in practice is to add very littl=
e
> innovation out of the fear that it will become stagnant and not-supported
> by the main author (say after publication). An extension system would
> accommodate both innovation whereas preserving the core quality.
>
> Currently we have a gigantic monolith that in practice imposes very
> conservative technologies and changes. I suspect that is why we do not se=
e
> anything really exciting with Biopython for the better part of the last
> decade,
>
> On 28 June 2017 at 04:25, Michiel de Hoon <[email protected]> wrote:
>
>> I agree with Joao here. I don't see an immediate and overriding problem
>> that modularity would solve, and I can see many drawbacks.
>>
>> Best,
>> -Michiel
>>
>>
>> On Monday, June 26, 2017 11:03 AM, Jo=C3=A3o Rodrigues <
>> [email protected]> wrote:
>>
>>
>> Copied from the other thread where I mistakenly posted:
>>
>> I think we should focus on other topics such as modularity. What do the
>> proponents of the said modularity say about it? What are its advantages?=
 I
>> personally think a big disadvantage is that with one package install you
>> get a wide array of tools for a variety of subjects. With a constellatio=
n
>> of modules you might end up with an up-to-date core and an out-of-date l=
one
>> module somewhere, which makes things much much harder not only to mainta=
in
>> but also to debug in case of issues.
>>
>>
>> _______________________________________________
>> Biopython-dev mailing list
>> [email protected]
>> http://mailman.open-bio.org/mailman/listinfo/biopython-dev
>>
>>
>>
>
>
> --
> Tiago Antao
> Scientific and HPC programmer
> http://tiago.org
> https://github.com/tiagoantao/
>

--089e08e5bdcdfc705d0553081669
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">So, in theory, you can end up with dozens of extensions th=
at are out of sync with the core in terms of &#39;core&#39; dependencies an=
d it&#39;s up to the maintainers of those extensions to keep them up-to-dat=
e. That&#39;s my main concern. We have a monolith that works. It is conserv=
ative and that&#39;s exactly why it works. Otherwise, for example, I would =
install the core and your Cython-based lightining fast PopGen code and then=
 you grow bored of it, a new version of Cython comes along that has some in=
compatibilities with your code, and now it doesn&#39;t work. Worse even, I =
release a blinding fast PDBParser using a new feature in Cython and now you=
 have two incompatible extensions.=C2=A0<div><br></div><div>Being very hone=
st, there is no barrier to anyone developing anything new here. If you want=
 to use Cython, go ahead and do it and then worst case scenario, we have a =
dependency warning and check that can simply skip compilation of that code.=
 It&#39;s what happens with Numpy at the moment IIRC. You can also bundle t=
he c code, instead of compiling the pyx on install, and compile it regularl=
y using GCC, skipping Cython altogether.=C2=A0</div><div><br></div><div>I a=
gree that we should &#39;refresh&#39; our dependencies to be able to do coo=
l things. And modularity is incredibly attractive. But the main advantage o=
f modularity - lowering the standards of biopython code - is also, in my op=
inion, it&#39;s main disadvantage.=C2=A0</div><div><br></div><div>Question:=
 How is scikit handling their different modules?</div></div><div class=3D"g=
mail_extra"><br><div class=3D"gmail_quote">2017-06-28 8:05 GMT-07:00 Tiago =
Ant=C3=A3o <span dir=3D"ltr">&lt;<a href=3D"mailto:[email protected]" ta=
rget=3D"_blank">[email protected]</a>&gt;</span>:<br><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex"><div dir=3D"ltr">It can see plenty of issues where it could h=
elp. In my specific case all the PopGen code is stopped for 10 years becaus=
e I would need to write very fast code (say in Cython). This would be an ex=
tension module, not a core module because it would impart a very big depend=
ency on the system.<div><br></div><div>Modules would allow a core with very=
 strict policies and dependencies _but_ extensions that could be way more r=
elaxed.</div><div><br></div><div>It would also lower the barrier of entry f=
or new content. Everyone could publish an extension. If the extension would=
 survive time (which most do not - creating a maintenance burden in the cor=
e) then it could eventually be made a core extension. Now the policy in pra=
ctice is to add very little innovation out of the fear that it will become =
stagnant and not-supported by the main author (say after publication). An e=
xtension system would accommodate both innovation whereas preserving the co=
re quality.=C2=A0<br></div><div><br></div><div>Currently we have a gigantic=
 monolith that in practice imposes very conservative technologies and chang=
es. I suspect that is why we do not see anything really exciting with Biopy=
thon for the better part of the last decade,</div></div><div class=3D"gmail=
_extra"><div><div class=3D"h5"><br><div class=3D"gmail_quote">On 28 June 20=
17 at 04:25, Michiel de Hoon <span dir=3D"ltr">&lt;<a href=3D"mailto:mjldeh=
[email protected]" target=3D"_blank">[email protected]</a>&gt;</span> wrote:<=
br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left=
:1px #ccc solid;padding-left:1ex"><div><div style=3D"color:#000;background-=
color:#fff;font-family:Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-se=
rif;font-size:10px"><div id=3D"m_-5291783102745619670m_4366980446051291029y=
ui_3_16_0_1_1498644225839_45791" dir=3D"ltr"><span id=3D"m_-529178310274561=
9670m_4366980446051291029yui_3_16_0_1_1498644225839_45863">I agree with Joa=
o here. I don&#39;t see an immediate and overriding problem that modularity=
 would solve, and I can see many drawbacks.</span></div><div dir=3D"ltr"><s=
pan id=3D"m_-5291783102745619670m_4366980446051291029yui_3_16_0_1_149864422=
5839_45863"><br></span></div><div dir=3D"ltr"><span id=3D"m_-52917831027456=
19670m_4366980446051291029yui_3_16_0_1_1498644225839_45863">Best,</span></d=
iv><div dir=3D"ltr"><span id=3D"m_-5291783102745619670m_4366980446051291029=
yui_3_16_0_1_1498644225839_45863">-Michiel</span></div> <div class=3D"m_-52=
91783102745619670m_4366980446051291029qtdSeparateBR"><br><br></div><div cla=
ss=3D"m_-5291783102745619670m_4366980446051291029yahoo_quoted" style=3D"dis=
play:block"> <div style=3D"font-family:Helvetica Neue,Helvetica,Arial,Lucid=
a Grande,sans-serif;font-size:10px"> <div style=3D"font-family:HelveticaNeu=
e,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif;font-size:16px"><=
span> <div dir=3D"ltr"><font size=3D"2" face=3D"Arial"> On Monday, June 26,=
 2017 11:03 AM, Jo=C3=A3o Rodrigues &lt;<a href=3D"mailto:j.p.g.l.m.rodrigu=
[email protected]" target=3D"_blank">[email protected]</a><wbr>&gt; =
wrote:<br></font></div>  <br><br> </span><div class=3D"m_-52917831027456196=
70m_4366980446051291029y_msg_container"><span><div id=3D"m_-529178310274561=
9670m_4366980446051291029yiv5820356184"><div><div dir=3D"ltr">Copied from t=
he other thread where I mistakenly posted:<br clear=3D"none"><br clear=3D"n=
one">I think we should focus on other topics such as=20
modularity. What do the proponents of the said modularity say about it?=20
What are its advantages? I personally think a big disadvantage is that=20
with one package install you get a wide array of tools for a variety of=20
subjects. With a constellation of modules you might end up with an=20
up-to-date core and an out-of-date lone module somewhere, which makes=20
things much much harder not only to maintain but also to debug in case=20
of issues. <div class=3D"m_-5291783102745619670m_4366980446051291029yiv5820=
356184yqt2693074091" id=3D"m_-5291783102745619670m_4366980446051291029yiv58=
20356184yqtfd23041"><br clear=3D"none"><br clear=3D"none"></div></div><div =
class=3D"m_-5291783102745619670m_4366980446051291029yiv5820356184yqt2693074=
091" id=3D"m_-5291783102745619670m_4366980446051291029yiv5820356184yqtfd499=
77">
</div></div></div></span><span><div class=3D"m_-5291783102745619670m_436698=
0446051291029yqt2693074091" id=3D"m_-5291783102745619670m_43669804460512910=
29yqtfd49057">______________________________<wbr>_________________<br clear=
=3D"none">Biopython-dev mailing list<br clear=3D"none"><a shape=3D"rect" hr=
ef=3D"mailto:[email protected]" target=3D"_blank">Biopytho=
[email protected]<wbr>.org</a><br clear=3D"none"><a shape=3D"rect" hre=
f=3D"http://mailman.open-bio.org/mailman/listinfo/biopython-dev" target=3D"=
_blank">http://mailman.open-bio.org/ma<wbr>ilman/listinfo/biopython-dev</a>=
</div><br><br></span></div>  </div> </div>  </div></div></div></blockquote>=
</div><br><br clear=3D"all"><div><br></div></div></div><span class=3D"">-- =
<br><div class=3D"m_-5291783102745619670gmail_signature" data-smartmail=3D"=
gmail_signature"><div dir=3D"ltr">Tiago Antao<div>Scientific and HPC progra=
mmer</div><div><a href=3D"http://tiago.org" target=3D"_blank">http://tiago.=
org</a></div><div><a href=3D"https://github.com/tiagoantao/" target=3D"_bla=
nk">https://github.com/tiagoantao/</a><br></div></div></div>
</span></div>
</blockquote></div><br></div>

--089e08e5bdcdfc705d0553081669--

--===============3693099315252760816==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Biopython-dev mailing list
[email protected]
http://mailman.open-bio.org/mailman/listinfo/biopython-dev
--===============3693099315252760816==--