Re: Proposition for Biopython 2
Patrick Kunzmann <[email protected]> Wed, 22 Nov 2017 12:28:43 +0100
| Newsgroups | gmane.comp.python.bio.devel |
|---|---|
| Message-ID | <[email protected]> |
This is a multi-part message in MIME format. --------------49C6254EED0EE0068C9DF658 Content-Type: multipart/alternative; boundary="------------6C1C1D3D2ACE1A62ACC835C3" --------------6C1C1D3D2ACE1A62ACC835C3 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Hello Shyam, thank you for your interest. 1. I made a benchmark for structure superimposition and global pairwise sequence alignments. In Biopython both functions are already accelerated using C-extensions. In Biotite the superimposition is accelerated via NumPy and the alignment via a Cython module. In both cases the timing was started after the creation of the respective internal representation (Bio.PDB.Structure, Bio.Seq.Seq and biotite.structure.AtomArray, biotite.sequence.ProteinSequence). For the superimposition, the structure of lysozyme (1aki) was superimposed on itself (1000 repeats). For the alignment, two 1000-bp polyalanine sequences were used (100 repeats). The benchmark script is attached to this mail. This was the output: 1000x Biopython coordinate superimposition: 5.644912210998882 s 1000x Biotite coordinate superimposition: 0.37393549399894255 s 100x Biopython pairwise global alignment: 6.318634318999102 s 100x Biotite pairwise global alignment: 0.5369201499997871 s In both cases Biotite was one order of magnitude faster than the respective implementation in Biopython. 2. Yes, porting already written Biopython code would be extremely difficult. This was one of my reasons for eventually putting the code in a separate project. Best regards, Patrick On 21.11.2017 21:36, Shyam Saladi wrote: > Hi Patrick, > > This looks pretty cool. I wonder two things: > > 1. Do you happen to have timing benchmarks comparing similar functions > in Biotite and Biopython? I'm curious about what functionality is > actually faster/sped up by numpy/cython code. > > 2. From a cursory look through the source and from reading your docs, > biotite seems like a significantly different module compared to the > current version of BioPython. If this sort of organization was used > for a Biopython 2, it seems like it would be very difficult to port > already-written, currently developed code to the new system. That > said, for those parts that are faster, as a user, I think it might be > useful to merge them into Biopython if it's possible to do so in > an API-compatible way. I wonder what others think about this... > > Thanks, > Shyam > > On Wed, Nov 15, 2017 at 12:40 AM, Patrick Kunzmann > <[email protected] <mailto:[email protected]>> wrote: > > Dear Biopython community, > > I decided to put the proposed Biopython 2 code base into an > separate project for the time being. The main reason for this are > the clarity issues that have bothered me lately: Although > distinguishing Biopython and the proposed Biopython 2 would be > easy on GitHub (different repos) and relatively easy on PyPI > (version identifier), I think confusions could occur in other > contexts (e.g. in the mailing list or StackOverflow). Therefore, > the project is continued unter the name 'Biotite'. The repository > was moved into a separate GitHub organisation and can be found at > https://github.com/biotite-dev/biotite > <https://github.com/biotite-dev/biotite> . The project is still > licensed under BSD 3-clause so potentially a project merge is > still possible at a later point. > > Best regards, > Patrick > > > On 02.11.2017 11:42, Patrick Kunzmann wrote: > > Dear Biopython community, > > here I present you a proposition for a potential Biopython 2.x > code base. But first things first: > > A few months ago I proposed an endeavor to rewrite Biopython > in order to bring it onto modern scientific Python standards > (http://lists.open-bio.org/pipermail/biopython-dev/2017-June/021740.html > <http://lists.open-bio.org/pipermail/biopython-dev/2017-June/021740.html>). > Arguably, the consensus was that this is something that should > be done, but those changes would require almost a complete > rewrite and barely anyone has time for it. Therefore, I took > the initiative some time later and created an experimental > repository for creating actual Biopython 2 code > (https://github.com/padix-key/biopython2experimental > <https://github.com/padix-key/biopython2experimental>). > Unfortunately it seems that the announcement mail for that did > not reach the mailing list, but went missing in the deep of > the web. Anyway, the repository is now at a presentable state. > The corresponding HTML documentation (including tutorial, API > reference and install instructions) can be found under > https://github.com/padix-key/biopython2/files/1437242/doc.zip > <https://github.com/padix-key/biopython2/files/1437242/doc.zip> > . So far it is not possible to install the package from PyPI, > since it is not the offical Biopython 2 package. Instead you > have to install it directly from the repo, if you want to test > the package. > > The package contains basic types and operations for working > with structure and sequence data, offers biological database > interaction with RCSB and NCBI Entrez and provides seamless > interfaces to external software. Although the package aims to > achieve similar area of application as Biopython 1.x, it is a > complete rewrite. > > The package is still in early development. I tried to > incorporate the ideas you and I brought up in the Biopython 2 > discussion and still everything is subject to changes in the > discussion with you. I already have some questions for discussion: > > 1. Should this package still be dual licensed? Since the BSD > 3-Clause and the Biopython license are quite similar, I would > suggest licensing Biopython 2 only under BSD 3-Clause for > clarity. But I do not have a strong opinion on that. > > 2. In our previous discussion some of you proposed putting > only core functionality into Biopython 2 and leaving > specialized code installable as plugins. This package does not > contain a mechanism for plugin packages, yet. I would rather > suggest a 'recommended packages' approach: Code that is based > on Biopython 2 and tackles a general biological problem would > be linked in a 'Recommended packages' section of the Biopython > 2 documentation. In my opinion, direct plugins in the > Biopython 2 package requires some confusing namespace > wizardry. Recommended packages would achieve almost the same, > with the slight difference, that the user writes 'import > recommendedpackage' rather than 'import biopython.someplugin'. > > If this package is accepted by the community, I would like to > hand over repository ownership to the 'Biopython' organisation > on GitHub and I would like to continue and supervise its > development as part of the GitHub 'Biopython' organisation. > > Best regards, > Patrick Kunzmann > > > _______________________________________________ > Biopython-dev mailing list > [email protected] > <mailto:[email protected]> > http://mailman.open-bio.org/mailman/listinfo/biopython-dev > <http://mailman.open-bio.org/mailman/listinfo/biopython-dev> > > --------------6C1C1D3D2ACE1A62ACC835C3 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <p>Hello Shyam,</p> <p>thank you for your interest.<br> </p> <p>1. I made a benchmark for structure superimposition and global pairwise sequence alignments. In Biopython both functions are already accelerated using C-extensions. In Biotite the superimposition is accelerated via NumPy and the alignment via a Cython module. In both cases the timing was started after the creation of the respective internal representation (Bio.PDB.Structure, Bio.Seq.Seq and biotite.structure.AtomArray, biotite.sequence.ProteinSequence). For the superimposition, the structure of lysozyme (1aki) was superimposed on itself (1000 repeats). For the alignment, two 1000-bp polyalanine sequences were used (100 repeats). The benchmark script is attached to this mail. This was the output:</p> <p><tt>1000x Biopython coordinate superimposition: 5.644912210998882 s<br> 1000x Biotite coordinate superimposition: 0.37393549399894255 s<br> 100x Biopython pairwise global alignment: 6.318634318999102 s<br> 100x Biotite pairwise global alignment: 0.5369201499997871 s</tt><tt></tt></p> <p>In both cases Biotite was one order of magnitude faster than the respective implementation in Biopython.<br> </p> <p>2. Yes, porting already written Biopython code would be extremely difficult. This was one of my reasons for eventually putting the code in a separate project.</p> <p>Best regards,</p> <p>Patrick<br> </p> <br> <br> <div class="moz-cite-prefix">On 21.11.2017 21:36, Shyam Saladi wrote:<br> </div> <blockquote type="cite" cite="mid:CAARX5cWe=w2SrtgS9AGJkvmvogt+-N-vMUUWggVxybQ7JitcqA@mail.gmail.com"> <div dir="ltr">Hi Patrick, <div><br> </div> <div>This looks pretty cool. I wonder two things:</div> <div><br> </div> <div>1. Do you happen to have timing benchmarks comparing similar functions in Biotite and Biopython? I'm curious about what functionality is actually faster/sped up by numpy/cython code. </div> <div><br> </div> <div>2. From a cursory look through the source and from reading your docs, biotite seems like a significantly different module compared to the current version of BioPython. If this sort of organization was used for a Biopython 2, it seems like it would be very difficult to port already-written, currently developed code to the new system. That said, for those parts that are faster, as a user, I think it might be useful to merge them into Biopython if it's possible to do so in an API-compatible way. I wonder what others think about this...</div> <div><br> </div> <div>Thanks,</div> <div>Shyam</div> </div> <div class="gmail_extra"><br> <div class="gmail_quote">On Wed, Nov 15, 2017 at 12:40 AM, Patrick Kunzmann <span dir="ltr"><<a href="mailto:[email protected]" target="_blank" moz-do-not-send="true">[email protected]</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Dear Biopython community,<br> <br> I decided to put the proposed Biopython 2 code base into an separate project for the time being. The main reason for this are the clarity issues that have bothered me lately: Although distinguishing Biopython and the proposed Biopython 2 would be easy on GitHub (different repos) and relatively easy on PyPI (version identifier), I think confusions could occur in other contexts (e.g. in the mailing list or StackOverflow). Therefore, the project is continued unter the name 'Biotite'. The repository was moved into a separate GitHub organisation and can be found at <a href="https://github.com/biotite-dev/biotite" rel="noreferrer" target="_blank" moz-do-not-send="true">https://github.com/biotite-dev<wbr>/biotite</a> . The project is still licensed under BSD 3-clause so potentially a project merge is still possible at a later point.<br> <br> Best regards,<br> Patrick<br> <br> <br> On 02.11.2017 11:42, Patrick Kunzmann wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Dear Biopython community,<br> <br> here I present you a proposition for a potential Biopython 2.x code base. But first things first:<br> <br> A few months ago I proposed an endeavor to rewrite Biopython in order to bring it onto modern scientific Python standards (<a href="http://lists.open-bio.org/pipermail/biopython-dev/2017-June/021740.html" rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.open-bio.org/pip<wbr>ermail/biopython-dev/2017-June<wbr>/021740.html</a>). Arguably, the consensus was that this is something that should be done, but those changes would require almost a complete rewrite and barely anyone has time for it. Therefore, I took the initiative some time later and created an experimental repository for creating actual Biopython 2 code (<a href="https://github.com/padix-key/biopython2experimental" rel="noreferrer" target="_blank" moz-do-not-send="true">https://github.com/padix-key/<wbr>biopython2experimental</a>). Unfortunately it seems that the announcement mail for that did not reach the mailing list, but went missing in the deep of the web. Anyway, the repository is now at a presentable state. The corresponding HTML documentation (including tutorial, API reference and install instructions) can be found under <a href="https://github.com/padix-key/biopython2/files/1437242/doc.zip" rel="noreferrer" target="_blank" moz-do-not-send="true">https://github.com/padix-key/b<wbr>iopython2/files/1437242/doc.zi<wbr>p</a> . So far it is not possible to install the package from PyPI, since it is not the offical Biopython 2 package. Instead you have to install it directly from the repo, if you want to test the package.<br> <br> The package contains basic types and operations for working with structure and sequence data, offers biological database interaction with RCSB and NCBI Entrez and provides seamless interfaces to external software. Although the package aims to achieve similar area of application as Biopython 1.x, it is a complete rewrite.<br> <br> The package is still in early development. I tried to incorporate the ideas you and I brought up in the Biopython 2 discussion and still everything is subject to changes in the discussion with you. I already have some questions for discussion:<br> <br> 1. Should this package still be dual licensed? Since the BSD 3-Clause and the Biopython license are quite similar, I would suggest licensing Biopython 2 only under BSD 3-Clause for clarity. But I do not have a strong opinion on that.<br> <br> 2. In our previous discussion some of you proposed putting only core functionality into Biopython 2 and leaving specialized code installable as plugins. This package does not contain a mechanism for plugin packages, yet. I would rather suggest a 'recommended packages' approach: Code that is based on Biopython 2 and tackles a general biological problem would be linked in a 'Recommended packages' section of the Biopython 2 documentation. In my opinion, direct plugins in the Biopython 2 package requires some confusing namespace wizardry. Recommended packages would achieve almost the same, with the slight difference, that the user writes 'import recommendedpackage' rather than 'import biopython.someplugin'.<br> <br> If this package is accepted by the community, I would like to hand over repository ownership to the 'Biopython' organisation on GitHub and I would like to continue and supervise its development as part of the GitHub 'Biopython' organisation.<br> <br> Best regards,<br> Patrick Kunzmann<br> <br> </blockquote> <br> ______________________________<wbr>_________________<br> Biopython-dev mailing list<br> <a href="mailto:[email protected]" target="_blank" moz-do-not-send="true">[email protected]<wbr>.org</a><br> <a href="http://mailman.open-bio.org/mailman/listinfo/biopython-dev" rel="noreferrer" target="_blank" moz-do-not-send="true">http://mailman.open-bio.org/ma<wbr>ilman/listinfo/biopython-dev</a><br> </blockquote> </div> <br> </div> </blockquote> <br> </body> </html> --------------6C1C1D3D2ACE1A62ACC835C3-- --------------49C6254EED0EE0068C9DF658 Content-Type: text/x-python; name="benchmark.py" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="benchmark.py" import timeit import biotite.structure as struc import biotite.structure.io as strucio import biotite.sequence as seq import biotite.sequence.align as align from Bio.PDB.MMCIFParser import MMCIFParser from Bio.PDB.Superimposer import Superimposer from Bio.Seq import Seq from Bio import pairwise2 parser = MMCIFParser() structure = parser.get_structure("1aki", "1aki.cif") def biopython_superimpose(): fixed = list(structure.get_atoms()) moving = list(structure.get_atoms()) imposer = Superimposer() imposer.set_atoms(fixed, moving) imposer.apply(moving) print("1000x Biopython coordinate superimposition:", timeit.timeit(biopython_superimpose, number=1000), "s") array = strucio.get_structure_from("1aki.cif") def biotite_superimpose(): imposed, transform = struc.superimpose(array, array, ca_only=False) rmsd = struc.rmsd(array, imposed) print("1000x Biotite coordinate superimposition:", timeit.timeit(biotite_superimpose, number=1000), "s") poly_ala_1 = Seq("A" * 1000) poly_ala_2 = Seq("A" * 1000) def biopython_align(): pairwise2.align.globalxx(poly_ala_1, poly_ala_2) print("100x Biopython pairwise global alignment:", timeit.timeit(biopython_align, number=100), "s") poly_ala_1 = seq.ProteinSequence("A" * 1000) poly_ala_2 = seq.ProteinSequence("A" * 1000) matrix = align.SubstitutionMatrix.std_protein_matrix() def biotite_align(): align.align_optimal(poly_ala_1, poly_ala_2, matrix) print("100x Biotite pairwise global alignment:", timeit.timeit(biotite_align, number=100), "s") --------------49C6254EED0EE0068C9DF658 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Biopython-dev mailing list [email protected] http://mailman.open-bio.org/mailman/listinfo/biopython-dev --------------49C6254EED0EE0068C9DF658--