Re: [Biopython] A possibility for speeding up FASTA/FASTQ reading in BioPython
Michiel de Hoon <[email protected]> Fri, 14 Nov 2025 00:17:04 +0000 (UTC)
| Newsgroups | gmane.comp.python.bio.general |
|---|---|
| Message-ID | <[email protected]> |
--===============6177179582732244821== Content-Type: multipart/alternative; boundary="----=_Part_7973658_348555842.1763079424255" ------=_Part_7973658_348555842.1763079424255 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Dear Terry, Thank you for contributing this code. A faster Fasta/Fastq parser would be = very welcome.Have you done any timings to find out why the current Fasta pa= rser is slower?My guess is that this is because of the SeqRecord constructo= r, which can create complex SeqRecord objects with annotations, letter_anno= tations and whatnot, while Fasta only needs the id attribute. Thanks,-Michiel On Wednesday, November 12, 2025 at 11:49:37 AM GMT+9, Jo= nes Kelly, Terence Carleton <[email protected]> wrote: =20 =20 Hi all I regularly process reasonably large FASTQ (hundreds of billions of sequenc= ing reads) and FASTA files using BioPython. For some years I've been meanin= g to implement a FASTQ/FASTA reader in a compiled language and add Python b= indings to improve the speed. I could've done this in C but I spent some de= cades writing C and I wanted to learn something new, so I considered a few = languages. Because Rust makes it very easy to create Python bindings, I dec= ided to give it a try. I thought I'd get going by asking the Claude CLI to = write me some Rust. That turned out to be a much, much better experience th= an I had anticipated. With Claude I played with several implementations, ke= eping track of timing. Claudealso=C2=A0wrote some tests. To compare what I = was seeing I got Claude to write a pure Python version, a pure C version, P= ython bindings to the C, and to create a benchmark suite. From what I can t= ell, the Rust/Python (and the C/Python) FASTA reading is twice as fast as B= ioPython and FASTQ reading is four times as fast.I didn't write a single li= ne of code. I just did some minimal cleaning up when things were already fa= r along.=C2=A0I've been using the code for the last month or two with no pr= oblems. The repo is at https://github.com/VirologyCharite/prseq=C2=A0 (prseq =3D Py= thon/Rust for sequences). You'll find the benchmark results on that page. = =C2=A0There are still some small things I would adjust in the API. =C2=A0BT= W, Claude also wrote the README (which should definitely be improved). I am wondering if there might be interest in incorporating this into BioPyt= hon. I don't know if there are any Rust dependencies in BioPython but I kno= w that there are some C extensions. We could use either, as their speeds ar= e comparable. If there's interest, I'd be happy to help (or to do it all, a= fter some discussion and maybe with some guidance). Thanks very much for all the work on BioPython. It's really been a pleasure= to use the code over the last dozen years or so. Terry Jones _______________________________________________ Biopython mailing list=C2=A0 -=C2=A0 [email protected] https://mailman.open-bio.org/mailman/listinfo/biopython =20 ------=_Part_7973658_348555842.1763079424255 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit <html><head></head><body><div class="ydp57a6f215yahoo-style-wrap" style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:10px;"><div></div> <div dir="ltr" data-setdir="false">Dear Terry,</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Thank you for contributing this code. A faster Fasta/Fastq parser would be very welcome.</div><div dir="ltr" data-setdir="false">Have you done any timings to find out why the current Fasta parser is slower?</div><div dir="ltr" data-setdir="false">My guess is that this is because of the SeqRecord constructor, which can create complex SeqRecord objects with annotations, letter_annotations and whatnot, while Fasta only needs the id attribute.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Thanks,</div><div dir="ltr" data-setdir="false">-Michiel</div> </div><div id="yahoo_quoted_4006301043" class="yahoo_quoted"> <div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;"> <div> On Wednesday, November 12, 2025 at 11:49:37 AM GMT+9, Jones Kelly, Terence Carleton <[email protected]> wrote: </div> <div><br></div> <div><br></div> <div><div id="yiv0139152588"> <div> <div style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);"> Hi all</div> <div dir="ltr" style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);"> <br> </div> <div><span style="font-size:16px;">I regularly process reasonably large FASTQ (hundreds of billions of sequencing reads) and FASTA files using BioPython. For some years I've been meaning to implement a FASTQ/FASTA reader in a compiled language and add Python bindings to improve the speed. I could've done this in C but I spent some decades writing C and I wanted to learn something new, so I considered a few languages. Because Rust makes it very easy to create Python bindings, I decided to give it a try. I thought I'd get going by asking the Claude CLI to write me some Rust. That turned out to be a much, much better experience than I had anticipated. With Claude I played with several implementations, keeping track of timing. Claude </span>also<span style="font-size:16px;"> wrote some tests. To compare what I was seeing I got Claude to write a pure Python version, a pure C version, Python bindings to the C, and to create a benchmark suite. From what I can tell, the Rust/Python (and the C/Python) FASTA reading is twice as fast as BioPython and FASTQ reading is four times as fast. </span><span style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:16px;color:rgb(0, 0, 0);background-color:rgb(255, 255, 255);">I didn't write a single line of code. I just did some minimal cleaning up when things were already far along.</span><span style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);"> I've been using the code for the last month or two with no problems.</span></div> <div dir="ltr" style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);"> <br> </div> <div style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);"> The repo is at <a rel="nofollow noopener noreferrer" target="_blank" href="https://github.com/VirologyCharite/prseq"> https://github.com/VirologyCharite/prseq</a> (prseq = Python/Rust for sequences). You'll find the benchmark results on that page. There are still some small things I would adjust in the API. BTW, Claude also wrote the README (which should definitely be improved).</div> <div dir="ltr" style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);"> <br> </div> <div dir="ltr" style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:16px;color:rgb(0, 0, 0);"> <span style="background-color:rgb(255, 255, 255);">I am wondering if there might be interest in incorporating this into BioPython. I don't know if there are any Rust dependencies in BioPython but I know that there are some C extensions. We could use either, as their speeds are comparable. </span><span style="font-size:12pt;">If there's interest, I'd be happy to help (or to do it all, after some discussion and maybe with some guidance).</span></div> <div dir="ltr" style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);"> <br> </div> <div style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);"> Thanks very much for all the work on BioPython. It's really been a pleasure to use the code over the last dozen years or so.</div> <div dir="ltr" style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);"> <br> </div> <div style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);"> Terry Jones</div> <div dir="ltr" style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);"> <br> </div> <div dir="ltr" style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);"> <br> </div> </div> </div>_______________________________________________<br>Biopython mailing list - <a ymailto="mailto:[email protected]" href="mailto:[email protected]">[email protected]</a><br><a href="https://mailman.open-bio.org/mailman/listinfo/biopython" target="_blank">https://mailman.open-bio.org/mailman/listinfo/biopython</a><br></div> </div> </div></body></html> ------=_Part_7973658_348555842.1763079424255-- --===============6177179582732244821== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Biopython mailing list - [email protected] https://mailman.open-bio.org/mailman/listinfo/biopython --===============6177179582732244821==--