[Biopython] A possibility for speeding up FASTA/FASTQ reading in BioPython
"Jones Kelly, Terence Carleton" <[email protected]> Tue, 11 Nov 2025 21:51:45 +0000
| Newsgroups | gmane.comp.python.bio.general |
|---|---|
| Message-ID | <[email protected]> |
--===============3569306717981501159== Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_e4626d66443342ba85127d98d91bb3a0charitede_" --_000_e4626d66443342ba85127d98d91bb3a0charitede_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi all I regularly process reasonably large FASTQ (hundreds of billions of sequenc= ing reads) and FASTA files using BioPython. For some years I've been meanin= g to implement a FASTQ/FASTA reader in a compiled language and add Python b= indings to improve the speed. I could've done this in C but I spent some de= cades writing C and I wanted to learn something new, so I considered a few = languages. Because Rust makes it very easy to create Python bindings, I dec= ided to give it a try. I thought I'd get going by asking the Claude CLI to = write me some Rust. That turned out to be a much, much better experience th= an I had anticipated. With Claude I played with several implementations, ke= eping track of timing. Claude also wrote some tests. To compare what I was = seeing I got Claude to write a pure Python version, a pure C version, Pytho= n bindings to the C, and to create a benchmark suite. From what I can tell,= the Rust/Python (and the C/Python) FASTA reading is twice as fast as BioPy= thon and FASTQ reading is four times as fast. I didn't write a single line = of code. I just did some minimal cleaning up when things were already far a= long. I've been using the code for the last month or two with no problems. The repo is at https://github.com/VirologyCharite/prseq (prseq =3D Python/= Rust for sequences). You'll find the benchmark results on that page. There= are still some small things I would adjust in the API. BTW, Claude also w= rote the README (which should definitely be improved). I am wondering if there might be interest in incorporating this into BioPyt= hon. I don't know if there are any Rust dependencies in BioPython but I kno= w that there are some C extensions. We could use either, as their speeds ar= e comparable. If there's interest, I'd be happy to help (or to do it all, a= fter some discussion and maybe with some guidance). Thanks very much for all the work on BioPython. It's really been a pleasure= to use the code over the last dozen years or so. Terry Jones --_000_e4626d66443342ba85127d98d91bb3a0charitede_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"= > </head> <body> <div style=3D"font-family: Aptos, Arial, Helvetica, sans-serif; font-size: = 12pt; color: rgb(0, 0, 0);"> Hi all</div> <div dir=3D"ltr" style=3D"font-family: Aptos, Arial, Helvetica, sans-serif;= font-size: 12pt; color: rgb(0, 0, 0);"> <br> </div> <div><span style=3D"font-size: 16px;">I regularly process reasonably large = FASTQ (hundreds of billions of sequencing reads) and FASTA files using BioP= ython. For some years I've been meaning to implement a FASTQ/FASTA reader i= n a compiled language and add Python bindings to improve the speed. I could've done this in C but I spent some = decades writing C and I wanted to learn something new, so I considered a fe= w languages. Because Rust makes it very easy to create Python bindings, I d= ecided to give it a try. I thought I'd get going by asking the Claude CLI to write me some Rust. That turned = out to be a much, much better experience than I had anticipated. With Claud= e I played with several implementations, keeping track of timing. Claude </span>also<span style=3D"font-size: 16px;"> wrote some tests. To comp= are what I was seeing I got Claude to write a pure Python version, a pure C= version, Python bindings to the C, and to create a benchmark suite. From w= hat I can tell, the Rust/Python (and the C/Python) FASTA reading is twice as fast as BioPython and FASTQ reading is= four times as fast. </span><span style=3D"font-family: Aptos, Arial, Helvetica, sans-serif; fon= t-size: 16px; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">I= didn't write a single line of code. I just did some minimal cleaning up wh= en things were already far along.</span><span style=3D"font-family: Aptos, = Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> = I've been using the code for the last month or two with no problems.</span></di= v> <div dir=3D"ltr" style=3D"font-family: Aptos, Arial, Helvetica, sans-serif;= font-size: 12pt; color: rgb(0, 0, 0);"> <br> </div> <div style=3D"font-family: Aptos, Arial, Helvetica, sans-serif; font-size: = 12pt; color: rgb(0, 0, 0);"> The repo is at <a href=3D"https://github.com/VirologyCharite/prseq" data-ou= tlook-id=3D"b95749ed-030e-4407-8496-334c7f335e75"> https://github.com/VirologyCharite/prseq</a> (prseq =3D Python/Rust f= or sequences). You'll find the benchmark results on that page. There = are still some small things I would adjust in the API. BTW, Claude al= so wrote the README (which should definitely be improved).</div> <div dir=3D"ltr" style=3D"font-family: Aptos, Arial, Helvetica, sans-serif;= font-size: 12pt; color: rgb(0, 0, 0);"> <br> </div> <div dir=3D"ltr" style=3D"font-family: Aptos, Arial, Helvetica, sans-serif;= font-size: 16px; color: rgb(0, 0, 0);"> <span style=3D"background-color: rgb(255, 255, 255);">I am wondering if the= re might be interest in incorporating this into BioPython. I don't know if = there are any Rust dependencies in BioPython but I know that there are some= C extensions. We could use either, as their speeds are comparable. </span><span style=3D"font-size: 12pt;">If= there's interest, I'd be happy to help (or to do it all, after some discus= sion and maybe with some guidance).</span></div> <div dir=3D"ltr" style=3D"font-family: Aptos, Arial, Helvetica, sans-serif;= font-size: 12pt; color: rgb(0, 0, 0);"> <br> </div> <div style=3D"font-family: Aptos, Arial, Helvetica, sans-serif; font-size: = 12pt; color: rgb(0, 0, 0);"> Thanks very much for all the work on BioPython. It's really been a pleasure= to use the code over the last dozen years or so.</div> <div dir=3D"ltr" style=3D"font-family: Aptos, Arial, Helvetica, sans-serif;= font-size: 12pt; color: rgb(0, 0, 0);"> <br> </div> <div style=3D"font-family: Aptos, Arial, Helvetica, sans-serif; font-size: = 12pt; color: rgb(0, 0, 0);"> Terry Jones</div> <div dir=3D"ltr" style=3D"font-family: Aptos, Arial, Helvetica, sans-serif;= font-size: 12pt; color: rgb(0, 0, 0);"> <br> </div> <div dir=3D"ltr" style=3D"font-family: Aptos, Arial, Helvetica, sans-serif;= font-size: 12pt; color: rgb(0, 0, 0);"> <br> </div> </body> </html> --_000_e4626d66443342ba85127d98d91bb3a0charitede_-- --===============3569306717981501159== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Biopython mailing list - [email protected] https://mailman.open-bio.org/mailman/listinfo/biopython --===============3569306717981501159==--