Re: [Biopython] A possibility for speeding up FASTA/FASTQ reading in BioPython

--===============7390494971566692494==
Content-Type: multipart/alternative; boundary="00000000000081bb7006445dd47c"

--00000000000081bb7006445dd47c
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On this topic, using an index or an alternative file format would be my
first thought for speed. Any decent benchmarks for different access
patterns / file / index formats out there?



On Mon, Nov 24, 2025, 9:41=E2=80=AFAM Peter Cock <[email protected]=
> wrote:

> Hello Terry,
>
> I just posted a blog about my thoughts on receiving generative AI
> contributions as an Open Source project maintainer:
>
>
> https://blastedbio.blogspot.com/2025/11/thoughts-on-generative-ai-contrib=
utions.html
>
> I am sceptical, and in this case adding a Rust dependency to Biopython
> seems too much to ask. I think you could get similar performance gains
> with C (which we do use) where at least the maintainers have some
> experience. However, even there, gains may not make the additional
> complexity and maintenance burden worthwhile.
>
> Thank you for writting and asking, rather than suprising everyone with
> a large pull request.
>
> Peter
>
> P.S. Cross reference https://github.com/biopython/biopython/pull/5085
>
> On Tue, Nov 11, 2025 at 10:00=E2=80=AFPM Jones Kelly, Terence Carleton
> <[email protected]> wrote:
> >
> > Hi all
> >
> > I regularly process reasonably large FASTQ (hundreds of billions of
> sequencing reads) and FASTA files using BioPython. For some years I've be=
en
> meaning to implement a FASTQ/FASTA reader in a compiled language and add
> Python bindings to improve the speed. I could've done this in C but I spe=
nt
> some decades writing C and I wanted to learn something new, so I consider=
ed
> a few languages. Because Rust makes it very easy to create Python binding=
s,
> I decided to give it a try. I thought I'd get going by asking the Claude
> CLI to write me some Rust. That turned out to be a much, much better
> experience than I had anticipated. With Claude I played with several
> implementations, keeping track of timing. Claude also wrote some tests. T=
o
> compare what I was seeing I got Claude to write a pure Python version, a
> pure C version, Python bindings to the C, and to create a benchmark suite=
.
> From what I can tell, the Rust/Python (and the C/Python) FASTA reading is
> twice as fast as BioPython and FASTQ reading is four times as fast. I
> didn't write a single line of code. I just did some minimal cleaning up
> when things were already far along. I've been using the code for the last
> month or two with no problems.
> >
> > The repo is at https://github.com/VirologyCharite/prseq  (prseq =3D
> Python/Rust for sequences). You'll find the benchmark results on that
> page.  There are still some small things I would adjust in the API.  BTW,
> Claude also wrote the README (which should definitely be improved).
> >
> > I am wondering if there might be interest in incorporating this into
> BioPython. I don't know if there are any Rust dependencies in BioPython b=
ut
> I know that there are some C extensions. We could use either, as their
> speeds are comparable. If there's interest, I'd be happy to help (or to d=
o
> it all, after some discussion and maybe with some guidance).
> >
> > Thanks very much for all the work on BioPython. It's really been a
> pleasure to use the code over the last dozen years or so.
> >
> > Terry Jones
> >
> >
> > _______________________________________________
> > Biopython mailing list  -  [email protected]
> > https://mailman.open-bio.org/mailman/listinfo/biopython
> _______________________________________________
> Biopython mailing list  -  [email protected]
> https://mailman.open-bio.org/mailman/listinfo/biopython
>

--00000000000081bb7006445dd47c
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"auto">On this topic, using an index or an alternative file form=
at would be my first thought for speed. Any decent benchmarks for different=
 access patterns / file / index formats out there?<div dir=3D"auto"><br></d=
iv><div dir=3D"auto"><br></div></div><br><div class=3D"gmail_quote gmail_qu=
ote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, Nov 24, 2025, =
9:41=E2=80=AFAM Peter Cock &lt;<a href=3D"mailto:[email protected]"=
>[email protected]</a>&gt; wrote:<br></div><blockquote class=3D"gma=
il_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-lef=
t:1ex">Hello Terry,<br>
<br>
I just posted a blog about my thoughts on receiving generative AI<br>
contributions as an Open Source project maintainer:<br>
<br>
<a href=3D"https://blastedbio.blogspot.com/2025/11/thoughts-on-generative-a=
i-contributions.html" rel=3D"noreferrer noreferrer" target=3D"_blank">https=
://blastedbio.blogspot.com/2025/11/thoughts-on-generative-ai-contributions.=
html</a><br>
<br>
I am sceptical, and in this case adding a Rust dependency to Biopython<br>
seems too much to ask. I think you could get similar performance gains<br>
with C (which we do use) where at least the maintainers have some<br>
experience. However, even there, gains may not make the additional<br>
complexity and maintenance burden worthwhile.<br>
<br>
Thank you for writting and asking, rather than suprising everyone with<br>
a large pull request.<br>
<br>
Peter<br>
<br>
P.S. Cross reference <a href=3D"https://github.com/biopython/biopython/pull=
/5085" rel=3D"noreferrer noreferrer" target=3D"_blank">https://github.com/b=
iopython/biopython/pull/5085</a><br>
<br>
On Tue, Nov 11, 2025 at 10:00=E2=80=AFPM Jones Kelly, Terence Carleton<br>
&lt;<a href=3D"mailto:[email protected]" target=3D"_blank" rel=3D"no=
referrer">[email protected]</a>&gt; wrote:<br>
&gt;<br>
&gt; Hi all<br>
&gt;<br>
&gt; I regularly process reasonably large FASTQ (hundreds of billions of se=
quencing reads) and FASTA files using BioPython. For some years I&#39;ve be=
en meaning to implement a FASTQ/FASTA reader in a compiled language and add=
 Python bindings to improve the speed. I could&#39;ve done this in C but I =
spent some decades writing C and I wanted to learn something new, so I cons=
idered a few languages. Because Rust makes it very easy to create Python bi=
ndings, I decided to give it a try. I thought I&#39;d get going by asking t=
he Claude CLI to write me some Rust. That turned out to be a much, much bet=
ter experience than I had anticipated. With Claude I played with several im=
plementations, keeping track of timing. Claude also wrote some tests. To co=
mpare what I was seeing I got Claude to write a pure Python version, a pure=
 C version, Python bindings to the C, and to create a benchmark suite. From=
 what I can tell, the Rust/Python (and the C/Python) FASTA reading is twice=
 as fast as BioPython and FASTQ reading is four times as fast. I didn&#39;t=
 write a single line of code. I just did some minimal cleaning up when thin=
gs were already far along. I&#39;ve been using the code for the last month =
or two with no problems.<br>
&gt;<br>
&gt; The repo is at <a href=3D"https://github.com/VirologyCharite/prseq" re=
l=3D"noreferrer noreferrer" target=3D"_blank">https://github.com/VirologyCh=
arite/prseq</a>=C2=A0 (prseq =3D Python/Rust for sequences). You&#39;ll fin=
d the benchmark results on that page.=C2=A0 There are still some small thin=
gs I would adjust in the API.=C2=A0 BTW, Claude also wrote the README (whic=
h should definitely be improved).<br>
&gt;<br>
&gt; I am wondering if there might be interest in incorporating this into B=
ioPython. I don&#39;t know if there are any Rust dependencies in BioPython =
but I know that there are some C extensions. We could use either, as their =
speeds are comparable. If there&#39;s interest, I&#39;d be happy to help (o=
r to do it all, after some discussion and maybe with some guidance).<br>
&gt;<br>
&gt; Thanks very much for all the work on BioPython. It&#39;s really been a=
 pleasure to use the code over the last dozen years or so.<br>
&gt;<br>
&gt; Terry Jones<br>
&gt;<br>
&gt;<br>
&gt; _______________________________________________<br>
&gt; Biopython mailing list=C2=A0 -=C2=A0 <a href=3D"mailto:Biopython@biopy=
thon.org" target=3D"_blank" rel=3D"noreferrer">[email protected]</a><=
br>
&gt; <a href=3D"https://mailman.open-bio.org/mailman/listinfo/biopython" re=
l=3D"noreferrer noreferrer" target=3D"_blank">https://mailman.open-bio.org/=
mailman/listinfo/biopython</a><br>
_______________________________________________<br>
Biopython mailing list=C2=A0 -=C2=A0 <a href=3D"mailto:Biopython@biopython.=
org" target=3D"_blank" rel=3D"noreferrer">[email protected]</a><br>
<a href=3D"https://mailman.open-bio.org/mailman/listinfo/biopython" rel=3D"=
noreferrer noreferrer" target=3D"_blank">https://mailman.open-bio.org/mailm=
an/listinfo/biopython</a><br>
</blockquote></div>

--00000000000081bb7006445dd47c--

--===============7390494971566692494==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Biopython mailing list  -  [email protected]
https://mailman.open-bio.org/mailman/listinfo/biopython

--===============7390494971566692494==--