Re: [Biopython] [ext] Re: A possibility for speeding up FASTA/FASTQ reading in BioPython
Michiel de Hoon <[email protected]> Fri, 28 Nov 2025 00:47:10 +0000 (UTC)
| Newsgroups | gmane.comp.python.bio.general |
|---|---|
| Message-ID | <[email protected]> |
--===============1325055932245067553==
Content-Type: multipart/alternative;
boundary="----=_Part_1876149_246379205.1764290830958"
------=_Part_1876149_246379205.1764290830958
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Hi all,
You may also want to try reading the Fasta/Fastq file in larger chunks, rat=
her than line-by-line. I have tried this in the past, and it gave me much b=
etter performance.
This may be a good opportunity to try a SeqRecord-free parser, i.e. a parse=
r that returns Seq objects instead of SeqRecord objects. I never quite unde=
rstood why we have SeqRecords alongside Seq objects.
Best,-MIchiel
On Wednesday, November 26, 2025 at 06:38:52 PM GMT+9, Peter Cock <p.j.a=
[email protected]> wrote: =20
=20
Hi Terry,
That's a very even handed reply. I hadn't considered speaking to AI
tools with Voice recognition, but Amazon Alexa isn't great and has
probably lowered my expectations - grin.
The SeqRecord overhead is big, especially making the ASCII quality
string into a list of integers. You often don't need this, and can
stick with strings. See eg:
https://www.open-bio.org/2009/09/25/biopython-fast-fastq/
In the short term, a much easier experiment to try is switching our
Pure Python FASTQ parser to work in bytes mode (binary handles) to
avoid the bytes to unicode conversion of the sequence and quality
strings.
Peter
On Tue, Nov 25, 2025 at 7:56=E2=80=AFPM Jones Kelly, Terence Carleton
<[email protected]> wrote:
>
> Hi all
>
> Thanks for the replies and I'm sorry to be so slow in replying.
>
> Peter - I=E2=80=99m sceptical, too.=C2=A0 I thought it highly unlikely th=
at BioPython would want to add a dependence on Rust. The C code could be us=
ed to get the same speed-up. The automatic detection of compressed input (i=
ncluding on stdin) would be lost, but perhaps could be added.=C2=A0 Dan - m=
aybe Claude could make the changes for a pr. I wouldn=E2=80=99t be surprise=
d, even with such a large codebase.
>
> Regarding the wider points made by Peter in his blog post, I agree with p=
robably all of it. I highly doubt that the current trajectory (in terms of =
energy use) of these tools is sustainable, and I regularly wonder about act=
ual cost (to the planet) I am incurring when using Claude, especially when =
it=E2=80=99s doing something I could do in emacs or with e.g., a perl one-l=
iner. But I get stuff done about 20x as fast when using Claude, and I just =
speak to it (using Dictation on OS X), barely typing a thing. That=E2=80=99=
s all an incredible change after 40+ years of having to get thoughts out of=
my brain and into the computer one keystroke at a time. It feels like magi=
c. BUT, I have to watch Claude very carefully, look at its code, make it wr=
ite many tests and explicitly prove that it has managed to do what was inte=
nded. That=E2=80=99s not so different from watching yourself or doing code =
review of a junior (very energetic ever-cheerful) programmer.
>
> I agree that an AI (Claude, certainly) could be used to very quickly make=
general improvements to a codebase with very little risk (assuming a test =
suite exists). I=E2=80=99ve done that quite a few times, e.g., telling Clau=
de I no longer need support for Python 3.X and to modernise the typing hint=
s for 3.X+1, or to add typing hints, etc.=C2=A0 It=E2=80=99s also good at p=
roducing documentation.
>
> I=E2=80=99m not sure where this leaves us.=C2=A0 If there=E2=80=99s agree=
ment that a C extension would be good to have, the one Claude made for my p=
rseq project benchmark suite is tested and could be used. I haven=E2=80=99t=
looked at the BioPython code closely enough to know how much work that wou=
ld be, though.
>
> Thanks again for the thoughtful replies. There was an earlier one asking =
why the BioPython code was slower than Claude=E2=80=99s minimal pure Python=
, and suggesting that would be due to the overhead of making SeqRecords. I =
think that must be the case. The impact is much more severe with FASTQ than=
FASTA, so I guess dealing with the quality strings must be quite expensive=
.
>
> Best,
> Terry
>
> From: Peter Cock <[email protected]>
> Date: Monday, 24. November 2025 at 10:41
> To: Jones Kelly, Terence Carleton <[email protected]>
> Cc: [email protected] <[email protected]>
> Subject: [ext] Re: [Biopython] A possibility for speeding up FASTA/FASTQ =
reading in BioPython
>
> Hello Terry,
>
> I just posted a blog about my thoughts on receiving generative AI
> contributions as an Open Source project maintainer:
>
> https://blastedbio.blogspot.com/2025/11/thoughts-on-generative-ai-contrib=
utions.html
>
> I am sceptical, and in this case adding a Rust dependency to Biopython
> seems too much to ask. I think you could get similar performance gains
> with C (which we do use) where at least the maintainers have some
> experience. However, even there, gains may not make the additional
> complexity and maintenance burden worthwhile.
>
> Thank you for writting and asking, rather than suprising everyone with
> a large pull request.
>
> Peter
>
> P.S. Cross reference https://github.com/biopython/biopython/pull/5085
>
> On Tue, Nov 11, 2025 at 10:00=E2=80=AFPM Jones Kelly, Terence Carleton
> <[email protected]> wrote:
> >
> > Hi all
> >
> > I regularly process reasonably large FASTQ (hundreds of billions of seq=
uencing reads) and FASTA files using BioPython. For some years I've been me=
aning to implement a FASTQ/FASTA reader in a compiled language and add Pyth=
on bindings to improve the speed. I could've done this in C but I spent som=
e decades writing C and I wanted to learn something new, so I considered a =
few languages. Because Rust makes it very easy to create Python bindings, I=
decided to give it a try. I thought I'd get going by asking the Claude CLI=
to write me some Rust. That turned out to be a much, much better experienc=
e than I had anticipated. With Claude I played with several implementations=
, keeping track of timing. Claude also wrote some tests. To compare what I =
was seeing I got Claude to write a pure Python version, a pure C version, P=
ython bindings to the C, and to create a benchmark suite. From what I can t=
ell, the Rust/Python (and the C/Python) FASTA reading is twice as fast as B=
ioPython and FASTQ reading is four times as fast. I didn't write a single l=
ine of code. I just did some minimal cleaning up when things were already f=
ar along. I've been using the code for the last month or two with no proble=
ms.
> >
> > The repo is at https://github.com/VirologyCharite/prseq=C2=A0 (prseq =
=3D Python/Rust for sequences). You'll find the benchmark results on that p=
age.=C2=A0 There are still some small things I would adjust in the API.=C2=
=A0 BTW, Claude also wrote the README (which should definitely be improved)=
.
> >
> > I am wondering if there might be interest in incorporating this into Bi=
oPython. I don't know if there are any Rust dependencies in BioPython but I=
know that there are some C extensions. We could use either, as their speed=
s are comparable. If there's interest, I'd be happy to help (or to do it al=
l, after some discussion and maybe with some guidance).
> >
> > Thanks very much for all the work on BioPython. It's really been a plea=
sure to use the code over the last dozen years or so.
> >
> > Terry Jones
> >
> >
> > _______________________________________________
> > Biopython mailing list=C2=A0 -=C2=A0 [email protected]
> > https://mailman.open-bio.org/mailman/listinfo/biopython
_______________________________________________
Biopython mailing list=C2=A0 -=C2=A0 [email protected]
https://mailman.open-bio.org/mailman/listinfo/biopython
=20
------=_Part_1876149_246379205.1764290830958
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<html><head></head><body><div class=3D"ydpa682210eyahoo-style-wrap" style=
=3D"font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:10px=
;"><div></div>
<div dir=3D"ltr" data-setdir=3D"false">Hi all,</div><div dir=3D"ltr=
" data-setdir=3D"false"><br></div><div dir=3D"ltr" data-setdir=3D"false">Yo=
u may also want to try reading the Fasta/Fastq file in larger chunks, rathe=
r than line-by-line. I have tried this in the past, and it gave me much bet=
ter performance.</div><div dir=3D"ltr" data-setdir=3D"false"><br></div><div=
dir=3D"ltr" data-setdir=3D"false">This may be a good opportunity to try a =
SeqRecord-free parser, i.e. a parser that returns Seq objects instead of Se=
qRecord objects. I never quite understood why we have SeqRecords alongside =
Seq objects.</div><div dir=3D"ltr" data-setdir=3D"false"><br></div><div dir=
=3D"ltr" data-setdir=3D"false">Best,</div><div dir=3D"ltr" data-setdir=3D"f=
alse">-MIchiel</div><div><br></div>
=20
</div><div id=3D"yahoo_quoted_5210870872" class=3D"yahoo_quoted">
<div style=3D"font-family:'Helvetica Neue', Helvetica, Arial, s=
ans-serif;font-size:13px;color:#26282a;">
=20
<div>
On Wednesday, November 26, 2025 at 06:38:52 PM GMT+=
9, Peter Cock <[email protected]> wrote:
</div>
<div><br></div>
<div><br></div>
=20
=20
<div><div dir=3D"ltr">Hi Terry,<br clear=3D"none"><br clear=
=3D"none">That's a very even handed reply. I hadn't considered speaking to =
AI<br clear=3D"none">tools with Voice recognition, but Amazon Alexa isn't g=
reat and has<br clear=3D"none">probably lowered my expectations - grin.<br =
clear=3D"none"><br clear=3D"none">The SeqRecord overhead is big, especially=
making the ASCII quality<br clear=3D"none">string into a list of integers.=
You often don't need this, and can<br clear=3D"none">stick with strings. S=
ee eg:<br clear=3D"none"><br clear=3D"none"><a shape=3D"rect" href=3D"https=
://www.open-bio.org/2009/09/25/biopython-fast-fastq/" target=3D"_blank">htt=
ps://www.open-bio.org/2009/09/25/biopython-fast-fastq/</a><br clear=3D"none=
"><br clear=3D"none">In the short term, a much easier experiment to try is =
switching our<br clear=3D"none">Pure Python FASTQ parser to work in bytes m=
ode (binary handles) to<br clear=3D"none">avoid the bytes to unicode conver=
sion of the sequence and quality<br clear=3D"none">strings.<br clear=3D"non=
e"><br clear=3D"none">Peter<br clear=3D"none"><div class=3D"yqt1432880540" =
id=3D"yqtfd65057"><br clear=3D"none">On Tue, Nov 25, 2025 at 7:56=E2=80=AFP=
M Jones Kelly, Terence Carleton<br clear=3D"none"><<a shape=3D"rect" yma=
ilto=3D"mailto:[email protected]" href=3D"mailto:terence.jones@chari=
te.de">[email protected]</a>> wrote:<br clear=3D"none">><br cl=
ear=3D"none">> Hi all<br clear=3D"none">><br clear=3D"none">> Than=
ks for the replies and I'm sorry to be so slow in replying.<br clear=3D"non=
e">><br clear=3D"none">> Peter - I=E2=80=99m sceptical, too. I =
thought it highly unlikely that BioPython would want to add a dependence on=
Rust. The C code could be used to get the same speed-up. The automatic det=
ection of compressed input (including on stdin) would be lost, but perhaps =
could be added. Dan - maybe Claude could make the changes for a pr. I=
wouldn=E2=80=99t be surprised, even with such a large codebase.<br clear=
=3D"none">><br clear=3D"none">> Regarding the wider points made by Pe=
ter in his blog post, I agree with probably all of it. I highly doubt that =
the current trajectory (in terms of energy use) of these tools is sustainab=
le, and I regularly wonder about actual cost (to the planet) I am incurring=
when using Claude, especially when it=E2=80=99s doing something I could do=
in emacs or with e.g., a perl one-liner. But I get stuff done about 20x as=
fast when using Claude, and I just speak to it (using Dictation on OS X), =
barely typing a thing. That=E2=80=99s all an incredible change after 40+ ye=
ars of having to get thoughts out of my brain and into the computer one key=
stroke at a time. It feels like magic. BUT, I have to watch Claude very car=
efully, look at its code, make it write many tests and explicitly prove tha=
t it has managed to do what was intended. That=E2=80=99s not so different f=
rom watching yourself or doing code review of a junior (very energetic ever=
-cheerful) programmer.<br clear=3D"none">><br clear=3D"none">> I agre=
e that an AI (Claude, certainly) could be used to very quickly make general=
improvements to a codebase with very little risk (assuming a test suite ex=
ists). I=E2=80=99ve done that quite a few times, e.g., telling Claude I no =
longer need support for Python 3.X and to modernise the typing hints for 3.=
X+1, or to add typing hints, etc. It=E2=80=99s also good at producing=
documentation.<br clear=3D"none">><br clear=3D"none">> I=E2=80=99m n=
ot sure where this leaves us. If there=E2=80=99s agreement that a C e=
xtension would be good to have, the one Claude made for my prseq project be=
nchmark suite is tested and could be used. I haven=E2=80=99t looked at the =
BioPython code closely enough to know how much work that would be, though.<=
br clear=3D"none">><br clear=3D"none">> Thanks again for the thoughtf=
ul replies. There was an earlier one asking why the BioPython code was slow=
er than Claude=E2=80=99s minimal pure Python, and suggesting that would be =
due to the overhead of making SeqRecords. I think that must be the case. Th=
e impact is much more severe with FASTQ than FASTA, so I guess dealing with=
the quality strings must be quite expensive.<br clear=3D"none">><br cle=
ar=3D"none">> Best,<br clear=3D"none">> Terry<br clear=3D"none">><=
br clear=3D"none">> From: Peter Cock <<a shape=3D"rect" ymailto=3D"ma=
ilto:[email protected]" href=3D"mailto:[email protected]">p=
[email protected]</a>><br clear=3D"none">> Date: Monday, 24. N=
ovember 2025 at 10:41<br clear=3D"none">> To: Jones Kelly, Terence Carle=
ton <<a shape=3D"rect" ymailto=3D"mailto:[email protected]" href=
=3D"mailto:[email protected]">[email protected]</a>><br cl=
ear=3D"none">> Cc: <a shape=3D"rect" ymailto=3D"mailto:biopython@biopyth=
on.org" href=3D"mailto:[email protected]">[email protected]</a>=
<<a shape=3D"rect" ymailto=3D"mailto:[email protected]" href=3D"m=
ailto:[email protected]">[email protected]</a>><br clear=3D"=
none">> Subject: [ext] Re: [Biopython] A possibility for speeding up FAS=
TA/FASTQ reading in BioPython<br clear=3D"none">><br clear=3D"none">>=
Hello Terry,<br clear=3D"none">><br clear=3D"none">> I just posted a=
blog about my thoughts on receiving generative AI<br clear=3D"none">> c=
ontributions as an Open Source project maintainer:<br clear=3D"none">><b=
r clear=3D"none">> <a shape=3D"rect" href=3D"https://blastedbio.blogspot=
.com/2025/11/thoughts-on-generative-ai-contributions.html" target=3D"_blank=
">https://blastedbio.blogspot.com/2025/11/thoughts-on-generative-ai-contrib=
utions.html</a><br clear=3D"none">><br clear=3D"none">> I am sceptica=
l, and in this case adding a Rust dependency to Biopython<br clear=3D"none"=
>> seems too much to ask. I think you could get similar performance gain=
s<br clear=3D"none">> with C (which we do use) where at least the mainta=
iners have some<br clear=3D"none">> experience. However, even there, gai=
ns may not make the additional<br clear=3D"none">> complexity and mainte=
nance burden worthwhile.<br clear=3D"none">><br clear=3D"none">> Than=
k you for writting and asking, rather than suprising everyone with<br clear=
=3D"none">> a large pull request.<br clear=3D"none">><br clear=3D"non=
e">> Peter<br clear=3D"none">><br clear=3D"none">> P.S. Cross refe=
rence <a shape=3D"rect" href=3D"https://github.com/biopython/biopython/pull=
/5085" target=3D"_blank">https://github.com/biopython/biopython/pull/5085</=
a><br clear=3D"none">><br clear=3D"none">> On Tue, Nov 11, 2025 at 10=
:00=E2=80=AFPM Jones Kelly, Terence Carleton<br clear=3D"none">> <<a =
shape=3D"rect" ymailto=3D"mailto:[email protected]" href=3D"mailto:t=
[email protected]">[email protected]</a>> wrote:<br clear=
=3D"none">> ><br clear=3D"none">> > Hi all<br clear=3D"none">&g=
t; ><br clear=3D"none">> > I regularly process reasonably large FA=
STQ (hundreds of billions of sequencing reads) and FASTA files using BioPyt=
hon. For some years I've been meaning to implement a FASTQ/FASTA reader in =
a compiled language and add Python bindings to improve the speed. I could'v=
e done this in C but I spent some decades writing C and I wanted to learn s=
omething new, so I considered a few languages. Because Rust makes it very e=
asy to create Python bindings, I decided to give it a try. I thought I'd ge=
t going by asking the Claude CLI to write me some Rust. That turned out to =
be a much, much better experience than I had anticipated. With Claude I pla=
yed with several implementations, keeping track of timing. Claude also wrot=
e some tests. To compare what I was seeing I got Claude to write a pure Pyt=
hon version, a pure C version, Python bindings to the C, and to create a be=
nchmark suite. From what I can tell, the Rust/Python (and the C/Python) FAS=
TA reading is twice as fast as BioPython and FASTQ reading is four times as=
fast. I didn't write a single line of code. I just did some minimal cleani=
ng up when things were already far along. I've been using the code for the =
last month or two with no problems.<br clear=3D"none">> ><br clear=3D=
"none">> > The repo is at <a shape=3D"rect" href=3D"https://github.co=
m/VirologyCharite/prseq" target=3D"_blank">https://github.com/VirologyChari=
te/prseq</a> (prseq =3D Python/Rust for sequences). You'll find the b=
enchmark results on that page. There are still some small things I wo=
uld adjust in the API. BTW, Claude also wrote the README (which shoul=
d definitely be improved).<br clear=3D"none">> ><br clear=3D"none">&g=
t; > I am wondering if there might be interest in incorporating this int=
o BioPython. I don't know if there are any Rust dependencies in BioPython b=
ut I know that there are some C extensions. We could use either, as their s=
peeds are comparable. If there's interest, I'd be happy to help (or to do i=
t all, after some discussion and maybe with some guidance).<br clear=3D"non=
e">> ><br clear=3D"none">> > Thanks very much for all the work =
on BioPython. It's really been a pleasure to use the code over the last doz=
en years or so.<br clear=3D"none">> ><br clear=3D"none">> > Ter=
ry Jones<br clear=3D"none">> ><br clear=3D"none">> ><br clear=
=3D"none">> > _______________________________________________<br clea=
r=3D"none">> > Biopython mailing list - <a shape=3D"rect"=
ymailto=3D"mailto:[email protected]" href=3D"mailto:Biopython@biopyt=
hon.org">[email protected]</a><br clear=3D"none">> > <a shape=
=3D"rect" href=3D"https://mailman.open-bio.org/mailman/listinfo/biopython" =
target=3D"_blank">https://mailman.open-bio.org/mailman/listinfo/biopython</=
a><br clear=3D"none">_______________________________________________<br cle=
ar=3D"none">Biopython mailing list - <a shape=3D"rect" ymailto=
=3D"mailto:[email protected]" href=3D"mailto:[email protected]"=
>[email protected]</a><br clear=3D"none"><a shape=3D"rect" href=3D"ht=
tps://mailman.open-bio.org/mailman/listinfo/biopython" target=3D"_blank">ht=
tps://mailman.open-bio.org/mailman/listinfo/biopython</a><br clear=3D"none"=
></div></div></div>
</div>
</div></body></html>
------=_Part_1876149_246379205.1764290830958--
--===============1325055932245067553==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Biopython mailing list - [email protected]
https://mailman.open-bio.org/mailman/listinfo/biopython
--===============1325055932245067553==--