Re: [Biopython] pd.df.to_csv(..., compression="bgzip")?
Dan Bolser <[email protected]> Thu, 14 Mar 2024 22:06:34 +0000
| Newsgroups | gmane.comp.python.bio.general |
|---|---|
| Message-ID | <CANs1yP+y6WWdPU1+qKBA8fFwXHqgdjMo_+duoDSGL1nE2_n+=Q@mail.gmail.com> |
--===============6068534878542868063== Content-Type: multipart/alternative; boundary="0000000000006052470613a617d8" --0000000000006052470613a617d8 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable For the record, I just used pysam.tabix_compress https://pysam.readthedocs.io/en/latest/api.html#pysam.tabix_compress On Wed, 13 Mar 2024 at 11:22, Dan Bolser <[email protected]> wrote: > Nice idea, I would never have thought of that. > > Thanks Peter! > > On Wed, Mar 13, 2024, 11:18 AM Peter Cock <[email protected]> > wrote: > >> Ah. I would give it a file handle then: >> >> with bgzf.open("example.txt.bgz", "w") as bgzf_handle: >> my_data_frame.to_csv(bgzf_handle, ...) >> >> I would expect that to work according to >> >> https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFr= ame.to_csv.html >> - possibly with an explicit compression=3DNone added? >> >> Peter >> >> >> On Wed, Mar 13, 2024 at 11:08=E2=80=AFAM Dan Bolser <[email protected]= o.uk> >> wrote: >> > >> > pandas.to_csv is the function that writes data. pandas.read_csv >> silently handles decompression as needed. >> > >> > >> > >> > On Wed, Mar 13, 2024, 10:49 AM Peter Cock <[email protected]> >> wrote: >> >> >> >> Yes. BGZF is just a special kind of GZIP file, if all you are doing i= s >> >> decompressing it for reading it then the standard gzip.open(...) >> >> is fine. >> >> >> >> Peter >> >> >> >> >> >> On Wed, Mar 13, 2024 at 10:03=E2=80=AFAM Dan Bolser <dan.bolser@outse= e.co.uk> >> wrote: >> >>> >> >>> bgzip is a 'bio' thing, so thought I'd ask here. It's perhaps not >> 'biopython', but it's bio/python. >> >>> >> >>> On Tue, 12 Mar 2024 at 19:11, Sean Brimer <[email protected]> wrote= : >> >>>> >> >>>> Hi Dan, >> >>>> >> >>>> This feels more like a panda's issue than a biopython issue. That >> said, I think you could just use gzip. I think. bgzip for samtools was >> built on top of gzip so it probably decompresses in a similar way. >> >>>> >> >>>> On Tue, Mar 12, 2024 at 12:52=E2=80=AFPM Dan Bolser <dan.bolser@out= see.co.uk> >> wrote: >> >>>>> >> >>>>> Hi, >> >>>>> >> >>>>> I can pass `compression=3D"gzip"` to pandas.DataFrame.to_csv, but = not >> bgzip... how to update pandas to support bgzip? >> >>>>> >> >>>>> >> >>>>> Thanks, >> >>>>> _______________________________________________ >> >>>>> Biopython mailing list - [email protected] >> >>>>> https://mailman.open-bio.org/mailman/listinfo/biopython >> >>> >> >>> _______________________________________________ >> >>> Biopython mailing list - [email protected] >> >>> https://mailman.open-bio.org/mailman/listinfo/biopython >> > --0000000000006052470613a617d8 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">For=C2=A0 the record, I just used=C2=A0pysam.tabix_compres= s <a href=3D"https://pysam.readthedocs.io/en/latest/api.html#pysam.tabix_co= mpress">https://pysam.readthedocs.io/en/latest/api.html#pysam.tabix_compres= s</a></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_a= ttr">On Wed, 13 Mar 2024 at 11:22, Dan Bolser <<a href=3D"mailto:dan.bol= [email protected]">[email protected]</a>> wrote:<br></div><blockquo= te class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px = solid rgb(204,204,204);padding-left:1ex"><div dir=3D"auto">Nice idea, I wou= ld never have thought of that.<div dir=3D"auto"><br></div><div dir=3D"auto"= >Thanks Peter!</div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" c= lass=3D"gmail_attr">On Wed, Mar 13, 2024, 11:18 AM Peter Cock <<a href= =3D"mailto:[email protected]" target=3D"_blank">p.j.a.cock@googlema= il.com</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"m= argin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left= :1ex">Ah. I would give it a file handle then:<br> <br> with bgzf.open("example.txt.bgz", "w") as bgzf_handle:<= br> =C2=A0 =C2=A0 my_data_frame.to_csv(bgzf_handle, ...)<br> <br> I would expect that to work according to<br> <a href=3D"https://pandas.pydata.org/pandas-docs/stable/reference/api/panda= s.DataFrame.to_csv.html" rel=3D"noreferrer noreferrer" target=3D"_blank">ht= tps://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.t= o_csv.html</a><br> - possibly with an explicit compression=3DNone added?<br> <br> Peter<br> <br> <br> On Wed, Mar 13, 2024 at 11:08=E2=80=AFAM Dan Bolser <<a href=3D"mailto:d= [email protected]" rel=3D"noreferrer" target=3D"_blank">dan.bolser@out= see.co.uk</a>> wrote:<br> ><br> > pandas.to_csv is the function that writes data. pandas.read_csv silent= ly handles decompression as needed.<br> ><br> ><br> ><br> > On Wed, Mar 13, 2024, 10:49 AM Peter Cock <<a href=3D"mailto:p.j.a.= [email protected]" rel=3D"noreferrer" target=3D"_blank">p.j.a.cock@google= mail.com</a>> wrote:<br> >><br> >> Yes. BGZF is just a special kind of GZIP file, if all you are doin= g is<br> >> decompressing it for reading it then the standard gzip.open(...)<b= r> >> is fine.<br> >><br> >> Peter<br> >><br> >><br> >> On Wed, Mar 13, 2024 at 10:03=E2=80=AFAM Dan Bolser <<a href=3D= "mailto:[email protected]" rel=3D"noreferrer" target=3D"_blank">dan.b= [email protected]</a>> wrote:<br> >>><br> >>> bgzip is a 'bio' thing, so thought I'd ask here. I= t's perhaps not 'biopython', but it's bio/python.<br> >>><br> >>> On Tue, 12 Mar 2024 at 19:11, Sean Brimer <<a href=3D"mailt= o:[email protected]" rel=3D"noreferrer" target=3D"_blank">[email protected]= om</a>> wrote:<br> >>>><br> >>>> Hi Dan,<br> >>>><br> >>>> This feels more like a panda's issue than a biopython = issue. That said, I think you could just use gzip. I think. bgzip for samto= ols was built on top of gzip so it probably decompresses in a similar way.<= br> >>>><br> >>>> On Tue, Mar 12, 2024 at 12:52=E2=80=AFPM Dan Bolser <<a= href=3D"mailto:[email protected]" rel=3D"noreferrer" target=3D"_blan= k">[email protected]</a>> wrote:<br> >>>>><br> >>>>> Hi,<br> >>>>><br> >>>>> I can pass `compression=3D"gzip"` to pandas.= DataFrame.to_csv, but not bgzip... how to update pandas to support bgzip?<b= r> >>>>><br> >>>>><br> >>>>> Thanks,<br> >>>>> _______________________________________________<br> >>>>> Biopython mailing list=C2=A0 -=C2=A0 <a href=3D"mailto= :[email protected]" rel=3D"noreferrer" target=3D"_blank">Biopython@bi= opython.org</a><br> >>>>> <a href=3D"https://mailman.open-bio.org/mailman/listin= fo/biopython" rel=3D"noreferrer noreferrer" target=3D"_blank">https://mailm= an.open-bio.org/mailman/listinfo/biopython</a><br> >>><br> >>> _______________________________________________<br> >>> Biopython mailing list=C2=A0 -=C2=A0 <a href=3D"mailto:Biopyth= [email protected]" rel=3D"noreferrer" target=3D"_blank">Biopython@biopython.= org</a><br> >>> <a href=3D"https://mailman.open-bio.org/mailman/listinfo/biopy= thon" rel=3D"noreferrer noreferrer" target=3D"_blank">https://mailman.open-= bio.org/mailman/listinfo/biopython</a><br> </blockquote></div> </blockquote></div> --0000000000006052470613a617d8-- --===============6068534878542868063== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Biopython mailing list - [email protected] https://mailman.open-bio.org/mailman/listinfo/biopython --===============6068534878542868063==--