Re: fast serialization of huge bit vectors?
Shiro Kawai <[email protected]> Tue, 28 Apr 2026 11:19:50 -1000
| Newsgroups | gmane.lisp.scheme.gauche |
|---|---|
| Message-ID | <CALN0JNHOX_B_RSX4C98gNjCappK=prtcYYDq9Hmkw0MtcwDx+g@mail.gmail.com> |
--===============6474995968248514789== Content-Type: multipart/alternative; boundary="0000000000003f391206508bcba0" --0000000000003f391206508bcba0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Binary I/O of bitvectors is an interesting idea. Yes, binary representation of bitvector has more parameters to consider: - Unit of 'word' --- do we dump bytewise, or 2/4/8-byte wordwise? If the latter, word endianness matter. - Bit endianness --- MSB first or LSB first - Data boundary --- If the data is a bitstream / bit-packed, the data boundary may not fall on the byte boundary (e.g. compressed data) I think bitstream is rather a special case and can be handled separately. For quick data dump/restore purpose, we should use natural word size and endianness, as byte/bit swapping defeats the purpose. Another possibility is to allow aliasing bitvector and uvector (like uvector-alias). Then you can use uvector binary I/O. Actually, aliasing may be generally useful for handling binary data and treating part of it as a bitmap. What do you think? --shiro On Tue, Apr 28, 2026 at 3:56=E2=80=AFAM Jens Thiele <[email protected]> wrote: > Hi, just some minor thoughts: > > I just found gauche.bitvector and SRFI-178. Until now I used a small > hack based on u8vectors for huge bit vectors. Using u8vectors I simply > used write-uvector and port->uvector for serialization (not sure whether > using port->uvector is really clever, maybe I should have added a size > header ...) > > Now I wanted to get rid of that hack and switch to gauche.bitvector > and/or SRFI-178, but I didn't find a good way to read/write a bitvector > as binary data. Likely this isn't by accident but to prevent > byte-order/word size/padding ugliness? Or did I miss something obvious? > > jens > > PS: playing around: > gosh> (use gauche.uvector) > gosh> (use gauche.bitvector) > gosh> (use gauche.generator) > ;; for comparision test with u8vector > gosh> (define v (make-u8vector (ash 1 29) 0)) > v > gosh> (time (with-output-to-file "/tmp/v" (cut write-uvector v))) > ;(time (with-output-to-file "/tmp/v" (cut write-uvector v))) > ; real 2.718 > ; user 2.490 > ; sys 0.220 > gosh> (define bv (make-bitvector (ash 1 29) 0)) > bv > ;; first very slow try to dump bitvector to file (no size/padding info > stored) > gosh> (time (with-output-to-file "/tmp/v" (lambda() (generator-for-each > (compose write-byte bitvector->integer (cut apply bitvector <>)) (ggroup > (bitvector->int-generator bv) 8))))) > ;(time (with-output-to-file "/tmp/v" (lambda () (generator-for-each (com > ... > ; real 167.931 > ; user 180.650 > ; sys 0.370 > > > _______________________________________________ > Gauche-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/gauche-devel > --0000000000003f391206508bcba0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:large">Bin= ary I/O of bitvectors is an interesting idea.=C2=A0 Yes, binary representat= ion of bitvector has more parameters to consider:<br><br>- Unit of 'wor= d' --- do we dump bytewise, or 2/4/8-byte wordwise? If the latter, word= endianness matter.</div><div class=3D"gmail_default" style=3D"font-size:la= rge">- Bit endianness --- MSB first or LSB first</div><div class=3D"gmail_d= efault" style=3D"font-size:large">- Data boundary --- If the data is a bits= tream / bit-packed, the data boundary may not fall on the byte boundary (e.= g. compressed data)</div><div class=3D"gmail_default" style=3D"font-size:la= rge"><br></div><div class=3D"gmail_default" style=3D"font-size:large">I thi= nk bitstream is rather a special case and can be handled separately.=C2=A0<= /div><div class=3D"gmail_default" style=3D"font-size:large"><br></div><div = class=3D"gmail_default" style=3D"font-size:large">For quick data dump/resto= re purpose, we should use natural word size and endianness, as byte/bit swa= pping defeats the purpose.=C2=A0 Another possibility is to allow aliasing b= itvector and uvector (like uvector-alias).=C2=A0 Then you can use uvector b= inary I/O.=C2=A0 Actually, aliasing may be generally useful for handling bi= nary data and treating part of it as a bitmap.=C2=A0 What do you think?</di= v><div class=3D"gmail_default" style=3D"font-size:large"><br></div><div cla= ss=3D"gmail_default" style=3D"font-size:large">--shiro</div><div class=3D"g= mail_default" style=3D"font-size:large"><br></div><div class=3D"gmail_defau= lt" style=3D"font-size:large"><br></div><div class=3D"gmail_default" style= =3D"font-size:large"><br></div><div class=3D"gmail_default" style=3D"font-s= ize:large"><br></div><div class=3D"gmail_default" style=3D"font-size:large"= ><br></div><div class=3D"gmail_default" style=3D"font-size:large"><br></div= ><div class=3D"gmail_default" style=3D"font-size:large"><br><br></div></div= ><br><div class=3D"gmail_quote gmail_quote_container"><div dir=3D"ltr" clas= s=3D"gmail_attr">On Tue, Apr 28, 2026 at 3:56=E2=80=AFAM Jens Thiele <<a= href=3D"mailto:[email protected]">[email protected]</a>> wrote:<br></div><blo= ckquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left= :1px solid rgb(204,204,204);padding-left:1ex">Hi, just some minor thoughts:= <br> <br> I just found gauche.bitvector and SRFI-178. Until now I used a small<br> hack based on u8vectors for huge bit vectors. Using u8vectors I simply<br> used write-uvector and port->uvector for serialization (not sure whether= <br> using port->uvector is really clever, maybe I should have added a size<b= r> header ...)<br> <br> Now I wanted to get rid of that hack and switch to gauche.bitvector<br> and/or SRFI-178, but I didn't find a good way to read/write a bitvector= <br> as binary data. Likely this isn't by accident but to prevent<br> byte-order/word size/padding ugliness? Or did I miss something obvious?<br> <br> jens<br> <br> PS: playing around:<br> gosh> (use gauche.uvector)<br> gosh> (use gauche.bitvector)<br> gosh> (use gauche.generator)<br> ;; for comparision test with u8vector<br> gosh> (define v (make-u8vector (ash 1 29) 0))<br> v<br> gosh> (time (with-output-to-file "/tmp/v" (cut write-uvector v= )))<br> ;(time (with-output-to-file "/tmp/v" (cut write-uvector v)))<br> ; real=C2=A0 =C2=A02.718<br> ; user=C2=A0 =C2=A02.490<br> ; sys=C2=A0 =C2=A0 0.220<br> gosh> (define bv (make-bitvector (ash 1 29) 0))<br> bv<br> ;; first very slow try to dump bitvector to file (no size/padding info stor= ed)<br> gosh> (time (with-output-to-file "/tmp/v" (lambda() (generator= -for-each (compose write-byte bitvector->integer (cut apply bitvector &l= t;>)) (ggroup (bitvector->int-generator bv) 8)))))<br> ;(time (with-output-to-file "/tmp/v" (lambda () (generator-for-ea= ch (com ...<br> ; real 167.931<br> ; user 180.650<br> ; sys=C2=A0 =C2=A0 0.370<br> <br> <br> _______________________________________________<br> Gauche-devel mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">Gau= [email protected]</a><br> <a href=3D"https://lists.sourceforge.net/lists/listinfo/gauche-devel" rel= =3D"noreferrer" target=3D"_blank">https://lists.sourceforge.net/lists/listi= nfo/gauche-devel</a><br> </blockquote></div> --0000000000003f391206508bcba0-- --===============6474995968248514789== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============6474995968248514789== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Gauche-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/gauche-devel --===============6474995968248514789==--