Re: Is there a pure JS implementation of the WOFF2 Brotli codec?
Mike Kamermans <[email protected]> Wed, 15 Jul 2015 12:07:39 -0700
| Newsgroups | gmane.comp.web.fonts |
|---|---|
| Message-ID | <CABhc0+JeMLB1aOxXSbobJE8s_=WTp9XcrojmWFNWgYL6u6jJzg@mail.gmail.com> |
--001a11c384ce2a8eeb051aeeabb8 Content-Type: text/plain; charset=UTF-8 Ah, those are good links, thanks! - Mike On Wed, Jul 15, 2015 at 10:07 AM, Jonathan Kew <[email protected]> wrote: > On 15/7/15 17:46, Mike Kamermans wrote: > >> Yeah,that's the conclusion we had to draw too - debugging the running >> process revealed that the emscripten'd library was allocating three >> massive array buffers (~313MB each), just by loading the library, so >> that's either a nasty emscripten bug, or could be a deferred allocation >> pattern that works in C++ but turns into instant allocation in JS. >> > > Note that Brotli makes use of a very large static array, of which there > are separate copies in the encoder and decoder sources; see > > https://github.com/google/brotli/blob/master/dec/dictionary.h > https://github.com/google/brotli/blob/master/enc/dictionary.h > > If the emscripten'd library is eagerly loading this into an array, and > doing so separately for the encoder and decoder, that could easily account > for at least a couple of massive array allocations. (Why three, though? No > idea offhand.) > > JK > > >> That said, a pure JS implementation would be great as "reality check" >> too, so that for raw speed a (fixed) emscripten'd library is great, but >> for sanity, a secondary implementation is available to hold emscripten >> accountable to. >> >> - Mike >> >> On Tue, Jul 14, 2015 at 7:37 PM, Robert O'Callahan <[email protected] >> <mailto:[email protected]>> wrote: >> >> On Wed, Jul 15, 2015 at 11:25 AM, Mike Kamermans <[email protected] >> <mailto:[email protected]>> wrote: >> >> On the off-chance that anyone knows, fontkit >> (https://github.com/devongovett/fontkit) is currently using an >> emscripten'ed version of the Brotli codec for handling WOFF2 >> decoding, which turns what is a 34MB footprint process without >> WOFF2 decoding into a ~1000MB footprint process with... making >> it rather unusable for dealing with WOFF2 files in any >> meaningful production setting. >> >> >> That overhead sounds way too high. I guess something is wrong with >> emscripten or some other part of the workflow. >> >> Rob >> -- >> lbir ye,ea yer.tnietoehr rdn rdsme,anea lurpr edna e hnysnenh hhe >> uresyf toD >> selthor stor edna siewaoeodm or v sstvr esBa kbvted,t >> rdsme,aoreseoouoto >> o l euetiuruewFa kbn e hnystoivateweh uresyf tulsa rehr rdm or >> rnea lurpr >> .a war hsrer holsa rodvted,t nenh hneireseoouot.tniesiewaoeivatewt >> sstvr esn >> >> >> > --001a11c384ce2a8eeb051aeeabb8 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Ah, those are good links, thanks!<br><br></div>- Mike= <br></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Wed,= Jul 15, 2015 at 10:07 AM, Jonathan Kew <span dir=3D"ltr"><<a href=3D"ma= ilto:[email protected]" target=3D"_blank">[email protected]</a>></span= > wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bo= rder-left:1px #ccc solid;padding-left:1ex"><span class=3D"">On 15/7/15 17:4= 6, Mike Kamermans wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Yeah,that's the conclusion we had to draw too - debugging the running<b= r> process revealed that the emscripten'd library was allocating three<br> massive array buffers (~313MB each), just by loading the library, so<br> that's either a nasty emscripten bug, or could be a deferred allocation= <br> pattern that works in C++ but turns into instant allocation in JS.<br> </blockquote> <br></span> Note that Brotli makes use of a very large static array, of which there are= separate copies in the encoder and decoder sources; see<br> <br> =C2=A0 <a href=3D"https://github.com/google/brotli/blob/master/dec/dictiona= ry.h" rel=3D"noreferrer" target=3D"_blank">https://github.com/google/brotli= /blob/master/dec/dictionary.h</a><br> =C2=A0 <a href=3D"https://github.com/google/brotli/blob/master/enc/dictiona= ry.h" rel=3D"noreferrer" target=3D"_blank">https://github.com/google/brotli= /blob/master/enc/dictionary.h</a><br> <br> If the emscripten'd library is eagerly loading this into an array, and = doing so separately for the encoder and decoder, that could easily account = for at least a couple of massive array allocations. (Why three, though? No = idea offhand.)<span class=3D"HOEnZb"><font color=3D"#888888"><br> <br> JK<br> <br> </font></span><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;= border-left:1px #ccc solid;padding-left:1ex"><span class=3D""> <br> That said, a pure JS implementation would be great as "reality check&q= uot;<br> too, so that for raw speed a (fixed) emscripten'd library is great, but= <br> for sanity, a secondary implementation is available to hold emscripten<br> accountable to.<br> <br> - Mike<br> <br> On Tue, Jul 14, 2015 at 7:37 PM, Robert O'Callahan <<a href=3D"mailt= o:[email protected]" target=3D"_blank">[email protected]</a><br></spa= n><span class=3D""> <mailto:<a href=3D"mailto:[email protected]" target=3D"_blank">robert= @ocallahan.org</a>>> wrote:<br> <br> =C2=A0 =C2=A0 On Wed, Jul 15, 2015 at 11:25 AM, Mike Kamermans <<a href= =3D"mailto:[email protected]" target=3D"_blank">[email protected]</a><br></= span><span class=3D""> =C2=A0 =C2=A0 <mailto:<a href=3D"mailto:[email protected]" target=3D"_bl= ank">[email protected]</a>>> wrote:<br> <br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 On the off-chance that anyone knows, fontkit<br= > =C2=A0 =C2=A0 =C2=A0 =C2=A0 (<a href=3D"https://github.com/devongovett/font= kit" rel=3D"noreferrer" target=3D"_blank">https://github.com/devongovett/fo= ntkit</a>) is currently using an<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 emscripten'ed version of the Brotli codec f= or handling WOFF2<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 decoding, which turns what is a 34MB footprint = process without<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 WOFF2 decoding into a ~1000MB footprint process= with... making<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 it rather unusable for dealing with WOFF2 files= in any<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 meaningful production setting.<br> <br> <br> =C2=A0 =C2=A0 That overhead sounds way too high. I guess something is wrong= with<br> =C2=A0 =C2=A0 emscripten or some other part of the workflow.<br> <br> =C2=A0 =C2=A0 Rob<br> =C2=A0 =C2=A0 --<br> =C2=A0 =C2=A0 lbir ye,ea yer.tnietoehr=C2=A0 rdn rdsme,anea lurpr=C2=A0 edn= a e hnysnenh hhe<br> =C2=A0 =C2=A0 uresyf toD<br> =C2=A0 =C2=A0 selthor=C2=A0 stor=C2=A0 edna=C2=A0 siewaoeodm=C2=A0 or v sst= vr=C2=A0 esBa=C2=A0 kbvted,t<br> =C2=A0 =C2=A0 rdsme,aoreseoouoto<br> =C2=A0 =C2=A0 o l euetiuruewFa=C2=A0 kbn e hnystoivateweh uresyf tulsa rehr= =C2=A0 rdm=C2=A0 or<br> =C2=A0 =C2=A0 rnea lurpr<br> =C2=A0 =C2=A0 .a war hsrer holsa rodvted,t=C2=A0 nenh hneireseoouot.tniesie= waoeivatewt<br> =C2=A0 =C2=A0 sstvr=C2=A0 esn<br> <br> <br> </span></blockquote> <br> </blockquote></div><br></div> --001a11c384ce2a8eeb051aeeabb8--