Re: [PROJ] LIBERTIFF / Thread-safe TIFF reader
Michael Smith via gdal-dev <[email protected]> Sun, 22 Dec 2024 15:19:47 -0500
| Newsgroups | gmane.comp.gis.gdal.devel,gmane.comp.gis.proj-4.devel |
|---|---|
| Message-ID | <[email protected]> |
> This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --===============0592666596466681796== Content-type: multipart/alternative; boundary="B_3817725587_3384165574" > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3817725587_3384165574 Content-type: text/plain; charset="UTF-8" Content-transfer-encoding: quoted-printable It=E2=80=99s a COG so it just decompresses and reads the tiles/cells needed for r= eprojection =20 =20 --=20 Michael Smith Remote Sensing/GIS Center US Army Corps of Engineers =20 =20 From: PROJ <[email protected]> on behalf of Rahkonen Jukka via P= ROJ <[email protected]> Reply-To: Rahkonen Jukka <[email protected]> Date: Sunday, December 22, 2024 at 3:12=E2=80=AFPM To: Howard Butler <[email protected]>, Even Rouault <[email protected]= m> Cc: gdal dev <[email protected]>, proj <[email protected]> Subject: Re: [PROJ] [gdal-dev] LIBERTIFF / Thread-safe TIFF reader =20 Hi, =20 The total size of the .tif files in my PROJ_DATA is 959 MB. I tried the LE= RC compression with MAX_Z_ERROR values 0.001 and 0.01 for /f %i in ('dir /b *.tif') do (gdal_translate -of GTiff -co compress=3Dler= c -co max_z_error=3D0.001 %i %i.lerc.tif) z_error 0.001: 682 MB (-29%) z_error 0.01: 549 MB (-43%) =20 The savings in space are real (though I think I could save 98 percent of sp= ace be deleting the grids which I never use). When it comes to compression/d= ecompression, I think that a proper test where Proj is used for doing someth= ing real would be needed. Does Proj decompress the whole grid file, or just = the tiles that it needs? How much time is used for opening the file and find= ing the tiles? =20 -Jukka Rahkonen- =20 =20 L=C3=A4hett=C3=A4j=C3=A4: gdal-dev <[email protected]> Puolesta Howard B= utler via gdal-dev L=C3=A4hetetty: sunnuntai 22. joulukuuta 2024 16.39 Vastaanottaja: Even Rouault <[email protected]> Kopio: gdal dev <[email protected]>; proj <[email protected]> Aihe: Re: [gdal-dev] [PROJ] LIBERTIFF / Thread-safe TIFF reader =20 =20 =20 On Dec 20, 2024, at 4:56=E2=80=AFPM, Even Rouault via gdal-dev <[email protected]= geo.org> wrote: =20 Besides reprocessing all of our existing grids,https://github.com/geotiffjs= /geotiff.js/tree/master/src/compression seems to show that it doesn't suppor= t ZSTD, which would break at least our wonderful cdn.proj.orginteractive map =20 To throw a wrench into it...=20 =20 geotiff.js has LERC support. Floating point LERC TIFFs compressed with a MA= X_Z_ERROR to one decimal better than the *stated precision* of the grids wou= ld be smaller and more efficiently compressed/decompressed. By maybe a 1/3rd= over zstd and deflate. =20 See this great write-up by LINZ about the topic https://github.com/linz/ele= vation/tree/master/docs/tiff-compression=20 =20 Howard _______________________________________________ PROJ mailing list PROJ@list= s.osgeo.org https://lists.osgeo.org/mailman/listinfo/proj=20 --B_3817725587_3384165574 Content-type: text/html; charset="UTF-8" Content-transfer-encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-microsof= t-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" xmlns:m= =3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http://www.w3.org= /TR/REC-html40"><head><meta http-equiv=3DContent-Type content=3D"text/html; char= set=3Dutf-8"><meta name=3DGenerator content=3D"Microsoft Word 15 (filtered medium)= "><style><!-- /* Font Definitions */ @font-face {font-family:Helvetica; panose-1:0 0 0 0 0 0 0 0 0 0;} @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:Aptos; panose-1:2 11 0 4 2 2 2 2 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; font-size:11.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} span.apple-converted-space {mso-style-name:apple-converted-space;} span.EmailStyle20 {mso-style-type:personal-reply; font-family:"Calibri",sans-serif; color:windowtext;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt; mso-ligatures:none;} @page WordSection1 {size:8.5in 11.0in; margin:70.85pt 56.7pt 70.85pt 56.7pt;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--></head><body lang=3DEN-US link=3Dblue vlink=3Dp= urple style=3D'word-wrap:break-word'><div class=3DWordSection1><p class=3DMsoNorma= l>It=E2=80=99s a COG so it just decompresses and reads the tiles/cells needed for = reprojection<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3D= MsoNormal><o:p> </o:p></p><div><div><p class=3DMsoNormal>-- <o:p></o= :p></p></div><div><p class=3DMsoNormal>Michael Smith<o:p></o:p></p><p class=3DMs= oNormal>Remote Sensing/GIS Center<o:p></o:p></p><p class=3DMsoNormal>US Army C= orps of Engineers<o:p></o:p></p></div></div><p class=3DMsoNormal><o:p> </= o:p></p><p class=3DMsoNormal><o:p> </o:p></p><div style=3D'border:none;bord= er-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=3DMsoNormal><b>= <span lang=3DFI style=3D'font-size:12.0pt;color:black'>From: </span></b><span la= ng=3DFI style=3D'font-size:12.0pt;color:black'>PROJ <[email protected]= .org> on behalf of Rahkonen Jukka via PROJ <[email protected]><b= r><b>Reply-To: </b>Rahkonen Jukka <[email protected]>= ;<br><b>Date: </b>Sunday, December 22, 2024 at 3:12=E2=80=AFPM<br><b>To: </b>Howar= d Butler <[email protected]>, Even Rouault <[email protected]= ><br><b>Cc: </b>gdal dev <[email protected]>, proj <proj@= lists.osgeo.org><br><b>Subject: </b>Re: [PROJ] [gdal-dev] LIBERTIFF / Thr= ead-safe TIFF reader<o:p></o:p></span></p></div><div><p class=3DMsoNormal><spa= n lang=3DFI style=3D'font-size:12.0pt;font-family:"Aptos",sans-serif'><o:p> = ;</o:p></span></p></div><p class=3DMsoNormal><span lang=3DFI>Hi,<o:p></o:p></spa= n></p><p class=3DMsoNormal><span lang=3DFI><o:p> </o:p></span></p><p class=3D= MsoNormal style=3D'margin-bottom:12.0pt'>The total size of the .tif files in m= y PROJ_DATA is 959 MB. I tried the LERC compression with MAX_Z_ERROR v= alues 0.001 and 0.01<o:p></o:p></p><p class=3DMsoNormal>for /f %i in ('dir /b = *.tif') do (gdal_translate -of GTiff -co compress=3Dlerc -co max_z_error=3D0.001= %i %i.lerc.tif)<br><br>z_error 0.001: 682 MB (-29%)<o:p></o:p></p><p class=3D= MsoNormal>z_error 0.01: 549 MB (-43%)<o:p></o:p></p><p class=3DMsoNormal><o:p>= </o:p></p><p class=3DMsoNormal>The savings in space are real (though I t= hink I could save 98 percent of space be deleting the grids which I never us= e). When it comes to compression/decompression, I think that a proper test w= here Proj is used for doing something real would be needed. Does Proj decomp= ress the whole grid file, or just the tiles that it needs? How much time is = used for opening the file and finding the tiles?<o:p></o:p></p><p class=3DMsoN= ormal><o:p> </o:p></p><p class=3DMsoNormal>-Jukka Rahkonen-<o:p></o:p></p= ><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal><o:p> </o:p= ></p><div><div style=3D'border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0= pt 0in 0in 0in'><p class=3DMsoNormal><b><span lang=3DFI>L=C3=A4hett=C3=A4j=C3=A4:</span></b>= <span lang=3DFI> gdal-dev <[email protected]> <b>Puolesta= </b>Howard Butler via gdal-dev<br><b>L=C3=A4hetetty:</b> sunnuntai 22. joulukuu= ta 2024 16.39<br><b>Vastaanottaja:</b> Even Rouault <even.rouault@spatial= ys.com><br><b>Kopio:</b> gdal dev <[email protected]>; proj = <[email protected]><br><b>Aihe:</b> Re: [gdal-dev] [PROJ] LIBERTIFF= / Thread-safe TIFF reader<o:p></o:p></span></p></div></div><p class=3DMsoNorm= al><span lang=3DFI><o:p> </o:p></span></p><p class=3DMsoNormal><span lang=3DF= I><o:p> </o:p></span></p><div><p class=3DMsoNormal style=3D'margin-bottom:1= 2.0pt'><span lang=3DFI><o:p> </o:p></span></p><blockquote style=3D'margin-t= op:5.0pt;margin-bottom:5.0pt'><div><p class=3DMsoNormal><span lang=3DFI>On Dec 2= 0, 2024, at 4:56=E2=80=AFPM, Even Rouault via gdal-dev <<a href=3D"mailto:gdal-de= [email protected]">[email protected]</a>> wrote:<o:p></o:p></span>= </p></div><p class=3DMsoNormal><span lang=3DFI><o:p> </o:p></span></p><div>= <p class=3DMsoNormal><span lang=3DFI style=3D'font-size:13.5pt;font-family:Helveti= ca'>Besides reprocessing all of our existing grids,</span><span lang=3DFI><a h= ref=3D"https://github.com/geotiffjs/geotiff.js/tree/master/src/compression"><s= pan style=3D'font-size:13.5pt;font-family:Helvetica'>https://github.com/geotif= fjs/geotiff.js/tree/master/src/compression</span></a></span><span class=3Dappl= e-converted-space><span lang=3DFI style=3D'font-size:13.5pt;font-family:Helvetic= a'> </span></span><span lang=3DFI style=3D'font-size:13.5pt;font-family:Hel= vetica'>seems to show that it doesn't support ZSTD, which would break at lea= st our wonderful<span class=3Dapple-converted-space> </span></span><span = lang=3DFI><a href=3D"http://cdn.proj.org/"><span style=3D'font-size:13.5pt;font-fa= mily:Helvetica'>cdn.proj.org</span></a></span><span lang=3DFI style=3D'font-size= :13.5pt;font-family:Helvetica'>interactive map</span><span lang=3DFI><o:p></o:= p></span></p></div></blockquote></div><p class=3DMsoNormal><span lang=3DFI><o:p>= </o:p></span></p><div><p class=3DMsoNormal><span lang=3DFI>To throw a wren= ch into it... <o:p></o:p></span></p></div><div><p class=3DMsoNormal><span= lang=3DFI><o:p> </o:p></span></p></div><div><p class=3DMsoNormal><span lan= g=3DFI>geotiff.js has LERC support. Floating point LERC TIFFs compressed with = a MAX_Z_ERROR to one decimal better than the *stated precision* of the grids= would be smaller and more efficiently compressed/decompressed. By maybe a 1= /3rd over zstd and deflate.<o:p></o:p></span></p></div><div><p class=3DMsoNorm= al><span lang=3DFI><o:p> </o:p></span></p></div><div><p class=3DMsoNormal><= span lang=3DFI>See this great write-up by LINZ about the topic <a href=3D"h= ttps://github.com/linz/elevation/tree/master/docs/tiff-compression">https://= github.com/linz/elevation/tree/master/docs/tiff-compression</a> <o:p></= o:p></span></p></div><div><p class=3DMsoNormal><span lang=3DFI><o:p> </o:p>= </span></p></div><div><p class=3DMsoNormal><span lang=3DFI>Howard<o:p></o:p></sp= an></p></div><p class=3DMsoNormal><span style=3D'font-size:12.0pt;font-family:"A= ptos",sans-serif'>_______________________________________________ PROJ maili= ng list [email protected] https://lists.osgeo.org/mailman/listinfo/proj <= o:p></o:p></span></p></div></body></html> --B_3817725587_3384165574-- --===============0592666596466681796== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ gdal-dev mailing list [email protected] https://lists.osgeo.org/mailman/listinfo/gdal-dev --===============0592666596466681796==--