Re: [PROJ] RFC 102: Embedding resource files into libgdal (and PROJ)

--===============6438371105975986659==
Content-Type: multipart/alternative; boundary="0000000000001b2d6e062396b7c5"

--0000000000001b2d6e062396b7c5
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

For my weird google production setup: I haven't heard of any bugs along
those lines. I only build cs2cs and gie for running tests and users have no
access to any of the proj binaries in production. I don't think folks do
much with sqlite (most db usage are things like Spanner or FireBase) and
most proj usage is pretty boring.

If they need them, they get them through debian on their desktops or
whatever cloud container they might use if they are allowed to do that, but
not in the normal production environment.

On Thu, Oct 3, 2024 at 10:30=E2=80=AFAM Even Rouault <even.rouault@spatialy=
s.com>
wrote:

> Hi,
>
> I've now a PROJ branch with proj.db embedding working at
> https://github.com/OSGeo/PROJ/pull/4265. And contrary to the current
> wording of the RFC text, I've actually invested in making it work with
> non-C23 compilers too (C23 mechanism used when available), by adapting
>
> https://gitlab.com/jhamberg/cmake-examples/-/blob/master/cmake/FileEmbed.=
cmake
> and optimizing it because its performance at generating a .c file from a
> binary was not acceptable on multi-megabyte large files like proj.db.
> Non-C23 embedding works fine with older gcc, clang, MSVC. The proj_db.c
> generation from proj.db runs in 8 seconds, and building it with gcc in
> 18 seconds (looking at build times in CI, that seems to be fast with
> MSVC too)
>
> But I won't (probably) support non-C23 embedding for GDAL: that would be
> too much complication w.r.t benefit. Only a "few" GDAL drivers need
> resource files, whereas access to proj.db is a strong requirement.
>
> Kurt, I had to modify the memOpen() and the memvfs_init() functions of
> the https://www.sqlite.org/src/doc/tip/ext/misc/memvfs.c (I did other
> changes in memvfs_init() to avoid making it the default VFS, and
> registering it per db-connection to avoid a global registration),
> otherwise some complex SQL requests that require creating a temporary
> file wouldn't work, such as the ones of 'bin/projinfo --list-crs' or
> 'bin/cct "this is a bogus CRS meant to trigger a syntax error in
> proj_create"'  (with sqlite 3.46 of fedora:rawhide). Is it something you
> ran into? Cf the changes in memvfs.c in
>
> https://github.com/rouault/PROJ/commit/e7fcf162f6968c48587836f537b5067081=
12e7d4
>
> With those changes, ctest pass
>
> Even
>
>
> http://www.spatialys.com
> My software is free, but my time generally not.
>
>

--0000000000001b2d6e062396b7c5
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><br></div><div><br></div>For my weird google producti=
on setup: I haven&#39;t heard of any bugs along those lines. I only build c=
s2cs and gie for running tests and users have no access to any of the proj =
binaries in production. I don&#39;t think folks do much with sqlite (most d=
b usage are things like Spanner or FireBase) and most proj usage is pretty =
boring.<div><br></div><div>If they need them, they get them through debian =
on their desktops or whatever cloud container they might use if they are al=
lowed to do that, but not in the normal production environment.</div></div>=
<br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Thu=
, Oct 3, 2024 at 10:30=E2=80=AFAM Even Rouault &lt;<a href=3D"mailto:even.r=
[email protected]">[email protected]</a>&gt; wrote:<br></div><b=
lockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-le=
ft:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
I&#39;ve now a PROJ branch with proj.db embedding working at <br>
<a href=3D"https://github.com/OSGeo/PROJ/pull/4265" rel=3D"noreferrer" targ=
et=3D"_blank">https://github.com/OSGeo/PROJ/pull/4265</a>. And contrary to =
the current <br>
wording of the RFC text, I&#39;ve actually invested in making it work with =
<br>
non-C23 compilers too (C23 mechanism used when available), by adapting <br>
<a href=3D"https://gitlab.com/jhamberg/cmake-examples/-/blob/master/cmake/F=
ileEmbed.cmake" rel=3D"noreferrer" target=3D"_blank">https://gitlab.com/jha=
mberg/cmake-examples/-/blob/master/cmake/FileEmbed.cmake</a> <br>
and optimizing it because its performance at generating a .c file from a <b=
r>
binary was not acceptable on multi-megabyte large files like proj.db.=C2=A0=
 <br>
Non-C23 embedding works fine with older gcc, clang, MSVC. The proj_db.c <br=
>
generation from proj.db runs in 8 seconds, and building it with gcc in <br>
18 seconds (looking at build times in CI, that seems to be fast with <br>
MSVC too)<br>
<br>
But I won&#39;t (probably) support non-C23 embedding for GDAL: that would b=
e <br>
too much complication w.r.t benefit. Only a &quot;few&quot; GDAL drivers ne=
ed <br>
resource files, whereas access to proj.db is a strong requirement.<br>
<br>
Kurt, I had to modify the memOpen() and the memvfs_init() functions of <br>
the <a href=3D"https://www.sqlite.org/src/doc/tip/ext/misc/memvfs.c" rel=3D=
"noreferrer" target=3D"_blank">https://www.sqlite.org/src/doc/tip/ext/misc/=
memvfs.c</a> (I did other <br>
changes in memvfs_init() to avoid making it the default VFS, and <br>
registering it per db-connection to avoid a global registration), <br>
otherwise some complex SQL requests that require creating a temporary <br>
file wouldn&#39;t work, such as the ones of &#39;bin/projinfo --list-crs&#3=
9; or <br>
&#39;bin/cct &quot;this is a bogus CRS meant to trigger a syntax error in <=
br>
proj_create&quot;&#39;=C2=A0 (with sqlite 3.46 of fedora:rawhide). Is it so=
mething you <br>
ran into? Cf the changes in memvfs.c in <br>
<a href=3D"https://github.com/rouault/PROJ/commit/e7fcf162f6968c48587836f53=
7b506708112e7d4" rel=3D"noreferrer" target=3D"_blank">https://github.com/ro=
uault/PROJ/commit/e7fcf162f6968c48587836f537b506708112e7d4</a><br>
<br>
With those changes, ctest pass<br>
<br>
Even<br>
<br>
<br>
<a href=3D"http://www.spatialys.com" rel=3D"noreferrer" target=3D"_blank">h=
ttp://www.spatialys.com</a><br>
My software is free, but my time generally not.<br>
<br>
</blockquote></div>

--0000000000001b2d6e062396b7c5--

--===============6438371105975986659==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
gdal-dev mailing list
[email protected]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

--===============6438371105975986659==--