Re: RFC4: last chance to comment before motion

--===============7345767326896056697==
Content-Type: multipart/alternative; boundary="00000000000062728d059b3ee545"

--00000000000062728d059b3ee545
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

In general, I think the RFC is awesome, suggesting a design that will
greatly
improve the interoperability (and ubiquity) of geodetic transformations.

I do, however, worry that we are rapidly entering a complexity level where
the
code base becomes write-only, or at least alarmingly dependent on a single
developer, although a highly competent one, but so much more worrying.

As we all know geo-programming competence is measured in the proposed SI
unit
=E2=80=9Crouault=E2=80=9D - abbreviated Rt. Currently the Rt cannot be expr=
essed based on
known
physical constants (although this is a matter of active research by leading
standards laboratories), so it is tentatively defined by the sample specime=
n
curated by the company Spatialys, and exposed to the world for calibration,
through Github.

Experiments have shown that the combined mass of geo-programming competence
in
the known universe is less than 2 Rt - and although analysis is ongoing, it
is
not even known whether geo-programming competence is conserved under state
changes.

Hence, there are good reasons to fear that PROJ, an essential piece of
geodetic
infrastructure, in a future where Spatialys has lost interest, will have to
be
maintained by lesser spirits, lacking the mental power to grasp the
multifaceted
monolith of PROJ source.

I never fully succeeded in my attempts to restructure the PROJ architecture=
,
before the WKT2 etc. efforts funded through https://gdalbarn.com/ suddenly
doubled the size of the source code, introduced SQLite as a direct
dependency,
and largely rendered my plans obsolete.

The switch to C++ as primary development language has, however, potentially
simplified much of that work, and I still think it will be a good idea to
redo the overall architecture of the code base in order to make it
maintainable
for a group of mental midgets, individually weighing less than 0.01 Rt,
rather
than the solitary 1.0 Rt mental giant currently doing such remarkable work.

I am fully aware that obtaining funding for such =E2=80=9Cinvisible=E2=80=
=9D work is very
hard,
and I am not at all volunteering to do it myself. I do, however, think it i=
s
worth giving it a few thoughts directly in the RFC - e.g. by including an
explicit list of additional dependencies introduced. If I read the RFC
correctly, these include:

libtiff
curl
zlib

But please elaborate on this - and perhaps add a few remarks about potentia=
l
internal interfaces (i.e. considering the grid handling and grid shift
functionality as a kind of server with a minimal interface, which in
principle - long term - could reside on a remote machine). Any comment of
this
kind in the RFC will ease the road forward for architectural restructuring
for
improved long term (decades, centuries) maintainability.

One thing I especially worry about is the potential loss of mind share:
Up until version 5.0, PROJ depended on the C standard library only, and
hence was quite easy to build and contribute to.

With the increasing build complexity, I fear that less new material
(projections
and other new operations) will be contributed to PROJ, unless we make it
easier
to integrate external code seamlessly with the library - a thing that is no=
t
easily done with the current static tables of operations.

The current low level architecture, however, mostly mimics in C the de fact=
o
workings of C++ inheritance and virtual functions, so a C++ redo of some lo=
w
level stuff will potentially simplify a lot of the basic stuff.
This is way beyond the scope of this RFC, but please try not to complicate
this
kind of work unnecessarily.

Apart from these, possibly unfounded, concerns, I find the RFC tasteful
and well balanced.

I especially enjoy the fact that you dismiss the use of HDF5 (which I
consider
a horrible reinvention of zip files) and NTv2 (which I consider a horrible,
and
limited, reinvention of TIFF files).

When TIFF is what is needed (despite all its arcane monstrosities),
TIFF is what we should use. Lets get rid of unnecessary specialized
file formats, if TIFF can support all our needs.

/thomas


Den tor. 2. jan. 2020 kl. 14.47 skrev Even Rouault <
[email protected]>:

> Hi,
>
> PROJ 6.3 being out, it is time to consider the next steps :-)
>
> RFC4
>
> https://github.com/rouault/PROJ/blob/rfc4_remote_and_geotiff_grid/docs/so=
urce/community/rfc/rfc-4.rst
> is hopefully now in its final state. Number of changes have been added to
> it since
> the initial version to synchronize it with the implementation, but nothin=
g
> dramatically different.
> You can the diff at
> https://github.com/rouault/PROJ/compare/5bbfc07..2b16de3
>
> It is now backed by a working implementation available at
> https://github.com/OSGeo/PROJ/pull/1817
>
> The CDN, storing grids converted to GeoTIFF, is now available at
> https://cdn.proj.org/
>
> As an example, the following will now work (using the
> OSTN15_NTv2_OSGBtoETRS.tif grid):
>
> $ echo 60 2 | PROJ_NETWORK=3DON cs2cs -d 8 OSGB36 ETRS89
> 59.99957003     1.99763761 0.00000000
>
> On next invocations, the parts of the grid that are needed and have
> already been
> queried will be retrieved from the cache (at ~/.local/share/proj/cache.db
> on Linux)
>
> For now, the RFC4 implementation works with a grid_alternatives database
> table still
> pointing at the NTv2 & GTX files of
> https://github.com/osgeo/proj-datumgrid and it will
> automagically try to fetch the correspond .tif files from the CDN when
> they are not locally
> available.
>
> In a next step, we'll also need to decide how/when we switch to
> https://github.com/osgeo/proj-datumgrid-geotiff (which is the master of
> cdn.proj.org) and
> if we provide a single or multiple packages of it (current total size of
> files is 486 MB),
> and/or a utility to retrieve as a post installation stage grids of
> interest from a given
> bounding box / producer / country from the CDN.
> I was initially thinking to a python utility, but there could have
> issues with Python dependencies, so perhaps a binary using the new curl
> dependency would be
> easier to deploy & use. This can be for follow-up discussions.
>
> Even
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
> _______________________________________________
> PROJ mailing list
> [email protected]
> https://lists.osgeo.org/mailman/listinfo/proj
>

--00000000000062728d059b3ee545
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">In general, I think the RFC is awesome, suggesting a desig=
n that will greatly<br>improve the interoperability (and ubiquity) of geode=
tic transformations.<br><br>I do, however, worry that we are rapidly enteri=
ng a complexity level where the<br>code base becomes write-only, or at leas=
t alarmingly dependent on a single<br>developer, although a highly competen=
t one, but so much more worrying.<br><br>As we all know geo-programming com=
petence is measured in the proposed SI unit<br>=E2=80=9Crouault=E2=80=9D - =
abbreviated Rt. Currently the Rt cannot be expressed based on known<br>phys=
ical constants (although this is a matter of active research by leading<br>=
standards laboratories), so it is tentatively defined by the sample specime=
n<br>curated by the company Spatialys, and exposed to the world for calibra=
tion,<br>through Github.<br><br>Experiments have shown that the combined ma=
ss of geo-programming competence in<br>the known universe is less than 2 Rt=
 - and although analysis is ongoing, it is<br>not even known whether geo-pr=
ogramming competence is conserved under state changes.<br><br>Hence, there =
are good reasons to fear that PROJ, an essential piece of geodetic<br>infra=
structure, in a future where Spatialys has lost interest, will have to be<b=
r>maintained by lesser spirits, lacking the mental power to grasp the multi=
faceted<br>monolith of PROJ source.<br><br>I never fully succeeded in my at=
tempts to restructure the PROJ architecture,<br>before the WKT2 etc. effort=
s funded through <a href=3D"https://gdalbarn.com/">https://gdalbarn.com/</a=
> suddenly<br>doubled the size of the source code, introduced SQLite as a d=
irect dependency,<br>and largely rendered my plans obsolete.<br><br>The swi=
tch to C++ as primary development language has, however, potentially<br>sim=
plified much of that work, and I still think it will be a good idea to<br>r=
edo the overall architecture of the code base in order to make it maintaina=
ble<br>for a group of mental midgets, individually weighing less than 0.01 =
Rt, rather<br>than the solitary 1.0 Rt mental giant currently doing such re=
markable work.<br><br>I am fully aware that obtaining funding for such =E2=
=80=9Cinvisible=E2=80=9D work is very hard,<br>and I am not at all voluntee=
ring to do it myself. I do, however, think it is<br>worth giving it a few t=
houghts directly in the RFC - e.g. by including an<br>explicit list of addi=
tional dependencies introduced. If I read the RFC<br>correctly, these inclu=
de:<br><br>libtiff<br>curl<br>zlib<br><br>But please elaborate on this - an=
d perhaps add a few remarks about potential<br>internal interfaces (i.e. co=
nsidering the grid handling and grid shift<br>functionality as a kind of se=
rver with a minimal interface, which in<br>principle - long term - could re=
side on a remote machine). Any comment of this<br>kind in the RFC will ease=
 the road forward for architectural restructuring for<br>improved long term=
 (decades, centuries) maintainability.<br><br>One thing I especially worry =
about is the potential loss of mind share:<br>Up until version 5.0, PROJ de=
pended on the C standard library only, and<br>hence was quite easy to build=
 and contribute to.<br><br>With the increasing build complexity, I fear tha=
t less new material (projections<br>and other new operations) will be contr=
ibuted to PROJ, unless we make it easier<br>to integrate external code seam=
lessly with the library - a thing that is not<br>easily done with the curre=
nt static tables of operations.<br><br>The current low level architecture, =
however, mostly mimics in C the de facto<br>workings of C++ inheritance and=
 virtual functions, so a C++ redo of some low<br>level stuff will potential=
ly simplify a lot of the basic stuff.<br>This is way beyond the scope of th=
is RFC, but please try not to complicate this<br>kind of work unnecessarily=
.<br><br>Apart from these, possibly unfounded, concerns, I find the RFC tas=
teful<br>and well balanced.<br><br>I especially enjoy the fact that you dis=
miss the use of HDF5 (which I consider<br>a horrible reinvention of zip fil=
es) and NTv2 (which I consider a horrible, and<br>limited, reinvention of T=
IFF files).<br><br>When TIFF is what is needed (despite all its arcane mons=
trosities),<br>TIFF is what we should use. Lets get rid of unnecessary spec=
ialized<br>file formats, if TIFF can support all our needs.<br><br>/thomas<=
br><br></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail=
_attr">Den tor. 2. jan. 2020 kl. 14.47 skrev Even Rouault &lt;<a href=3D"ma=
ilto:[email protected]">[email protected]</a>&gt;:<br></d=
iv><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bord=
er-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
PROJ 6.3 being out, it is time to consider the next steps :-)<br>
<br>
RFC4<br>
<a href=3D"https://github.com/rouault/PROJ/blob/rfc4_remote_and_geotiff_gri=
d/docs/source/community/rfc/rfc-4.rst" rel=3D"noreferrer" target=3D"_blank"=
>https://github.com/rouault/PROJ/blob/rfc4_remote_and_geotiff_grid/docs/sou=
rce/community/rfc/rfc-4.rst</a><br>
is hopefully now in its final state. Number of changes have been added to i=
t since<br>
the initial version to synchronize it with the implementation, but nothing =
dramatically different.<br>
You can the diff at <a href=3D"https://github.com/rouault/PROJ/compare/5bbf=
c07..2b16de3" rel=3D"noreferrer" target=3D"_blank">https://github.com/rouau=
lt/PROJ/compare/5bbfc07..2b16de3</a><br>
<br>
It is now backed by a working implementation available at<br>
<a href=3D"https://github.com/OSGeo/PROJ/pull/1817" rel=3D"noreferrer" targ=
et=3D"_blank">https://github.com/OSGeo/PROJ/pull/1817</a><br>
<br>
The CDN, storing grids converted to GeoTIFF, is now available at <a href=3D=
"https://cdn.proj.org/" rel=3D"noreferrer" target=3D"_blank">https://cdn.pr=
oj.org/</a><br>
<br>
As an example, the following will now work (using the OSTN15_NTv2_OSGBtoETR=
S.tif grid):<br>
<br>
$ echo 60 2 | PROJ_NETWORK=3DON cs2cs -d 8 OSGB36 ETRS89<br>
59.99957003=C2=A0 =C2=A0 =C2=A01.99763761 0.00000000<br>
<br>
On next invocations, the parts of the grid that are needed and have already=
 been<br>
queried will be retrieved from the cache (at ~/.local/share/proj/cache.db o=
n Linux)<br>
<br>
For now, the RFC4 implementation works with a grid_alternatives database ta=
ble still<br>
pointing at the NTv2 &amp; GTX files of <a href=3D"https://github.com/osgeo=
/proj-datumgrid" rel=3D"noreferrer" target=3D"_blank">https://github.com/os=
geo/proj-datumgrid</a> and it will<br>
automagically try to fetch the correspond .tif files from the CDN when they=
 are not locally<br>
available.<br>
<br>
In a next step, we&#39;ll also need to decide how/when we switch to<br>
<a href=3D"https://github.com/osgeo/proj-datumgrid-geotiff" rel=3D"noreferr=
er" target=3D"_blank">https://github.com/osgeo/proj-datumgrid-geotiff</a> (=
which is the master of <a href=3D"http://cdn.proj.org" rel=3D"noreferrer" t=
arget=3D"_blank">cdn.proj.org</a>) and<br>
if we provide a single or multiple packages of it (current total size of fi=
les is 486 MB),<br>
and/or a utility to retrieve as a post installation stage grids of interest=
 from a given<br>
bounding box / producer / country from the CDN.<br>
I was initially thinking to a python utility, but there could have<br>
issues with Python dependencies, so perhaps a binary using the new curl dep=
endency would be<br>
easier to deploy &amp; use. This can be for follow-up discussions.<br>
<br>
Even<br>
<br>
-- <br>
Spatialys - Geospatial professional services<br>
<a href=3D"http://www.spatialys.com" rel=3D"noreferrer" target=3D"_blank">h=
ttp://www.spatialys.com</a><br>
_______________________________________________<br>
PROJ mailing list<br>
<a href=3D"mailto:[email protected]" target=3D"_blank">[email protected].=
org</a><br>
<a href=3D"https://lists.osgeo.org/mailman/listinfo/proj" rel=3D"noreferrer=
" target=3D"_blank">https://lists.osgeo.org/mailman/listinfo/proj</a><br>
</blockquote></div>

--00000000000062728d059b3ee545--

--===============7345767326896056697==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Disposition: inline

X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KUFJPSiBtYWls
aW5nIGxpc3QKUFJPSkBsaXN0cy5vc2dlby5vcmcKaHR0cHM6Ly9saXN0cy5vc2dlby5vcmcvbWFp
bG1hbi9saXN0aW5mby9wcm9qCg==

--===============7345767326896056697==--