Re: mapcache seed speed optimization

--===============8075883366139786610==
Content-Type: multipart/alternative; boundary="0000000000008111de059713238d"

--0000000000008111de059713238d
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

I may not be unrealistic for it take mapserver to 4-6 seconds to generate
4096px image depending on the amount data being rendered and the complexity
of your styles / expressions / labels.  If your mapserver is able to
generate a 1024x1024 in under a second that is pretty good.  Like Jukka and
other suggest I would look at translating the files to shp files with
spatial indexes or postgis tables with spatial and attribute indexes if
necessary.  Preprocessing the data (simplification / prefiltering / point
thinning cluster) and using SCALETOKENs to reference different source
layers at different scales can lead to big performance boosts at the
mapserver end.

For us the biggest limiting factor has been the write blocking to sqlite
cache when seeding tiles, if you increase the number of threads / process
in mapcache seed those process will just end up waiting to write to the
cache.  You can generally figure out what the max number of processes the
cache can handle by running a seed on a test area and look at the number of
tiles seeded per second, as some point -n will have no effect which your
process are just waiting to write the cache.  I would look at using this
method to test sqlite vs disk or other backends.  The GeoTiff cache may be
promising as it appears to be non-blocking but experimental...

We use a riak cache with a riak leveldb backend which works well for us but
is a bit of a pain to manage as it difficult to delete objects so it's not
great for caches that need to be refreshed.

Finally you may want to test MapProxy as an alternative to MapCache (we use
both).  MapProxy supports a non-blocking compact cache that could solve
your inode problem

Regards









On Fri, 8 Nov 2019 at 11:48, Sebastiano Laini <
[email protected]> wrote:

> We don=E2=80=99t create the data, we rely on OS (ordnance survey) to supp=
ly us the
> maps files and then we publish them in our service but I assume that
> probably we will need to create some flow to improve it or download it in
> other format that is faster.
>
>
>
> Sebastiano Laini
>
> Web Developer
>
> Buchanan Computing
>
>
>
> *From:* Fawcett, David (MNIT) [mailto:[email protected]]
> *Sent:* 08 November 2019 16:41
> *To:* Sebastiano Laini <[email protected]>;
> 'Rahkonen Jukka (MML)' <[email protected]>; '
> [email protected]' <[email protected]>
> *Subject:* RE: mapcache seed speed optimization
>
>
>
> For data formats that are slower to read, would it add fit your workflow
> to convert it to different data format before creating the tiles?
> _______________________________________________
> mapserver-users mailing list
> [email protected]
> https://lists.osgeo.org/mailman/listinfo/mapserver-users

--0000000000008111de059713238d
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>I may not be unrealistic for it take mapserver to 4-6=
 seconds to generate 4096px image depending on the amount data being render=
ed and the complexity of your styles / expressions / labels.=C2=A0 If your =
mapserver is able to generate a 1024x1024 in under a second that is pretty =
good.=C2=A0 Like Jukka and other suggest I would look at translating the fi=
les to shp files with spatial indexes or postgis tables with spatial and at=
tribute indexes if necessary.=C2=A0 Preprocessing the data (simplification =
/ prefiltering / point thinning cluster) and using SCALETOKENs to reference=
 different source layers at different scales can lead to big performance bo=
osts at the mapserver end.</div><div><br></div><div>For us the biggest limi=
ting factor has been the write blocking to sqlite cache when seeding tiles,=
 if you increase the number of threads / process in mapcache seed those pro=
cess will just end up waiting to write to the cache.=C2=A0 You can generall=
y figure out what the max number of processes the cache can handle by runni=
ng a seed on a test area and look at the number of tiles seeded per second,=
 as some point -n will have no effect which your process are just waiting t=
o write the cache.=C2=A0 I would look at using this method to test sqlite v=
s disk or other backends.=C2=A0 The GeoTiff cache may be promising as it ap=
pears to be non-blocking but experimental...</div><div><br></div><div>We us=
e a riak cache with a riak leveldb backend which works well for us but is a=
 bit of a pain to manage as it difficult to delete objects so it&#39;s not =
great for caches that need to be refreshed.</div><div><br></div><div>Finall=
y you may want to test MapProxy as an alternative to MapCache (we use both)=
.=C2=A0 MapProxy supports a non-blocking compact cache that could solve you=
r inode problem=C2=A0 <br></div><div><br></div><div>Regards<br></div><div><=
br></div><div><br></div><div>=C2=A0 =C2=A0 </div><div><br></div><div><br></=
div><div><br></div><div><br></div><div><br> </div></div><br><div class=3D"g=
mail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Fri, 8 Nov 2019 at 11:=
48, Sebastiano Laini &lt;<a href=3D"mailto:Sebastiano.Laini@buchanancomputi=
ng.co.uk" target=3D"_blank">[email protected]</a>&gt=
; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px=
 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">





<div lang=3D"EN-GB">
<div>
<p class=3D"MsoNormal"><span style=3D"color:rgb(31,73,125)">We don=E2=80=99=
t create the data, we rely on OS (ordnance survey) to supply us the maps fi=
les and then we publish them in our service but I assume that probably we w=
ill need to create some
 flow to improve it or download it in other format that is faster.<u></u><u=
></u></span></p>
<p class=3D"MsoNormal"><span style=3D"color:rgb(31,73,125)"><u></u>=C2=A0<u=
></u></span></p>
<p class=3D"MsoNormal"><span style=3D"color:rgb(31,73,125)">Sebastiano Lain=
i<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"color:rgb(31,73,125)">Web Developer<u=
></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"color:rgb(31,73,125)">Buchanan Comput=
ing<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"color:rgb(31,73,125)"><u></u>=C2=A0<u=
></u></span></p>
<div>
<div style=3D"border-color:rgb(225,225,225) currentcolor currentcolor;borde=
r-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm"=
>
<p class=3D"MsoNormal"><b><span lang=3D"EN-US">From:</span></b><span lang=
=3D"EN-US"> Fawcett, David (MNIT) [mailto:<a href=3D"mailto:david.fawcett@s=
tate.mn.us" target=3D"_blank">[email protected]</a>]
<br>
<b>Sent:</b> 08 November 2019 16:41<br>
<b>To:</b> Sebastiano Laini &lt;<a href=3D"mailto:Sebastiano.Laini@Buchanan=
Computing.co.uk" target=3D"_blank">[email protected]=
</a>&gt;; &#39;Rahkonen Jukka (MML)&#39; &lt;<a href=3D"mailto:jukka.rahkon=
[email protected]" target=3D"_blank">jukka.rahkonen@maanmittauslaitos=
.fi</a>&gt;; &#39;<a href=3D"mailto:[email protected]" target=
=3D"_blank">[email protected]</a>&#39; &lt;<a href=3D"mailto:=
[email protected]" target=3D"_blank">[email protected]=
geo.org</a>&gt;<br>
<b>Subject:</b> RE: mapcache seed speed optimization<u></u><u></u></span></=
p>
</div>
</div>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal"><span style=3D"color:rgb(31,73,125)" lang=3D"EN-US">=
For data formats that are slower to read, would it add fit your workflow to=
 convert it to different data format before creating the tiles?
<u></u><u></u></span></p>
</div>
</div>

_______________________________________________<br>
mapserver-users mailing list<br>
<a href=3D"mailto:[email protected]" target=3D"_blank">mapser=
[email protected]</a><br>
<a href=3D"https://lists.osgeo.org/mailman/listinfo/mapserver-users" rel=3D=
"noreferrer" target=3D"_blank">https://lists.osgeo.org/mailman/listinfo/map=
server-users</a></blockquote></div>

--0000000000008111de059713238d--

--===============8075883366139786610==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Disposition: inline

X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbWFwc2VydmVy
LXVzZXJzIG1haWxpbmcgbGlzdAptYXBzZXJ2ZXItdXNlcnNAbGlzdHMub3NnZW8ub3JnCmh0dHBz
Oi8vbGlzdHMub3NnZW8ub3JnL21haWxtYW4vbGlzdGluZm8vbWFwc2VydmVyLXVzZXJz

--===============8075883366139786610==--