Re: mapcache seed speed optimization

--===============0299715973488954194==
Content-Type: multipart/alternative;
 boundary="--=_RainLoop_758_590447393.1573502842"


----=_RainLoop_758_590447393.1573502842
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

Hi=0A=0AI have had a similar situation as You with large national data se=
ts in shape format. One of the biggest improvements for me has been to sp=
lit large shape files into smaller and=0Athen use vector tile indexes in =
Mapserver.=0AThe tool shp2tile has been working well for me. Stephen Wood=
bridge has written that tool. It can be found at http://imaptools.com/dow=
nload-software.html  (http://imaptools.com/download-software.html)=0AI th=
ink that You had to compile yourself on linux, but that was easy.=0A=0AI =
have an example where I have automated this in a script that runs through=
 all shape file in a directory and splits shape files larger than a certa=
in size and then creates the vector tile index files.=0AThe code snippet =
that is part of larger import script can be found here:=0Ahttps://gist.gi=
thub.com/LarsSchy/644b37ab2b1e58b48808bfb2b57d0359 (https://gist.github.c=
om/LarsSchy/644b37ab2b1e58b48808bfb2b57d0359)=0A=0AThe script snippet als=
o shows the normal optimizations we do.=0A=0AHope this could give You som=
e ideas how optimize and make Mapserver and Mapcache run faster. I also r=
an into the problem with to many inodes. =0AThat got solved by using sqli=
te as backend in Mapcache.=0A=0ABest regards / Lars Schylberg=0A11 novemb=
er 2019 kl. 15:42, "Travis Kirstine" <[email protected] (mailto:tr=
[email protected]?to=3D%22Travis%20Kirstine%22%20<traviskirstine@gma=
il.com>)> skrev:=0AI may not be unrealistic for it take mapserver to 4-6 =
seconds to generate 4096px image depending on the amount data being rende=
red and the complexity of your styles / expressions / labels. If your map=
server is able to generate a 1024x1024 in under a second that is pretty g=
ood. Like Jukka and other suggest I would look at translating the files t=
o shp files with spatial indexes or postgis tables with spatial and attri=
bute indexes if necessary. Preprocessing the data (simplification / prefi=
ltering / point thinning cluster) and using SCALETOKENs to reference diff=
erent source layers at different scales can lead to big performance boost=
s at the mapserver end. =0AFor us the biggest limiting factor has been th=
e write blocking to sqlite cache when seeding tiles, if you increase the =
number of threads / process in mapcache seed those process will just end =
up waiting to write to the cache. You can generally figure out what the m=
ax number of processes the cache can handle by running a seed on a test a=
rea and look at the number of tiles seeded per second, as some point -n w=
ill have no effect which your process are just waiting to write the cache=
. I would look at using this method to test sqlite vs disk or other backe=
nds. The GeoTiff cache may be promising as it appears to be non-blocking =
but experimental... =0AWe use a riak cache with a riak leveldb backend wh=
ich works well for us but is a bit of a pain to manage as it difficult to=
 delete objects so it's not great for caches that need to be refreshed. =
=0AFinally you may want to test MapProxy as an alternative to MapCache (w=
e use both). MapProxy supports a non-blocking compact cache that could so=
lve your inode problem =0ARegards =0A On Fri, 8 Nov 2019 at 11:48, Sebast=
iano Laini <[email protected] (mailto:Sebastiano.L=
[email protected])> wrote:  =0A	We don=E2=80=99t create the da=
ta, we rely on OS (ordnance survey) to supply us the maps files and then =
we publish them in our service but I assume that probably we will need to=
 create some flow to improve it or download it in other format that is fa=
ster. =0A=0A	Sebastiano Laini =0A=0A	Web Developer =0A=0A	Buchanan Comput=
ing =0A	From: Fawcett, David (MNIT) [mailto:[email protected] (ma=
ilto:[email protected])]=0ASent: 08 November 2019 16:41=0ATo: Seb=
astiano Laini <[email protected] (mailto:Sebastian=
[email protected])>; 'Rahkonen Jukka (MML)' <jukka.rahkonen=
@maanmittauslaitos.fi (mailto:[email protected])>; 'map=
[email protected] (mailto:[email protected])' <m=
[email protected] (mailto:[email protected])>=
=0ASubject: RE: mapcache seed speed optimization =0A=0A	For data formats =
that are slower to read, would it add fit your workflow to convert it to =
different data format before creating the tiles?  _______________________=
________________________=0Amapserver-users mailing list=0Amapserver-users=
@lists.osgeo.org (mailto:[email protected])=0Ahttps://lists=
.osgeo.org/mailman/listinfo/mapserver-users (https://lists.osgeo.org/mail=
man/listinfo/mapserver-users)

----=_RainLoop_758_590447393.1573502842
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE html><html><head><meta http-equiv=3D"Content-Type" content=3D"t=
ext/html; charset=3Dutf-8" /></head><body><div data-html-editor-font-wrap=
per=3D"true" style=3D"font-family: arial, sans-serif; font-size: 13px;">H=
i<br><br>I have had a similar situation as You with large national data s=
ets in shape format. One of the biggest improvements for me has been to s=
plit large shape files into smaller and<br>then use vector tile indexes i=
n Mapserver.<br>The tool shp2tile has been working well for me. Stephen W=
oodbridge has written that tool. It can be found at <a href=3D"http://ima=
ptools.com/download-software.html">http://imaptools.com/download-software=
.html </a><br>I think that You had to compile yourself on linux, but that=
 was easy.<br><br>I have an example where I have automated this in a scri=
pt that runs through all shape file in a directory and splits shape files=
 larger than a certain size and then creates the vector tile index files.=
<br>The code snippet that is part of larger import script can be found he=
re:<br><a href=3D"https://gist.github.com/LarsSchy/644b37ab2b1e58b48808bf=
b2b57d0359">https://gist.github.com/LarsSchy/644b37ab2b1e58b48808bfb2b57d=
0359</a><br><br>The script snippet also shows the normal optimizations we=
 do.<br><br>Hope this could give You some ideas how optimize and make Map=
server and Mapcache run faster. I also ran into the problem with to many =
inodes. <br>That got solved by using sqlite as backend in Mapcache.<br><b=
r>Best regards / Lars Schylberg<br><br><br><br>11 november 2019 kl. 15:42=
, "Travis Kirstine" &lt;<a target=3D"_blank" tabindex=3D"-1" href=3D"mail=
to:[email protected]?to=3D%22Travis%20Kirstine%22%20&lt;traviskirs=
[email protected]&gt;">[email protected]</a>&gt; skrev:<br> <blockquo=
te><div><div> <div dir=3D"ltr"> <div>I may not be unrealistic for it take=
 mapserver to 4-6 seconds to generate 4096px image depending on the amoun=
t data being rendered and the complexity of your styles / expressions / l=
abels. If your mapserver is able to generate a 1024x1024 in under a secon=
d that is pretty good. Like Jukka and other suggest I would look at trans=
lating the files to shp files with spatial indexes or postgis tables with=
 spatial and attribute indexes if necessary. Preprocessing the data (simp=
lification / prefiltering / point thinning cluster) and using SCALETOKENs=
 to reference different source layers at different scales can lead to big=
 performance boosts at the mapserver end.</div> <div></div> <div>For us t=
he biggest limiting factor has been the write blocking to sqlite cache wh=
en seeding tiles, if you increase the number of threads / process in mapc=
ache seed those process will just end up waiting to write to the cache. Y=
ou can generally figure out what the max number of processes the cache ca=
n handle by running a seed on a test area and look at the number of tiles=
 seeded per second, as some point -n will have no effect which your proce=
ss are just waiting to write the cache. I would look at using this method=
 to test sqlite vs disk or other backends. The GeoTiff cache may be promi=
sing as it appears to be non-blocking but experimental...</div> <div></di=
v> <div>We use a riak cache with a riak leveldb backend which works well =
for us but is a bit of a pain to manage as it difficult to delete objects=
 so it's not great for caches that need to be refreshed.</div> <div></div=
> <div>Finally you may want to test MapProxy as an alternative to MapCach=
e (we use both). MapProxy supports a non-blocking compact cache that coul=
d solve your inode problem</div> <div></div> <div>Regards</div> <div></di=
v> <div></div> <div></div> <div></div> <div></div> <div></div> <div></div=
> <div></div> </div> <div> <div dir=3D"ltr">On Fri, 8 Nov 2019 at 11:48, =
Sebastiano Laini &lt;<a target=3D"_blank" rel=3D"external nofollow noopen=
er noreferrer" tabindex=3D"-1" href=3D"mailto:Sebastiano.Laini@buchananco=
mputing.co.uk">[email protected]</a>&gt; wrote:</d=
iv> <blockquote style=3D"margin: 0px 0px 0px 0.8ex;border-left: 1px solid=
 rgb(204,204,204);padding-left: 1ex"> <div lang=3D"EN-GB"><div> <p><span =
style=3D"color: rgb(31,73,125)">We don=E2=80=99t create the data, we rely=
 on OS (ordnance survey) to supply us the maps files and then we publish =
them in our service but I assume that probably we will need to create som=
e flow to improve it or download it in other format that is faster.</span=
></p> <p><span style=3D"color: rgb(31,73,125)">Sebastiano Laini</span></p=
> <p><span style=3D"color: rgb(31,73,125)">Web Developer</span></p> <p><s=
pan style=3D"color: rgb(31,73,125)">Buchanan Computing</span></p> <div><d=
iv style=3D"border-color: rgb(225,225,225) currentcolor currentcolor;bord=
er-style: solid none none;border-width: 1pt medium medium;padding: 3pt 0c=
m 0cm"><p><b><span lang=3D"EN-US">From:</span></b><span lang=3D"EN-US"> F=
awcett, David (MNIT) [mailto:<a target=3D"_blank" rel=3D"external nofollo=
w noopener noreferrer" tabindex=3D"-1" href=3D"mailto:david.fawcett@state=
.mn.us">[email protected]</a>]<br><b>Sent:</b> 08 November 2019 1=
6:41<br><b>To:</b> Sebastiano Laini &lt;<a target=3D"_blank" rel=3D"exter=
nal nofollow noopener noreferrer" tabindex=3D"-1" href=3D"mailto:Sebastia=
[email protected]">[email protected]=
k</a>&gt;; 'Rahkonen Jukka (MML)' &lt;<a target=3D"_blank" rel=3D"externa=
l nofollow noopener noreferrer" tabindex=3D"-1" href=3D"mailto:jukka.rahk=
[email protected]">[email protected]</a>&gt;; '=
<a target=3D"_blank" rel=3D"external nofollow noopener noreferrer" tabind=
ex=3D"-1" href=3D"mailto:[email protected]">mapserver-users=
@lists.osgeo.org</a>' &lt;<a target=3D"_blank" rel=3D"external nofollow n=
oopener noreferrer" tabindex=3D"-1" href=3D"mailto:mapserver-users@lists.=
osgeo.org">[email protected]</a>&gt;<br><b>Subject:</b> RE:=
 mapcache seed speed optimization</span></p></div></div> <p><span style=
=3D"color: rgb(31,73,125)" lang=3D"EN-US">For data formats that are slowe=
r to read, would it add fit your workflow to convert it to different data=
 format before creating the tiles? </span></p> </div></div>______________=
_________________________________<br>mapserver-users mailing list<br><a t=
arget=3D"_blank" rel=3D"external nofollow noopener noreferrer" tabindex=
=3D"-1" href=3D"mailto:[email protected]">mapserver-users@l=
ists.osgeo.org</a><br><a rel=3D"external nofollow noopener noreferrer" ta=
rget=3D"_blank" tabindex=3D"-1" href=3D"https://lists.osgeo.org/mailman/l=
istinfo/mapserver-users">https://lists.osgeo.org/mailman/listinfo/mapserv=
er-users</a> </blockquote> </div> </div></div></blockquote> <br><br><sign=
ature></signature> </div></body></html>

----=_RainLoop_758_590447393.1573502842--

--===============0299715973488954194==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Disposition: inline

X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbWFwc2VydmVy
LXVzZXJzIG1haWxpbmcgbGlzdAptYXBzZXJ2ZXItdXNlcnNAbGlzdHMub3NnZW8ub3JnCmh0dHBz
Oi8vbGlzdHMub3NnZW8ub3JnL21haWxtYW4vbGlzdGluZm8vbWFwc2VydmVyLXVzZXJz

--===============0299715973488954194==--