mapcache seed speed optimization
Sebastiano Laini <[email protected]> Fri, 8 Nov 2019 12:34:40 +0000
| Newsgroups | gmane.comp.gis.mapserver.user |
|---|---|
| Message-ID | <bba2445c47974ff7ade03c9660c88d2d@BCExchange.BuchananComputing.co.uk> |
--===============4012511239791285099==
Content-Language: en-US
Content-Type: multipart/alternative;
boundary="_000_bba2445c47974ff7ade03c9660c88d2dBCExchangeBuchananCompu_"
--_000_bba2445c47974ff7ade03c9660c88d2dBCExchangeBuchananCompu_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Hi all,
I'm evaluating new stacks but seems that I'm stuck with mapserver and mapca=
che due to the input source of our maps, so I'm trying to improve the speed=
of the mapserver/mapcache stack and what I cannot improve is the seed spee=
d.
The -n and -p parameters seems to be useless.
-n | -nthreads: number of parallel threads that should be used to request t=
iles from the WMS source. The default is 1, but can be set higher if the WM=
S server can withstand parallel requests. (As a rule of thumb, the value ch=
osen here should never be much higher than the number of CPUs on the WMS se=
rver.)
-p | -nprocesses: number of parallel processes that should be used to reque=
st tiles from the WMS source.
I've tried to use -n 20000 and -p 60000 but for some reason the CPU load is=
max 100% in 1 core, no way to make it use more power and speed up the seed=
process.
Max memory used is 470MB between Centos 7 (with all the services running) a=
nd the seed process.
I've setup apache to use HTT/2 and it make quite a difference when the cach=
e is already seeded but during the seed process doesn't seems to be affecte=
d.
My MapServer build is:
./configure \
--with-ogr=3D/usr/local/bin/gdal-config \
--with-gdal=3D/usr/local/bin/gdal-config \
--with-wfsclient \
--with-wmsclient \
--with-wfs \
--enable-debug \
--with-curl-config=3D/usr/bin/curl-config \
--with-proj \
--with-jpeg \
--with-freetype \
--with-postgis=3D/usr/pgsql-9.3/bin/pg_config \
--with-geos=3D/usr/local/bin/geos-config
I know there is mapserver 7 and yet I'm using 6 for the test.
While the MapCache is:
cmake -DCMAKE_PREFIX_PATH=3D"/usr/bin/sqlite3" \
-DWITH_SQLITE=3D1 \
-DWITH_BERKELEY_DB=3D0 \
-DWITH_TIFF=3D0 \
-DWITH_GEOTIFF=3D0 \
-DWITH_FCGI=3D0 \
-DWITH_PCRE=3D0 \
-DWITH_PIXMAN=3D1 \
-DWITH_OGR=3D1 \
-DWITH_GEOS=3D1 \
../
MapCache is the latest version and this is my mapcache.xml file, we use sql=
ite as a cache method
<cache name=3D"cache_sqlite" type=3D"sqlite3">
<dbfile>/home/www/html/maps/cache/{tileset}/{grid}/{z}/{x}-{y}.sqlite3</d=
bfile>
<xcount>10000</xcount>
<ycount>10000</ycount>
<pragma name=3D"max_page_count">1573741823</pragma>
</cache>
<tileset name=3D"bccache">
<source>bcmaps</source>
<cache>cache_sqlite</cache>
<grid>bcgrid</grid>
<format>PNG</format>
<metatile>16 16</metatile>
<metabuffer>0</metabuffer>
<expires>2628000</expires>
</tileset>
<grid name=3D"bcgrid">
<metadata>
<title>BC custom grid</title>
</metadata>
<srs>EPSG:27700</srs>
<size>256 256</size>
<extent>0 0 700000 1250000</extent>
<resolutions>70 28 14 7 2.8 1.4 0.7</resolutions>
</grid>
<format name=3D"mypng" type=3D"PNG">
<compression>best</compression>
</format>
<service type=3D"wms" enabled=3D"true">
<full_wms>assemble</full_wms>
<resample_mode>bilinear</resample_mode>
<format>mypng</format>
<maxsize>4096</maxsize>
</service>
I've also tried to use -DWITH_MAPSERVER but seems that the integration is n=
ot working and they doen't care? https://github.com/mapserver/mapcache/issu=
es/169 this issue was open 2 years ago and still open, not even a reply fro=
m the developers.
How can I improve the seed process?
Should I increase the maxsize and metatile to requests more tiles and cache=
faster?
Regards,
Sebastiano Laini
Web Developer
Buchanan Computing
--_000_bba2445c47974ff7ade03c9660c88d2dBCExchangeBuchananCompu_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
code
{mso-style-priority:99;
font-family:"Courier New";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3D"EN-GB" link=3D"#0563C1" vlink=3D"#954F72">
<div class=3D"WordSection1">
<p class=3D"MsoNormal">Hi all,<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">I’m evaluating new stacks but seems that IR=
17;m stuck with mapserver and mapcache due to the input source of our maps,=
so I’m trying to improve the speed of the mapserver/mapcache stack a=
nd what I cannot improve is the seed speed.<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">The –n and –p parameters seems to be use=
less.<o:p></o:p></p>
<p style=3D"line-height:18.0pt;background:white"><strong><span style=3D"fon=
t-size:9.5pt;font-family:"Arial",sans-serif;color:#3E4349">-n | &=
#8211;nthreads</span></strong><span style=3D"font-size:9.5pt;font-family:&q=
uot;Arial",sans-serif;color:#3E4349">: number of parallel threads
that should be used to request tiles from the WMS source. The default is 1=
, but can be set higher if the WMS server can withstand parallel requests. =
(As a rule of thumb, the value chosen here should never be much higher than=
the number of CPUs on the WMS server.)<o:p></o:p></span></p>
<p style=3D"line-height:18.0pt;background:white"><strong><span style=3D"fon=
t-size:9.5pt;font-family:"Arial",sans-serif;color:#3E4349">-p | &=
#8211;nprocesses</span></strong><span style=3D"font-size:9.5pt;font-family:=
"Arial",sans-serif;color:#3E4349">: number of parallel
processes that should be used to request tiles from the WMS source.<o:p></=
o:p></span></p>
<p class=3D"MsoNormal">I’ve tried to use –n 20000 and –p =
60000 but for some reason the CPU load is max 100% in 1 core, no way to mak=
e it use more power and speed up the seed process.<o:p></o:p></p>
<p class=3D"MsoNormal">Max memory used is 470MB between Centos 7 (with all =
the services running) and the seed process.<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">I’ve setup apache to use HTT/2 and it make qui=
te a difference when the cache is already seeded but during the seed proces=
s doesn’t seems to be affected.<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">My MapServer build is:<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"><code><span style=3D"font-size:10.0pt">./configure \=
<o:p></o:p></span></code></p>
<p class=3D"MsoNormal"><code><span style=3D"font-size:10.0pt">--with-ogr=3D=
/usr/local/bin/gdal-config \<o:p></o:p></span></code></p>
<p class=3D"MsoNormal"><code><span style=3D"font-size:10.0pt">--with-gdal=
=3D/usr/local/bin/gdal-config \<o:p></o:p></span></code></p>
<p class=3D"MsoNormal"><code><span style=3D"font-size:10.0pt">--with-wfscli=
ent \<o:p></o:p></span></code></p>
<p class=3D"MsoNormal"><code><span style=3D"font-size:10.0pt">--with-wmscli=
ent \<o:p></o:p></span></code></p>
<p class=3D"MsoNormal"><code><span style=3D"font-size:10.0pt">--with-wfs \<=
o:p></o:p></span></code></p>
<p class=3D"MsoNormal"><code><span style=3D"font-size:10.0pt">--enable-debu=
g \<o:p></o:p></span></code></p>
<p class=3D"MsoNormal"><code><span style=3D"font-size:10.0pt">--with-curl-c=
onfig=3D/usr/bin/curl-config \<o:p></o:p></span></code></p>
<p class=3D"MsoNormal"><code><span style=3D"font-size:10.0pt">--with-proj \=
<o:p></o:p></span></code></p>
<p class=3D"MsoNormal"><code><span style=3D"font-size:10.0pt">--with-jpeg \=
<o:p></o:p></span></code></p>
<p class=3D"MsoNormal"><code><span style=3D"font-size:10.0pt">--with-freety=
pe \<o:p></o:p></span></code></p>
<p class=3D"MsoNormal"><code><span style=3D"font-size:10.0pt">--with-postgi=
s=3D/usr/pgsql-9.3/bin/pg_config \<o:p></o:p></span></code></p>
<p class=3D"MsoNormal"><code><span style=3D"font-size:10.0pt">--with-geos=
=3D/usr/local/bin/geos-config<o:p></o:p></span></code></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">I know there is mapserver 7 and yet I’m using =
6 for the test.<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">While the MapCache is: <o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">cmake -DCMAKE_PREFIX_PATH=3D"/usr/b=
in/sqlite3" \<o:p></o:p></p>
<p class=3D"MsoNormal"> -DWITH_SQ=
LITE=3D1 \<o:p></o:p></p>
<p class=3D"MsoNormal"> -DWITH_BE=
RKELEY_DB=3D0 \<o:p></o:p></p>
<p class=3D"MsoNormal"> -DWITH_TI=
FF=3D0 \<o:p></o:p></p>
<p class=3D"MsoNormal"> -DWITH_GE=
OTIFF=3D0 \<o:p></o:p></p>
<p class=3D"MsoNormal"> -DWITH_FC=
GI=3D0 \<o:p></o:p></p>
<p class=3D"MsoNormal"> -DWITH_PC=
RE=3D0 \<o:p></o:p></p>
<p class=3D"MsoNormal"> -DWITH_PI=
XMAN=3D1 \<o:p></o:p></p>
<p class=3D"MsoNormal"> -DWITH_OG=
R=3D1 \<o:p></o:p></p>
<p class=3D"MsoNormal"> -DWITH_GE=
OS=3D1 \<o:p></o:p></p>
<p class=3D"MsoNormal"> ../=
<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">MapCache is the latest version and this is my mapcac=
he.xml file, we use sqlite as a cache method<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"> <cache name=3D"cache_sqlite" typ=
e=3D"sqlite3"><o:p></o:p></p>
<p class=3D"MsoNormal"> <dbfile>/home/www/html/maps/cache/{tile=
set}/{grid}/{z}/{x}-{y}.sqlite3</dbfile><o:p></o:p></p>
<p class=3D"MsoNormal"> <xcount>10000</xcount><o:p></o:p>=
</p>
<p class=3D"MsoNormal"> <ycount>10000</ycount><o:p></o:p>=
</p>
<p class=3D"MsoNormal"> <pragma name=3D"max_page_count"&=
gt;1573741823</pragma><o:p></o:p></p>
<p class=3D"MsoNormal"> </cache><o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"> <tileset name=3D"bccache"><o:=
p></o:p></p>
<p class=3D"MsoNormal"> <source>bcmaps</source&g=
t;<o:p></o:p></p>
<p class=3D"MsoNormal"> <cache>cache_sqlite</cac=
he><o:p></o:p></p>
<p class=3D"MsoNormal"> <grid>bcgrid</grid><o=
:p></o:p></p>
<p class=3D"MsoNormal"> <format>PNG</format><=
o:p></o:p></p>
<p class=3D"MsoNormal"> <metatile>16 16</metatil=
e><o:p></o:p></p>
<p class=3D"MsoNormal"> <metabuffer>0</metabuffe=
r><o:p></o:p></p>
<p class=3D"MsoNormal"> <expires>2628000</expire=
s><o:p></o:p></p>
<p class=3D"MsoNormal"> </tileset><o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"> <grid name=3D"bcgrid"><o:p></=
o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; <metadata><o:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; <title>BC custom grid</title><=
o:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; </metadata><o:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; <srs>EPSG:27700</srs><o:p></o:=
p></p>
<p class=3D"MsoNormal"> &nbs=
p; <size>256 256</size><o:p></o:p=
></p>
<p class=3D"MsoNormal"> &nbs=
p; <extent>0 0 700000 1250000</exten=
t><o:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; <resolutions>70 28 14 7 2.8 1.4 0.7&=
lt;/resolutions><o:p></o:p></p>
<p class=3D"MsoNormal"> </grid><o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"> <format name=3D"mypng" type=3D&q=
uot;PNG"><o:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; <compression>best</compression>=
;<o:p></o:p></p>
<p class=3D"MsoNormal"> </format><o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"> <service type=3D"wms" enabled=3D=
"true"><o:p></o:p></p>
<p class=3D"MsoNormal"> <full_wms>assemble</full=
_wms><o:p></o:p></p>
<p class=3D"MsoNormal"> <resample_mode>bilinear<=
/resample_mode><o:p></o:p></p>
<p class=3D"MsoNormal"> <format>mypng</format>=
;<o:p></o:p></p>
<p class=3D"MsoNormal"> <maxsize>4096</maxsize&g=
t;<o:p></o:p></p>
<p class=3D"MsoNormal"> </service><o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">I’ve also tried to use –DWITH_MAPSERVER =
but seems that the integration is not working and they doen’t care?
<a href=3D"https://github.com/mapserver/mapcache/issues/169">https://github=
.com/mapserver/mapcache/issues/169</a> this issue was open 2 years ago and =
still open, not even a reply from the developers.<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">How can I improve the seed process?<o:p></o:p></p>
<p class=3D"MsoNormal">Should I increase the maxsize and metatile to reques=
ts more tiles and cache faster?<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Regards,<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D;mso-fareast-language:EN=
-GB">Sebastiano Laini<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D;mso-fareast-language:EN=
-GB">Web Developer<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D;mso-fareast-language:EN=
-GB">Buchanan Computing<o:p></o:p></span></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>
--_000_bba2445c47974ff7ade03c9660c88d2dBCExchangeBuchananCompu_--
--===============4012511239791285099==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Disposition: inline
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbWFwc2VydmVy
LXVzZXJzIG1haWxpbmcgbGlzdAptYXBzZXJ2ZXItdXNlcnNAbGlzdHMub3NnZW8ub3JnCmh0dHBz
Oi8vbGlzdHMub3NnZW8ub3JnL21haWxtYW4vbGlzdGluZm8vbWFwc2VydmVyLXVzZXJz
--===============4012511239791285099==--