Re: mapcache seed speed optimization
Yves Jacolin <[email protected]> Fri, 8 Nov 2019 14:42:03 +0100
| Newsgroups | gmane.comp.gis.mapserver.user |
|---|---|
| Message-ID | <CAO8EQxbrPQ0eYf71JnLLqXGqEt7aA32a86R5+HNLjkvqoxgDqQ@mail.gmail.com> |
--===============0178363968483909200== Content-Type: multipart/alternative; boundary="00000000000024770b0596d5f36a" --00000000000024770b0596d5f36a Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Sebastiano, Some though in my mind: * =E2=80=93n 20000 : do you really have 2'000 CPU on your WMS server? if no= t, it could be worst than better, flooding your WMS server. * what is the bottleneck? If WMS server is slow you can't make MapCache faster, you should optimise WMS Server. Multi-domain can help, data improvement, etc. also. * Does your WMS is on the same server? same network? Does you network fast enough? * metatile can help definitively! * "we use sqlite as a cache method" =3D> I don't know if sqlite is really fast or not, but this could be another bottleneck :) What do you mean exactly by improve seed speed? :) This is matter of personal point of view some time. Y. Le ven. 8 nov. 2019 =C3=A0 13:43, Sebastiano Laini < [email protected]> a =C3=A9crit : > Hi all, > > > > I=E2=80=99m evaluating new stacks but seems that I=E2=80=99m stuck with m= apserver and > mapcache due to the input source of our maps, so I=E2=80=99m trying to im= prove the > speed of the mapserver/mapcache stack and what I cannot improve is the se= ed > speed. > > > > The =E2=80=93n and =E2=80=93p parameters seems to be useless. > > *-n | =E2=80=93nthreads*: number of parallel threads that should be used = to > request tiles from the WMS source. The default is 1, but can be set highe= r > if the WMS server can withstand parallel requests. (As a rule of thumb, t= he > value chosen here should never be much higher than the number of CPUs on > the WMS server.) > > *-p | =E2=80=93nprocesses*: number of parallel processes that should be u= sed to > request tiles from the WMS source. > > I=E2=80=99ve tried to use =E2=80=93n 20000 and =E2=80=93p 60000 but for s= ome reason the CPU load > is max 100% in 1 core, no way to make it use more power and speed up the > seed process. > > Max memory used is 470MB between Centos 7 (with all the services running) > and the seed process. > > > > I=E2=80=99ve setup apache to use HTT/2 and it make quite a difference whe= n the > cache is already seeded but during the seed process doesn=E2=80=99t seems= to be > affected. > > > > My MapServer build is: > > > > ./configure \ > > --with-ogr=3D/usr/local/bin/gdal-config \ > > --with-gdal=3D/usr/local/bin/gdal-config \ > > --with-wfsclient \ > > --with-wmsclient \ > > --with-wfs \ > > --enable-debug \ > > --with-curl-config=3D/usr/bin/curl-config \ > > --with-proj \ > > --with-jpeg \ > > --with-freetype \ > > --with-postgis=3D/usr/pgsql-9.3/bin/pg_config \ > > --with-geos=3D/usr/local/bin/geos-config > > > > I know there is mapserver 7 and yet I=E2=80=99m using 6 for the test. > > > > While the MapCache is: > > > > cmake -DCMAKE_PREFIX_PATH=3D"/usr/bin/sqlite3" \ > > -DWITH_SQLITE=3D1 \ > > -DWITH_BERKELEY_DB=3D0 \ > > -DWITH_TIFF=3D0 \ > > -DWITH_GEOTIFF=3D0 \ > > -DWITH_FCGI=3D0 \ > > -DWITH_PCRE=3D0 \ > > -DWITH_PIXMAN=3D1 \ > > -DWITH_OGR=3D1 \ > > -DWITH_GEOS=3D1 \ > > ../ > > > > MapCache is the latest version and this is my mapcache.xml file, we use > sqlite as a cache method > > > > <cache name=3D"cache_sqlite" type=3D"sqlite3"> > > > <dbfile>/home/www/html/maps/cache/{tileset}/{grid}/{z}/{x}-{y}.sqlite3</d= bfile> > > <xcount>10000</xcount> > > <ycount>10000</ycount> > > <pragma name=3D"max_page_count">1573741823</pragma> > > </cache> > > > > <tileset name=3D"bccache"> > > <source>bcmaps</source> > > <cache>cache_sqlite</cache> > > <grid>bcgrid</grid> > > <format>PNG</format> > > <metatile>16 16</metatile> > > <metabuffer>0</metabuffer> > > <expires>2628000</expires> > > </tileset> > > > > <grid name=3D"bcgrid"> > > <metadata> > > <title>BC custom grid</title> > > </metadata> > > <srs>EPSG:27700</srs> > > <size>256 256</size> > > <extent>0 0 700000 1250000</extent> > > <resolutions>70 28 14 7 2.8 1.4 0.7</resolutions> > > </grid> > > > > <format name=3D"mypng" type=3D"PNG"> > > <compression>best</compression> > > </format> > > > > <service type=3D"wms" enabled=3D"true"> > > <full_wms>assemble</full_wms> > > <resample_mode>bilinear</resample_mode> > > <format>mypng</format> > > <maxsize>4096</maxsize> > > </service> > > > > I=E2=80=99ve also tried to use =E2=80=93DWITH_MAPSERVER but seems that th= e integration is > not working and they doen=E2=80=99t care? > https://github.com/mapserver/mapcache/issues/169 this issue was open 2 > years ago and still open, not even a reply from the developers. > > > > How can I improve the seed process? > > Should I increase the maxsize and metatile to requests more tiles and > cache faster? > > > > Regards, > > > > Sebastiano Laini > > Web Developer > > Buchanan Computing > > > _______________________________________________ > mapserver-users mailing list > [email protected] > https://lists.osgeo.org/mailman/listinfo/mapserver-users --=20 Yves Jacolin Training and support manager - Team Manager Camptocamp Tel (France) : +33 4 58 48 20 43 Tel (Switzerland) : +41 21 619 10 43 Mob. : +33 6 18 75 42 21 email : [email protected] http://www.camptocamp.com --00000000000024770b0596d5f36a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Sebastiano,</div><div><br></div><div>Some though in m= y mind: <br></div><div>* =E2=80=93n 20000 : do you really have 2'000 CP= U on your WMS server? if not, it could be worst than better, flooding your = WMS server.<br></div><div>* what is the bottleneck?=C2=A0 If WMS server is = slow you can't make MapCache faster, you should optimise WMS Server. Mu= lti-domain can help, data improvement, etc. also.</div><div>* Does your WMS= is on the same server? same network? Does you network fast enough?</div><d= iv>* metatile can help definitively!<br></div><div>* "we use sqlite as= a cache method" =3D> I don't know if sqlite is really fast or = not, but this could be another bottleneck :)</div><div><br></div><div>What = do you mean exactly by improve seed speed? :) This is matter of personal po= int of view some time.</div><div><br></div><div>Y.<br></div></div><br><div = class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">Le=C2=A0ven. 8 = nov. 2019 =C3=A0=C2=A013:43, Sebastiano Laini <<a href=3D"mailto:Sebasti= [email protected]">[email protected]= </a>> a =C3=A9crit=C2=A0:<br></div><blockquote class=3D"gmail_quote" sty= le=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);paddi= ng-left:1ex"> <div lang=3D"EN-GB"> <div class=3D"gmail-m_5762177471845286621WordSection1"> <p class=3D"MsoNormal">Hi all,<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">I=E2=80=99m evaluating new stacks but seems that I= =E2=80=99m stuck with mapserver and mapcache due to the input source of our= maps, so I=E2=80=99m trying to improve the speed of the mapserver/mapcache= stack and what I cannot improve is the seed speed.<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">The =E2=80=93n and =E2=80=93p parameters seems to be= useless.<u></u><u></u></p> <p style=3D"line-height:18pt;background:white none repeat scroll 0% 0%"><st= rong><span style=3D"font-size:9.5pt;font-family:"Arial",sans-seri= f;color:rgb(62,67,73)">-n | =E2=80=93nthreads</span></strong><span style=3D= "font-size:9.5pt;font-family:"Arial",sans-serif;color:rgb(62,67,7= 3)">: number of parallel threads that should be used to request tiles from the WMS source. The default is 1= , but can be set higher if the WMS server can withstand parallel requests. = (As a rule of thumb, the value chosen here should never be much higher than= the number of CPUs on the WMS server.)<u></u><u></u></span></p> <p style=3D"line-height:18pt;background:white none repeat scroll 0% 0%"><st= rong><span style=3D"font-size:9.5pt;font-family:"Arial",sans-seri= f;color:rgb(62,67,73)">-p | =E2=80=93nprocesses</span></strong><span style= =3D"font-size:9.5pt;font-family:"Arial",sans-serif;color:rgb(62,6= 7,73)">: number of parallel processes that should be used to request tiles from the WMS source.<u></u>= <u></u></span></p> <p class=3D"MsoNormal">I=E2=80=99ve tried to use =E2=80=93n 20000 and =E2= =80=93p 60000 but for some reason the CPU load is max 100% in 1 core, no wa= y to make it use more power and speed up the seed process.<u></u><u></u></p= > <p class=3D"MsoNormal">Max memory used is 470MB between Centos 7 (with all = the services running) and the seed process.<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">I=E2=80=99ve setup apache to use HTT/2 and it make q= uite a difference when the cache is already seeded but during the seed proc= ess doesn=E2=80=99t seems to be affected.<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">My MapServer build is:<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal"><code><span style=3D"font-size:10pt">./configure \<u= ></u><u></u></span></code></p> <p class=3D"MsoNormal"><code><span style=3D"font-size:10pt">--with-ogr=3D/u= sr/local/bin/gdal-config \<u></u><u></u></span></code></p> <p class=3D"MsoNormal"><code><span style=3D"font-size:10pt">--with-gdal=3D/= usr/local/bin/gdal-config \<u></u><u></u></span></code></p> <p class=3D"MsoNormal"><code><span style=3D"font-size:10pt">--with-wfsclien= t \<u></u><u></u></span></code></p> <p class=3D"MsoNormal"><code><span style=3D"font-size:10pt">--with-wmsclien= t \<u></u><u></u></span></code></p> <p class=3D"MsoNormal"><code><span style=3D"font-size:10pt">--with-wfs \<u>= </u><u></u></span></code></p> <p class=3D"MsoNormal"><code><span style=3D"font-size:10pt">--enable-debug = \<u></u><u></u></span></code></p> <p class=3D"MsoNormal"><code><span style=3D"font-size:10pt">--with-curl-con= fig=3D/usr/bin/curl-config \<u></u><u></u></span></code></p> <p class=3D"MsoNormal"><code><span style=3D"font-size:10pt">--with-proj \<u= ></u><u></u></span></code></p> <p class=3D"MsoNormal"><code><span style=3D"font-size:10pt">--with-jpeg \<u= ></u><u></u></span></code></p> <p class=3D"MsoNormal"><code><span style=3D"font-size:10pt">--with-freetype= \<u></u><u></u></span></code></p> <p class=3D"MsoNormal"><code><span style=3D"font-size:10pt">--with-postgis= =3D/usr/pgsql-9.3/bin/pg_config \<u></u><u></u></span></code></p> <p class=3D"MsoNormal"><code><span style=3D"font-size:10pt">--with-geos=3D/= usr/local/bin/geos-config<u></u><u></u></span></code></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">I know there is mapserver 7 and yet I=E2=80=99m usin= g 6 for the test.<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">While the MapCache is: <u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">cmake=C2=A0=C2=A0 -DCMAKE_PREFIX_PATH=3D"/usr/b= in/sqlite3" \<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -DWITH_SQ= LITE=3D1 \<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -DWITH_BE= RKELEY_DB=3D0 \<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -DWITH_TI= FF=3D0 \<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -DWITH_GE= OTIFF=3D0 \<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -DWITH_FC= GI=3D0 \<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -DWITH_PC= RE=3D0 \<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -DWITH_PI= XMAN=3D1 \<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -DWITH_OG= R=3D1 \<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -DWITH_GE= OS=3D1 \<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ../= <u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">MapCache is the latest version and this is my mapcac= he.xml file, we use sqlite as a cache method<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">=C2=A0 <cache name=3D"cache_sqlite" typ= e=3D"sqlite3"><u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0 <dbfile>/home/www/html/maps/cache/{tile= set}/{grid}/{z}/{x}-{y}.sqlite3</dbfile><u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0 <xcount>10000</xcount><u></u><u><= /u></p> <p class=3D"MsoNormal">=C2=A0 <ycount>10000</ycount><u></u><u><= /u></p> <p class=3D"MsoNormal">=C2=A0 <pragma name=3D"max_page_count"&= gt;1573741823</pragma><u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0 </cache><u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">=C2=A0 <tileset name=3D"bccache"><u>= </u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0 <source>bcmaps</source&g= t;<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0 <cache>cache_sqlite</cac= he><u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0 <grid>bcgrid</grid><u= ></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0 <format>PNG</format><= u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0 <metatile>16 16</metatil= e><u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0 <metabuffer>0</metabuffe= r><u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0 <expires>2628000</expire= s><u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0 </tileset><u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">=C2=A0 <grid name=3D"bcgrid"><u></u>= <u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <metadata><u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <title>BC custom grid</title>= <u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 </metadata><u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <srs>EPSG:27700</srs><u></u><= u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <size>256 256</size><u></u><u= ></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <extent>0 0 700000 1250000</exte= nt><u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <resolutions>70 28 14 7 2.8 1.4 0.7= </resolutions><u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0 </grid><u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">=C2=A0 <format name=3D"mypng" type=3D&q= uot;PNG"><u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <compression>best</compression&g= t;<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0 </format><u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">=C2=A0 <service type=3D"wms" enabled=3D= "true"><u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0 <full_wms>assemble</full= _wms><u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0 <resample_mode>bilinear<= /resample_mode><u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0 <format>mypng</format>= ;<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0 <maxsize>4096</maxsize&g= t;<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0 </service><u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">I=E2=80=99ve also tried to use =E2=80=93DWITH_MAPSER= VER but seems that the integration is not working and they doen=E2=80=99t c= are? <a href=3D"https://github.com/mapserver/mapcache/issues/169" target=3D"_bla= nk">https://github.com/mapserver/mapcache/issues/169</a> this issue was ope= n 2 years ago and still open, not even a reply from the developers.<u></u><= u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">How can I improve the seed process?<u></u><u></u></p= > <p class=3D"MsoNormal">Should I increase the maxsize and metatile to reques= ts more tiles and cache faster?<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">Regards,<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal"><span style=3D"color:rgb(31,73,125)">Sebastiano Lain= i<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"color:rgb(31,73,125)">Web Developer<u= ></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"color:rgb(31,73,125)">Buchanan Comput= ing<u></u><u></u></span></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> </div> </div> _______________________________________________<br> mapserver-users mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">mapser= [email protected]</a><br> <a href=3D"https://lists.osgeo.org/mailman/listinfo/mapserver-users" rel=3D= "noreferrer" target=3D"_blank">https://lists.osgeo.org/mailman/listinfo/map= server-users</a></blockquote></div><br clear=3D"all"><br>-- <br><div dir=3D= "ltr" class=3D"gmail_signature"><div dir=3D"ltr"><div><div dir=3D"ltr"><div= ><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir= =3D"ltr"><div><div dir=3D"ltr"><div>Yves Jacolin</div><div>Training and sup= port manager - Team Manager<br>Camptocamp<br><br>Tel (France) : +33 4 58 48= 20 43<br>Tel (Switzerland) : +41 21 619 10 43<br>Mob. : +33 6 18 75 42 21<= br><br>email : <a href=3D"mailto:[email protected]" target=3D"_bl= ank">[email protected]</a><br><a href=3D"http://www.camptocamp.co= m" target=3D"_blank">http://www.camptocamp.com</a></div></div></div></div><= /div></div></div></div></div></div></div></div></div></div></div> --00000000000024770b0596d5f36a-- --===============0178363968483909200== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbWFwc2VydmVy LXVzZXJzIG1haWxpbmcgbGlzdAptYXBzZXJ2ZXItdXNlcnNAbGlzdHMub3NnZW8ub3JnCmh0dHBz Oi8vbGlzdHMub3NnZW8ub3JnL21haWxtYW4vbGlzdGluZm8vbWFwc2VydmVyLXVzZXJz --===============0178363968483909200==--