Re: Higher Memory Usage with vsicurl and gdal > 2.2.4

I solved my problem... I found a way to reduce memory utilization and
allocations when using MapServer with GDAL >= 2.3.0:

     CONFIG "GDAL_HTTP_MULTIRANGE"       "SERIAL"

With this setting, the memory footprint matches that of GDAL <= 2.2.4.

With the default (YES) has mapserv allocates much more memory. Running
mapserv with supervisor & configured over FCGI results in a much higher
memory footprint over time. My best guess is that with the default parallel
mode has many HTTP connections kept alive and each connection consumes some
non-trivial amount of memory.  With CONFIG "CPL_CURL_VERBOSE" "ON", I see 9
HTTP connections kept around only with the default GDAL_HTTP_MULTIRANGE=ON
setting:

* Curl_http_done: called premature == 0
* Connection #17 to host s3.amazonaws.com left intact
* Curl_http_done: called premature == 0
* Connection #11 to host s3.amazonaws.com left intact
* Curl_http_done: called premature == 0
* Connection #13 to host s3.amazonaws.com left intact
* Curl_http_done: called premature == 0
* Connection #14 to host s3.amazonaws.com left intact
* Curl_http_done: called premature == 0
* Connection #15 to host s3.amazonaws.com left intact
* Curl_http_done: called premature == 0
* Connection #19 to host s3.amazonaws.com left intact
* Curl_http_done: called premature == 0
* Connection #16 to host s3.amazonaws.com left intact
* Curl_http_done: called premature == 0
* Connection #18 to host s3.amazonaws.com left intact
* Curl_http_done: called premature == 0
* Connection #12 to host s3.amazonaws.com left intact

Side-note: the comments in cpl_vsil_curl.cpp for GDAL_HTTP_MULTIRANGE do
not match the ConfigOptions documentation
https://github.com/OSGeo/gdal/blob/1d799cf5d8c399d9ac3feb7ef9f05dc6e3297a82/gdal/port/cpl_vsil_curl.cpp#L2691-L2696
https://trac.osgeo.org/gdal/wiki/ConfigOptions#GDAL_HTTP_MULTIRANGE
The valid settings are actually SINGLE_GET, SERIAL or _anything_ else for
"parallel" mode

It's great to see all the vsicurl improvements to GDAL which MapServer can
leverage!  Time for me to experiment with the other gdal-2.3 config options
for libcurl >= 7.33 built with nghttp2.

Cheers,
Pete

On Fri, Mar 29, 2019 at 3:02 PM Peter Schmitt <[email protected]> wrote:

> Hi,
>
> I run MapServer in a Docker container. I noticed its memory utilization
> climbed much higher than expected. This seems to happen only under the
> following conditions:
>
> * Files are accessed network-based filesystems (I tried both /vsicurl/ and
> /vsis3/)
> * Using gdal  > 2.2.4 (tested with 2.3.0, 2.3.2 & 2.4.1.  I see the same
> behavior with both MapServer-7.0.7 and MapServer-7.2.2.
>
> I had a hard time isolating this to GDAL and could not reproduce with
> similar gdal_translate commands.  So I put together a little example to try
> and demonstrate the problem.
> https://github.com/pedros007/gdal_mapserver_memory
>
> Does anyone have any ideas as to what might be going on? I tried various
> cache settings, which do not seem to reduce memory utilization:
>
> https://github.com/pedros007/gdal_mapserver_memory/blob/master/mapfiles/indonesian_redcross.map#L15-L18
>
> Thanks,
> Pete
>


-- 
Pete

_______________________________________________
mapserver-users mailing list
[email protected]
https://lists.osgeo.org/mailman/listinfo/mapserver-users