File upload memory usage
Dries Deschout <[email protected]> Mon, 26 Mar 2018 13:21:39 +0000
| Newsgroups | gmane.comp.gcc.cgicc.general |
|---|---|
| Message-ID | <DB4PR04MB268B2487616FF352D7BF822DCAD0@DB4PR04MB268.eurprd04.prod.outlook.com> |
--_004_DB4PR04MB268B2487616FF352D7BF822DCAD0DB4PR04MB268eurprd_
Content-Type: multipart/alternative;
boundary="_000_DB4PR04MB268B2487616FF352D7BF822DCAD0DB4PR04MB268eurprd_"
--_000_DB4PR04MB268B2487616FF352D7BF822DCAD0DB4PR04MB268eurprd_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Hello,
While upgrading compilers from GCC 4.7 to GCC 7.2 on an embedded device, I =
noticed a substantial increase in memory usage while parsing a file upload =
request.
This seems to be caused by changed behavior in std::string. As GCC 4.7 was =
using copy-on-write and refcounting for std::string, a string copy did not =
cause a copy of the data. Now with GCC 7.2, a string copy also copies the d=
ata. For some large (+30MB) requests, this now makes our application go out=
-of-memory on a embedded device with only 128MB of RAM.
To bring memory usage back to the GCC 4.7 values, I patched Cgicc::parseMIM=
E to std::move the data to the FormFile and FormEntry (see attached patch).
As this patch is using some C++ 11 features, I was wondering if you do acce=
pt patches that (conditionally) use C++ 11 (or higher) features?
I could rework the patch to only use those features when supported by the _=
_cplusplus version.
Best Regards,
Dries
Secure your future - Meet Newtec Dialog=AE<http://www.newtec.eu/product/new=
tec-dialog>, the platform that embraces change - with award-winning Mx-DMA=
=AE<http://www.newtec.eu/technology/mx-dma>!
***mail confidentiality footer ***
This message and any attachments thereto are confidential. They may also be=
privileged or otherwise protected by work product immunity or other legal =
rules. If you have received it by mistake, please let us know by e-mail rep=
ly and delete it from your system; you may not copy this message or disclos=
e its contents to anyone. E-mail transmission cannot be guaranteed to be se=
cure or error free as information could be intercepted, corrupted, lost, de=
stroyed, arrive late or incomplete, or contain viruses. The sender therefor=
e is in no way liable for any errors or omissions in the content of this me=
ssage, which may arise as a result of e-mail transmission. If verification =
is required, please request a hard copy.
--_000_DB4PR04MB268B2487616FF352D7BF822DCAD0DB4PR04MB268eurprd_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
<style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style>
</head>
<body dir=3D"ltr">
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Helvetica,sans-serif;" dir=3D"ltr">
<p style=3D"margin-top:0;margin-bottom:0">Hello,</p>
<p style=3D"margin-top:0;margin-bottom:0"><br>
</p>
<p style=3D"margin-top:0;margin-bottom:0">While upgrading compilers from GC=
C 4.7 to GCC 7.2 on an embedded device, I noticed a substantial increase in=
memory usage while parsing a file upload request.<br>
</p>
<p style=3D"margin-top:0;margin-bottom:0"><br>
</p>
<p style=3D"margin-top:0;margin-bottom:0">This seems to be caused by change=
d behavior in std::string. As GCC 4.7 was using copy-on-write and refcounti=
ng for std::string, a string copy did not cause a copy of the data. Now wit=
h GCC 7.2, a string copy also copies
the data. For some large (+30MB) requests, this now makes our applicat=
ion go out-of-memory on a embedded device with only 128MB of RAM.</p>
<p style=3D"margin-top:0;margin-bottom:0"><br>
</p>
<p style=3D"margin-top:0;margin-bottom:0">To bring memory usage back to the=
GCC 4.7 values, I patched Cgicc::parseMIME to std::move the data to the Fo=
rmFile and FormEntry (see attached patch).<br>
</p>
<p style=3D"margin-top:0;margin-bottom:0"><br>
</p>
<p style=3D"margin-top:0;margin-bottom:0">As this patch is using some C+=
;+ 11 features, I was wondering if you do accept patches that (conditio=
nally) use C++ 11 (or higher) features?
<br>
</p>
<p style=3D"margin-top:0;margin-bottom:0"><span>I could rework the patch to=
only use those features when supported by the __cplusplus version.</span><=
br>
</p>
<br>
<p style=3D"margin-top:0;margin-bottom:0">Best Regards,</p>
<p style=3D"margin-top:0;margin-bottom:0">Dries<br>
</p>
</div>
Secure your future – <a href=3D"http://www.newtec.eu/product/newtec-d=
ialog">Meet Newtec Dialog=AE</a>, the platform that embraces change –=
with award-winning
<a href=3D"http://www.newtec.eu/technology/mx-dma">Mx-DMA=AE</a>! <br>
***mail confidentiality footer *** <br>
This message and any attachments thereto are confidential. They may also be=
privileged or otherwise protected by work product immunity or other legal =
rules. If you have received it by mistake, please let us know by e-mail rep=
ly and delete it from your system;
you may not copy this message or disclose its contents to anyone. E-mail t=
ransmission cannot be guaranteed to be secure or error free as information =
could be intercepted, corrupted, lost, destroyed, arrive late or incomplete=
, or contain viruses. The sender
therefore is in no way liable for any errors or omissions in the content o=
f this message, which may arise as a result of e-mail transmission. If veri=
fication is required, please request a hard copy.
</body>
</html>
--_000_DB4PR04MB268B2487616FF352D7BF822DCAD0DB4PR04MB268eurprd_--
--_004_DB4PR04MB268B2487616FF352D7BF822DCAD0DB4PR04MB268eurprd_
Content-Type: text/x-patch; name="01-move-form-data.patch"
Content-Description: 01-move-form-data.patch
Content-Disposition: attachment; filename="01-move-form-data.patch";
size=2282; creation-date="Mon, 26 Mar 2018 12:44:09 GMT";
modification-date="Mon, 26 Mar 2018 12:44:09 GMT"
Content-Transfer-Encoding: base64
ZGlmZiAtdXJOIGNnaWNjLm9yaWcvQ2dpY2MuY3BwIGNnaWNjL0NnaWNjLmNwcAotLS0gY2dpY2Mu
b3JpZy9DZ2ljYy5jcHAJMjAxNy0wNi0yNyAyMTozODowMi4wMDAwMDAwMDAgKzAyMDAKKysrIGNn
aWNjL0NnaWNjLmNwcAkyMDE4LTAzLTI2IDA4OjMwOjEzLjUwNzU0NzAwMCArMDIwMApAQCAtNDk3
LDEwICs0OTcsMTAgQEAKICAgTXVsdGlwYXJ0SGVhZGVyIGhlYWQgPSBwYXJzZUhlYWRlcihkYXRh
LnN1YnN0cigwLCB2YWx1ZVN0YXJ0KSk7CiAKICAgaWYoaGVhZC5nZXRGaWxlbmFtZSgpLmVtcHR5
KCkpCi0gICAgZkZvcm1EYXRhLnB1c2hfYmFjayhGb3JtRW50cnkoaGVhZC5nZXROYW1lKCksIHZh
bHVlKSk7CisgICAgZkZvcm1EYXRhLmVtcGxhY2VfYmFjayhoZWFkLmdldE5hbWUoKSwgc3RkOjpt
b3ZlKHZhbHVlKSk7CiAgIGVsc2UKLSAgICBmRm9ybUZpbGVzLnB1c2hfYmFjayhGb3JtRmlsZSho
ZWFkLmdldE5hbWUoKSwgCisgICAgZkZvcm1GaWxlcy5lbXBsYWNlX2JhY2soaGVhZC5nZXROYW1l
KCksIAogCQkJCSAgaGVhZC5nZXRGaWxlbmFtZSgpLCAKIAkJCQkgIGhlYWQuZ2V0Q29udGVudFR5
cGUoKSwgCi0JCQkJICB2YWx1ZSkpOworCQkJCSAgc3RkOjptb3ZlKHZhbHVlKSk7CiB9CmRpZmYg
LXVyTiBjZ2ljYy5vcmlnL0Zvcm1FbnRyeS5oIGNnaWNjL0Zvcm1FbnRyeS5oCi0tLSBjZ2ljYy5v
cmlnL0Zvcm1FbnRyeS5oCTIwMTctMDYtMjcgMjE6Mzg6MDIuMDAwMDAwMDAwICswMjAwCisrKyBj
Z2ljYy9Gb3JtRW50cnkuaAkyMDE4LTAzLTI2IDA4OjI5OjE1Ljc2NzYzODAwMCArMDIwMApAQCAt
OTcsNiArOTcsMTEgQEAKIAkgICAgICBjb25zdCBzdGQ6OnN0cmluZyYgdmFsdWUpCiAgICAgICA6
IGZOYW1lKG5hbWUpLCBmVmFsdWUodmFsdWUpCiAgICAge30KKyAgICBpbmxpbmUKKyAgICBGb3Jt
RW50cnkoc3RkOjpzdHJpbmcmJiBuYW1lLCAKKwkgICAgICBzdGQ6OnN0cmluZyYmIHZhbHVlKQor
ICAgICAgOiBmTmFtZShzdGQ6OmZvcndhcmQ8c3RkOjpzdHJpbmc+KG5hbWUpKSwgZlZhbHVlKHN0
ZDo6Zm9yd2FyZDxzdGQ6OnN0cmluZz4odmFsdWUpKQorICAgIHt9CiAgICAgCiAgICAgLyohCiAg
ICAgICogXGJyaWVmIENvcHkgY29uc3RydWN0b3IuCmRpZmYgLXVyTiBjZ2ljYy5vcmlnL0Zvcm1G
aWxlLmNwcCBjZ2ljYy9Gb3JtRmlsZS5jcHAKLS0tIGNnaWNjLm9yaWcvRm9ybUZpbGUuY3BwCTIw
MTctMDYtMjcgMjE6Mzg6MDIuMDAwMDAwMDAwICswMjAwCisrKyBjZ2ljYy9Gb3JtRmlsZS5jcHAJ
MjAxOC0wMy0yNiAwOToxNDo0My43NTA5NTczMDkgKzAyMDAKQEAgLTM5LDYgKzM5LDE3IEBACiAg
IGZEYXRhVHlwZSA9IGRhdGFUeXBlLmVtcHR5KCkgPyBzdGQ6OnN0cmluZygidGV4dC9wbGFpbiIp
IDogZGF0YVR5cGU7CiB9CiAKK2NnaWNjOjpGb3JtRmlsZTo6Rm9ybUZpbGUoc3RkOjpzdHJpbmcm
JiBuYW1lLCAKKwkJCSAgc3RkOjpzdHJpbmcmJiBmaWxlbmFtZSwgCisJCQkgIHN0ZDo6c3RyaW5n
JiYgZGF0YVR5cGUsIAorCQkJICBzdGQ6OnN0cmluZyYmIGRhdGEpCisgIDogZk5hbWUoc3RkOjpm
b3J3YXJkPHN0ZDo6c3RyaW5nPihuYW1lKSksCisgICAgZkZpbGVuYW1lKHN0ZDo6Zm9yd2FyZDxz
dGQ6OnN0cmluZz4oZmlsZW5hbWUpKSwKKyAgICBmRGF0YShzdGQ6OmZvcndhcmQ8c3RkOjpzdHJp
bmc+KGRhdGEpKQoreworICBmRGF0YVR5cGUgPSBkYXRhVHlwZS5lbXB0eSgpID8gc3RkOjpzdHJp
bmcoInRleHQvcGxhaW4iKSA6IGRhdGFUeXBlOworfQorCiBib29sCiBjZ2ljYzo6Rm9ybUZpbGU6
Om9wZXJhdG9yPT0gKGNvbnN0IEZvcm1GaWxlJiBmaWxlKSAJCWNvbnN0CiB7CmRpZmYgLXVyTiBj
Z2ljYy5vcmlnL0Zvcm1GaWxlLmggY2dpY2MvRm9ybUZpbGUuaAotLS0gY2dpY2Mub3JpZy9Gb3Jt
RmlsZS5oCTIwMTctMDYtMjcgMjE6Mzg6MDIuMDAwMDAwMDAwICswMjAwCisrKyBjZ2ljYy9Gb3Jt
RmlsZS5oCTIwMTgtMDMtMjYgMDg6NTk6MDEuODEzNzAxMDAwICswMjAwCkBAIC05Miw2ICs5Miwx
MCBAQAogCSAgICAgY29uc3Qgc3RkOjpzdHJpbmcmIGZpbGVuYW1lLCAKIAkgICAgIGNvbnN0IHN0
ZDo6c3RyaW5nJiBkYXRhVHlwZSwgCiAJICAgICBjb25zdCBzdGQ6OnN0cmluZyYgZGF0YSk7Cisg
ICAgRm9ybUZpbGUoc3RkOjpzdHJpbmcmJiBuYW1lLCAKKwkgICAgIHN0ZDo6c3RyaW5nJiYgZmls
ZW5hbWUsIAorCSAgICAgc3RkOjpzdHJpbmcmJiBkYXRhVHlwZSwgCisJICAgICBzdGQ6OnN0cmlu
ZyYmIGRhdGEpOwogICAgIAogICAgIC8qIQogICAgICAqIFxicmllZiBDb3B5IGNvbnN0cnVjdG9y
Lgo=
--_004_DB4PR04MB268B2487616FF352D7BF822DCAD0DB4PR04MB268eurprd_
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
help-cgicc mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/help-cgicc
--_004_DB4PR04MB268B2487616FF352D7BF822DCAD0DB4PR04MB268eurprd_--