Re: Performance woes - larger JSON payloads with CRS

Hey Henri,

You are in a bad situation and as far as I can see you are right, you might
have to drop modsec/CRS in this situation.

I've had a customer with a similar problem and we did a deep dive
investigation and I had to strike colors in the end.

The point is not the JSON parser. That has shown to be really fast. The point
is several hundred variables that go into CRS afterwards. If you run CRS on a
standard web application you get forms with a few parameters and that's easy.
But several megabytes of JSON means hundreds of arguments and CRS parses them
all.

So we tried to work with rule exclusions and skip the parameters we did not
think dangerous, but here comes the bummer: ModSec 2.9 grew substantially
slower the longer the ignore-lists of parameters became. This and a few very
odd behaviors.

Given the customer wanted a generic WAF without tuning of individual APIs we
got to a dead end.

However, if tuning was an option, then I would probably edit-CRS with
msc_pyparser and replace the target lists with arguments I was interested in.

https://coreruleset.org/20200901/introducing-msc_pyparser/

As a complementary practice, one could think of performing allowlist checks on
some / most of the JSON. Say you have a huge JSON payload with 500 parameters.
You examine it and discover that 300 of them actually contain simple digits
and asciii characters and neither special chars nor escape sequences.
So you do a regex allowlist and apply it to these 300 parameters of said
API. And the rest you can push into CRS. Or a subset of CRS.

I have not done this and the problem is if ModSec is able to handle the large
target lists in a speedy manner.


Now you can turn to a CDN or alternative WAF. I would do an extensive security
tests of such a system. As I said, the JSON parser can be really fast. The
difficult thing is to check several hundred parameters without losing
performance.

Good luck!

Christian


On Sun, Apr 25, 2021 at 08:47:06PM +0100, Henri Cook wrote:
> Hi all,
> 
> I'm in a situation where the only solution seems to be to drop modsec/CRS
> and look at something like Cloudflare's WAF (and change our security model
> out of necessity). I'm hoping the esteemed membership of this list might
> have some thoughts.
> 
> I've got about 1MB of JSON, payloads in our app might run to 20 or even
> 30MB ultimately.
> This 1MB of somewhat nested JSON (7 or 8 levels deep) can take 40 seconds
> to process in mod sec 3.0.4 with CRS 3.2.0
> 
> It takes 1 second to process in our API so the WAF element is a 39x slow
> down. I appreciate there'll be some delays in WAF. Cloudflare's WAF takes 5
> seconds to scan this payload - and that's my target.
> 
> Has anyone got any idea how to improve performance? Reading blog posts
> about the development of cloudflare's waf I see that memoization of common
> function calls was one of their absolute best performance improvements over
> their modsec implementation (e.g. strlen(response_body) so it's only
> calculated once instead of once per rule OR contains('somestring',
> response_body)... you get the drift). Do we have anything like this in
> modsec today? Is that already in place and my 39 seconds is after that?
> 
> I appreciate that mod sec is fast on its own and adding complex rules can
> be said to slow it down. With CRS being by far the most common use case for
> mod sec (based on my googling) I'm surprised it's this slow, do you think
> i've missed something?
> 
> To note: I'm only scanning JSON payloads, typically much less than 0.5MB
> but new, irregular ones that we need scanned in ideally <10 seconds that
> can range from 1MB-30MB
> 
> Best regards,
> 
> Henri Cook


> _______________________________________________
> mod-security-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/



_______________________________________________
mod-security-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/mod-security-users
Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
http://www.modsecurity.org/projects/commercial/rules/
http://www.modsecurity.org/projects/commercial/support/