Re: Bad logins bogging down server

Follow up....

The botnet is still hammering away, checking those old accounts.  But 
the bottleneck appears to have been saslauthd threads.  Doubling the 
thread count from 5 to 10 has resolved the problem for now.  (And, might 
even explain the occasional slow response from IMAP I've observed.)  I 
will run more experiments to see just how high the thread count should 
be, and I've got a list of other optimizations I will try.  This note is 
  in case somebody else sees the same problem.

Mike

On 09/16/2017 07:41 AM, Michael D. Sofka wrote:
> I'm seeking help from the collective wisdom of the Cyrus world.
> 
> In the past two days we have seen first a doubling, and then a 
> quadrupling+ of badlogins to Cyrus.  These appear to be coming from a 
> botnet, in that the IPs are spread around in a way that evades fail2ban. 
>   It got so bad Friday afternoon, that we took the extraordinary step of 
> blocking off-campus connections to IMAP (email can still be read via 
> Webmail and the VPN).
> 
> The symptoms are that connections grow, and grow and grow until 
> authentication slows, holding open connections longer and longer.  It 
> takes about 15 minutes for the connection number to be at a point at 
> which service is interrupted.  Friday night at attempt was made to 
> re-enable off-campus IMAP, and the bots were still at it, service was 
> again disrupted.
> 
> But the number of connections does not appear close to max permitted by 
> Cyrus.
> 
> We have a Murder cluster:  Three front-end servers, Two back-end 
> servers, Two replication servers.
> The front-end servers are Ubuntu 14.04, Cyrus 2.4.17.  The back-end and 
> replication servers are Ubuntu 16.04, Cyrus 2.4.18.  (Upgrading 
> front-ends on the short list.)
> 
> Authentication is via saslauthd, configured to use PAM, which is using 
> krb5.  Kerberos is running on three different kerberos servers. Load on 
> the kerberos servers is light, and the kerb-admin says nowhere close to 
> saturated.  In fact, it handled much higher numbers of authentications 
> before imapproxy on the Webmail service. (That was years ago, previous 
> kerb servers, so there is still the possibility the kerberos servers are 
> somehow slowed....)
> 
> Each Front-end server is configured for 5000 imapd on 143, and 5000 on 
> port 993.  Netstat shows about 4-5,000 imap connections per front-end 
> server when authentication slows.  There are well under 5000 imapd 
> processes of either type.  And after the Friday evening test re-allowing 
> off-campus IMAP, the network admin reported about 1600 connections to 
> port 993 total as IMAP authentication is slowed to a crawl.
> 
> We are not close to file-max on any of the servers.
> 
> imapd.conf has a 10 second delay for a badlogin.
> 
> There are some mupdate log entries
> 
>     Thread timed out waiting for listener_lock
>     Worker thread finished, for a total of 3 (2 spare)
> 
> Around the time of the Friday afternoon problems, when I was restarting 
> Front-end servers to recover. And no mupdate log entries since.  What 
> does this mean?  There are entries in syslog when mupdate is restarted, 
> stating that it could not reset the file limit to 5k. 
> mupdate_connections_max is 1024, so the failure to reset has no affect, 
> unless that is the limitation.  But I see no log entries indicating that.
> 
> Any other resources or limits in either Cyrus or Linux (Debian) that I 
> should look at?
> 
> Thank you in advance for any help.
> 
> Mike
> 
> 

-- 
Michael D. Sofka               [email protected]
ITI Sr. Systems Programmer,   Email, TeX, Epistemology
Rensselaer Polytechnic Institute, Troy, NY.  http://www.rpi.edu/~sofkam/
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus