Rate limit API - database storage question

Hi all:

Given my last email, I tried to add the GCRA
(https://en.wikipedia.org/wiki/Generic_cell_rate_algorithm) rate
limiting to the rest interface.

The config file allows the admin to set the number of api calls. It
also (unlike web login protection) allows the admin to set the
period.

By default it allows a validated user one api call/second (3600
attempts in 3600 seconds). The user can use all 3600 if they want in a
minute, but will have to wait another minute before they get another
API call, and they will get all 3600 calls back if they wait an hour.
The limit is the same for every user.

To allow flexibility at the tracker level, the limit object is created
by a method call in the RestfulInstance class. An admin should be
able to replace the method with their own version. If the admin was
change the user schema and define attempts and period properties, the
new method could use those to provide different tiers of service to
different users.

However there is an issue that I am looking for feedback on. Currently
I store the data I need in the one time key database. However I am
running into a problem. When hitting the api quickly, I get:

  _gdbm.error: [Errno 11] Resource temporarily unavailable

which I assume is a file lock conflict since the gdbm database does
not allow parallel access.

Unlike the login rate limiting, only authenticated users are able to
use the API.

So I could add a: GCRA_data property to the user schema and store the
data there. However this means the user schema has to change in order
for somebody to upgrade.  Also depending on the backend I may still
have an issue, but for postgres/mysql I don't expect a problem.

Alternatively, does it make sense to trap the temporary unavailable
gdbm error and retry it X number of times before returning an error to
the client? If so, does anybody have a feel for X and if it needs to
be user settable?

Also does anybody want to try adding rate limiting to the xmlrpc API
endpoint? The code is pretty much the same as what is in
cgi/actions.py::LoginAction.

Thanks.

Have a great week all.
--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.