Re: Introduction + question (I thought I understood throttling, but I don't...)

On Sat, Apr 04, 2015 at 05:14:30PM +0200, Fabrizio Giudici wrote:
> Hello.
>

Hi. I'll do what I can to answer :)

> This is my first post here. Short introduction: I'm Fabrizio Giudici, I'm
>  from Italy and I've started developing a media center (yet another). I
> want to integrate it with MB.
>
> Ok, so I've written the initial integration code in the past days. Of
> course I immediately ran into the HTTP 503 thing. I read the docs and I
> believed I understood. I applied a throttle to my code so it makes only
> one request per second. It worked until yesterday. Today I'm always seeing
> almost exclusively 503 errors with:
>
> 	Your requests are exceeding the allowable rate limit (150.4, limit is
> 22.0 per 20 seconds)
>
> I'm not understanding the first number.

It should list the number of requests within the period, so that's 150.4
requests in 20 seconds.

> Actually I was able to get just a
> handful of answers from MB today, and then I see the 503 again. How can it
> be that I made 150 request per seconds if I'm immediately blocked? Perhaps
> is it a problem related to my IP address? It's a home internet provider,
> so it could be shared with other people using the service. (?)
>

During the period where you're getting 503s, I do in fact see a large
number of requests with both "blueMarine2" and "blueMarine2-1"
useragents -- 7 or 8 per second, which is consistent with 150 requests
in 20 seconds. The requests it seems to be making at
that time are for recordings, and it seems that at least some are
multiple requests for the same entity (e.g., three requests in a row for
/ws/2/recording/b061125b-28c6-40fc-9239-11fb0260695a?inc=artists).

You seem to have ended up ratelimited several distinct times, the last
of which is the largest continuous block (around 11:31 UTC today). The
only user-agents making requests from your IP are blueMarine, so it's
not another user or program doing things.

Some things I'd check:

a.) Is your code multithreaded such that it might make simultaneous
requests?
b.) Were you perhaps running two instances (blueMarine2 and
blueMarine2-1 presumably not being the same instance of your software)
c.) When you get a 503 error, do you retry too quickly or using a
separate queue of requests than the normal requests, such that it might
exacerbate its own problem by making yet more requests?
d.) If you're really stumped, you could try installing a local copy of
the server ( https://bitbucket.org/metabrainz/musicbrainz-server/ ) or
install our VM and make requests against that, checking the logs
yourself.

> For the record, I'm using the user agent "blueMarine".
>

We'd appreciate if you'd read
https://musicbrainz.org/doc/XML_Web_Service/Rate_Limiting#Provide_meaningful_User-Agent_strings
for advice on a better user-agent string as well, but perhaps you
simply haven't gotten to that stage of your development :)

> Please let me know.
>
> Thanks.
>

I hope this helps!

> --
> Fabrizio Giudici - Java Architect @ Tidalwave s.a.s.
> "We make Java work. Everywhere."
> http://tidalwave.it/fabrizio/blog - [email protected]
>
> _______________________________________________
> MusicBrainz-devel mailing list
> [email protected]
> http://lists.musicbrainz.org/mailman/listinfo/musicbrainz-devel

_______________________________________________
MusicBrainz-devel mailing list
[email protected]
http://lists.musicbrainz.org/mailman/listinfo/musicbrainz-devel

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlUgSMMACgkQq+0avXExp2+xugCeJQYo63qDgDErJMeNu8F+/v4O
7XoAnRXpYe3U0G/ZpRG4dtHkEv4A2CSr
=vgx/
-----END PGP SIGNATURE-----