Re: Robocode and reinforcement learning

Hi Pavel -

On Sun, Mar 14, 2021 at 7:59 AM Pavel Šavara <[email protected]> wrote:

> Re feature selection, I would be interested to see if you could also learn
> the "Has bearing on opponent" and "Square root of degree deviation from gun
> to opponent" functions.
>

Feature / representation learning is an interesting (and popular) topic
these days, particularly in the neural network community. My rlai package
is agnostic to the value-function approximation model, and it would be
possible to swap the current scikit-learn model (SGD regressor) for a deep
neural network package. Perhaps I'll head that direction at some point, but
for now I'm enjoying manual feature specification -- it's the "art" side of
RL agent design, and it's fun to see extended tactics emerge from a small
set of features.

I think that it would be even more interesting when you let it aim at
> moving target. For example, If it could extrapolate where the enemy would
> be at the time when the bullet arrives.
>

This is next, and "aiming ahead" will definitely be key!


> Regarding collaboration, I would be willing to collaborate on making
> remote robot API as a canonical part of Robocode.
> Because network latency is a significant issue here, there needs to be an
> API which is not chatty. It needs to have one call per turn only.
>

In my first attempt to integrate Robocode with my RL server, I tried to
implement a REST API for the RL server. In this setup, the robot made
REST/PUT requests to send game state to the RL server and retrieve actions
to execute. It was possible to do this in a single REST/PUT command; but it
was too slow (REST overhead). So I ditched the REST approach and stripped
it down to a simple TCP socket, which is much faster. At this point I can
get about 500 turns per second. This is good enough for learning simple
skills (e.g., my aiming example), but it's not very fast. One reason it's
slow is that my RL server is sending high-resolution actions to the game
(e.g., rotate the gun 5 degrees per turn). If the actions specified larger
movements (e.g., rotate the gun 100 degrees), then there would be less
back-and-forth (chattiness) with the RL server, and turns/second would
increase dramatically. Considerations of smaller/larger actions lead to the
need for continuous actions spaces, which I also haven't done yet. There
are so many interesting questions!



> 1) Somewhat similar to OpenAI gym conceptually.
>
> reset() -> observations
> step(actions) -> observations, rewards
>

I previously integrated rlai with OpenAI Gym (also on the website
<https://matthewgerber.github.io/rlai/case_studies/inverted_pendulum.html>),
and I patterned the Robocode integration similarly. Here's how each battle
round proceeds:


   1. RL server opens TCP listener on 127.0.0.1:54321
   2. Robocode robot (client) opens socket to 127.0.0.1:54321
   3. RL server accepts client connection
   4. Robocode robot sends initial game state and RL server reads it (this
   is your "reset" operation above)
   5. Loop until round ends (this is your "step" operation above)
      1. Robocode robot reads next action from RL server
      2. Robocode robot executes the action and sends the updated game
      state to RL server

By design, the Robocode robot does not send a reward value to the RL
server. This allows the RL server to specify its own reward function. The
reward function is a fundamental (and interesting) aspect of RL agent
design, and I wanted to provide full flexibility to the RL agent designer.


>
>
> 2) Also figure out how to make it possible for such robots to enter
> competition.
> - Adding a new competition category, with different robot turn timeouts.
> - Solve how to host such remote robots.
>

Hosting remote robots is indeed the central problem that I see, too (more
on this below).


>
> I think that we would like to enable a wide technological landscape for
> remote robot authors (like Python).
>

My original goal was to create a native Java wrapper around my Python RL
code, so that the RL routines could be invoked directly by the Robocode
engine; however, this sort of wrapper seems experimental and problematic.
So I went with a TCP connection to achieve interprocess (i.e., Java <-->
Python) communication. This is slower, but it has two significant
benefits:  (1) TCP-based IPC works for any program/language, and (2) it
opens up the possibility of running remote-hosted robot implementations,
like we're discussing here.


> Therefore we could not expect that the robot would be hosted by the
> community, which is running the competition on compute they donated.
>

I'm not so sure. Perhaps I'm missing something specific to Robocode, but it
seems like the community might be able to host their own remote Robot
implementations (i.e., servers as I've described above).


> I'm considering inverted server/client roles. We would let the robocode
> engine be the client of the robot's server.
>

I think this is essentially how I have architected my integration of
Robocode and the RL server. The Java robot is the TCP client, and the RL
server is a TCP server.


> So I imagine that the Java "robot" would only configure the Robocode
> engine with a URL of the robot's API.
>

Exactly. Right now, my Robocode robot (Java) opens a connection to
127.0.0.1:54321; however, the author of the robot would be free to direct
the connection to any host:port. Or, if *we* can control the robot hosting
service, then we could specify all of the remote robot hosts/ports.


>
> The API the robot would have to implement would look like:
> reset(observations) -> actions
> turn(observations, rewards) -> actions
> roundEnd(rewards)
>
> What do you think ?
>

This is pretty close to what I'm doing now. I think it's a workable
approach.


> Would it be possible/interesting to create such 1) API and contribute it
> to OpenAI gym as a new environment ?
>

OpenAI Gym supports many <https://gym.openai.com/envs/#atari> games, but
none of them involve multiple RL agents running concurrently. I think a
Robocode environment where multiple distributed RL agents could learn and
compete would be a really nice addition. I'm not aware of anything like
this.


> How difficult it would be to run RL with the inverted 2) API shape ?
>

I think my current architecture is pretty close to what you have in mind.
The most obvious difficulty (to me) would be figuring out where to host the
remote robot servers, as you point out. Let's think about that.

Matt

-- 
You received this message because you are subscribed to the Google Groups "robocode" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit https://groups.google.com/d/msgid/robocode/CAOcpbfoJW%3D%3DM2hRPKSGcJzo1YsagDbOXC5PpbB6Pn2cHkN0KFQ%40mail.gmail.com.