Re: Robocode and reinforcement learning

On Sun, Mar 14, 2021 at 1:07 PM Pavel Šavara <[email protected]> wrote:

>
>
> In my first attempt to integrate Robocode with my RL server, I tried to
>> implement a REST API for the RL server. In this setup, the robot made
>> REST/PUT requests to send game state to the RL server and retrieve actions
>> to execute. It was possible to do this in a single REST/PUT command; but it
>> was too slow (REST overhead).
>>
> Interesting, I would not think that REST would make enough difference.
>

Perhaps it was just my (incorrect) use of it. In any case, it was overly
complicated (it needed a separate thread, synchronization locks, etc.) for
the very simple thing I needed (synchronous read/write over TCP).


> We already have internal/custom binary protocol for that:
> actions == ExecCommands
> observations == ExecResults
>
> https://github.com/robo-code/robocode/blob/master/robocode.core/src/main/java/net/sf/robocode/peer/ExecCommands.java#L294
>
> https://github.com/robo-code/robocode/blob/master/robocode.core/src/main/java/net/sf/robocode/peer/ExecResults.java#L86
>
> I implemented it years ago when I wanted to run a .NET robot and I did
> some in-process IPC.
>
> Is that good enough or we should prefer something more formal ?
>

Personally, I much prefer a serialization format like JSON. But
serialization/deserialization with JSON is probably slower versus binary
due to parsing.


>
>> By design, the Robocode robot does not send a reward value to the RL
>> server. This allows the RL server to specify its own reward function. The
>> reward function is a fundamental (and interesting) aspect of RL agent
>> design, and I wanted to provide full flexibility to the RL agent designer.
>>
> Definitely your RL needs to re-model the rewards. I'm just talking about
> Robocode's natural score. It evolves with each turn.
> Maybe that's philosophically part of RL `observations` ?
>

Indeed, I think it's reasonable to view the reward as a function of the
state (observations), so as a practical consideration it should be
sufficient to only pass observations (states) back to the agents and let
them calculate reward signals as they wish. It's certainly convenient when
the environment (OpenAI Gym or Robocode) precomputes a useful reward value.
Formally (e.g., per the textbook), the agent's scope is limited to things
that it can directly control (e.g., taking actions, sensing states, and
processing rewards). So in this sense, allowing the agent to control the
reward definition could be weird, since the agent might simply set it to
infinity. This is certainly getting philosophical about what an agent is.
Ultimately, the measure of a Robocode agent's success will be the official
ranking provided by the Robocode engine, and this is beyond control of the
agents. So there's no harm in allowing RL agents to dictate the internal
reward signals they use for the purpose of learning good behavior policies.


>
>> I'm not so sure. Perhaps I'm missing something specific to Robocode, but
>> it seems like the community might be able to host their own remote Robot
>> implementations (i.e., servers as I've described above).
>>
> Unless every author with the remote robot is willing to provide docker
> image, sharing my-robot-how-to-host-it on diverse platforms would not fly.
> That's why I'm suggesting that any robot author who wants to have his
> robot in the competition would have his server open and available to
> incoming battles all the time. On authors own servers.
>

Is there a reason why we couldn't just adopt the standard client/server
model used in online games, where a player sets up a server, other players
join as clients, and off we go?

Matt

-- 
You received this message because you are subscribed to the Google Groups "robocode" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit https://groups.google.com/d/msgid/robocode/CAOcpbfp3jZa3OOAwG20VXiNk-e3_z6-t_wiaU-ncCndyhjjzzw%40mail.gmail.com.