Re: Robocode and reinforcement learning

Re feature selection, I would be interested to see if you could also learn
the "Has bearing on opponent" and "Square root of degree deviation from gun
to opponent" functions.
I think that it would be even more interesting when you let it aim at
moving target. For example, If it could extrapolate where the enemy would
be at the time when the bullet arrives.

Regarding collaboration, I would be willing to collaborate on making remote
robot API as a canonical part of Robocode.
Because network latency is a significant issue here, there needs to be an
API which is not chatty. It needs to have one call per turn only.

1) Somewhat similar to OpenAI gym conceptually.

reset() -> observations
step(actions) -> observations, rewards


2) Also figure out how to make it possible for such robots to enter
competition.
- Adding a new competition category, with different robot turn timeouts.
- Solve how to host such remote robots.

I think that we would like to enable a wide technological landscape for
remote robot authors (like Python).
Therefore we could not expect that the robot would be hosted by the
community, which is running the competition on compute they donated.
I'm considering inverted server/client roles. We would let the robocode
engine be the client of the robot's server.
So I imagine that the Java "robot" would only configure the Robocode engine
with a URL of the robot's API.

The API the robot would have to implement would look like:
reset(observations) -> actions
turn(observations, rewards) -> actions
roundEnd(rewards)

What do you think ?
Would it be possible/interesting to create such 1) API and contribute it to
OpenAI gym as a new environment ?
How difficult it would be to run RL with the inverted 2) API shape ?

Thank you!


On Sun, Mar 14, 2021 at 12:07 PM Matthew Gerber <[email protected]>
wrote:

> Hi Pavel -
>
> I've added answers below.
>
> On Saturday, March 13, 2021 at 3:49:03 PM UTC-5 Pavel Savara wrote:
>
>>
>> Do I see it well that it took 3000 rounds to learn to lock the gun on the
>> enemy ?
>>
>
> It takes about 5 rounds for the agent to settle into a decent aiming
> policy. The graph on the right shows time steps ("turns" in Robocode) for a
> single round. I've updated the webpage
> <https://matthewgerber.github.io/rlai/case_studies/robocode.html> with
> additional details about this. Now you can see separate videos for training
> and testing. The training rounds tend to be chaotic because the learning
> agent is mixing its aiming policy with random behavior to maintain
> exploration and learning. In the testing video, the agent has no random
> behavior, so it's clear what the policy is:  rotate radar --> obtain
> bearing --> rotate gun --> fire. It was neat to see this tactic emerge from
> the specified features.
>
> How much of that is done by you in feature selection ?
>>
>
> The updated webpage explains the features used. It looks like 5 features
> is sufficient to do radar-driven aiming against a stationary opponent.
>
>
>> What's coming next ?
>>
>
> I also added a couple TBD sections:  aiming against a mobile robot, and
> evasive movement.
>
> Are you interested in working on this, or know anyone who might be? This
> is just a personal hobby project of mine, but I'd welcome collaboration.
>
> Matt
>
> --
> You received this message because you are subscribed to the Google Groups
> "robocode" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/robocode/b4bc3fb7-d97b-4d30-9ba5-c9930f1146a6n%40googlegroups.com
> <https://groups.google.com/d/msgid/robocode/b4bc3fb7-d97b-4d30-9ba5-c9930f1146a6n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups "robocode" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit https://groups.google.com/d/msgid/robocode/CABB%2B072EmsrbYnKQPMy4Fs%3DSAiSbZyEbkRm247M9LRU1SCweMA%40mail.gmail.com.