Re: 30% faster with a batch size of 63 instead of 64!

--===============6864381162719473909==
Content-Type: multipart/alternative; boundary="0000000000000caf4905a53fe1f0"

--0000000000000caf4905a53fe1f0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Very nice. :)
And thanks for the note about batch sizing. Specifically tuning parameters
for this level of strength on 9x9 seems like it could be quite valuable,
Kata definitely hasn't done that either.

But it feels like bots are very very close to optimal on 9x9. With some
dedicated work, more months or years of training, it might be possible to
reach unbeatable, for all practical purposes, and as you mentioned in the
other thread, adaptively building out an opening book could be a part of
getting.there - I'd love to see an "unbeatable 9x9 crazystone" a year down
the line.

One fundamental issue that I've been noticing in a variety of domains is
precisely that self-play under AlphaZero and generally reinforcement
learning in environments like these doesn't explore enough, and it's very,
very difficult to get it to do so in a way that's still robust and
efficient enough. And unless you plan to do something like AlphaStar's
internal self-play training league, which would seem to nontrivally
multiply the cost, it seems like playing other opponents instead of just
selfplay can't entirely be the solution... because once you reach enough
better than the best other opponent, it's hard to usefully continue doing
that. And the league *still* didn't entirely fix the problem for AlphaStar,
in that humans were still able to sometimes find exploitative strategies
that it hadn't learned any idea of how to handle via selfplay, and reacted
very poorly to. It feels like there's something unsolved and "missing" from
current algorithms.


On Sat, May 9, 2020 at 7:17 PM R=C3=A9mi Coulom <[email protected]> wro=
te:

> Yeaaaah! first win against Kata!
> http://www.yss-aya.com/cgos/viewer.cgi?9x9/SGF/2020/05/09/999849.sgf
>
> In addition to the optimized batch size, I did two other things:
>  - I use two batches of 63 instead of one, with double buffering, so that
> the GPU is kept 100% busy. About 14k nodes per second now.
>  - I make the search less selective, by using a bigger exploration
> constant in the MCTS formula.
> I should download Katago and CLOP my search parameters against it.
>
> So far I have tried to keep the "Zero" philosophy of using self-play only=
,
> but playing against other opponents is very likely to be a better approac=
h
> at making progress.
>
> R=C3=A9mi
> _______________________________________________
> Computer-go mailing list
> [email protected]
> http://computer-go.org/mailman/listinfo/computer-go
>

--0000000000000caf4905a53fe1f0
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Very nice. :)<div>And thanks for the note about batch sizi=
ng. Specifically=C2=A0tuning parameters for this level of strength on 9x9 s=
eems like it could be quite valuable, Kata definitely hasn&#39;t done that =
either.=C2=A0</div><div><br></div><div>But it feels like bots are very very=
 close to optimal on 9x9. With some dedicated work, more months or years of=
 training, it might be possible to reach unbeatable, for all practical purp=
oses, and as you mentioned in the other thread, adaptively building out an =
opening book could be a part of getting.there - I&#39;d love to see an &quo=
t;unbeatable 9x9 crazystone&quot; a year down the line.</div><div><br></div=
><div>One fundamental issue that I&#39;ve been noticing in a variety of dom=
ains is precisely that self-play under AlphaZero and generally reinforcemen=
t learning in environments like these doesn&#39;t explore enough, and it&#3=
9;s very, very difficult to get it to do so in a way that&#39;s still robus=
t and efficient enough. And unless you plan to do something like AlphaStar&=
#39;s internal self-play training league, which would seem to nontrivally m=
ultiply the cost, it seems like playing other opponents instead of just sel=
fplay can&#39;t entirely be the solution... because once you reach enough b=
etter than the best other opponent, it&#39;s hard to usefully continue doin=
g that. And the league *still* didn&#39;t entirely fix the problem for Alph=
aStar, in that humans were still able to sometimes find exploitative strate=
gies that it hadn&#39;t learned any idea of how to handle via selfplay, and=
 reacted very poorly to. It feels like there&#39;s something unsolved and &=
quot;missing&quot; from current algorithms.</div><div><br></div></div><br><=
div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Sat, May=
 9, 2020 at 7:17 PM R=C3=A9mi Coulom &lt;<a href=3D"mailto:remi.coulom@gmai=
l.com" target=3D"_blank">[email protected]</a>&gt; wrote:<br></div><blo=
ckquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left=
:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><div>Yeaaaah=
! first win against Kata!</div><div><a href=3D"http://www.yss-aya.com/cgos/=
viewer.cgi?9x9/SGF/2020/05/09/999849.sgf" target=3D"_blank">http://www.yss-=
aya.com/cgos/viewer.cgi?9x9/SGF/2020/05/09/999849.sgf</a></div><div><br></d=
iv><div>In addition to the optimized batch size, I did two other things:</d=
iv><div>=C2=A0- I use two batches of 63 instead of one, with double bufferi=
ng, so that the GPU is kept 100% busy. About 14k nodes per second now.<br><=
/div><div>=C2=A0- I make the search less selective, by using a bigger explo=
ration constant in the MCTS formula.<br></div><div></div><div>I should down=
load Katago and CLOP my search parameters against it.</div><div><br></div><=
div>So far I have tried to keep the &quot;Zero&quot; philosophy of using se=
lf-play only, but playing against other opponents is very likely to be a be=
tter approach at making progress.<br></div><div><br></div><div>R=C3=A9mi<br=
></div></div>
_______________________________________________<br>
Computer-go mailing list<br>
<a href=3D"mailto:[email protected]" target=3D"_blank">Computer-g=
[email protected]</a><br>
<a href=3D"http://computer-go.org/mailman/listinfo/computer-go" rel=3D"nore=
ferrer" target=3D"_blank">http://computer-go.org/mailman/listinfo/computer-=
go</a><br>
</blockquote></div>

--0000000000000caf4905a53fe1f0--

--===============6864381162719473909==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Computer-go mailing list
[email protected]
http://computer-go.org/mailman/listinfo/computer-go

--===============6864381162719473909==--