Re: Training an AlphaGo Zero-like algorithm with limited hardware on 7x7 boards
Igor Polyakov <[email protected]> Mon, 27 Jan 2020 12:06:09 +0800
| Newsgroups | gmane.games.devel.go |
|---|---|
| Message-ID | <CAPMj7mbfaW+TEAEvxYy33yspWbjeN53LFAnkULqG7m=Xa0qyZQ@mail.gmail.com> |
--===============1717623306273912684== Content-Type: multipart/alternative; boundary="0000000000001f78b4059d173bd9" --0000000000001f78b4059d173bd9 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I would be surprised if my model ever lost to GNU Go on 9x9. It's a lot stronger than Fuego, which already stomps GNU Go. It would be a waste of time to test it vs. GNU Go or even MCTS bots. I only plan on running tests vs. current best models to see how it does against the state of the art 9x9 nets On Mon, Jan 27, 2020, 06:39 cody2007 via Computer-go < [email protected]> wrote: > Thanks again for your thoughts and experiences R=C3=A9mi and Igor. > > I'm still puzzled by what is making training slower for me than R=C3=A9mi > (although I wouldn't be surprised if Igor's results were faster when > matched for hardware, model size, strength etc-- see below). Certainly ko= mi > sounds like it might help a lot. I'm going to have to check out the code > from David Wu. > > It takes me longer than a day for "training" to actually start with my > code -- because I first generate 128*2*32*35 =3D 285k training samples be= fore > even running the first round of backprop. After the first day, therefore, > my model is always still entirely random. So, possibly: > > (1) either your and David Wu's implementations are faster in wall clock > time computationally > (2) backprop is being started before the initial training buffer is fille= d > (the Wu paper used 250k but it's not 100% clear to me if training did not > start until that initial buffer was filled) > (3) "training" time is being counted as the time when backprop starts > regardless of how long the initial training buffer took to create. > > Another thing is that I'm not using any of the techniques beyond AlphaGo > Zero that David Wu used. So, depending on if you guys are using some or a= ll > of those additional features and/or loss functions, it'd be expected that > you're getting much faster training than me. I was actually starting to > test adding some of his ideas from that paper to my code a while back but > then coincidentally discovered the models I was training weren't as > horrible as I had first thought. > > Have either of you ever benchmarked your 7x7 (or 9x9) models against GNU > Go? > > By the way, all benchmarking against GNU Go that I've reported was in > single-pass mode only (i.e., I was not running the tree search on top of > the net outputs) > > Thanks, > Cody > > =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original = Message =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 > On Sunday, January 26, 2020 11:22 AM, Igor Polyakov < > [email protected]> wrote: > > I trained using David Wu's code for a few months on 9x9 only and it's bee= n > superhuman after a few months. > > I'm not sure if anyone's interested, but I can release my network to the > world. It's around the strength of KataGo, but only on 9x9. I could do a > final test before releasing it into the wild > > On Mon, Jan 27, 2020, 00:17 R=C3=A9mi Coulom <[email protected]> wrot= e: > >> Yes, using komi would help a lot. Still, I feel that something else must >> be wrong, because winning 100% of the games as Black without komi should= be >> very easy on 7x7. >> >> I have not written anything about what I did with Crazy Stone. But my >> experiments and ideas were really very similar to what David Wu did: >> https://blog.janestreet.com/accelerating-self-play-learning-in-go/ >> >> To clarify what I wrote in my previous message: "strong from scratch in = a >> single day" was for 7x7. I like testing new ideas with small networks on >> small boards, because training is very fast, and what works on small boa= rds >> with small networks usually also works on large boards with big networks= . >> >> R=C3=A9mi >> >> On Sun, Jan 26, 2020 at 12:30 AM cody2007 <[email protected]> >> wrote: >> >>> Hi R=C3=A9mi, >>> >>> Thanks for your comments! I am not using any komi and had not given muc= h >>> thought to it. Although, I suppose by having black win most games, I'm >>> depriving the network of its only learning signal. I will have to try w= ith >>> an appropriately set komi next... >>> >>> >When I started to develop the Zero version of Crazy Stone, I spend a >>> lot of time optimizing my method on a single (V100) GPU >>> Any chance you've written about it somewhere? I'd be interested to lear= n >>> more but wasn't able to find anything on the Crazy Stone website. >>> >>> Thanks, >>> Cody >>> >>> =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Origina= l Message =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 >>> On Saturday, January 25, 2020 5:49 PM, R=C3=A9mi Coulom < >>> [email protected]> wrote: >>> >>> Hi, >>> >>> Thanks for sharing your experiments. >>> >>> Your match results are strange. Did you use a komi? You should use a >>> komi of 9: >>> https://senseis.xmp.net/?7x7 >>> >>> The final strength of your network looks surprisingly weak. When I >>> started to develop the Zero version of Crazy Stone, I spend a lot of ti= me >>> optimizing my method on a single (V100) GPU. I could train a strong net= work >>> from scratch in a single day. Using a wrong komi might have hurt you. A= lso, >>> on such a small board, it is not so easy to make sure that the self-pla= y >>> games have enough variety. You'd have to find many balanced random init= ial >>> positions in order to avoid replicating the same game again and again. >>> >>> R=C3=A9mi >>> >>> >>> _______________________________________________ >> Computer-go mailing list >> [email protected] >> http://computer-go.org/mailman/listinfo/computer-go >> > > _______________________________________________ > Computer-go mailing list > [email protected] > http://computer-go.org/mailman/listinfo/computer-go > --0000000000001f78b4059d173bd9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"auto">I would be surprised if my model ever lost to GNU Go on 9= x9. It's a lot stronger than Fuego, which already stomps GNU Go. It wou= ld be a waste of time to test it vs. GNU Go or even MCTS bots. I only plan = on running tests vs. current best models to see how it does against the sta= te of the art 9x9 nets</div><br><div class=3D"gmail_quote"><div dir=3D"ltr"= class=3D"gmail_attr">On Mon, Jan 27, 2020, 06:39 cody2007 via Computer-go = <<a href=3D"mailto:[email protected]" target=3D"_blank" rel=3D= "noreferrer">[email protected]</a>> wrote:<br></div><blockquot= e class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc sol= id;padding-left:1ex"><div>Thanks again for your thoughts and experiences R= =C3=A9mi and Igor.<br></div><div><br></div><div>I'm still puzzled by wh= at is making training slower for me than R=C3=A9mi (although I wouldn't= be surprised if Igor's results were faster when matched for hardware, = model size, strength etc-- see below). Certainly komi sounds like it might = help a lot. I'm going to have to check out the code from David Wu.<br><= /div><div><br></div><div>It takes me longer than a day for "training&q= uot; to actually start with my code -- because I first generate 128*2*32*35= =3D 285k training samples before even running the first round of backprop.= After the first day, therefore, my model is always still entirely random.= =C2=A0 So, possibly:<br></div><div><br></div><div>(1) either your and David= Wu's implementations are faster in wall clock time computationally<br>= </div><div>(2) backprop is being started before the initial training buffer= is filled (the Wu paper used 250k but it's not 100% clear to me if tra= ining did not start until that initial buffer was filled)<br></div><div>(3)= "training" time is being counted as the time when backprop start= s regardless of how long the initial training buffer took to create.<br></d= iv><div><br></div><div>Another thing is that I'm not using any of the t= echniques beyond AlphaGo Zero that David Wu used. So, depending on if you g= uys are using some or all of those additional features and/or loss function= s, it'd be expected that you're getting much faster training than m= e. I was actually starting to test adding some of his ideas from that paper= to my code a while back but then coincidentally discovered the models I wa= s training weren't as horrible as I had first thought.<br></div><div><b= r></div><div>Have either of you ever benchmarked your 7x7 (or 9x9) models a= gainst GNU Go?<br></div><div><br></div><div>By the way, all benchmarking ag= ainst GNU Go that I've reported was in single-pass mode only (i.e., I w= as not running the tree search on top of the net outputs)<br></div><div><br= ></div><div>Thanks,<br></div><div>Cody<br></div><div><br></div><div>=E2=80= =90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original Message = =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90<br></div><d= iv> On Sunday, January 26, 2020 11:22 AM, Igor Polyakov <<a href=3D"mail= to:[email protected]" rel=3D"noreferrer noreferrer" target=3D"_bla= nk">[email protected]</a>> wrote:<br></div><div> <br></div><blo= ckquote type=3D"cite"><div dir=3D"auto"><div>I trained using David Wu's= code for a few months on 9x9 only and it's been superhuman after a few= months.<br></div><div dir=3D"auto"><br></div><div dir=3D"auto">I'm not= sure if anyone's interested, but I can release my network to the world= . It's around the strength of KataGo, but only on 9x9. I could do a fin= al test before releasing it into the wild<br></div></div><div><br></div><di= v class=3D"gmail_quote"><div dir=3D"ltr">On Mon, Jan 27, 2020, 00:17 R=C3= =A9mi Coulom <<a href=3D"mailto:[email protected]" rel=3D"noreferrer= noreferrer" target=3D"_blank">[email protected]</a>> wrote:<br></di= v><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:= 1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div dir=3D"ltr"><div>Yes= , using komi would help a lot. Still, I feel that something else must be wr= ong, because winning 100% of the games as Black without komi should be very= easy on 7x7.<br></div><div><br></div><div>I have not written anything abou= t what I did with Crazy Stone. But my experiments and ideas were really ver= y similar to what David Wu did:<br></div><div><a href=3D"https://blog.janes= treet.com/accelerating-self-play-learning-in-go/" rel=3D"noreferrer norefer= rer noreferrer" target=3D"_blank">https://blog.janestreet.com/accelerating-= self-play-learning-in-go/</a><br></div><div><br></div><div>To clarify what = I wrote in my previous message: "strong from scratch in a single day&q= uot; was for 7x7. I like testing new ideas with small networks on small boa= rds, because training is very fast, and what works on small boards with sma= ll networks usually also works on large boards with big networks.<br></div>= <div><br></div><div>R=C3=A9mi<br></div></div><div><br></div><div class=3D"g= mail_quote"><div dir=3D"ltr">On Sun, Jan 26, 2020 at 12:30 AM cody2007 <= <a href=3D"mailto:[email protected]" rel=3D"noreferrer noreferrer nor= eferrer" target=3D"_blank">[email protected]</a>> wrote:<br></div>= <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left:1px solid rgb(204,204,204);padding-left:1ex"><div>Hi R=C3=A9mi,<br></d= iv><div><br></div><div>Thanks for your comments! I am not using any komi an= d had not given much thought to it. Although, I suppose by having black win= most games, I'm depriving the network of its only learning signal. I w= ill have to try with an appropriately set komi next...<br></div><div><br></= div><div>>When I started to develop the Zero version of Crazy Stone, I s= pend a lot of time optimizing my method on a single (V100) GPU<br></div><di= v>Any chance you've written about it somewhere? I'd be interested t= o learn more but wasn't able to find anything on the Crazy Stone websit= e.<br></div><div><br></div><div>Thanks,<br></div><div>Cody<br></div><div><b= r></div><div>=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80= =90 Original Message =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90= =E2=80=90<br></div><div>On Saturday, January 25, 2020 5:49 PM, R=C3=A9mi Co= ulom <<a href=3D"mailto:[email protected]" rel=3D"noreferrer norefer= rer noreferrer" target=3D"_blank">[email protected]</a>> wrote:<br><= /div><div><br></div><blockquote type=3D"cite"><div dir=3D"ltr"><div>Hi,<br>= </div><div><br></div><div>Thanks for sharing your experiments.<br></div><di= v><br></div><div>Your match results are strange. Did you use a komi? You sh= ould use a komi of 9:<br></div><div><a href=3D"https://senseis.xmp.net/?7x7= " rel=3D"noreferrer noreferrer noreferrer" target=3D"_blank">https://sensei= s.xmp.net/?7x7</a><br></div><div><br></div><div>The final strength of your = network looks surprisingly weak. When I started to develop the Zero version= of Crazy Stone, I spend a lot of time optimizing my method on a single (V1= 00) GPU. I could train a strong network from scratch in a single day. Using= a wrong komi might have hurt you. Also, on such a small board, it is not s= o easy to make sure that the self-play games have enough variety. You'd= have to find many balanced random initial positions in order to avoid repl= icating the same game again and again.<br></div><div><br></div><div>R=C3=A9= mi<br></div></div></blockquote><div><br></div></blockquote></div></div><div= >_______________________________________________<br></div><div> Computer-go= mailing list<br></div><div> <a href=3D"mailto:[email protected]"= rel=3D"noreferrer noreferrer noreferrer" target=3D"_blank">Computer-go@com= puter-go.org</a><br></div><div> <a href=3D"http://computer-go.org/mailman/l= istinfo/computer-go" rel=3D"noreferrer noreferrer noreferrer noreferrer" ta= rget=3D"_blank">http://computer-go.org/mailman/listinfo/computer-go</a><br>= </div></blockquote></div></blockquote><div><br></div>______________________= _________________________<br> Computer-go mailing list<br> <a href=3D"mailto:[email protected]" rel=3D"noreferrer noreferrer= " target=3D"_blank">[email protected]</a><br> <a href=3D"http://computer-go.org/mailman/listinfo/computer-go" rel=3D"nore= ferrer noreferrer noreferrer" target=3D"_blank">http://computer-go.org/mail= man/listinfo/computer-go</a><br> </blockquote></div> --0000000000001f78b4059d173bd9-- --===============1717623306273912684== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Computer-go mailing list [email protected] http://computer-go.org/mailman/listinfo/computer-go --===============1717623306273912684==--