Ladder and zeroness (was: Re: CGOS source on github)
David Wu <[email protected]> Sat, 23 Jan 2021 10:25:18 -0500
| Newsgroups | gmane.games.devel.go |
|---|---|
| Message-ID | <CAGEydYttA2UTdBWc1vwAPCpkiDCOSLcvn+s3i6WqAWpOWkkhew@mail.gmail.com> |
--00000000000014287f05b992ea61 Content-Type: multipart/alternative; boundary="00000000000014287d05b992ea5f" --00000000000014287d05b992ea5f Content-Type: text/plain; charset="UTF-8" On Sat, Jan 23, 2021 at 5:34 AM Darren Cook <[email protected]> wrote: > Each convolutional layer should spread the information across the board. > I think alpha zero used 20 layers? So even 3x3 filters would tell you > about the whole board - though the signal from the opposite corner of > the board might end up a bit weak. > > I think we can assume it is doing that successfully, because otherwise > we'd hear about it losing lots of games in ladders. > Unfortunately, we can't assume that based on that observation. If you observe what is going on with both Leela Zero and ELF, and MiniGo and SAI as well - all of which are reproductions of AlphaZero with different hyperparameters and infrastructure that do not include a ladder feature, I think you can find *all* of them have at least some trouble with ladders. So this is empirical evidence that the vanilla AlphaZero algorithm when applied to Go with a convolutional resnet, often has ladder problems. And by seeing how these reproductions behave, it also becomes clear how your observation can still be true at the same time. Which is: with enough playouts, for all these bots MCTS is able to solve ladders well enough at the root position and the upper levels of the tree to avoid losing outright - usually a few tens of thousands of playouts are plenty. So it just affects the strength by causing harm to the evaluation quality deeper in the tree in ways that are harder to see. The kind of thing that might cost you more like 20-50 Elo (pure guess, just my intuition for the *very* rough order of magnitude with this much search on top), rather than losing you every game. The bigger problem happens when you try any of these bots on weaker hardware that only gets few playouts - low-end GPUs, mobile hardware, etc. for example.... *or the numbers of playouts that people often run CGOS bots with*, namely 200 playouts, or 800 playouts, etc. You will find that they are still clearly top-pro-level or superhuman at almost all aspects of the game... except for ladders! And now at these low numbers of playouts, it does include outright losing games due to ladders, or making major misjudgments about a sequence that will depend on a ladder in 1-3 moves in the future. Sometimes, this even happens in the low-thousands of playouts. For example, attached SGF shows such a case, where Leela Zero using almost the latest 40-block network (LZ285) with 2k playouts per move (plus tree reuse) attempted to break a ladder, failed, and then played out the ladder anyways and lost on the spot. It is also true that neural nets *are* capable of learning judgments related to ladders given the right data. Some time back, I found with some visualizations for KataGo's net that it actually is tracing a width-6 diagonal band across the board from ladders! But the inductive bias is weak enough, plus the structure of the game tree for ladders is hard (it's like the classic "cliff walking" problem in RL turned up to the max), that it's a chicken-and-egg problem. Starting from a net that doesn't understand ladders yet, the "MCTS policy/value-improvement operator" is empirically very poor at bootstrapping the net into understanding them. > > something the first version of AlphaGo did (before they tried to make it > > "zero") and something that many other bots do as well. But Leela Zero and > > ELF do not do this, because of attempting to remain "zero", ... > > I know that zero-ness was very important to DeepMind, but I thought the > open source dedicated go bots that have copied it did so because AlphaGo > Zero was stronger than AlphaGo Master after 21-40 days of training. > I.e. in the rarefied atmosphere of super-human play that starter package > of human expert knowledge was considered a weight around its neck. > The PR and public press around AlphaZero may give one this impression generally - it certainly sounds like a more impressive discovery if not only can you learn from Zero, but doing so is actually better! But I'm confident that this is not true in general, and that it also depends on what "expert knowledge" you add, and how you add it. You may note that the AlphaGo Zero paper makes no mention of how long or with how many TPUs AlphaGo Master was trained (or if it does, I can't find it) - so it's hard to say what Master vs Zero shows. Also, it claims that AlphaGo Master still made use of handcrafted Monte-Carlo rollouts, which I can easily believe that jettisoning could lead to a big improvement. And it's at least plausible to me that not-pretraining on human pro games might give better final results (*but* this is unclear - at least I don't know of any paper that actually runs this as a controlled test).. But there are other bits of "expert knowledge" that do provide an improvement over being pure-zero if done correctly, including: * Predicting the final ownership of the board, not just the win/loss. * Adding a small/mild term for caring about score, rather than just win/loss. * Seeding a percentage of the self-play training games to start in positions based on external or expert-supplied game or board positions (this is the main way KataGo went from being highly vulnerable to MiYuting's flying dagger like other zero bots, to playing it decently well and now often winning games based on it depending on if the other side happens to shoot themselves in the foot with one of the trap variations or not). And yes, for now it also includes: * Adding ladder status as an input to the neural net. --00000000000014287d05b992ea5f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"></div><div class=3D"gmail_quote"><div dir= =3D"ltr" class=3D"gmail_attr">On Sat, Jan 23, 2021 at 5:34 AM Darren Cook &= lt;<a href=3D"mailto:[email protected]">[email protected]</a>> wrote:<br><= /div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bo= rder-left:1px solid rgb(204,204,204);padding-left:1ex"> Each convolutional layer should spread the information across the board.<br= > I think alpha zero used 20 layers? So even 3x3 filters would tell you<br> about the whole board - though the signal from the opposite corner of<br> the board might end up a bit weak.<br> <br> I think we can assume it is doing that successfully, because otherwise<br> we'd hear about it losing lots of games in ladders.<br></blockquote><di= v><br></div><div>Unfortunately, we can't assume that based on that obse= rvation.</div><div><br></div><div>If you observe what is going on with both= Leela Zero and ELF, and MiniGo and SAI as well - all of which are reproduc= tions of AlphaZero with different hyperparameters and infrastructure that d= o not include a ladder feature, I think you can find *all* of them have at = least some trouble with ladders.=C2=A0So this is empirical evidence that th= e vanilla AlphaZero algorithm when applied to Go with a convolutional resne= t, often has ladder problems.=C2=A0</div><div><br></div><div>And by seeing = how these reproductions behave, it also becomes clear how your observation = can still be true at the same time.</div><div><br></div><div>Which is: with= enough playouts, for all these bots MCTS is able to solve ladders well eno= ugh at the root position and the upper levels of the tree to avoid losing o= utright - usually a few tens of thousands of playouts are plenty. So it jus= t affects the strength by causing harm to the evaluation quality deeper in = the tree in ways that are harder to see. The kind of thing that might cost = you more like 20-50 Elo (pure guess, just my intuition for the *very* rough= order of magnitude with this much search on top), rather than losing you e= very game.</div><div><br></div><div>The bigger problem happens when you try= any of these bots on weaker hardware that only gets few playouts - low-end= GPUs, mobile hardware, etc. for example.... <i><b>or the numbers of playou= ts that people often run CGOS bots with</b></i>, namely 200 playouts, or 80= 0 playouts, etc. You will find that they are still clearly top-pro-level or= superhuman at almost all aspects of the game... except for ladders! And no= w at these low numbers of playouts, it does include outright losing games d= ue to ladders, or making major misjudgments about a sequence that will depe= nd on a ladder in 1-3 moves in the future.</div><div><br></div><div>Sometim= es, this even happens in the low-thousands of playouts. For example, attach= ed SGF shows such a case, where Leela Zero using almost the latest 40-block= network (LZ285) with 2k playouts per move (plus tree reuse) attempted to b= reak a ladder, failed, and then played out the ladder anyways and lost on t= he spot.</div><div><br></div><div>It is also true that neural nets *are* ca= pable of learning judgments related to ladders given the right data. Some t= ime back, I found with some visualizations for KataGo's net that it act= ually is tracing a width-6 diagonal band across the board from ladders! But= the inductive bias is weak enough, plus the structure of the game tree for= ladders is hard (it's like the classic "cliff walking" probl= em in RL turned up to the max), that it's a chicken-and-egg problem. St= arting from a net that doesn't understand ladders yet, the "MCTS p= olicy/value-improvement operator" is empirically very poor at bootstra= pping the=C2=A0net into understanding them.</div><div>=C2=A0</div><blockquo= te class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px = solid rgb(204,204,204);padding-left:1ex"> > something the first version of AlphaGo did (before they tried to make = it<br> > "zero") and something that many other bots do as well. But L= eela Zero and<br> > ELF do not do this, because of attempting to remain "zero", = ...<br> <br> I know that zero-ness was very important to DeepMind, but I thought the<br> open source dedicated go bots that have copied it did so because AlphaGo<br= > Zero was stronger than AlphaGo Master after 21-40 days of training.<br> I.e. in the rarefied atmosphere of super-human play that starter package<br= > of human expert knowledge was considered a weight around its neck.<br></blo= ckquote><div><br></div><div>The PR and public press around AlphaZero may gi= ve one this impression generally - it certainly sounds like a more impressi= ve discovery if not only can you learn from Zero, but doing so is actually = better! But I'm confident that this is not true in general, and that it= also depends on what "expert knowledge" you add, and how you add= it.</div><div><br></div><div>You may note that the AlphaGo Zero paper make= s no mention of how long or with how many TPUs AlphaGo Master was trained (= or if it does, I can't find it) - so it's hard to say what Master v= s Zero shows. Also, it claims that AlphaGo Master still made use of handcra= fted Monte-Carlo rollouts, which I can easily believe that jettisoning coul= d lead to a big improvement. And it's at least plausible to me that not= -pretraining on human pro games might give better final results (*but* this= is unclear - at least I don't know of any paper that actually runs thi= s as a controlled test)..</div><div><br></div><div>But there are other bits= of "expert knowledge" that do provide an improvement over being = pure-zero if done correctly, including:</div><div>* Predicting the final ow= nership of the board, not just the win/loss.</div><div>* Adding a small/mil= d term for caring about score, rather than just win/loss.</div><div></div><= div>* Seeding a percentage of the self-play training games to start in posi= tions based on external or expert-supplied game or board positions (this is= the main way KataGo went from being highly vulnerable to MiYuting's fl= ying dagger like other zero bots, to playing it decently well and now often= winning games based on it depending on if the other side happens to shoot = themselves in the foot with one of the trap variations or not).</div><div><= br></div><div>And yes, for now it also includes:</div><div>* Adding ladder = status as an input to the neural net.</div><div><br></div></div></div> --00000000000014287d05b992ea5f-- --00000000000014287f05b992ea61 Content-Type: application/octet-stream; name="0_94.sgf" Content-Disposition: attachment; filename="0_94.sgf" Content-Transfer-Encoding: base64 Content-ID: <f_kk9sxwxy0> X-Attachment-Id: f_kk9sxwxy0 KDtGRls0XUFQW2dvbWlsbDowLjguMl0KQ1tFdmVudDogdjE4dnNsegpHYW1lIGlkIDBfOTQKRGF0 ZSAyMDIxLTAxLTEyIDE0OjQzClJlc3VsdCB2MTggYmVhdCBsejI4NS0yayBCK1IKdjE4IGNwdSB0 aW1lOiAyNC41M3MKbHoyODUtMmsgY3B1IHRpbWU6IDEwMi4yMHMKQmxhY2sgdjE4IEthdGFHbzox LjcuMApXaGl0ZSBsejI4NS0yayBMZWVsYSBaZXJvOjAuMTddCkNBW1VURi04XURUWzIwMjEtMDEt MTJdRVZbdjE4dnNsel1HTVsxXUdOW3YxOHZzbHogMF85NF1LTVs3LjVdUEJbS2F0YUdvOjEuNy4w XQpQV1tMZWVsYSBaZXJvOjAuMTddUkVbQitSXVNaWzE5XTtCW3BwXTsKQ1ttb3ZlIDEsIFcgPT4g RDQKUGxheW91dHM6IDIwNjgsIFdpbjogNTcuMTQlLCBQVjogUTE2IFIxNyBSMTYgUTE3IFAxNiBP MTggRDE2IEQ0IEMzIEQzIEM0IEM2IEM1IEQ1XQpXW2RwXTtCW2RkXTsKQ1ttb3ZlIDMsIFcgPT4g UTE2ClBsYXlvdXRzOiAyMTQ4LCBXaW46IDU3Ljc3JSwgUFY6IFExNiBDMyBEMyBDNCBDNiBDNSBE NSBCNiBDMTcgQzE2IEQxNyBFMTYgRTE3IEYxNyBGMTggUjE3IFIxNiBRMTcgUDE3IFAxOCBPMTgg TzE3XQpXW3BkXTtCW3FmXTsKQ1ttb3ZlIDUsIFcgPT4gUjMKUGxheW91dHM6IDIzMTAsIFdpbjog NTguMDclLCBQVjogQzE3IEQxNyBDMTYgRDE1IEMxNSBDMTQgQjE0IE8xNyBRMTQgUTEzIFAxNCBT MTUgUjE3IEwxNyBOMTQgUjEyIEQxNCBDMTMgQjEzIEMxMiBFMTQgRTEyXQpXW3FxXTtCW3FwXTsK Q1ttb3ZlIDcsIFcgPT4gUTMKUGxheW91dHM6IDI3MTcsIFdpbjogNTcuOTQlLCBQVjogUTMgTzMg UDMgUDQgTzIgTzE3IE8xNiBOMTYgTzE1IFAxOCBSMTcgTjE1IE4xNCBNMTQgTjEzIFIxMSBNMTMg TDE0IEYxNyBFMTcgRjE2IEQxNCBRMTNdCldbcHFdO0JbbnFdOwpDW21vdmUgOSwgVyA9PiBQMwpQ bGF5b3V0czogMzA4MSwgV2luOiA1Ny45MiUsIFBWOiBQMyBQNCBPMiBPMTcgTzE2IE4xNiBPMTUg UDE4IFIxNyBSMTEgTzEzIE0xNyBPNCBOMyBOMiBNMyBPNSBNNSBROCBTMyBTMiBTNSBUMyBQNiBO NiBPOCBNNiBPMTAgSzUgSzMgSDRdCldbb3FdO0Jbb3BdOwpDW21vdmUgMTEsIFcgPT4gTzIKUGxh eW91dHM6IDUxMzMsIFdpbjogNTcuOTclLCBQVjogTzIgTzE3IE8xNiBOMTYgTzE1IFAxOCBSMTcg UjExIE8xMyBGMTcgTDE2IE0xNSBHMTcgRzE2IEgxNiBIMTcgRzE4XQpXW25yXTtCW2NxXTsKQ1tt b3ZlIDEzLCBXID0+IEM0ClBsYXlvdXRzOiA0ODkzLCBXaW46IDU3Ljc3JSwgUFY6IEM0IEQzIEUz IEUyIEYyIEYzIEU0IEcyIEQyIEYxIEMyIE8xNyBPMTYgQjMgQjIgTjE2IE8xNSBQMTggUjE3IFIx MSBPMTMgRjE3IEwxNiBNMTUgRzE3IEcxOCBIMTcgSDE4IEYxNiBFMTddCldbY3BdO0JbZHFdOwpD W21vdmUgMTUsIFcgPT4gRTQKUGxheW91dHM6IDUzOTQsIFdpbjogNTcuODAlLCBQVjogRTMgRTIg RjIgRjMgRTQgRzIgRDIgRjEgQzIgTzE3IE8xNiBOMTYgTzE1IFAxOCBSMTcgTjE1IE4xNCBNMTQg TzEzIFIxMSBNMTMgTDE0IEYxNyBFMTggTDE2IEwxMyBNMTJdCldbZXBdO0JbZXFdOwpDW21vdmUg MTcsIFcgPT4gRjMKUGxheW91dHM6IDI2MDQsIFdpbjogNTguMjAlLCBQVjogRjMgRjIgUjEyIEYx NyBSMTUgUzMgUzIgTjIgTzQgTjMgUzQgTzUgQzE0IEMxMiBFMTRdCldbZnFdO0JbZnJdOwpDW21v dmUgMTksIFcgPT4gUjEyClBsYXlvdXRzOiA0NDk2LCBXaW46IDU4LjI2JSwgUFY6IFIxMiBGMTcg UjE1IFMzIFMyIE4yIE80IE4zIFM0IE81IEMxNCBDMTIgRTE0IEUxMiBDMTAgRjQgRzMgRzIgQjNd CldbcWhdO0JbZGxdOwpDW21vdmUgMjEsIFcgPT4gQjMKUGxheW91dHM6IDIxNzksIFdpbjogNjAu MTIlLCBQVjogQjMgQjIgRzIgRjQgRzMgQjQgRjUgUDE0IFAxMiBOMiBOMyBPNCBNMiBPMTYgUzE2 IE8xMiBPMTEgTjEyIE4xMSBNMTEgTTEyIE0xMCBOOSBOMTQgTDggUzNdCldbYnFdO0JbYnJdOwpD W21vdmUgMjMsIFcgPT4gRzIKUGxheW91dHM6IDM5MjYsIFdpbjogNjAuODAlLCBQVjogRzIgRjQg RzMgQjQgRjUgTjIgTjMgTzQgTTIgUDE0IFAxMiBPMTYgUzE2IE8xMiBQMTEgTzEzIE4xMCBTMyBO MSBSMiBRMiBRMSBQMl0KV1tncl07QltmcF07CkNbbW92ZSAyNSwgVyA9PiBHMwpQbGF5b3V0czog NTg5MywgV2luOiA2MC45NCUsIFBWOiBHMyBCNCBGNSBOMiBOMyBPNCBNMiBQMTQgUDEyIE8xNiBT MTYgTzEyIFAxMSBPMTMgTjEwIFMzIE4xIFIyIFEyIFExIFAyIFI5IE8xNyBOMTcgUDE3XQpXW2dx XTtCW2JwXTsKQ1ttb3ZlIDI3LCBXID0+IEY1ClBsYXlvdXRzOiA3OTQ4LCBXaW46IDYxLjA2JSwg UFY6IEY1IE4yIE4zIE80IE0yIFAxNCBQMTIgTzE2IFMxNiBPMTIgUDExIE8xMyBOMTAgUzMgTjEg UjIgUTIgUTEgUDIgUjkgTzE3IE4xNyBQMTddCldbZm9dO0JbZ3BdOwpDW21vdmUgMjksIFcgPT4g RTIKUGxheW91dHM6IDIwNDAsIFdpbjogNTEuMjAlLCBQVjogRTIgRTUgQzIgQTMgRDIgQjMgRDYg QzYgRTYgSDQgRzYgRjcgRzcgSjMgRjEgRjggTjMgTzQgTTJdCldbZXJdO0JbYXFdOwpDW21vdmUg MzEsIFcgPT4gRDIKUGxheW91dHM6IDIwNDYsIFdpbjogNTQuOTElLCBQVjogRDIgRTUgQzIgQjMg RDYgQzYgRTYgSDQgRzYgRjcgRzcgRjggRzggTjMgTjIgTTMgSjMgSjQgSzMgSzQgSzcgTDYgTDcg TTddCldbZHJdO0JbY3JdOwpDW21vdmUgMzMsIFcgPT4gSjQKUGxheW91dHM6IDIyMDMsIFdpbjog NDQuMTYlLCBQVjogSjQgSDQgSDMgRzYgTzQgTjMgTzUgSDYgUTYgUzMgUzIgUzUgVDMgVDQgUzRd CldbaXBdO0JbaHBdOwpDW21vdmUgMzUsIFcgPT4gSjMKUGxheW91dHM6IDIyNzAsIFdpbjogNDIu MzMlLCBQVjogSjMgTjIgSDUgUDIgTzQgTjMgUzQgUzUgUTIgTzEgUzMgUjYgTzUgTTUgUjkgUTgg UTkgUDggUzcgUjcgUjE3IFA5IFAxMCBRMTEgUjExIFExMCBTOCBPMTAgUDEzXQpXW2lxXTtCW21y XTsKQ1ttb3ZlIDM3LCBXID0+IEg1ClBsYXlvdXRzOiAyOTUzLCBXaW46IDQyLjc1JSwgUFY6IEg1 IFAyIE80IE4zIFEyIE81IFM0IFM1IEMxNyBFNSBHNSBEMTcgQzE2IEMxNSBCMTUgQjE0IEMxNF0K V1tob107Qltvcl07CkNbbW92ZSAzOSwgVyA9PiBTNApQbGF5b3V0czogNDIyMywgV2luOiA0My4z NiUsIFBWOiBTNCBTNSBPNCBOMyBRMiBPNSBDMTcgRTUgRzUgRDE3IEMxNiBDMTUgQjE1IEIxNCBD MTQgRDE1IEIxMyBCMTYgQTE0IEIxNyBSMTcgTjE3IFExMyBLMTYgUjkgQzUgSDMgRTEzIEMxMV0K V1tycF07Qltyb107CkNbbW92ZSA0MSwgVyA9PiBPNApQbGF5b3V0czogMzIxOCwgV2luOiA0My4w NSUsIFBWOiBPNCBOMyBTMyBPNSBTNiBSNSBSOSBTNyBRMiBPMSBSNyBTOCBSNl0KV1tucF07Qltt cV07CkNbbW92ZSA0MywgVyA9PiBTMwpQbGF5b3V0czogNDg3NSwgV2luOiA0MS42MyUsIFBWOiBT MyBPNSBRMiBFNSBHNSBPMTcgUTE0IFIxNyBRMTcgUjE2IFExNSBTMTQgUTE4IFIxOCBMMTcgUTEz IE4xNiBQMTMgTjE0XQpXW3JxXTtCW25vXTsKQ1ttb3ZlIDQ1LCBXID0+IFEyClBsYXlvdXRzOiAy MzUyLCBXaW46IDM2LjgyJSwgUFY6IFEyIEU1IEc1IE8xNyBRMTQgUjE3IFExNyBSMTYgUTE1IFMx NCBRMTggUjE4IEwxNyBRMTMgTjE2IFAxMyBRMTEgTzE2IE8xNSBOMTUgTzE0XQpXW3ByXTtCW3Fu XTsKQ1ttb3ZlIDQ3LCBXID0+IEMxNwpQbGF5b3V0czogMjQ4NywgV2luOiAzOC4yNyUsIFBWOiBD MTcgRTUgRzUgRDE3IEMxNiBDMTQgQzE1IEQxNSBCMTQgTzEgTzE3IEMxOCBCMTggQjE1IEIxNiBC MTMgUjkgUjE2IFIxNyBTMTcgUzE4IFIxMV0KV1tjY107QltjZF07CkNbbW92ZSA0OSwgVyA9PiBE MTcKUGxheW91dHM6IDIyMTYsIFdpbjogMzkuNTElLCBQVjogRDE3IEU1IEc1IEYxNyBFMTcgRTE2 IEYxOCBPMSBSOSBPMTcgTzE2IE4xNiBPMTUgUjE3IFAxNyBSMTZdCldbZGNdO0JbZW9dOwpDW21v dmUgNTEsIFcgPT4gRzUKUGxheW91dHM6IDI3OTAsIFdpbjogMzcuMTQlLCBQVjogRzUgRjE3IEUx NyBFMTYgRjE4IE8xIEcxNyBGMTYgQjE3IEM1IE8xNiBSMTcgUjE2IFMxNiBTMTUgUzE3IFMxNCBQ MTggQzEzIFMyXQpXW2dvXTtCW2VkXTsKQ1ttb3ZlIDUzLCBXID0+IEYxOApQbGF5b3V0czogMzAw MiwgV2luOiAzNy41MiUsIFBWOiBGMTggTzEgUjkgTzE2IE8xNSBOMTUgTzE0IFIxNyBSMTYgUTE3 IFAxNiBQMTddCldbZmJdO0JbY29dOwpDW21vdmUgNTUsIFcgPT4gUjE3ClBsYXlvdXRzOiAyMTQ2 LCBXaW46IDQxLjM0JSwgUFY6IFIxNyBQMTQgTjE2IFIxMCBRMTEgTjE1IE0xNSBOMTQgTzE2IE0x NCBRMTAgUjkgTjEwXQpXW3FjXTtCW29mXTsKQ1ttb3ZlIDU3LCBXID0+IE4xNgpQbGF5b3V0czog MjQ0NywgV2luOiA0MS4wMyUsIFBWOiBOMTYgUjEwIE02IEw1IFExMSBOMTUgTTE1IE4xNCBPMTYg TzExXQpXW21kXTtCW3FqXTsKQ1ttb3ZlIDU5LCBXID0+IE02ClBsYXlvdXRzOiAzNTQ1LCBXaW46 IDQyLjM5JSwgUFY6IE02IE01IE00IE40IFAxMiBOMTQgTjEyIE0xNiBNMTcgTDE2IE4xNyBNMTIg TjExXQpXW2xuXTtCW2xtXTsKQ1ttb3ZlIDYxLCBXID0+IE40ClBsYXlvdXRzOiAyMzI1LCBXaW46 IDU5LjIxJSwgUFY6IE40IEw2IE00IE42IE0zIE8xIFExIE4xNSBPMTYgTTE1IFAxMiBOMTIgUDEw IE4xMF0KV1ttcF07QltscF07CkNbbW92ZSA2MywgVyA9PiBONQpQbGF5b3V0czogMjUxMCwgV2lu OiA2Ny41NSUsIFBWOiBONSBONyBMNiBMNyBLNiBPMSBPNiBONiBNNSBQNSBRMSBMMiBQMTIgTjE1 IE8xNiBNMTUgUDEwIE0xNiBHMTcgUjggTTExXQpXW21vXTtCW21uXTsKQ1ttb3ZlIDY1LCBXID0+ IE01ClBsYXlvdXRzOiAyODkwLCBXaW46IDc4LjY0JSwgUFY6IE01IEw1IEw2IEs2IEw3IEw4IEs3 IEo3IEs4IEs5IEo4IEg4IEo5IEoxMCBIOSBHOSBIMTAgSDExIEcxMCBGMTAgRzExIEcxMiBGMTEg RTExIEsxMCBGMTIgSDMgTzEgUzEgRDUgQjE2IEIxNSBCMTcgRjQgQjE0IEExNSBDMTUgQzE0XQpX W2xvXTtCW2tvXTsKQ1ttb3ZlIDY3LCBXID0+IEw2ClBsYXlvdXRzOiA0NjE1LCBXaW46IDczLjQ2 JSwgUFY6IEw2IEs2IEw3IEw4IEs3IEo3IEs4IEs5IEo4IEg4IEo5IEoxMCBIOSBHOSBIMTAgSDEx IEcxMCBGMTAgRzExIEcxMiBKMTEgRjExIEgzIEQ1IFAxIEY0IEIxNiBCMTUgQjE3IEMxNCBOMTQg TzEzIE4xMyBPMTIgTjEyIE8xMSBHMTYgUzE2IFMxNyBTMTVdCldba25dO0Jbam5dOwpDW21vdmUg NjksIFcgPT4gTDcKUGxheW91dHM6IDYzNzksIFdpbjogNjAuNDQlLCBQVjogTDcgTDggSzcgSjcg SzggSzkgSjggSDggSjkgSjEwIEg5IEc5IEgxMCBIMTEgRzEwIEYxMCBHMTEgRzEyIEgxMiBGMTEg SDMgTzEgUzEgTjE0IEQ1IEM2IEIxNiBCMTUgQjE3IEMxNCBHMTYgUTE1XQpXW2ttXTtCW2tsXTsK Q1ttb3ZlIDcxLCBXID0+IEs3ClBsYXlvdXRzOiA4MTIxLCBXaW46IDUzLjk4JSwgUFY6IEs3IEo3 IEs4IEs5IEo4IEg4IEo5IEoxMCBIOSBHOSBIMTAgSDExIEcxMCBGMTAgRzExIEcxMiBGMTEgRTEx IE04IEYxMiBIMyBENSBQMSBGNCBCMTYgQjE1IEIxNyBDMTQgTjE0IE8xMyBOMTMgTzEyIE4xMl0K V1tqbV07QltpbV07CkNbbW92ZSA3MywgVyA9PiBLOApQbGF5b3V0czogOTg1OSwgV2luOiA0Ny4y NyUsIFBWOiBLOCBLOSBKOCBIOCBKOSBKMTAgSDkgRzkgSDEwIEgxMSBHMTAgRjEwIEcxMSBHMTIg SDcgRjExIEgzIEQ1IFAxIEY0IEIxNiBCMTUgQjE3IEMxNCBOMTQgTzEzIE4xMyBPMTIgTjEyIE8x MSBHMTZdCldbamxdO0JbamtdOwpDW21vdmUgNzUsIFcgPT4gSjgKUGxheW91dHM6IDExNjEyLCBX aW46IDQxLjgyJSwgUFY6IEo4IEg4IEo5IEoxMCBIOSBHOSBIMTAgSDExIEcxMCBGMTAgRzExIEcx MiBGMTEgRTExIEYxMiBGMTMgSDcgRTEyIEgzIEQ1IFAxIEY0IEIxNiBCMTUgQjE3IE4xNCBCMTQg QTE1XQpXW2lsXTtCW2hsXTsKQ1ttb3ZlIDc3LCBXID0+IEo5ClBsYXlvdXRzOiAxMzM1NCwgV2lu OiA0MC43MyUsIFBWOiBKOSBKMTAgSDkgRzkgSDEwIEgxMSBHMTAgRjEwIEcxMSBHMTIgRjExIEUx MSBGMTIgRjEzIEUxMCBFMTIgSDMgRDUgUDEgRjQgQjE2IEIxNSBCMTcgTjE0IEIxNCBBMTUgQzE1 IEMxNCBEMTUgRDE0IEUxNV0KV1tpa107Qltpal07CkNbbW92ZSA3OSwgVyA9PiBIOQpQbGF5b3V0 czogMTUwOTcsIFdpbjogMzcuOTMlLCBQVjogSDkgRzkgSDEwIEgxMSBHMTAgRjEwIEcxMSBHMTIg RjExIEUxMSBGMTIgRjEzIEo2IEUxMiBIMyBENSBQMSBGNCBOMTQgTzEzIE4xMyBPMTIgTjEyIE8x MSBCMTYgQjE1IEIxNyBDMTRdCldbaGtdO0JbZ2tdOwpDW21vdmUgODEsIFcgPT4gSDEwClBsYXlv dXRzOiAxNjgzOSwgV2luOiAzNS41MiUsIFBWOiBIMTAgSDExIEcxMCBGMTAgRzExIEcxMiBGMTEg RTExIEYxMiBGMTMgRTEyIEQxMiBINyBFMTMgSDMgRDUgUDEgRjQgQjE2IEIxNSBCMTcgTjE0IEIx NCBBMTUgQzE1IEMxNF0KV1toal07QltoaV07CkNbbW92ZSA4MywgVyA9PiBHMTAKUGxheW91dHM6 IDE4NjAzLCBXaW46IDM1LjcyJSwgUFY6IEcxMCBGMTAgRzExIEcxMiBGMTEgRTExIEYxMiBGMTMg RTEyIEQxMiBMOSBFMTMgSDMgRDUgUDEgRjQgQjE2IEIxNSBCMTcgTjE0IEIxNCBBMTUgQzE1IEMx NF0KV1tnal07Qltmal07CkNbbW92ZSA4NSwgVyA9PiBHMTEKUGxheW91dHM6IDIwMzA3LCBXaW46 IDMzLjE1JSwgUFY6IEcxMSBHMTIgRjExIEUxMSBGMTIgRjEzIEUxMiBEMTIgTTggRTEzIEgzIEQ1 IEY0IE8xIFMxIE4xNCBCMTYgQjE1IEIxNyBTMTUgUzE2IFIxNSBCMTQgQTE1IEMxNSBDMTQgRDE1 IEQxNCBFMTUgRjE1IEUxN10KV1tnaV07QltnaF07CkNbbW92ZSA4NywgVyA9PiBGMTEKUGxheW91 dHM6IDE5NTQ1LCBXaW46IDM0LjE1JSwgUFY6IEYxMSBFMTEgRjEyIEYxMyBFMTIgRDEyIEUxMyBF MTQgRzEzIEQxMyBIMyBENSBQMSBGNCBOMTQgTzEzIE4xMyBPMTIgTjEyIE8xMSBOMTEgTjEwIE0x MF0KV1tmaV07QltlaV07CkNbbW92ZSA4OSwgVyA9PiBGMTIKUGxheW91dHM6IDE3MTE1LCBXaW46 IDMyLjg0JSwgUFY6IEYxMiBGMTMgRTEyIEQxMiBFMTMgRTE0IE43IEQxMyBIMyBENSBQMSBGNCBO MTQgTzEzIE4xMyBPMTIgTjEyIE8xMSBOMTEgTjEwIE0xMF0KV1tmaF07QltmZ107CkNbbW92ZSA5 MSwgVyA9PiBFMTIKUGxheW91dHM6IDE2MTE1LCBXaW46IDI4Ljc5JSwgUFY6IEUxMiBEMTIgRTEz IEUxNCBKNiBEMTMgSDMgRDUgUDEgRjQgTjE0IE8xMyBOMTMgTzEyIE4xMiBPMTEgTjExIE4xMCBN MTAgTjkgQjE3IEIxNiBBMTYgQTE1XQpXW2VoXTtCW2RoXTsKQ1ttb3ZlIDkzLCBXID0+IEUxMwpQ bGF5b3V0czogMTI2NzQsIFdpbjogMjQuNzIlLCBQVjogRTEzIEUxNCBLNSBEMTMgSDMgRDUgUDEg RjQgTjE0IE8xMyBOMTMgTzEyIE4xMiBPMTEgQjE3IEIxNl0KV1tlZ107QltlZl07CkNbbW92ZSA5 NSwgVyA9PiBEMTMKUGxheW91dHM6IDg2NzEsIFdpbjogMjUuMzklLCBQVjogRDEzIEMxMyBNOCBE MTQgSDMgRDUgUDEgRjQgTjE0IE8xMyBOMTNdCldbZGddO0JbY2ddOwpDW21vdmUgOTcsIFcgPT4g TTgKUGxheW91dHM6IDMwMTYsIFdpbjogMzUuMDUlLCBQVjogTTggRDE0IEgzIEQ1IFAxIEY0IE4x NCBPMTMgTjEzIE8xMiBOMTIgTzExIE4xMSBPMTAgTjEwIE45IE05IE44IEwxMCBMOSBNMTBdCldb bGxdO0JbZGZdCkNbZmluYWwgbWVzc2FnZSBmcm9tIHc6IDw8PAptb3ZlIDk5LCBXID0+IHJlc2ln bgpQbGF5b3V0czogMjQ2NiwgV2luOiAgMC4zOSUsIFBWOiBIMyBENSBQMSBGNCBOMTQgTzEzIE4x MyBPMTIgTjEyIE8xMSBOMTEgTzEwIE4xMCBOOSBNOSBOOCBMMTAgTDkgTTEwIFMxNiBTMTcgUzE1 IFAxNSBRMTUKPj4+Cgp2MTggYmVhdCBsejI4NS0yayBCK1JdCikK --00000000000014287f05b992ea61 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Computer-go mailing list [email protected] http://computer-go.org/mailman/listinfo/computer-go --00000000000014287f05b992ea61--