Re: CGOS source on github
Dan Schmidt <[email protected]> Fri, 22 Jan 2021 11:22:44 -0500
| Newsgroups | gmane.games.devel.go |
|---|---|
| Message-ID | <CAPMsxv+Mi7TCGQREKX6dix8wMJLVPz5f6Fp55Vf15Ma=1oJheQ@mail.gmail.com> |
--===============5836125722911268965== Content-Type: multipart/alternative; boundary="000000000000ce3f8f05b97f9913" --000000000000ce3f8f05b97f9913 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable The primary purpose of a rating system is to predict the results of future games accurately (this is the usual axiom, at least). In a one-dimensional rating system, such as Elo, where each player's skill is represented by a single number, it is impossible to have a (non-wacky) system where A is expected to beat B in a two-player match, B is expected to beat C in a two-player match, and C is expected to beat A in a two-player match. So if the players are eccentric in that respect, a one-dimensional rating system is always going to have real problems with accurate predictions. Dan On Fri, Jan 22, 2021 at 10:54 AM uurtamo <[email protected]> wrote: > also frankly not a problem for a rating system to handle. > > a rating system shouldn't be tweaked to handle eccentricities of its > players other than the general assumptions of how a game's result is > determined (like, does it allow for "win" and "draw" and "undetermined" o= r > just "win"). > > s. > > > On Fri, Jan 22, 2021 at 6:29 AM David Wu <[email protected]> wrote: > >> On Fri, Jan 22, 2021 at 8:08 AM R=C3=A9mi Coulom <[email protected]> >> wrote: >> >>> You are right that non-determinism and bot blind spots are a source of >>> problems with Elo ratings. I add randomness to the openings, but it is >>> still difficult to avoid repeating some patterns. I have just noticed t= hat >>> the two wins of CrazyStone-81-15po against LZ_286_e6e2_p400 were caused= by >>> very similar ladders in the opening: >>> http://www.yss-aya.com/cgos/viewer.cgi?19x19/SGF/2021/01/21/733333.sgf >>> http://www.yss-aya.com/cgos/viewer.cgi?19x19/SGF/2021/01/21/733301.sgf >>> Such a huge blind spot in such a strong engine is likely to cause ratin= g >>> compression. >>> R=C3=A9mi >>> >> >> I agree, ladders are definitely the other most noticeable way that Elo >> model assumptions may be broken, since pure-zero bots have a hard time w= ith >> them, and can easily cause difference(A,B) + difference(B,C) to be very >> inconsistent with difference(A,C). If some of A,B,C always handle ladder= s >> very well and some are blind to them, then you are right that probably n= o >> amount of opening randomization can smooth it out. >> >> _______________________________________________ >> Computer-go mailing list >> [email protected] >> http://computer-go.org/mailman/listinfo/computer-go >> > _______________________________________________ > Computer-go mailing list > [email protected] > http://computer-go.org/mailman/listinfo/computer-go > --000000000000ce3f8f05b97f9913 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>The primary purpose of a rating system is to predict = the results of future games accurately (this is the usual axiom, at least).= <br></div><div><br></div><div>In a one-dimensional rating system, such as E= lo, where each player's skill is represented by a single number, it is = impossible to have a (non-wacky) system where A is expected to beat B in a = two-player match, B is expected to beat C in a two-player match, and C is e= xpected to beat A in a two-player match.</div><div><br></div><div>So if the= players are eccentric in that respect, a one-dimensional rating system is = always going to have real problems with accurate predictions.</div><div><br= ></div><div>Dan<br></div><div><br></div><div><br></div><div class=3D"gmail_= quote"><div dir=3D"ltr" class=3D"gmail_attr">On Fri, Jan 22, 2021 at 10:54 = AM uurtamo <<a href=3D"mailto:[email protected]">[email protected]</a>&g= t; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0p= x 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div d= ir=3D"ltr"><div style=3D"font-family:arial,helvetica,sans-serif">also frank= ly not a problem for a rating system to handle.</div><div style=3D"font-fam= ily:arial,helvetica,sans-serif"><br></div><div style=3D"font-family:arial,h= elvetica,sans-serif">a rating system shouldn't be tweaked to handle ecc= entricities of its players other than the general assumptions of how a game= 's result is determined (like, does it allow for "win" and &q= uot;draw" and "undetermined" or just "win").</div>= <div style=3D"font-family:arial,helvetica,sans-serif"><br></div><div style= =3D"font-family:arial,helvetica,sans-serif">s.</div><div style=3D"font-fami= ly:arial,helvetica,sans-serif"><br></div></div><br><div class=3D"gmail_quot= e"><div dir=3D"ltr" class=3D"gmail_attr">On Fri, Jan 22, 2021 at 6:29 AM Da= vid Wu <<a href=3D"mailto:[email protected]" target=3D"_blank">light= [email protected]</a>> wrote:<br></div><blockquote class=3D"gmail_quote" = style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);pa= dding-left:1ex"><div dir=3D"ltr"><div dir=3D"ltr">On Fri, Jan 22, 2021 at 8= :08 AM R=C3=A9mi Coulom <<a href=3D"mailto:[email protected]" target= =3D"_blank">[email protected]</a>> wrote:<br></div><div class=3D"gma= il_quote"><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8= ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr= "><div>You are right that non-determinism and bot blind spots are a source = of problems with Elo ratings. I add randomness to the openings, but it is s= till difficult to avoid repeating some patterns. I have just noticed that t= he two wins of CrazyStone-81-15po against=20 LZ_286_e6e2_p400 were caused by very similar ladders in the opening:</div><= div><a href=3D"http://www.yss-aya.com/cgos/viewer.cgi?19x19/SGF/2021/01/21/= 733333.sgf" target=3D"_blank">http://www.yss-aya.com/cgos/viewer.cgi?19x19/= SGF/2021/01/21/733333.sgf</a></div><div><a href=3D"http://www.yss-aya.com/c= gos/viewer.cgi?19x19/SGF/2021/01/21/733301.sgf" target=3D"_blank">http://ww= w.yss-aya.com/cgos/viewer.cgi?19x19/SGF/2021/01/21/733301.sgf</a></div><div= >Such a huge blind spot in such a strong engine is likely to cause rating c= ompression.</div><div>R=C3=A9mi</div></div></blockquote><div><br></div><div= >I agree, ladders are definitely the other most noticeable way that Elo mod= el assumptions may be broken, since pure-zero bots have a hard time with th= em, and can easily cause difference(A,B)=C2=A0+ difference(B,C) to be very = inconsistent with difference(A,C). If some of A,B,C always handle ladders v= ery well and some are blind to them, then you are=C2=A0right that probably = no amount of opening randomization can smooth it out.</div><div><br></div><= /div></div> _______________________________________________<br> Computer-go mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">Computer-g= [email protected]</a><br> <a href=3D"http://computer-go.org/mailman/listinfo/computer-go" rel=3D"nore= ferrer" target=3D"_blank">http://computer-go.org/mailman/listinfo/computer-= go</a><br> </blockquote></div> _______________________________________________<br> Computer-go mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">Computer-g= [email protected]</a><br> <a href=3D"http://computer-go.org/mailman/listinfo/computer-go" rel=3D"nore= ferrer" target=3D"_blank">http://computer-go.org/mailman/listinfo/computer-= go</a><br> </blockquote></div></div> --000000000000ce3f8f05b97f9913-- --===============5836125722911268965== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Computer-go mailing list [email protected] http://computer-go.org/mailman/listinfo/computer-go --===============5836125722911268965==--