Re: Time for amber2?
Marek Olšák <[email protected]> Wed, 20 May 2026 16:45:33 -0400
| Newsgroups | gmane.comp.video.mesa3d.devel |
|---|---|
| Message-ID | <CAAxE2A6J3+TpkCQ4vx4o8cjyjJL1qci0ugTnA+fTzhD1De020w@mail.gmail.com> |
--0000000000004bc4f2065245e1ca Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I'd say we'll deal with any CI issues discretely and individually once we have the list of driver maintainers and CI job maintainers. There may be a sense of urgency, but I don't think we need a specific time limit. Marek On Wed, May 20, 2026, 01:36 Iago Toral <[email protected]> wrote: > El jue, 14-05-2026 a las 19:13 -0400, Marek Ol=C5=A1=C3=A1k escribi=C3=B3= : > > Here's a more detailed description of the problem and a possible > > solution. > > > > First, the worst case scenario: A small one-line commit that=E2=80=99s > > correct > > and trivial causes a test failure in the CI. The maintainer of the > > affected driver is asked for help, who concludes that it=E2=80=99s like= ly a > > HW > > bug and is forwarded to the HW team of the corresponding GPU company. > > Now the management of the GPU company has to allocate staff to > > investigate the failure. 3 months later, we may have a workaround. Or > > not. > > > > Second, the scale: The CI has lots of undocumented devices with > > undocumented erratas and drivers with hacks and incomplete > > implementations. (that=E2=80=99s normal for any project) Any of those d= evices > > can fail at any time for reasons that might not make sense, and any > > of > > the drivers can fail for random reasons too. It=E2=80=99s not fair to a= sk the > > contributor to keep everything conformant at every MR. Even if the > > devices were documented with open source implementations (e.g. uarch > > specs, HDL, RTL) and well documented drivers, it=E2=80=99s not reasonab= le to > > ask the contributor to study them all. > > > > Thus, we can=E2=80=99t expect the contributor to be solely responsible = for > > conformance of all devices at every MR in main. > > > > It=E2=80=99s useful to keep drivers that have regular contributors conf= ormant > > at most commits in main, but why do we need to keep drivers without > > contributors conformant? If somebody cares about those drivers but > > not > > enough to contribute in main, they can contribute fixes during the RC > > window or on their own schedule. > > > > We need a two-tier system: > > > > Tier 1: > > - Devices are tested by the CI pre-merge. > > - A contact person is required for CI failure assessment and closure > > within a reasonable time. (if the person is on leave, a backup person > > must be available, or else the device is moved to Tier 2) > > > I think this makes sense, but we need to agree on what "reasonable > time" means to make sure everyone is on the same page. > > Iago > > > - Highly recommended: A fully functional drm-shim for each CI job > > with > > a user guide, how to print compiled shaders, etc. > > - Links to HW documentation if available. > > - If maintainers end up xfailing a significant number of failures > > regularly, the device is moved to Tier 2. (due to not using the CI to > > maintain conformance) > > > > Tier 2: > > - Pre-merge CI can=E2=80=99t run on the target devices / implementation= s. > > main > > doesn=E2=80=99t have to work. The quality of release branches is up to > > maintainers. The RC window can be extended. > > - Only unit tests can run per-merge, as well as any deviceless driver > > tests, like the following. > > - Optionally develop deviceless driver validation tests that verify > > driver output (shader instructions, command buffers). LLVM LIT tests > > are the perfect example - they validate all LLVM backends and prevent > > regressions without any physical devices. > > > > > > Marek > > > > On Fri, May 1, 2026 at 5:21=E2=80=AFAM Daniel Stone <[email protected]= rg> > > wrote: > > > > > > Hi, > > > > > > On Thu, 30 Apr 2026 at 23:34, Timur Krist=C3=B3f > > > <[email protected]> wrote: > > > > On 2026. =C3=A1prilis 30., cs=C3=BCt=C3=B6rt=C3=B6k 23:07:12 k=C3= =B6z=C3=A9p-eur=C3=B3pai ny=C3=A1ri id=C5=91 > > > > Marek Ol=C5=A1=C3=A1k > > > > wrote: > > > > > First of all, no contributor to shared code is required to fix > > > > > issues > > > > > in all drivers that their commit breaks. The goal is to stop > > > > > using the > > > > > pre-merge CI as a justification to force unrelated contributors > > > > > to > > > > > work on all drivers just because they are contributors. It > > > > > would be a > > > > > bit exploitative to assume that every contributor must debug > > > > > all > > > > > drivers that turn red due to a change. I think I understand > > > > > that well > > > > > because I have debugged 5+ drivers by myself in the past that > > > > > are not > > > > > my responsibility to maintain, and it does feel exploitative. > > > > > > There's a bit more nuance in this though. If one set of people is > > > breaking 17 drivers every day because they can't be bothered to do > > > the > > > basics to keep things working and just want to yolo whatever they > > > just > > > thought of into the tree, it's 'unethical' and unfair on the rest > > > of > > > the people who then spend their entire time bisecting and fixing up > > > what the others broke. (Those people then probably get accused of > > > being freeloaders and exploiting the labour of the people breaking > > > everything, because they don't get to spend any time on fun new > > > stuff, > > > given all their time is spent fixing what the others broke.) > > > > > > I think we've all taken it as axiomatic that there's a balance to > > > be > > > struck there: don't make others miserable because you can't be > > > bothered spending five minutes thinking about why your new code > > > breaks > > > existing users, but on the other hand you absolutely should expect > > > support from the relevant people to help work it out and resolve > > > it. > > > > > > I'm pretty sure no-one is suggesting ripping up that social > > > contract, > > > but we should be clear about what we mean. > > > > > > > > Therefore, we could establish that each driver/HW combo in pre- > > > > > merge > > > > > CI has the following options: > > > > > 1) a contact person for prompt CI issue resolution > > > > > 2) unconditional xfail by the author (or removal from pre-merge > > > > > CI if > > > > > logs lack the information necessary to add xfail) > > > > > > > > I think we should establish both of those, in that order. > > > > That is, if the contact person does not reply promptly, just > > > > let's add the > > > > expected failure. > > > > > > Yeah, that's a pretty obvious baseline. So far it seems to have > > > worked > > > out in the usual way (people know who works on what so it's easy to > > > ping them however), but if that's not working out, maybe someone > > > could > > > suggest a more formal document along the lines of MAINTAINERS or > > > CODEOWNERS or whatever? > > > > > > Cheers, > > > Daniel > > > > --0000000000004bc4f2065245e1ca Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"auto"><div>I'd say we'll deal with any CI issues discre= tely and individually once we have the list of driver maintainers and CI jo= b maintainers. There may be a sense of urgency, but I don't think we ne= ed a specific time limit.</div><div dir=3D"auto"><br></div><div dir=3D"auto= ">Marek<br><br><div class=3D"gmail_quote gmail_quote_container" dir=3D"auto= "><div dir=3D"ltr" class=3D"gmail_attr">On Wed, May 20, 2026, 01:36 Iago To= ral <<a href=3D"mailto:[email protected]">[email protected]</a>> wrot= e:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bo= rder-left:1px #ccc solid;padding-left:1ex">El jue, 14-05-2026 a las 19:13 -= 0400, Marek Ol=C5=A1=C3=A1k escribi=C3=B3:<br> > Here's a more detailed description of the problem and a possible<b= r> > solution.<br> > <br> > First, the worst case scenario: A small one-line commit that=E2=80=99s= <br> > correct<br> > and trivial causes a test failure in the CI. The maintainer of the<br> > affected driver is asked for help, who concludes that it=E2=80=99s lik= ely a<br> > HW<br> > bug and is forwarded to the HW team of the corresponding GPU company.<= br> > Now the management of the GPU company has to allocate staff to<br> > investigate the failure. 3 months later, we may have a workaround. Or<= br> > not.<br> > <br> > Second, the scale: The CI has lots of undocumented devices with<br> > undocumented erratas and drivers with hacks and incomplete<br> > implementations. (that=E2=80=99s normal for any project) Any of those = devices<br> > can fail at any time for reasons that might not make sense, and any<br= > > of<br> > the drivers can fail for random reasons too. It=E2=80=99s not fair to = ask the<br> > contributor to keep everything conformant at every MR. Even if the<br> > devices were documented with open source implementations (e.g. uarch<b= r> > specs, HDL, RTL) and well documented drivers, it=E2=80=99s not reasona= ble to<br> > ask the contributor to study them all.<br> > <br> > Thus, we can=E2=80=99t expect the contributor to be solely responsible= for<br> > conformance of all devices at every MR in main.<br> > <br> > It=E2=80=99s useful to keep drivers that have regular contributors con= formant<br> > at most commits in main, but why do we need to keep drivers without<br= > > contributors conformant? If somebody cares about those drivers but<br> > not<br> > enough to contribute in main, they can contribute fixes during the RC<= br> > window or on their own schedule.<br> > <br> > We need a two-tier system:<br> > <br> > Tier 1:<br> > - Devices are tested by the CI pre-merge.<br> > - A contact person is required for CI failure assessment and closure<b= r> > within a reasonable time. (if the person is on leave, a backup person<= br> > must be available, or else the device is moved to Tier 2)<br> <br> <br> I think this makes sense, but we need to agree on what "reasonable<br> time" means to make sure everyone is on the same page.<br> <br> Iago<br> <br> > - Highly recommended: A fully functional drm-shim for each CI job<br> > with<br> > a user guide, how to print compiled shaders, etc.<br> > - Links to HW documentation if available.<br> > - If maintainers end up xfailing a significant number of failures<br> > regularly, the device is moved to Tier 2. (due to not using the CI to<= br> > maintain conformance)<br> > <br> > Tier 2:<br> > - Pre-merge CI can=E2=80=99t run on the target devices / implementatio= ns.<br> > main<br> > doesn=E2=80=99t have to work. The quality of release branches is up to= <br> > maintainers. The RC window can be extended.<br> > - Only unit tests can run per-merge, as well as any deviceless driver<= br> > tests, like the following.<br> > - Optionally develop deviceless driver validation tests that verify<br= > > driver output (shader instructions, command buffers). LLVM LIT tests<b= r> > are the perfect example - they validate all LLVM backends and prevent<= br> > regressions without any physical devices.<br> > <br> > <br> > Marek<br> > <br> > On Fri, May 1, 2026 at 5:21=E2=80=AFAM Daniel Stone <<a href=3D"mai= lto:[email protected]" target=3D"_blank" rel=3D"noreferrer">daniel@foois= hbar.org</a>><br> > wrote:<br> > > <br> > > Hi,<br> > > <br> > > On Thu, 30 Apr 2026 at 23:34, Timur Krist=C3=B3f<br> > > <<a href=3D"mailto:[email protected]" target=3D"_blank" = rel=3D"noreferrer">[email protected]</a>> wrote:<br> > > > On 2026. =C3=A1prilis 30., cs=C3=BCt=C3=B6rt=C3=B6k 23:07:12= k=C3=B6z=C3=A9p-eur=C3=B3pai ny=C3=A1ri id=C5=91<br> > > > Marek Ol=C5=A1=C3=A1k<br> > > > wrote:<br> > > > > First of all, no contributor to shared code is required= to fix<br> > > > > issues<br> > > > > in all drivers that their commit breaks. The goal is to= stop<br> > > > > using the<br> > > > > pre-merge CI as a justification to force unrelated cont= ributors<br> > > > > to<br> > > > > work on all drivers just because they are contributors.= It<br> > > > > would be a<br> > > > > bit exploitative to assume that every contributor must = debug<br> > > > > all<br> > > > > drivers that turn red due to a change. I think I unders= tand<br> > > > > that well<br> > > > > because I have debugged 5+ drivers by myself in the pas= t that<br> > > > > are not<br> > > > > my responsibility to maintain, and it does feel exploit= ative.<br> > > <br> > > There's a bit more nuance in this though. If one set of peopl= e is<br> > > breaking 17 drivers every day because they can't be bothered = to do<br> > > the<br> > > basics to keep things working and just want to yolo whatever they= <br> > > just<br> > > thought of into the tree, it's 'unethical' and unfair= on the rest<br> > > of<br> > > the people who then spend their entire time bisecting and fixing = up<br> > > what the others broke. (Those people then probably get accused of= <br> > > being freeloaders and exploiting the labour of the people breakin= g<br> > > everything, because they don't get to spend any time on fun n= ew<br> > > stuff,<br> > > given all their time is spent fixing what the others broke.)<br> > > <br> > > I think we've all taken it as axiomatic that there's a ba= lance to<br> > > be<br> > > struck there: don't make others miserable because you can'= ;t be<br> > > bothered spending five minutes thinking about why your new code<b= r> > > breaks<br> > > existing users, but on the other hand you absolutely should expec= t<br> > > support from the relevant people to help work it out and resolve<= br> > > it.<br> > > <br> > > I'm pretty sure no-one is suggesting ripping up that social<b= r> > > contract,<br> > > but we should be clear about what we mean.<br> > > <br> > > > > Therefore, we could establish that each driver/HW combo= in pre-<br> > > > > merge<br> > > > > CI has the following options:<br> > > > > 1) a contact person for prompt CI issue resolution<br> > > > > 2) unconditional xfail by the author (or removal from p= re-merge<br> > > > > CI if<br> > > > > logs lack the information necessary to add xfail)<br> > > > <br> > > > I think we should establish both of those, in that order.<br= > > > > That is, if the contact person does not reply promptly, just= <br> > > > let's add the<br> > > > expected failure.<br> > > <br> > > Yeah, that's a pretty obvious baseline. So far it seems to ha= ve<br> > > worked<br> > > out in the usual way (people know who works on what so it's e= asy to<br> > > ping them however), but if that's not working out, maybe some= one<br> > > could<br> > > suggest a more formal document along the lines of MAINTAINERS or<= br> > > CODEOWNERS or whatever?<br> > > <br> > > Cheers,<br> > > Daniel<br> > <br> <br> </blockquote></div></div></div> --0000000000004bc4f2065245e1ca--