Re: Are incident reports still being published to Wikitech?
Sohom Datta via Wikitech-l <[email protected]> Wed, 13 May 2026 16:20:18 -0400
| Newsgroups | gmane.science.linguistics.wikipedia.technical |
|---|---|
| Message-ID | <CAFf9UfyHZ5mZ1RqbOs-Bc=VmyEUGgh2xm+TtgrbL9GT0Nw45Nw@mail.gmail.com> |
--===============3775201647149154240== Content-Type: multipart/alternative; boundary="000000000000892c770651b8b75a" --000000000000892c770651b8b75a Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I've filed T426137 <https://phabricator.wikimedia.org/T426137> regarding the issues I outlined in my previous email. Regards, Sohom Datta --- Open-source contributor @Wikimedia On Tue, May 12, 2026, 9:19=E2=80=AFAM Sohom Datta <[email protected]> w= rote: > One of the main aspects is that the majority of infrastructure-related >> incidents that we=E2=80=99ve been dealing with over the last few months = have been >> of a sensitive nature. As per WP:DNFTT >> <https://en.wikipedia.org/wiki/Wikipedia:Deny_recognition> and our >> policies around sensitive information, we don=E2=80=99t publish informat= ion on our >> response to these outages even if some of the action items are fine to b= e >> public. > > > Another factor is the need to scale our incident tracking process. We >> now coordinate all incidents using Corto >> <https://wikitech.wikimedia.org/wiki/Corto>, which creates Phabricator >> tickets automatically - defaulting to closed. > > > From looking at the #wikimedia-incidents > <https://phabricator.wikimedia.org/project/board/2143/query/all/> > dashboard, even with volunteer NDA access, only 3 tasks remain non-public= ; > the "rest" are public. This means either the vast majority of such tasks > are being filed in a way that even community members with appropriate > access have very little visibility into the state of incidents on Wikimed= ia > services (which, to remind you, previously used to be an open on-wiki > process), or y'all really aren't having that many cases where DENY is bei= ng followed > and I cannot make up my mind which in this instance is the correct answer= . > > The last factor is our incident review process. To account for the >> diversity of incidents and involvement of many teams, we have moved to a >> less rigid process where the filling out of the wikitech document and th= e >> scorecard etc don=E2=80=99t feature. This is the main reason for the lac= k of >> updates to the incident status page. >> > > >> The process of filling out the wikitech incident reports has been a >> fairly arduous process of copying information out of Phabricator tasks a= nd >> Google docs which duplicates effort. > > > If the main thrust is "we would like our newer crop of engineers/managers > to not use wikitext and instead use an IRC bot" (which, imo, is definitel= y > not a great take to have given that this is Wikimedia), why not just code > the bot to transparently automatically copy everything to a wikitext page > (similar to SAL)? > > The reason I say this is because there appears to be no current easy way > to chronologically look at incident reports per year in a nice and > organized manner, and the current approach being used is very hard to > scale if I want to query the question, "So, the day before yesterday, > somebody on VPT saw a ton of broken thumbnails, was that an incident or a > me problem ?" > > > Regards, > Sohom Datta > --- > Open-source contributor @Wikimedia > > > On Tue, May 12, 2026 at 7:29=E2=80=AFAM Hugh Nowlan via Wikitech-l < > [email protected]> wrote: > >> Hello, >> >> On Sun, 26 Apr 2026 at 21:11, a smart kitten via Wikitech-l < >> [email protected]> wrote: >> >>> Hey all, >>> >>> Just a question -- do folks know if incident reports are still being >>> completed & published on Wikitech (such that they're listed at [0])? >>> >>> I just thought to ask given that - for me - the Phabricator search >>> results at [1] list 9 tasks created since the start of this year that a= re >>> tagged with `#Wikimedia-Incident` (& some folks with an NDA may be able= to >>> see more tasks than that), but the Wikitech page at [0] currently only >>> lists there as having been one incident that's occurred since the start= of >>> this year [2]. >>> >> >> The short answer is no, they are not being published regularly any more = - >> there have been a few shifts in incident response that we need to docume= nt >> and align our processes on. >> >> There are a few factors at work here: >> * One of the main aspects is that the majority of infrastructure-related >> incidents that we=E2=80=99ve been dealing with over the last few months = have been >> of a sensitive nature. As per WP:DNFTT >> <https://en.wikipedia.org/wiki/Wikipedia:Deny_recognition> and our >> policies around sensitive information, we don=E2=80=99t publish informat= ion on our >> response to these outages even if some of the action items are fine to b= e >> public. >> * Another factor is the need to scale our incident tracking process. We >> now coordinate all incidents using Corto >> <https://wikitech.wikimedia.org/wiki/Corto>, which creates Phabricator >> tickets automatically - defaulting to closed. >> * The last factor is our incident review process. To account for the >> diversity of incidents and involvement of many teams, we have moved to a >> less rigid process where the filling out of the wikitech document and th= e >> scorecard etc don=E2=80=99t feature. This is the main reason for the lac= k of >> updates to the incident status page. >> >> The process of filling out the wikitech incident reports has been a >> fairly arduous process of copying information out of Phabricator tasks a= nd >> Google docs which duplicates effort. >> For the time being, given the more concerted use of Phabricator tasks in >> the era of Corto, we will try to update and publish Phabricator tickets >> wherever possible. When an incident is being closed out by the incident >> coordinator, they will transfer relevant information from the Google doc >> (if needed) and open up the task if it is suitable to do so in order to >> provide context on our outages and any interesting technical lessons >> learned. We=E2=80=99ll also work towards formalising a process to replac= e the >> existing process officially. >> >> Cheers, >> Hugh >> _______________________________________________ >> Wikitech-l mailing list -- [email protected] >> To unsubscribe send an email to [email protected] >> >> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.o= rg/ > > --000000000000892c770651b8b75a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"auto"><div>I<span class=3D"gmail_default" styl= e=3D"font-family:verdana,sans-serif">'ve filed=C2=A0</span><a href=3D"h= ttps://phabricator.wikimedia.org/T426137" target=3D"_blank"><span class=3D"= gmail_default" style=3D"font-family:verdana,sans-serif"></span>T426137</a>= =C2=A0<span class=3D"gmail_default" style=3D"font-family:verdana,sans-serif= ">regarding the issues I outlined in my=C2=A0previous email.</span></div><d= iv><br></div><div data-smartmail=3D"gmail_signature"><div dir=3D"ltr"><div>= <div dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir= =3D"ltr"><div><div dir=3D"ltr"><div><font face=3D"verdana, sans-serif" colo= r=3D"#666666">Regards,</font></div><div><font face=3D"verdana, sans-serif" = color=3D"#666666">Sohom Datta</font><span></span></div><div><font face=3D"v= erdana, sans-serif" color=3D"#666666">---</font></div><div><font face=3D"ve= rdana, sans-serif" color=3D"#666666">Open-source contributor @Wikimedia</fo= nt></div></div></div></div></div></div></div></div></div></div></div></div>= </div></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"= gmail_attr">On Tue, May 12, 2026, 9:19=E2=80=AFAM Sohom Datta <<a href= =3D"mailto:[email protected]" target=3D"_blank">[email protected]</= a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0p= x 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><d= iv dir=3D"ltr"><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0p= x 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span clas= s=3D"gmail_default" style=3D"font-family:verdana,sans-serif"></span>One of = the main aspects is that the majority of infrastructure-related incidents t= hat we=E2=80=99ve been dealing with over the last few months have been of a= sensitive nature. As per=C2=A0<a href=3D"https://en.wikipedia.org/wiki/Wik= ipedia:Deny_recognition" rel=3D"noreferrer" target=3D"_blank">WP:DNFTT</a>= =C2=A0and our policies around sensitive information, we don=E2=80=99t publi= sh information on our response to these outages even if some of the action = items are fine to be public.</blockquote><div><br></div><blockquote class= =3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg= b(204,204,204);padding-left:1ex">=C2=A0<span class=3D"gmail_default" style= =3D"font-family:verdana,sans-serif"></span>Another factor is the need to sc= ale our incident tracking process. We now coordinate all incidents using=C2= =A0<a href=3D"https://wikitech.wikimedia.org/wiki/Corto" rel=3D"noreferrer"= target=3D"_blank">Corto</a>, which creates Phabricator tickets automatical= ly - defaulting to closed.</blockquote><div><br></div><div><div class=3D"gm= ail_default" style=3D"font-family:verdana,sans-serif"><span style=3D"border= -style:none;background:none">From looking</span> at the <a href=3D"https://= phabricator.wikimedia.org/project/board/2143/query/all/" rel=3D"noreferrer"= target=3D"_blank">#wikimedia-incidents</a> dashboard, <span style=3D"borde= r-style:none;background:none">even with volunteer NDA access, only 3 tasks = remain non-public;</span> the "rest" are public. This means eithe= r the vast majority of such tasks are being filed in a way that even commun= ity members with appropriate access have very little visibility into the st= ate of incidents on Wikimedia services (which, to remind you, previously us= ed to be an open on-wiki process), or y'all really aren't having th= at many cases where DENY is being <span style=3D"border-style:none;backgrou= nd:none">followed and</span> I cannot <span style=3D"border-style:none;back= ground:none">make up my mind which in this instance is the correct answer.<= /span></div><div class=3D"gmail_default" style=3D"font-family:verdana,sans-= serif"><br></div><blockquote style=3D"margin:0px 0px 0px 0.8ex;border-left:= 1px solid rgb(204,204,204);padding-left:1ex" class=3D"gmail_quote"><span cl= ass=3D"gmail_default" style=3D"font-family:verdana,sans-serif"></span>The l= ast factor is our incident review process. To account for the diversity of = incidents and involvement of many teams, we have moved to a less rigid proc= ess where the filling out of the wikitech document and the scorecard etc do= n=E2=80=99t feature. This is the main reason for the lack of updates to the= incident status page.<br></blockquote><div>=C2=A0</div><blockquote style= =3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding= -left:1ex" class=3D"gmail_quote"><span class=3D"gmail_default" style=3D"fon= t-family:verdana,sans-serif">The process of filling out the wikitech incide= nt reports has been a fairly arduous process of copying information out of = Phabricator tasks and Google docs which duplicates effort.</span></blockquo= te><div><br></div><div class=3D"gmail_default" style=3D"font-family:verdana= ,sans-serif">If the main thrust is "we would like our newer crop of en= gineers/managers to not use wikitext and instead use an IRC bot" (whic= h, imo, is definitely not a great take <span style=3D"border-style:none;bac= kground:none">to have </span>given that this is Wikimedia), why not just co= de the bot to transparently automatically copy everything to a wikitext pag= e (similar to SAL)?</div><br></div><div><div class=3D"gmail_default" style= =3D"font-family:verdana,sans-serif"><span style=3D"border-style:none;backgr= ound:none">The reason I say this is</span> because there appears to be <spa= n style=3D"border-style:none;background:none">no current easy </span>way to= chronologically <span style=3D"border-style:none;background:none">look at<= /span> incident reports per year in a nice and organized <span style=3D"bor= der-style:none;background:none">manner, and the</span> current approach <sp= an style=3D"border-style:none;background:none">being used </span>is very ha= rd to scale <span style=3D"border-style:none;background:none">if I want</sp= an> to<span style=3D"border-style:none;background:none"> query the question= , "So, the</span> day before yesterday, somebody on VPT saw<span style= =3D"border-style:none;background:none"> a ton of</span> broken<span style= =3D"border-style:none;background:none"> thumbnails, </span>was that an inci= dent or a<span style=3D"border-style:none;background:none"> me problem ?</s= pan>"=C2=A0=C2=A0</div><br></div><div>=C2=A0</div><div><div dir=3D"ltr= " class=3D"gmail_signature"><div dir=3D"ltr"><div><div dir=3D"ltr"><div><di= v dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir=3D"= ltr"><div><font face=3D"verdana, sans-serif" color=3D"#666666">Regards,</fo= nt></div><div><font face=3D"verdana, sans-serif" color=3D"#666666">Sohom Da= tta</font><span></span></div><div><font face=3D"verdana, sans-serif" color= =3D"#666666">---</font></div><div><font face=3D"verdana, sans-serif" color= =3D"#666666">Open-source contributor @Wikimedia</font></div></div></div></d= iv></div></div></div></div></div></div></div></div></div></div><br></div><b= r><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Tue, = May 12, 2026 at 7:29=E2=80=AFAM Hugh Nowlan via Wikitech-l <<a href=3D"m= ailto:[email protected]" rel=3D"noreferrer" target=3D"_blank">= [email protected]</a>> wrote:<br></div><blockquote class=3D= "gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(2= 04,204,204);padding-left:1ex"><div dir=3D"ltr"><div>Hello,=C2=A0</div><div>= <br></div><div dir=3D"ltr">On Sun, 26 Apr 2026 at 21:11, a smart kitten via= Wikitech-l <<a href=3D"mailto:[email protected]" rel=3D"no= referrer" target=3D"_blank">[email protected]</a>> wrote:</= div><div dir=3D"ltr"><div class=3D"gmail_quote"><blockquote class=3D"gmail_= quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,= 204);padding-left:1ex">Hey all,<br> <br> Just a question -- do folks know if incident reports are still being comple= ted & published on Wikitech (such that they're listed at [0])?<br> <br> I just thought to ask given that - for me - the Phabricator search results = at [1] list 9 tasks created since the start of this year that are tagged wi= th `#Wikimedia-Incident` (& some folks with an NDA may be able to see m= ore tasks than that), but the Wikitech page at [0] currently only lists the= re as having been one incident that's occurred since the start of this = year [2].<br></blockquote><div><br></div><div>The short answer is no, they = are not being published regularly any more - there have been a few shifts i= n incident response that we need to document and align our processes on.=C2= =A0</div><div><br></div><div>There are a few factors at work here:</div><di= v>* One of the main aspects is that the majority of infrastructure-related = incidents that we=E2=80=99ve been dealing with over the last few months hav= e been of a sensitive nature. As per <a href=3D"https://en.wikipedia.org/wi= ki/Wikipedia:Deny_recognition" rel=3D"noreferrer" target=3D"_blank">WP:DNFT= T</a> and our policies around sensitive information, we don=E2=80=99t publi= sh information on our response to these outages even if some of the action = items are fine to be public.</div><div>* Another factor is the need to scal= e our incident tracking process. We now coordinate all incidents using <a h= ref=3D"https://wikitech.wikimedia.org/wiki/Corto" rel=3D"noreferrer" target= =3D"_blank">Corto</a>, which creates Phabricator tickets automatically - de= faulting to closed.</div><div>* The last factor is our incident review proc= ess. To account for the diversity of incidents and involvement of many team= s, we have moved to a less rigid process where the filling out of the wikit= ech document and the scorecard etc don=E2=80=99t feature. This is the main = reason for the lack of updates to the incident status page.</div><div><br><= /div><div>The process of filling out the wikitech incident reports has been= a fairly arduous process of copying information out of Phabricator tasks a= nd Google docs which duplicates effort.</div><div>For the time being, given= the more concerted use of Phabricator tasks in the era of Corto, we will t= ry to update and publish Phabricator tickets wherever possible. When an inc= ident is being closed out by the incident coordinator, they will transfer r= elevant information from the Google doc (if needed) and open up the task if= it is suitable to do so in order to provide context on our outages and any= interesting=C2=A0technical lessons learned. We=E2=80=99ll also work toward= s formalising a process to replace the existing process officially.</div><d= iv><br></div><div>Cheers,</div><div>Hugh</div></div></div> </div> _______________________________________________<br> Wikitech-l mailing list -- <a href=3D"mailto:[email protected]= " rel=3D"noreferrer" target=3D"_blank">[email protected]</a><b= r> To unsubscribe send an email to <a href=3D"mailto:[email protected]= kimedia.org" rel=3D"noreferrer" target=3D"_blank">[email protected]= kimedia.org</a><br> <a href=3D"https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wik= imedia.org/" rel=3D"noreferrer noreferrer" target=3D"_blank">https://lists.= wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/</a></blockquo= te></div> </blockquote></div> --000000000000892c770651b8b75a-- --===============3775201647149154240== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Wikitech-l mailing list -- [email protected] To unsubscribe send an email to [email protected] https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ --===============3775201647149154240==--