Re: Are incident reports still being published to Wikitech?

--===============4329714827089791224==
Content-Type: multipart/alternative; boundary="0000000000003e273d06519eb8e7"

--0000000000003e273d06519eb8e7
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

>
> One of the main aspects is that the majority of infrastructure-related
> incidents that we=E2=80=99ve been dealing with over the last few months h=
ave been
> of a sensitive nature. As per WP:DNFTT
> <https://en.wikipedia.org/wiki/Wikipedia:Deny_recognition> and our
> policies around sensitive information, we don=E2=80=99t publish informati=
on on our
> response to these outages even if some of the action items are fine to be
> public.


 Another factor is the need to scale our incident tracking process. We now
> coordinate all incidents using Corto
> <https://wikitech.wikimedia.org/wiki/Corto>, which creates Phabricator
> tickets automatically - defaulting to closed.


>From looking at the #wikimedia-incidents
<https://phabricator.wikimedia.org/project/board/2143/query/all/>
dashboard, even with volunteer NDA access, only 3 tasks remain non-public;
the "rest" are public. This means either the vast majority of such tasks
are being filed in a way that even community members with appropriate
access have very little visibility into the state of incidents on Wikimedia
services (which, to remind you, previously used to be an open on-wiki
process), or y'all really aren't having that many cases where DENY is
being followed
and I cannot make up my mind which in this instance is the correct answer.

The last factor is our incident review process. To account for the
> diversity of incidents and involvement of many teams, we have moved to a
> less rigid process where the filling out of the wikitech document and the
> scorecard etc don=E2=80=99t feature. This is the main reason for the lack=
 of
> updates to the incident status page.
>


> The process of filling out the wikitech incident reports has been a fairl=
y
> arduous process of copying information out of Phabricator tasks and Googl=
e
> docs which duplicates effort.


If the main thrust is "we would like our newer crop of engineers/managers
to not use wikitext and instead use an IRC bot" (which, imo, is definitely
not a great take to have given that this is Wikimedia), why not just code
the bot to transparently automatically copy everything to a wikitext page
(similar to SAL)?

The reason I say this is because there appears to be no current easy way to
chronologically look at incident reports per year in a nice and
organized manner,
and the current approach being used is very hard to scale if I want to
query the question, "So, the day before yesterday, somebody on VPT saw a
ton of broken thumbnails, was that an incident or a me problem ?"


Regards,
Sohom Datta
---
Open-source contributor @Wikimedia


On Tue, May 12, 2026 at 7:29=E2=80=AFAM Hugh Nowlan via Wikitech-l <
[email protected]> wrote:

> Hello,
>
> On Sun, 26 Apr 2026 at 21:11, a smart kitten via Wikitech-l <
> [email protected]> wrote:
>
>> Hey all,
>>
>> Just a question -- do folks know if incident reports are still being
>> completed & published on Wikitech (such that they're listed at [0])?
>>
>> I just thought to ask given that - for me - the Phabricator search
>> results at [1] list 9 tasks created since the start of this year that ar=
e
>> tagged with `#Wikimedia-Incident` (& some folks with an NDA may be able =
to
>> see more tasks than that), but the Wikitech page at [0] currently only
>> lists there as having been one incident that's occurred since the start =
of
>> this year [2].
>>
>
> The short answer is no, they are not being published regularly any more -
> there have been a few shifts in incident response that we need to documen=
t
> and align our processes on.
>
> There are a few factors at work here:
> * One of the main aspects is that the majority of infrastructure-related
> incidents that we=E2=80=99ve been dealing with over the last few months h=
ave been
> of a sensitive nature. As per WP:DNFTT
> <https://en.wikipedia.org/wiki/Wikipedia:Deny_recognition> and our
> policies around sensitive information, we don=E2=80=99t publish informati=
on on our
> response to these outages even if some of the action items are fine to be
> public.
> * Another factor is the need to scale our incident tracking process. We
> now coordinate all incidents using Corto
> <https://wikitech.wikimedia.org/wiki/Corto>, which creates Phabricator
> tickets automatically - defaulting to closed.
> * The last factor is our incident review process. To account for the
> diversity of incidents and involvement of many teams, we have moved to a
> less rigid process where the filling out of the wikitech document and the
> scorecard etc don=E2=80=99t feature. This is the main reason for the lack=
 of
> updates to the incident status page.
>
> The process of filling out the wikitech incident reports has been a fairl=
y
> arduous process of copying information out of Phabricator tasks and Googl=
e
> docs which duplicates effort.
> For the time being, given the more concerted use of Phabricator tasks in
> the era of Corto, we will try to update and publish Phabricator tickets
> wherever possible. When an incident is being closed out by the incident
> coordinator, they will transfer relevant information from the Google doc
> (if needed) and open up the task if it is suitable to do so in order to
> provide context on our outages and any interesting technical lessons
> learned. We=E2=80=99ll also work towards formalising a process to replace=
 the
> existing process officially.
>
> Cheers,
> Hugh
> _______________________________________________
> Wikitech-l mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.or=
g/

--0000000000003e273d06519eb8e7
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px =
0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span cl=
ass=3D"gmail_default" style=3D"font-family:verdana,sans-serif"></span>One o=
f the main aspects is that the majority of infrastructure-related incidents=
 that we=E2=80=99ve been dealing with over the last few months have been of=
 a sensitive nature. As per=C2=A0<a href=3D"https://en.wikipedia.org/wiki/W=
ikipedia:Deny_recognition" target=3D"_blank">WP:DNFTT</a>=C2=A0and our poli=
cies around sensitive information, we don=E2=80=99t publish information on =
our response to these outages even if some of the action items are fine to =
be public.</blockquote><div><br></div><blockquote class=3D"gmail_quote" sty=
le=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);paddi=
ng-left:1ex">=C2=A0<span class=3D"gmail_default" style=3D"font-family:verda=
na,sans-serif"></span>Another factor is the need to scale our incident trac=
king process. We now coordinate all incidents using=C2=A0<a href=3D"https:/=
/wikitech.wikimedia.org/wiki/Corto" target=3D"_blank">Corto</a>, which crea=
tes Phabricator tickets automatically - defaulting to closed.</blockquote><=
div><br></div><div><div class=3D"gmail_default" style=3D"font-family:verdan=
a,sans-serif"><span class=3D"Asgive ng" style=3D"border-style:none;backgrou=
nd:none">From looking</span> at the <a href=3D"https://phabricator.wikimedi=
a.org/project/board/2143/query/all/">#wikimedia-incidents</a> dashboard, <s=
pan class=3D"Asgive ng" style=3D"border-style:none;background:none">even wi=
th volunteer NDA access, only 3 tasks remain non-public;</span> the &quot;r=
est&quot; are public. This means either the vast majority of such tasks are=
 being filed in a way that even community members with appropriate access h=
ave very little visibility into the state of incidents on Wikimedia service=
s (which, to remind you, previously used to be an open on-wiki process), or=
 y&#39;all really aren&#39;t having that many cases where DENY is being <sp=
an class=3D"Asgive ng" style=3D"border-style:none;background:none">followed=
 and</span> I cannot <span class=3D"Asgive ng" style=3D"border-style:none;b=
ackground:none">make up my mind which in this instance is the correct answe=
r.</span></div><div class=3D"gmail_default" style=3D"font-family:verdana,sa=
ns-serif"><br></div><blockquote style=3D"margin:0px 0px 0px 0.8ex;border-le=
ft:1px solid rgb(204,204,204);padding-left:1ex" class=3D"gmail_quote"><span=
 class=3D"gmail_default" style=3D"font-family:verdana,sans-serif"></span>Th=
e last factor is our incident review process. To account for the diversity =
of incidents and involvement of many teams, we have moved to a less rigid p=
rocess where the filling out of the wikitech document and the scorecard etc=
 don=E2=80=99t feature. This is the main reason for the lack of updates to =
the incident status page.<br></blockquote><div>=C2=A0</div><blockquote styl=
e=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);paddin=
g-left:1ex" class=3D"gmail_quote"><span class=3D"gmail_default" style=3D"fo=
nt-family:verdana,sans-serif">The process of filling out the wikitech incid=
ent reports has been a fairly arduous process of copying information out of=
 Phabricator tasks and Google docs which duplicates effort.</span></blockqu=
ote><div><br></div><div class=3D"gmail_default" style=3D"font-family:verdan=
a,sans-serif">If the main thrust is &quot;we would like our newer crop of e=
ngineers/managers to not use wikitext and instead use an IRC bot&quot; (whi=
ch, imo, is definitely not a great take <span class=3D"Asgive ng" style=3D"=
border-style:none;background:none">to have </span>given that this is Wikime=
dia), why not just code the bot to transparently automatically copy everyth=
ing to a wikitext page (similar to SAL)?</div><br></div><div><div class=3D"=
gmail_default" style=3D"font-family:verdana,sans-serif"><span class=3D"Asgi=
ve ng" style=3D"border-style:none;background:none">The reason I say this is=
</span> because there appears to be <span class=3D"Q6ibn ng" style=3D"borde=
r-style:none;background:none">no current easy </span>way to chronologically=
 <span class=3D"Asgive ng" style=3D"border-style:none;background:none">look=
 at</span> incident reports per year in a nice and organized <span class=3D=
"Asgive ng" style=3D"border-style:none;background:none">manner, and the</sp=
an> current approach <span class=3D"Asgive ng" style=3D"border-style:none;b=
ackground:none">being used </span>is very hard to scale <span class=3D"Asgi=
ve ng" style=3D"border-style:none;background:none">if I want</span> to<span=
 class=3D"Asgive ng" style=3D"border-style:none;background:none"> query the=
 question, &quot;So, the</span> day before yesterday, somebody on VPT saw<s=
pan class=3D"Asgive ng" style=3D"border-style:none;background:none"> a ton =
of</span> broken<span class=3D"Q6ibn ng" style=3D"border-style:none;backgro=
und:none"> thumbnails, </span>was that an incident or a<span class=3D"Asgiv=
e ng" style=3D"border-style:none;background:none"> me problem ?</span>&quot=
;=C2=A0=C2=A0</div><br></div><div>=C2=A0</div><div><div dir=3D"ltr" class=
=3D"gmail_signature" data-smartmail=3D"gmail_signature"><div dir=3D"ltr"><d=
iv><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div d=
ir=3D"ltr"><div><div dir=3D"ltr"><div><font face=3D"verdana, sans-serif" co=
lor=3D"#666666">Regards,</font></div><div><font face=3D"verdana, sans-serif=
" color=3D"#666666">Sohom Datta</font><span></span></div><div><font face=3D=
"verdana, sans-serif" color=3D"#666666">---</font></div><div><font face=3D"=
verdana, sans-serif" color=3D"#666666">Open-source contributor @Wikimedia</=
font></div></div></div></div></div></div></div></div></div></div></div></di=
v></div></div><br></div><br><div class=3D"gmail_quote gmail_quote_container=
"><div dir=3D"ltr" class=3D"gmail_attr">On Tue, May 12, 2026 at 7:29=E2=80=
=AFAM Hugh Nowlan via Wikitech-l &lt;<a href=3D"mailto:[email protected]=
imedia.org">[email protected]</a>&gt; wrote:<br></div><blockqu=
ote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px=
 solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><div>Hello,=C2=
=A0</div><div><br></div><div dir=3D"ltr">On Sun, 26 Apr 2026 at 21:11, a sm=
art kitten via Wikitech-l &lt;<a href=3D"mailto:[email protected].=
org" target=3D"_blank">[email protected]</a>&gt; wrote:</div><=
div dir=3D"ltr"><div class=3D"gmail_quote"><blockquote class=3D"gmail_quote=
" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);=
padding-left:1ex">Hey all,<br>
<br>
Just a question -- do folks know if incident reports are still being comple=
ted &amp; published on Wikitech (such that they&#39;re listed at [0])?<br>
<br>
I just thought to ask given that - for me - the Phabricator search results =
at [1] list 9 tasks created since the start of this year that are tagged wi=
th `#Wikimedia-Incident` (&amp; some folks with an NDA may be able to see m=
ore tasks than that), but the Wikitech page at [0] currently only lists the=
re as having been one incident that&#39;s occurred since the start of this =
year [2].<br></blockquote><div><br></div><div>The short answer is no, they =
are not being published regularly any more - there have been a few shifts i=
n incident response that we need to document and align our processes on.=C2=
=A0</div><div><br></div><div>There are a few factors at work here:</div><di=
v>* One of the main aspects is that the majority of infrastructure-related =
incidents that we=E2=80=99ve been dealing with over the last few months hav=
e been of a sensitive nature. As per <a href=3D"https://en.wikipedia.org/wi=
ki/Wikipedia:Deny_recognition" target=3D"_blank">WP:DNFTT</a> and our polic=
ies around sensitive information, we don=E2=80=99t publish information on o=
ur response to these outages even if some of the action items are fine to b=
e public.</div><div>* Another factor is the need to scale our incident trac=
king process. We now coordinate all incidents using <a href=3D"https://wiki=
tech.wikimedia.org/wiki/Corto" target=3D"_blank">Corto</a>, which creates P=
habricator tickets automatically - defaulting to closed.</div><div>* The la=
st factor is our incident review process. To account for the diversity of i=
ncidents and involvement of many teams, we have moved to a less rigid proce=
ss where the filling out of the wikitech document and the scorecard etc don=
=E2=80=99t feature. This is the main reason for the lack of updates to the =
incident status page.</div><div><br></div><div>The process of filling out t=
he wikitech incident reports has been a fairly arduous process of copying i=
nformation out of Phabricator tasks and Google docs which duplicates effort=
.</div><div>For the time being, given the more concerted use of Phabricator=
 tasks in the era of Corto, we will try to update and publish Phabricator t=
ickets wherever possible. When an incident is being closed out by the incid=
ent coordinator, they will transfer relevant information from the Google do=
c (if needed) and open up the task if it is suitable to do so in order to p=
rovide context on our outages and any interesting=C2=A0technical lessons le=
arned. We=E2=80=99ll also work towards formalising a process to replace the=
 existing process officially.</div><div><br></div><div>Cheers,</div><div>Hu=
gh</div></div></div>
</div>
_______________________________________________<br>
Wikitech-l mailing list -- <a href=3D"mailto:[email protected]=
" target=3D"_blank">[email protected]</a><br>
To unsubscribe send an email to <a href=3D"mailto:[email protected]=
kimedia.org" target=3D"_blank">[email protected]</a><br>
<a href=3D"https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wik=
imedia.org/" rel=3D"noreferrer" target=3D"_blank">https://lists.wikimedia.o=
rg/postorius/lists/wikitech-l.lists.wikimedia.org/</a></blockquote></div>

--0000000000003e273d06519eb8e7--

--===============4329714827089791224==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Wikitech-l mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
--===============4329714827089791224==--