[pr-preview] PR Preview outage post-mortem

--0000000000001c8bd505a9112b33
Content-Type: text/plain; charset="UTF-8"

Hi folks,

Here's a post-mortem following the recent outage of PR Preview

## Current situation ##
The problem is fully resolved and PR PReview is working normally.

To retrigger a build, please edit the body of the pull request on GitHub
(e.g. just a line break).

## Duration of outage ##
Multiple days. The outage was so long because the alert system failed to
handle this as an error case.

## Cause of outage ##
The outage was due to Github having deprecated an API endpoint way in
advance of the announced plan[1].

The problem was compounded by the error message being logged not containing
the keyword "error". As a result, it wasn't picked by the email alert
system, and I wasn't made aware of it until Anne filed an issue against the
repository[2].

## Solution ##
The cause of the outage was quickly identified. The solution was to upgrade
to the new API endpoint. This resolved the issue right away.

A support ticket was filed with GitHub to let them know about this issue.

## Follow-ups ##

- Issues were filed to (1) make sure that error handling would catch such
cases in the future[3], and (2) improve logging so that failed builds could
be re-run once the problem was solved[4].
- Searching for funding to pay for the development of these issues and
others, and cover maintenance costs should be re-prioritized.

Thanks,

--tobie

---
[1]:
https://developer.github.com/changes/2020-04-15-replacing-create-installation-access-token-endpoint/
[2]: https://github.com/tobie/pr-preview/issues/73
[3]: https://github.com/tobie/pr-preview/issues/74
[4]: https://github.com/tobie/pr-preview/issues/75

--0000000000001c8bd505a9112b33
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi folks,<div><br></div><div>Here&#39;s a post-mortem foll=
owing the recent outage of PR Preview<br><div><br></div><div>## Current sit=
uation ##</div><div>The problem is fully resolved and PR PReview is working=
 normally.</div><div><br></div><div>To retrigger a build, please edit the b=
ody of the pull request on GitHub (e.g. just a line break).</div><div><br><=
/div><div>## Duration of outage ##</div><div>Multiple days. The outage was =
so long because the alert system failed to handle this as an error case.</d=
iv><div><br></div><div>## Cause of outage ##</div><div>The outage was due t=
o Github having deprecated an API endpoint way in advance of the announced =
plan[1].</div><div><br></div><div>The problem was compounded by the error m=
essage being logged not containing the keyword &quot;error&quot;. As a=C2=
=A0result, it wasn&#39;t picked by the email alert system, and I wasn&#39;t=
 made=C2=A0aware of it until Anne filed an issue against=C2=A0the repositor=
y[2].</div><div><br></div><div>## Solution ##</div><div>The cause of the ou=
tage was quickly identified. The solution was to upgrade to the new API end=
point. This resolved the issue right away.</div><div><br></div><div>A suppo=
rt ticket was filed with GitHub to let them know about this=C2=A0issue.</di=
v><div><br></div><div>## Follow-ups ##</div><div><br></div><div>- Issues we=
re filed to (1) make sure that error handling would catch such cases in the=
 future[3], and (2) improve logging so that failed builds could be re-run o=
nce the problem was solved[4].</div><div>- Searching for funding to pay for=
 the development of these issues and others, and cover maintenance costs sh=
ould be re-prioritized.</div><div><br></div><div>Thanks,</div><div><br></di=
v><div>--tobie</div><div><br></div><div>---</div><div>[1]:=C2=A0<a href=3D"=
https://developer.github.com/changes/2020-04-15-replacing-create-installati=
on-access-token-endpoint/">https://developer.github.com/changes/2020-04-15-=
replacing-create-installation-access-token-endpoint/</a></div><div>[2]:=C2=
=A0<a href=3D"https://github.com/tobie/pr-preview/issues/73">https://github=
.com/tobie/pr-preview/issues/73</a></div></div><div>[3]:=C2=A0<a href=3D"ht=
tps://github.com/tobie/pr-preview/issues/74">https://github.com/tobie/pr-pr=
eview/issues/74</a></div><div>[4]:=C2=A0<a href=3D"https://github.com/tobie=
/pr-preview/issues/75">https://github.com/tobie/pr-preview/issues/75</a></d=
iv></div>

--0000000000001c8bd505a9112b33--