[Tools-discuss] Analysis of 4 June failed upgrade to Datatrack er infrastructure

# Summary


The 4 June upgrade attempt [1] did not go as planned and was rolled 
back.[2] We over-committed our production cluster, which supports the 
IETF Datatracker, mailarchive and other services. We also uncovered 
issues with our intended Redis configuration that we need to address 
before making that change to the Datatracker.


We will be adding resources, adjusting constraints, and improving our 
pre-deploy testing to avoid this kind of failure going forward.


We will be scheduling two windows in the near term to accomplish the 
required upgrades, the first of which will require a brief outage for 
the Datatracker.


# Details


The upgrade sequence failed at the first deploy step, which would have 
moved the Datatracker from using memcached to using Redis for local 
cache, including the session cache (which holds login state).


As a reminder, deploying the Datatracker with Redis will make persistent 
login more durable, will provide more opportunities for future 
performance improvements, and will provide a more stable platform for 
future upgrades.


This deployment involved scaling up a second instance of the main 
Datatracker container and a deployment of the Redis operator with new 
nodes to support it. The deployment failed due to resource constraints. 
During the rolling upgrade to the Datatracker, the scheduler was unable 
to place the next Datatracker instance without conflicts and placed it 
on the same node as an instance of mailarchive, despite constraints 
intended to keep that from happening. That node was quickly overwhelmed, 
resulting in several other automatic adjustments to the cluster.


At the same time, we were observing unexpected behavior with the Redis 
operator itself. What was different from our testing in staging was that 
we had immediately prior added nodes to the pool Redis would be 
scheduled into, and they were still settling as Redis was initializing. 
This triggered Redis' high-availability function (we are using Sentinel) 
to start readjusting. The readjustments were veryslow, and may have 
failed outright if we had not shut them down. We learned several things 
from analysing this and will be making adjustments and testing 
additional failure conditions in staging before attempting this part of 
the upgrade again.


Unfortunately, rolling back leaves us unable to upgrade the underlying 
Kubernetes versions without interrupting the Datatracker. During our 
focus on the initial RPC modernization deployment, we have deferred 
upgrading this cluster too long and our deployed version has fallen out 
of support.  We were not upgrading because, given the current way the 
Datatracker is deployed, each upgrade requires several minutes of 
datatracker downtime. We will be scheduling an outage for the 
Datatracker soon to upgrade Kubernetes one version to be back in the 
supported range.


Once we solve the problem that this week's failed deployment was 
intended to solve, we'll be able to upgrade the underlying Kubernetes 
versions with no Datatracker downtime. At that point, we will upgrade 
the cluster's version to the current highest supported one (upgrades 
have to go through each version sequentially), and continue to update as 
new versions come into support as a regular part of operations.


Robert Sparks

Senior Director of IT for the IETF LLC.


[1] 
https://mailarchive.ietf.org/arch/msg/ietf-announce/s9DhiXhZSzQZ4ffVbgoytyzu0eI/

[2] 
https://mailarchive.ietf.org/arch/msg/ietf-announce/jtFUp0fGdwN5xJWSTvGsmJJu-_s/

-----------------------------------------------
Tools-discuss mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mailarchive.ietf.org/arch/browse/tools-discuss/