Re: [Wikitech-l] fr, nl, ja wikipedias locked

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enig872C855F6A5869E306241A7C
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Brion Vibber wrote:
> A combination of a couple configuration errors during restoration of one
> of our database servers may have corrupted data on the French, Dutch,
> and Japanese wikipedias, sending updates to the wrong server during at
> least some portion of the last 24 hours.
>
> I've temporarily locked these three wikis while we work it out.

About 11 hours of updates (from 11:20 to 22:08 UTC) were sent to the
wrong server on these three wikis. It appears to have been consistent
site-wide so there should be no data loss; we just need to move the data
back to the master server and things should be back to normal.

I'm currently backing up all the data on these wikis in preparation for
the data move; they should be back online within a few hours with no
data lost.


Details:

MySQL has a 'read-only' mode which we enable on our replicated slave
servers to protect against misconfigurations where data might be sent to
the wrong server. In the rush to get one of our two main servers back
online after a disk failure last week, this switch was left out when it
was brought online.

A subsequent typo in configuration left three wikis sending updates to
that server instead of the primary master, so there were strange
inconsistencies depending on whether the false 'master' or one of the
regular slave servers was read from. This was manifested as weird edit
conflict problems, inconsistent history, etc.

-- brion vibber (brion @ pobox.com)

--------------enig872C855F6A5869E306241A7C
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (Darwin)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFDKf/AwRnhpk1wk44RAra/AJ9RcJ8HQ8uEvjBrRU6DRN8HP5VfawCgue0p
Y2+0J7hC91wd3xW3q4r2D7c=
=QPWO
-----END PGP SIGNATURE-----

--------------enig872C855F6A5869E306241A7C--