[Bug 73472] [OPS] Jenkins: puppet master fills /var on labs with yaml reports
| Newsgroups | gmane.org.wikimedia.mediawiki.bugs |
|---|---|
| Message-ID | <[email protected]/> |
https://bugzilla.wikimedia.org/show_bug.cgi?id=73472 --- Comment #11 from Greg Grossmeier <[email protected]> --- Just keeping the heat on this bug, we had an outage this morning (times in Eastern US): 07:49 < icinga-wm> PROBLEM - BetaLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki01.diskspace._var.byte_avail.value (33.33%) That probably caused the outage (the only other thing around that time is bug 73567, which hasn't been fixed/reverted yet beta is back up). I *really really really* want to just throw hardware at the problem, but it's a pain given how OpenStack/Beta work, but I'm getting annoyed by all the warnings that we can't do anything else about. Our (Release Engineering's) job is not to rework prod logging policies on a case-by-case basis to make it work in Beta. Continued diff creation for reasons like that only complexify (it's a word) things. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. _______________________________________________ Wikibugs-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikibugs-l