Re: [EP-tech] Sysadmin Best Practices

Ah, we've just set up something that uses icinga to check that cron jobs
are running regularly. Our cron jobs all end with something like &&
touch /usr/share/eprints/var/timestamps/import_thingy

and then we use this plus some simple config files to check if all the
cron jobs completed recently enough.
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcgutteridge%2Fnagios-cron-monitor&data=01%7C01%7C%7C9dc3f49fc3824de1713d08d6b6c3ecc5%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&sdata=rngk17Dlt5aZPct%2F21qLfZmw9TGzjv%2BMgGdUTaRTFgI%3D&reserved=0

It means it's a single *active* nagios/icinga check.


On 01/04/2019 17:53, Newman D.R. via Eprints-tech wrote:
> Hi Adam,
>
> Here are some things I do:
>
> * I use Ansible to manage OS (YUM) package upgrades to systems on a
> fortnightly cycle. (I have a number of systems to update and don't want
> more regular or internally automated updates, where if there was
> problematic upgrade, I would either have a lot of broken systems all at
> once or systems randomly breaking at unexpected times).
>
> * I make sure I reboot systems after any kernel upgrade to ensure a
> vulnerable kernel is not left in place indefinitely.
>
> * I monitor (using Icinga) packages available for upgrade, so I can
> make sure security updates are applied more urgently.
>
> * I have written a systemd service file to ensure the indexer starts
> after a reboot.
>
> * I have a root @reboot cron job for making sure OpenOffice starts for
> coversheeting after a reboot.  I tried to write a systemd service file
> for this but is was unreliable.
>
> * I have an @reboot tidy_pids cron job to ensure there are no stale pid
>   (or tick) files in /opt/eprints3/var/ after a reboot that may stop the
> indexer or OpenOffice from restarting automatically.
>
> * I have a script that regenerates the sitemap on a daily basis.  the
> default sitemap seems to not be compatible when I tried to register
> this on Google Webmaster portal, so I wrote a modified version that
> just lists all the abstract pages and sets the changefreq to weekly.
>
> * I have lots of monitoring of the server itself.  This is the standard
> stuff like ping, load, disk space, number of processes, packages to
> upgrade, reboot required, etc. as well as some more EPrints specific
> stuff like the size like response on http, checking MySQL tables are
> not corrupted, checking if there are config files not committed to
> version control, any mail generated for the eprints user (e.g. by cron
> jobs), is the indexer/OpenOffice running, what is the size of the
> indexer queue.
>
> I am sure there are other things I do that do not spring to mind on
> first thought.
>
> Regards
>
> David Newman
>
> On Mon, 2019-04-01 at 16:46 +0100, Adam Field via Eprints-tech wrote:
>> Hi
>>
>>                  Can anyone share what they do on the systems side
>> (e.g. cron jobs, regular restarts, monitoring) that they have set up
>> on EPrints?  I’m interested in finding out what the best-practice is
>> for making an EPrints installation as reliable as possible.
>>
>> Regards
>>
>> --
>> Adam
>> *** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-
>> tech
>> *** Archive: https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.eprints.org%2Ftech.php%2F&data=01%7C01%7C%7C9dc3f49fc3824de1713d08d6b6c3ecc5%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&sdata=lElV59ilZiLsD14yJEvI0WmvfwSICIMHiRdvcFzeVDo%3D&reserved=0
>> *** EPrints community wiki: https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwiki.eprints.org%2F&data=01%7C01%7C%7C9dc3f49fc3824de1713d08d6b6c3ecc5%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&sdata=lz6tudQ4tVXm%2F99QKcPt114rOppyWpwLlRemndYUVHY%3D&reserved=0
>> *** EPrints developers Forum: https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fforum.eprints.org%2F&data=01%7C01%7C%7C9dc3f49fc3824de1713d08d6b6c3ecc5%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&sdata=kXhSVGgiv9t6zuWxUzPidr3TUGflypFr6sWBGZ0YAOs%3D&reserved=0
> *** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
> *** Archive: https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.eprints.org%2Ftech.php%2F&data=01%7C01%7C%7C9dc3f49fc3824de1713d08d6b6c3ecc5%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&sdata=lElV59ilZiLsD14yJEvI0WmvfwSICIMHiRdvcFzeVDo%3D&reserved=0
> *** EPrints community wiki: https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwiki.eprints.org%2F&data=01%7C01%7C%7C9dc3f49fc3824de1713d08d6b6c3ecc5%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&sdata=lz6tudQ4tVXm%2F99QKcPt114rOppyWpwLlRemndYUVHY%3D&reserved=0
> *** EPrints developers Forum: https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fforum.eprints.org%2F&data=01%7C01%7C%7C9dc3f49fc3824de1713d08d6b6c3ecc5%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&sdata=kXhSVGgiv9t6zuWxUzPidr3TUGflypFr6sWBGZ0YAOs%3D&reserved=0

--
Christopher Gutteridge <[email protected]>
You should read our team blog at http://blog.soton.ac.uk/webteam/


*** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
*** Archive: http://www.eprints.org/tech.php/
*** EPrints community wiki: http://wiki.eprints.org/
*** EPrints developers Forum: http://forum.eprints.org/