ibdata1 growed almost to the size of the partition space
"Heinz Hartfiel" <[email protected]> Thu, 01 Oct 2009 09:44:43 +0200
| Newsgroups | gmane.comp.security.ids.prelude.user |
|---|---|
| Message-ID | <[email protected]> |
As a new member of the list, first of all "Hello to everybody!" Some weeks ago we've installed prelude-ids on debian lenny 5 for testing purposes. After some tests i was pleasantly surprised about the possibilities of prelude (Great work!). Last week we had the following issue on our prelude-ids system: /var/lib/mysql/ibdata1 growed almost to the size of the partition space in our environment. Size Used Avail Use% 9.2G 8.4G 342M 97% We were unable to delete the alerts inside the agent configuration of prewikka. An internal server error occured while trying to delete alerts or heartbeats inside prewikka. We've used the following procudere to delete the alerts and to reduce the size of ibdata1 down to 1GB. I would like to share this information. Maybe it's usefull for somebody. How did we shrink the size of ibdata1? The following procedure works for us: 1.Stopping of all prelude related sensors on all involved hosts and apache2/prewikka on the prelude manager host. /etc/init.d/snort stop /etc/init.d/snort prelude-manager stop /etc/init.d/snort prelude-lml stop /etc/init.d/snort prelude-correlator stop /etc/init.d/apache2 stop 2.Quick and dirty remote filesystem mount via sshfs sshfs root@serverxyz:/tmp/remote/ /tmp/remote/ (e.g. sshfs remoteuser@remotehost:/path/to/remote_dir local_mountpoint) 3.preludedb-admin command to delete alle alerts before a specific date preludedb-admin delete alert "type=mysql name=prelude user=xxxxx pass=xxxxx" --criteria "alert.create_time < yyyy-mm-dd" 4.During the preludedb-admin delete process we've got the following error: "The total number of locks exceeds the lock table size". We've changed the innodb_buffer_pool_size to solve this inside /etc/mysql/my.cnf from the default value and did a restart of mysqld innodb_buffer_pool_size = 8388608 #default value -> should be changed to innodb_buffer_pool_size = 83886080 5.Run the preludedb-admin command again (depending on the size of the db, in our case ibdata1 was about 8GB in size, the following command runs several hours): preludedb-admin delete alert "type=mysql name=prelude user=xxxxx pass=xxxxx" --criteria "alert.create_time <= yyyy-mm-dd" 6.Dump of all databases / removal of /var/lib/mysql/ibdata1 and ib_logfiles mysqldump --verbose --user=xxxxx --password=xxxxx --extended-insert --all-databases --add-drop-database --disable-keys --flush-privileges --quick --routines --triggers > /tmp/remote/all_serverxyz_dbs.sql /etc/init.d/mysqld stop rm -i /var/lib/mysql/ib* 7.Re-initialize of the database mysql_install_db /etc/init.d/mysqld start 8.DB dump import mysql --user=xxxxx -p SET FOREIGN_KEY_CHECKS=0; SOURCE /tmp/remote/all_serverxyz_dbs.sql; SET FOREIGN_KEY_CHECKS=1; /etc/init.d/mysqld restart 9. Remote file system unmount fusermount -u /tmp/remote/ 10. Restart of all snort & prelude sensors and apache2. Furthermore it is planned to use the following script via cron to remove all alerts which are older than 1 month via cron: #!/bin/sh DATE=$(date -d "last month" +"%Y-%m-%d") echo "All prelude alerts before $DATE will be removed!" preludedb-admin delete alert "type=mysql name=prelude user=xxxxx pass=xxxxx" --criteria "alert.create_time < $DATE" It would be great if someone could verify if the above approach is correct or if we miss something important. Any help to optimize the prelude and mysql environment would be appreciated _______________________________________________ Prelude-user site list [email protected] http://lists.prelude-ids.org/mailman/listinfo/prelude-user