Re: another problem with conversations db
Michael Menge <[email protected]>
| Newsgroups | gmane.mail.imap.cyrus |
|---|---|
| Message-ID | <20190204101712.Horde.33ManGeIHA8D5TVzSiQ7825@webmail.uni-tuebingen.de> |
Hi, Quoting Bron Gondwana <[email protected]>: > Hi Michael, > > Sorry about the delay in looking at this - I was mad crazy busy > getting ready to go overseas. At Fosdem now, about to give a talk > about JMAP! > > OK, let's start with the things that give me a little bit of hives... > > configdirectory: /srv/cyrus-be > partition-default: /srv/cyrus-be > partition-ssd: /srv/cyrus-be/ssd-part > > Ouch. There's a couple of things I wouldn't do there - having the > partition be the same as the config directory, and having a separate > partition be a subdirectory of a partition. They're both asking for > trouble. I would probably lay my system out like: > > configdirectory: /srv/cyrus-be/conf > partition-default: /srv/cyrus-be/default-part > partition-ssd: /srv/cyrus-be/ssd-part > partition-default isn't used any more. To use the metapartition we moved all accounts form the default partition to the ssd partition which is the the new defaultpartition ("defaultpartition: ssd") > And then each tree would only have one type of thing in it. > > Anyway, I don't think that would break anything. > > metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part > metapartition_files: header index cache expunge squat annotations > lock dav archivecache > > Ooh, I haven't tested having cache and archivecache on the same > location. That's really interesting. Again, I'd be in favour of > separation here, give them different paths. That might be tricky > with ssd though, the way this is laid out. I assume you have some > kind of symlink farm going on? > I didn't know that there could be a problem with cache and archivecache. At the time we decided on the configuration for cyrus 3.0 I looked at the imapd.conf man page and for metapartition_files decided that I want all meta files on the ssd storage. There was no indication in the man page that there could be a problem. How do I separate location of archivecache from the other metapartition path? And fix the cache and archivecache files? No there is no sysmlink farm. We have mounted different iSCSI volumes to /srv/cyrus-ssd-be, /srv/cyrus-hdd-be and /srv/cyrus-be > Otherwise it all looks OK. Are you getting other IOERRORs in your > log files which could show things aborting? It really looks like > your conversations DB is getting out of sync due to other failures. I found a few instances of 3 related errors. Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening /srv/cyrus-be/ssd-part/L/user/XXXX/2185.: No such file or directory Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening /srv/cyrus-be/ssd-part/L/user/XXXX/2185.: No such file or directory Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR archive user.XXXX 2185 failed to copyfile (/srv/cyrus-be/ssd-part/L/user/XXXX/2185. => /srv/cyrus-hdd-be/archive/ssd-part/L/user/XXXX/2185.): Unknown code ____ 255 The file was already at /srv/cyrus-hdd-be/archive/ssd-part/L/user/XXXX/2185. stat 2185. File: '2185.' Size: 424542 Blocks: 832 IO Block: 4096 regular file Device: 860h/2144d Inode: 4677438462 Links: 1 Access: (0600/-rw-------) Uid: ( 96/ cyrus) Gid: ( 12/ mail) Context: system_u:object_r:unlabeled_t:s0 Access: 2018-11-27 01:12:46.585864239 +0100 Modify: 2018-11-27 01:12:46.585864239 +0100 Change: 2018-11-27 01:12:46.585864239 +0100 Birth: - > > Cheers, > > Bron. > > > > On Thu, Jan 31, 2019, at 19:36, Michael Menge wrote: >> >> Quoting Bron Gondwana <[email protected]>: >> >> > On Fri, Jan 25, 2019, at 20:00, Michael Menge wrote: >> >> Hi Bron >> >> >> >> >> >> Quoting Bron Gondwana <[email protected]>: >> >> >> >> > Sorry I haven't been following along with this earlier. Can you post >> >> > your imapd.conf and cyrus.conf as well as let her know if you run >> >> > anything which directly messes with files on disk. >> >> >> >> Thanks for looking into it. I have attached the backend and replica >> >> configs. There should be nothing messing with the files on disk >> >> directly. Some times we need to restore mails from file based >> >> backup (bacula) followed by a reconstruct but this was not the >> >> case this time. >> > >> > Do you always rebuild the conversations db after doing the >> > reconstruct? That will be necessary now. We switched to doing all >> > our restores using IMAP append a while back so we're never fiddling >> > the file system under Cyrus. >> > >> >> > >> >> > Also, what operating system is this on and what Cyrus version? >> >> > >> >> >> >> We are running a cyrus 3.0.8 compiled with the following Options >> >> (./configure --enable-murder --enable-http --enable-calalarmd >> >> --enable-replication --enable-backup --enable-idled >> >> --enable-autocreate CFLAGS="-fPIC -g") >> >> in a murder configuration on a RHEL 7.5 System. As filesystem we use xfs. >> > >> > They should be fine. I'll have a read of the config when I'm at a >> > real computer. >> > >> >> did you find anything in the config that would explain the problems? >> >> >> >> >> > Bron. >> >> > >> >> > On Fri, Jan 25, 2019, at 04:08, Michael Menge wrote: >> >> >> Hi, >> >> >> >> >> >> I have discovered an other problem with the conversations db: >> >> >> >> >> >> Thousends of lines with "IOERROR: conversations_audit on load:" and >> >> >> "IOERROR: conversations_audit on store:" >> >> >> A look at the source code shows that these errors are logged after >> >> >> "_sanity_check_counts" is called. >> >> >> The log level LOG_ERR and the prefix IOERROR indicate that I have a >> >> >> serious problem. Do I? >> >> >> >> >> >> This problem occurred for accounts where the rebuild of the >> >> >> conversations db was successful. >> >> >> >> >> >> I don't want to rebuild the conversations db every few days. >> >> >> >> >> >> Any help is appreciated. >> >> >> >> >> >> Kind regards >> >> >> >> >> >> >> >> >> Michael Menge >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -------------------------------------------------------------------------------- >> >> >> M.Menge Tel.: (49) 7071/29-70316 >> >> >> Universität Tübingen Fax.: (49) 7071/29-5912 >> >> >> Zentrum für Datenverarbeitung mail: >> >> >> [email protected] >> >> >> Wächterstraße 76 >> >> >> 72074 Tübingen >> >> >> >> >> >> ---- >> >> >> Cyrus Home Page: http://www.cyrusimap.org/ >> >> >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> >> >> To Unsubscribe: >> >> >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >> >> > >> >> > -- >> >> > Bron Gondwana >> >> > [email protected] >> >> >> >> >> >> >> >> >> -------------------------------------------------------------------------------- >> >> M.Menge Tel.: (49) 7071/29-70316 >> >> Universität Tübingen Fax.: (49) 7071/29-5912 >> >> Zentrum für Datenverarbeitung mail: >> >> [email protected] >> >> Wächterstraße 76 >> >> 72074 Tübingen >> >> >> >> ---- >> >> Cyrus Home Page: http://www.cyrusimap.org/ >> >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> >> To Unsubscribe: >> >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >> >> >> >> *Attachments:* >> >> * imapd_be.conf >> >> * cyrus_be.conf >> >> * cyrus_re.conf >> >> * imapd_re.conf >> > >> > -- >> > Bron Gondwana >> > [email protected] >> >> >> >> -------------------------------------------------------------------------------- >> M.Menge Tel.: (49) 7071/29-70316 >> Universität Tübingen Fax.: (49) 7071/29-5912 >> Zentrum für Datenverarbeitung mail: >> [email protected] >> Wächterstraße 76 >> 72074 Tübingen >> >> ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > -- > Bron Gondwana, CEO, FastMail Pty Ltd > [email protected] -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912 Zentrum für Datenverarbeitung mail: [email protected] Wächterstraße 76 72074 Tübingen ---- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus