Re: (1) dumping twoskip file and (2) getting folder UIDs
"Bron Gondwana via Info" <[email protected]> Tue, 30 Jun 2026 15:06:19 -0400
| Newsgroups | gmane.mail.imap.cyrus |
|---|---|
| Message-ID | <[email protected]> |
--2647c3651a24f0e09b9862d31933ba765467568b Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Ahh, for her own mailboxes, the seen state isn't stored in the seen databas= e, it's stored in the system flags inside cyrus.index. That's a little mor= e painful to read unless you know the cyrus.index format. There is a Cyrus= ::IndexFile module in the current Cyrus upstream which can read index files= going all the way back to 2.2, so it would have no difficulty giving you t= he data from a snapshotted cyrus.index file from the time. Simplest however might just be to restore a complete snapshot of her accoun= t for that time and select via IMAP and read the seen state by just connect= ing via IMAP and doing a FETCH FLAGS or "SEARCH SEEN". Bron. On Tue, Jun 30, 2026, at 13:51, Jay Sekora wrote: > Hi and thank you! >=20 > On Tue, Jun 30, 2026 at 11:09:02AM -0400, Bron Gondwana via Info wrote: > > You can use either cyr_dbtool or cvt_cyrusdb to get the text version - > > cyr_dbtool is much more powerful. >=20 > This got me part of the way there; I was able to convert mailboxes.db > and the users .seen file to plain text. >=20 > Interestingly, mailboxes.db has uniqueids for *MANY* of the mailboxes, > but not for the specific one I'm looking for (which was the user's INBOX). > I was able to get the uniqueid for the relevant INBOX from the cyrus.head= er > file, but that uniqueid does not show up in the user's *.seen file (either > the original twoskip file or after converting to flat format). This was = both > on the user's .seen file for her current inbox, which I know she's read > recently, and also on a restored copy from before her accidental > deletion/marking-read. >=20 > For reference, here's the (only matching) line in /tmp/mailboxes.flat > converted from mailboxes.db: >=20 > user.REDACTED %(A %(REDACTED lrswipkxtecdan cyrus lrswipkxtecda) P facu= lty F 124565 M 1748834865) >=20 > (although my copypasting doesn't preserve tabs.) >=20 > And the same sort of thing shows up for my own INBOX: >=20 > user.jsekora %(A %(jsekora lrswipcda cyrus lrswipkxtecda) P staff M 17= 41542865) >=20 > although there *are* other mailboxes that have uniqueids listed in=20 > mailboxes.db (in a separate section at the front, which doesn't list > anything for my own INBOX or the user's in question): >=20 > [...] > $RUNQ$ngb736k4jbddz7wreeovkcyt$user.OTHERUSER=20=20=20 > [...] > user.OTHERUSER %(A %(OTHERUSER lrswipkxtecdan) I ngb736k4jbddz7wreeovkcy= t P staff V 1777900694 C 1 F 2 M 1782827859) >=20 > I'm guessing the distinction here is the age of the mailbox; many of our > mailboxes were migrated (via `imapsync`) from our previous IMAP server > about a year and a half ago; the few I've spot-checked that *do* have > uniqueids are ones that were created more recently. >=20 > In any case the weirder problem seems to be that the flat file I get > from this users' twoskip .seen file doesn't seem to have any reference > either to the name (user.$USERNAME) or uniqueid (f13e2zfxso7s3trbkkz4ddqv) > of her old inbox. >=20 > I'm really puzzled by this! >=20 > She uses Thunderbird to read her mail. I'll ask her how she marks things > as seen and whether she has any Thunderbird extensions installed. >=20 > I'm looking for her seen/unseen state in /var/lib/cyrus/user/U/USERNAME.s= een, > and for her INBOX's uniqueid in /var/lib/cyrus/mailboxes.db (where it doe= sn't > show up although other users' do) as well as in cyrus.header in the mailb= ox > store (where it does show up). Is there someplace else I'm missing? >=20 > Jay >=20 > > HOWEVER - in either case, you're looking at mailbox uniqueids in the > > .seen folder. So you're going to need to find those our. With new > > enough Cyrus, they will be in the mailboxes.db file. With older cyrus, > > you're going to have to read them from the cyrus.header files in each > > mailbox. 3.4.3 might be late enough to have put the uniqueids into the > > mailboxes.db, I don't remember! Of course - you can also use cyr_dbtool > > or cvt_cyrusdb to read the contents of mailboxes.db. > > NOTE: work on copies of the files, not the original. >=20 > Thanks! >=20 > Jay >=20 -- Bron Gondwana, CEO, Fastmail Pty Ltd / Fastmail US LLC [email protected] ------------------------------------------ Cyrus: Info Permalink: https://cyrus.topicbox.com/groups/info/T643419685ae5113c-M68c204= 2ad6c35c092e556893 Delivery options: https://cyrus.topicbox.com/groups/info/subscription --2647c3651a24f0e09b9862d31933ba765467568b Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <!DOCTYPE html> <html><html><html><head><title></title></head><body><div style=3D"font-fami= ly:Arial;">Ahh, for her own mailboxes, the seen state isn't stored in t= he seen database, it's stored in the system flags inside cyrus.index.&n= bsp; That's a little more painful to read unless you know the cyrus.ind= ex format. There is a Cyrus::IndexFile module in the current Cyrus up= stream which can read index files going all the way back to 2.2, so it woul= d have no difficulty giving you the data from a snapshotted cyrus.index fil= e from the time.</div><div style=3D"font-family:Arial;"><br /></div><div st= yle=3D"font-family:Arial;">Simplest however might just be to restore a comp= lete snapshot of her account for that time and select via IMAP and read the= seen state by just connecting via IMAP and doing a FETCH FLAGS or "SE= ARCH SEEN".</div><div style=3D"font-family:Arial;"><br /></div><div st= yle=3D"font-family:Arial;">Bron.<br /></div><div style=3D"font-family:Arial= ;"><br /></div><div style=3D"font-family:Arial;"><br /></div><div>On Tue, J= un 30, 2026, at 13:51, Jay Sekora wrote:</div><blockquote id=3D"qt" style= =3D"" type=3D"cite"><div style=3D"font-family:Arial;">Hi and thank you!</di= v><div style=3D"font-family:Arial;"><br /></div><div style=3D"font-family:A= rial;">On Tue, Jun 30, 2026 at 11:09:02AM -0400, Bron Gondwana via Info wro= te:</div><div style=3D"font-family:Arial;">> You can use either cyr_dbto= ol or cvt_cyrusdb to get the text version -</div><div style=3D"font-family:= Arial;">> cyr_dbtool is much more powerful.</div><div style=3D"font-fami= ly:Arial;"><br /></div><div style=3D"font-family:Arial;">This got me part o= f the way there; I was able to convert mailboxes.db</div><div style=3D"font= -family:Arial;">and the users .seen file to plain text.</div><div style=3D"= font-family:Arial;"><br /></div><div style=3D"font-family:Arial;">Interesti= ngly, mailboxes.db has uniqueids for *MANY* of the mailboxes,</div><div sty= le=3D"font-family:Arial;">but not for the specific one I'm looking for = (which was the user's INBOX).</div><div style=3D"font-family:Arial;">I = was able to get the uniqueid for the relevant INBOX from the <a href= =3D"http://cyrus.header">cyrus.header</a></div><div style=3D"font-family:Ar= ial;">file, but that uniqueid does not show up in the user's *.seen fil= e (either</div><div style=3D"font-family:Arial;">the original twoskip file = or after converting to flat format). This was both</div><div style=3D= "font-family:Arial;">on the user's .seen file for her current inbox, wh= ich I know she's read</div><div style=3D"font-family:Arial;">recently, = and also on a restored copy from before her accidental</div><div style=3D"f= ont-family:Arial;">deletion/marking-read.</div><div style=3D"font-family:Ar= ial;"><br /></div><div style=3D"font-family:Arial;">For reference, here'= ;s the (only matching) line in /tmp/mailboxes.flat</div><div style=3D"font-= family:Arial;">converted from mailboxes.db:</div><div style=3D"font-family:= Arial;"><br /></div><div style=3D"font-family:Arial;"><a href=3D"http://use= r.REDACTED">user.REDACTED</a> %(A %(REDACTED lrswipkxtecdan cyr= us lrswipkxtecda) P faculty F 124565 M 1748834865)</div><div style=3D"font-= family:Arial;"><br /></div><div style=3D"font-family:Arial;">(although my c= opypasting doesn't preserve tabs.)</div><div style=3D"font-family:Arial= ;"><br /></div><div style=3D"font-family:Arial;">And the same sort of thing= shows up for my own INBOX:</div><div style=3D"font-family:Arial;"><br /></= div><div style=3D"font-family:Arial;"><a href=3D"http://user.jsekora">user.= jsekora</a> %(A %(jsekora lrswipcda cyrus lrswipkxtecda) = P staff M 1741542865)</div><div style=3D"font-family:Arial;"><br /></div><d= iv style=3D"font-family:Arial;">although there *are* other mailboxes that h= ave uniqueids listed in </div><div style=3D"font-family:Arial;">mailbo= xes.db (in a separate section at the front, which doesn't list</div><di= v style=3D"font-family:Arial;">anything for my own INBOX or the user's = in question):</div><div style=3D"font-family:Arial;"><br /></div><div style= =3D"font-family:Arial;">[...]</div><div style=3D"font-family:Arial;">$RUNQ$= ngb736k4jbddz7wreeovkcyt$user.OTHERUSER </div><div style= =3D"font-family:Arial;">[...]</div><div style=3D"font-family:Arial;"><a hre= f=3D"http://user.OTHERUSER">user.OTHERUSER</a> %(A %(OTHERUSER lrswip= kxtecdan) I ngb736k4jbddz7wreeovkcyt P staff V 1777900694 C 1 F 2 M 1782827= 859)</div><div style=3D"font-family:Arial;"><br /></div><div style=3D"font-= family:Arial;">I'm guessing the distinction here is the age of the mail= box; many of our</div><div style=3D"font-family:Arial;">mailboxes were migr= ated (via `imapsync`) from our previous IMAP server</div><div style=3D"font= -family:Arial;">about a year and a half ago; the few I've spot-checked = that *do* have</div><div style=3D"font-family:Arial;">uniqueids are ones th= at were created more recently.</div><div style=3D"font-family:Arial;"><br /= ></div><div style=3D"font-family:Arial;">In any case the weirder problem se= ems to be that the flat file I get</div><div style=3D"font-family:Arial;">f= rom this users' twoskip .seen file doesn't seem to have any referen= ce</div><div style=3D"font-family:Arial;">either to the name (user.$USERNAM= E) or uniqueid (f13e2zfxso7s3trbkkz4ddqv)</div><div style=3D"font-family:Ar= ial;">of her old inbox.</div><div style=3D"font-family:Arial;"><br /></div>= <div style=3D"font-family:Arial;">I'm really puzzled by this!</div><div= style=3D"font-family:Arial;"><br /></div><div style=3D"font-family:Arial;"= >She uses Thunderbird to read her mail. I'll ask her how she mark= s things</div><div style=3D"font-family:Arial;">as seen and whether she has= any Thunderbird extensions installed.</div><div style=3D"font-family:Arial= ;"><br /></div><div style=3D"font-family:Arial;">I'm looking for her se= en/unseen state in /var/lib/cyrus/user/U/USERNAME.seen,</div><div style=3D"= font-family:Arial;">and for her INBOX's uniqueid in /var/lib/cyrus/mail= boxes.db (where it doesn't</div><div style=3D"font-family:Arial;">show = up although other users' do) as well as in <a href=3D"http://cyrus= .header">cyrus.header</a> in the mailbox</div><div style=3D"font-family:Ari= al;">store (where it does show up). Is there someplace else I'm m= issing?</div><div style=3D"font-family:Arial;"><br /></div><div style=3D"fo= nt-family:Arial;">Jay</div><div style=3D"font-family:Arial;"><br /></div><d= iv style=3D"font-family:Arial;">> HOWEVER - in either case, you're l= ooking at mailbox uniqueids in the</div><div style=3D"font-family:Arial;">&= gt; .seen folder. So you're going to need to find those our. = ; With new</div><div style=3D"font-family:Arial;">> enough Cyrus, they w= ill be in the mailboxes.db file. With older cyrus,</div><div style=3D= "font-family:Arial;">> you're going to have to read them from the&nb= sp;<a href=3D"http://cyrus.header">cyrus.header</a> files in each</div><div= style=3D"font-family:Arial;">> mailbox. 3.4.3 might be late enoug= h to have put the uniqueids into the</div><div style=3D"font-family:Arial;"= >> mailboxes.db, I don't remember! Of course - you can also us= e cyr_dbtool</div><div style=3D"font-family:Arial;">> or cvt_cyrusdb to = read the contents of mailboxes.db.</div><div style=3D"font-family:Arial;">&= gt; NOTE: work on copies of the files, not the original.</div><div style=3D= "font-family:Arial;"><br /></div><div style=3D"font-family:Arial;">Thanks!<= /div><div style=3D"font-family:Arial;"><br /></div><div style=3D"font-famil= y:Arial;">Jay</div><div style=3D"font-family:Arial;"><br /></div><div style= =3D"font-family:Arial;"><br /></div><div style=3D"font-family:Arial;">-----= -------------------------------------</div><div style=3D"font-family:Arial;= ">Cyrus: Info</div><div style=3D"font-family:Arial;">Permalink: <a hre= f=3D"https://cyrus.topicbox.com/groups/info/T643419685ae5113c-Me593cde58637= d031ed5ba72d">https://cyrus.topicbox.com/groups/info/T643419685ae5113c-Me59= 3cde58637d031ed5ba72d</a></div><div style=3D"font-family:Arial;">Delivery o= ptions: <a href=3D"https://cyrus.topicbox.com/groups/info/subscription= ">https://cyrus.topicbox.com/groups/info/subscription</a></div><div style= =3D"font-family:Arial;"><br /></div></blockquote><div style=3D"font-family:= Arial;"><br /></div><div id=3D"sig56629417"><div class=3D"signature">--</di= v><div class=3D"signature"> Bron Gondwana, CEO, Fastmail Pty Ltd / Fa= stmail US LLC</div><div class=3D"signature"> [email protected]</= div><div class=3D"signature"><br /></div></div><div style=3D"font-family:Ar= ial;"><br /></div><div id=3D"topicbox-footer" style=3D"margin:10px 0 0;bord= er-top:1px solid #ddd;border-color:rgba(0,0,0,.15);padding:7px 0;"> <strong><a href=3D"https://cyrus.topicbox.com/latest" style=3D"color:inheri= t;text-decoration:none">Cyrus</a></strong> / Info / see <a href=3D"https://cyrus.topicbox.com/groups/info">discussions</a> + <a href=3D"https://cyrus.topicbox.com/groups/info/members">participants</a> + <a href=3D"https://cyrus.topicbox.com/groups/info/subscription">delivery&nb= sp;options</a> <a href=3D"https://cyrus.topicbox.com/groups/info/T643419685ae5113c-M68c204= 2ad6c35c092e556893" style=3D"float:right">Permalink</a> </div> </body></html></html></html>= --2647c3651a24f0e09b9862d31933ba765467568b--