Re: (1) dumping twoskip file and (2) getting folder UIDs
"Bron Gondwana via Info" <[email protected]> Tue, 30 Jun 2026 11:09:02 -0400
| Newsgroups | gmane.mail.imap.cyrus |
|---|---|
| Message-ID | <[email protected]> |
--4cb22db3ac1de1916ca15f79c8ef6d22701413e5 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable You can use either cyr_dbtool or cvt_cyrusdb to get the text version - cyr_= dbtool is much more powerful. HOWEVER - in either case, you're looking at mailbox uniqueids in the .seen = folder. So you're going to need to find those our. With new enough Cyrus,= they will be in the mailboxes.db file. With older cyrus, you're going to = have to read them from the cyrus.header files in each mailbox. 3.4.3 might= be late enough to have put the uniqueids into the mailboxes.db, I don't re= member! Of course - you can also use cyr_dbtool or cvt_cyrusdb to read the= contents of mailboxes.db. NOTE: work on copies of the files, not the original. Bron. On Tue, Jun 30, 2026, at 09:35, Jay Sekora wrote: > Hi! I have a user who accidentally marked all her mail as "read" and > needs read state of her messages restored from backups. At the same > time, she also accidentally deleted a bunch of messages, but also has > gotten lots of new mail since then. >=20 > So I think what I need to do is restore her old inbox as of just before > this happened to a new folder location (that part is easy), and then get > a copy of her $username.seen file from the same backup, extract the > seen/unseen data from the .seen file, and apply seen/unseen state to > the corresponding messages in the new (restored) folder location. >=20 > So the two pieces of this that I don't know how to do are (1) dumping > the twoskip file to a text format that I can get the seen/unseen message > UIDs out of, and (2) correlating the folder IDs in the twoskip file with > folder names I can select in `cyradm`. (If there's an easier way to do > this I'd love to hear it!) >=20 > So, (1) is there a script available somewhere to dump twoskip files to > a text format? And (2) how can I get a list of folder names and > corresponding folder IDs as found (apparently) in the twoskip file? > (2) seems like something I ought to know, but I don't. :-) >=20 > (Running `strings` on the twoskip file suggests the folders are stored > as IDs like "xhz89kuj0c4m8w0ey4tafenk1" rather than folder names like > "user.so-and-so". But presumably the output of `strings` might include > values that are deleted or superceded, and in any case it looks like > the actual message UIDs are in binary rather than textual format in > the twoskip file.) >=20 > Jay >=20 > PS -- This is on Ubuntu 22.04 with Cyrus 3.4.3 as packaged by Ubuntu. > But I can't imagine the package maintainers broke the twoskip file > format. :-) >=20 > PPS -- I found > https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb_twoskip.c > but my C is pretty rusty and I don't feel confident trying to convert that > to perl or python or something on my own. The other way I've thought > of to do this is to bring up an entire second IMAP server, restore the > user's inbox and .seen file to it as of a consistent date, and use > `imapsync` to synchronize seen/unseen state from the restore with the > production IMAP server =E2=80=94 assuming the folder ID is actually stored > *in* the folder directory itself. That sounds even worse than trying > to parse the twoskip file. -- Bron Gondwana, CEO, Fastmail Pty Ltd / Fastmail US LLC [email protected] ------------------------------------------ Cyrus: Info Permalink: https://cyrus.topicbox.com/groups/info/T643419685ae5113c-M57ae47= 210c17b347367077c5 Delivery options: https://cyrus.topicbox.com/groups/info/subscription --4cb22db3ac1de1916ca15f79c8ef6d22701413e5 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <!DOCTYPE html> <html><html><html><head><title></title></head><body><div style=3D"font-fami= ly:Arial;">You can use either cyr_dbtool or cvt_cyrusdb to get the text ver= sion - cyr_dbtool is much more powerful.</div><div style=3D"font-family:Ari= al;"><br /></div><div style=3D"font-family:Arial;">HOWEVER - in either case= , you're looking at mailbox uniqueids in the .seen folder. So you= 're going to need to find those our. With new enough Cyrus, they = will be in the mailboxes.db file. With older cyrus, you're going = to have to read them from the cyrus.header files in each mailbox. 3.4= .3 might be late enough to have put the uniqueids into the mailboxes.db, I = don't remember! Of course - you can also use cyr_dbtool or cvt_cy= rusdb to read the contents of mailboxes.db.</div><div style=3D"font-family:= Arial;"><br /></div><div style=3D"font-family:Arial;">NOTE: work on copies = of the files, not the original.</div><div style=3D"font-family:Arial;"><br = /></div><div style=3D"font-family:Arial;">Bron.</div><div style=3D"font-fam= ily:Arial;"><br /></div><div>On Tue, Jun 30, 2026, at 09:35, Jay Sekora wro= te:</div><blockquote id=3D"qt" style=3D"" type=3D"cite"><div style=3D"font-= family:Arial;">Hi! I have a user who accidentally marked all her mail= as "read" and</div><div style=3D"font-family:Arial;">needs read = state of her messages restored from backups. At the same</div><div st= yle=3D"font-family:Arial;">time, she also accidentally deleted a bunch of m= essages, but also has</div><div style=3D"font-family:Arial;">gotten lots of= new mail since then.</div><div style=3D"font-family:Arial;"><br /></div><d= iv style=3D"font-family:Arial;">So I think what I need to do is restore her= old inbox as of just before</div><div style=3D"font-family:Arial;">this ha= ppened to a new folder location (that part is easy), and then get</div><div= style=3D"font-family:Arial;">a copy of her $username.seen file from the sa= me backup, extract the </div><div style=3D"font-family:Arial;">seen/un= seen data from the .seen file, and apply seen/unseen state to</div><div sty= le=3D"font-family:Arial;">the corresponding messages in the new (restored) = folder location.</div><div style=3D"font-family:Arial;"><br /></div><div st= yle=3D"font-family:Arial;">So the two pieces of this that I don't know = how to do are (1) dumping</div><div style=3D"font-family:Arial;">the twoski= p file to a text format that I can get the seen/unseen message</div><div st= yle=3D"font-family:Arial;">UIDs out of, and (2) correlating the folder IDs = in the twoskip file with</div><div style=3D"font-family:Arial;">folder name= s I can select in `cyradm`. (If there's an easier way to do</div>= <div style=3D"font-family:Arial;">this I'd love to hear it!)</div><div = style=3D"font-family:Arial;"><br /></div><div style=3D"font-family:Arial;">= So, (1) is there a script available somewhere to dump twoskip files to</div= ><div style=3D"font-family:Arial;">a text format? And (2) how can I g= et a list of folder names and </div><div style=3D"font-family:Arial;">= corresponding folder IDs as found (apparently) in the twoskip file?</div><d= iv style=3D"font-family:Arial;">(2) seems like something I ought to know, b= ut I don't. :-)</div><div style=3D"font-family:Arial;"><br /></div><div= style=3D"font-family:Arial;">(Running `strings` on the twoskip file sugges= ts the folders are stored</div><div style=3D"font-family:Arial;">as IDs lik= e "xhz89kuj0c4m8w0ey4tafenk1" rather than folder names like</div>= <div style=3D"font-family:Arial;">"user.so-and-so". But pre= sumably the output of `strings` might include</div><div style=3D"font-famil= y:Arial;">values that are deleted or superceded, and in any case it looks l= ike </div><div style=3D"font-family:Arial;">the actual message UIDs ar= e in binary rather than textual format in</div><div style=3D"font-family:Ar= ial;">the twoskip file.)</div><div style=3D"font-family:Arial;"><br /></div= ><div style=3D"font-family:Arial;">Jay</div><div style=3D"font-family:Arial= ;"><br /></div><div style=3D"font-family:Arial;">PS -- This is on Ubuntu 22= .04 with Cyrus 3.4.3 as packaged by Ubuntu.</div><div style=3D"font-family:= Arial;">But I can't imagine the package maintainers broke the twoskip f= ile</div><div style=3D"font-family:Arial;">format. :-)</div><div style=3D"f= ont-family:Arial;"><br /></div><div style=3D"font-family:Arial;">PPS -- I f= ound</div><div style=3D"font-family:Arial;"><a href=3D"https://github.com/c= yrusimap/cyrus-imapd/blob/master/lib/cyrusdb_twoskip.c">https://github.com/= cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb_twoskip.c</a></div><div style= =3D"font-family:Arial;">but my C is pretty rusty and I don't feel confi= dent trying to convert that</div><div style=3D"font-family:Arial;">to perl = or python or something on my own. The other way I've thought</div= ><div style=3D"font-family:Arial;">of to do this is to bring up an entire s= econd IMAP server, restore the</div><div style=3D"font-family:Arial;">user&= #39;s inbox and .seen file to it as of a consistent date, and use </di= v><div style=3D"font-family:Arial;">`imapsync` to synchronize seen/unseen s= tate from the restore with the</div><div style=3D"font-family:Arial;">produ= ction IMAP server — assuming the folder ID is actually stored</div><d= iv style=3D"font-family:Arial;">*in* the folder directory itself. Tha= t sounds even worse than trying</div><div style=3D"font-family:Arial;">to p= arse the twoskip file.</div><div style=3D"font-family:Arial;"><br /></div><= div style=3D"font-family:Arial;">------------------------------------------= </div><div style=3D"font-family:Arial;">Cyrus: Info</div><div style=3D"font= -family:Arial;">Permalink: <a href=3D"https://cyrus.topicbox.com/group= s/info/T643419685ae5113c-Mf5c7c089518c1a2ec60f8bc2">https://cyrus.topicbox.= com/groups/info/T643419685ae5113c-Mf5c7c089518c1a2ec60f8bc2</a></div><div s= tyle=3D"font-family:Arial;">Delivery options: <a href=3D"https://cyrus= .topicbox.com/groups/info/subscription">https://cyrus.topicbox.com/groups/i= nfo/subscription</a></div><div style=3D"font-family:Arial;"><br /></div></b= lockquote><div style=3D"font-family:Arial;"><br /></div><div id=3D"sig56629= 417"><div class=3D"signature">--</div><div class=3D"signature"> Bron = Gondwana, CEO, Fastmail Pty Ltd / Fastmail US LLC</div><div class=3D"signat= ure"> [email protected]</div><div class=3D"signature"><br /></di= v></div><div style=3D"font-family:Arial;"><br /></div><div id=3D"topicbox-f= ooter" style=3D"margin:10px 0 0;border-top:1px solid #ddd;border-color:rgba= (0,0,0,.15);padding:7px 0;"> <strong><a href=3D"https://cyrus.topicbox.com/latest" style=3D"color:inheri= t;text-decoration:none">Cyrus</a></strong> / Info / see <a href=3D"https://cyrus.topicbox.com/groups/info">discussions</a> + <a href=3D"https://cyrus.topicbox.com/groups/info/members">participants</a> + <a href=3D"https://cyrus.topicbox.com/groups/info/subscription">delivery&nb= sp;options</a> <a href=3D"https://cyrus.topicbox.com/groups/info/T643419685ae5113c-M57ae47= 210c17b347367077c5" style=3D"float:right">Permalink</a> </div> </body></html></html></html>= --4cb22db3ac1de1916ca15f79c8ef6d22701413e5--