Backup compaction optimization in a block-level replication environment
Deborah Pickett <[email protected]>
| Newsgroups | gmane.mail.imap.cyrus |
|---|---|
| Message-ID | <[email protected]> |
(Sorry, that's a lot of big words. I'll try explaining what I want to do.) On my LAN I have a Cyrus IMAP server (3.0.11), and a dedicated Cyrus backup server (patched with Ellie's shared-mailbox and 64-bit fseek fixes). These are connected by a nice fat link so backups happen fast and often. A scheduled compaction occurs each morning thanks to an event in cyrus.conf. I now want to back up the backups to an off-site server over a much slower link. The off-site server doesn't speak the Cyrus sync protocol. What it does do well is block-level backups: if only a part of a file has changed, only that part needs to be transferred over the slow link. [I haven't decided whether my technology will be the rsync --checksum protocol, or Synology NAS XFS replication, or Microsoft Server VFS snapshots. They all do block-level backups well.] Since Cyrus backup files are append-only, they should behave well with block-level backups. But—correct me if I'm wrong—compaction is going to ruin my day because a reduction in the size of chunk (say) 5 moves the start offset of chunk 6 (and so on). Even if chunk 6 doesn't change it'll have to be retransmitted in its entirety. Right now I've set backup_compact_minsize and backup_compact_maxsize to zero but I'm not sure if even that is sufficient to prevent chunk offsets moving. Perhaps I need to disable the compaction event in cyrus.conf entirely. I really want compaction, though, or else my backups are going to get very, very big. Which leads me to my idea. What if compaction could be friendlier towards block-level backups, by deliberately avoiding changing chunk offsets in the backup file, even if that means gaps of unused bytes when (the aforementioned) chunk 5 shrinks? It won't always work out, for instance when a chunk grows in size. A compromise would need to be struck between keeping chunk offsets fixed and wasted fragmented space between chunks as they shrink. I haven't collected enough data to know if I am making the right assumptions about how chunk size evolves over time and how effective compaction is at removing cruft from a backup file. Has anyone thought about doing something like this with Cyrus backups? -- Deborah Pickett System Administrator Polyfoam Australia Pty Ltd ---- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus