Re: HD synch?

Robert Baer wrote:
>   Must be a law, data grows.
> 
>   I never image a drive for backup; I clone it.
>   AFAIK the process takes the same amount of time.
>   But usage differs greatly; a clone is immediately usable, like the 
> original.
>   An image must be "restored" to yet another place/media, and _that_ 
> restoration then is usable.
> 
>   Now as data grows, time needed for backup grows, now eating 
> TERRORbytes of space and eating hourzz of time.
> 
> 
>   I would love an edge by syncing the copy to the original on a regular 
> basis instead of cloning on a regular basis.
> 
>   A program in the background could be ideal; copy incremental data as 
> and when needed, or chunk away on a regular basis.
> 
>   Suggestions, please?
> 
> 
>     Thanks, R. Baer

When you Image a drive (convert a whole drive to a .container file),
there is

    Full
    Differential
    Incremental

It is the incremental option,

    Full Incr Incr Incr...    Full Incr Incr Incr...

that gives the small addition to the Full each day.

During restoration to the third day, you'd use three files

    Full Incr Incr

to get to the "state" of the second Incr. Because an Incr
is never rewritten, your file "notes.txt" could be stored
in all three files, and the metadata knows that the
third one is the one to keep during a restore.

On a backup product that has Incremental Forever, there
don't have to be quite as many file groups. The Full
and a set of Incr, can be squashed together (synthetically)
to make a new Full, ready to have more Incr collected.

*******

The other type is Differential. Backups are measured
against the Full. Each Differential tends to stand on its
own two legs.

    Full Diff1 Diff2 Diff3...  Full Diff1 Diff2 Diff3...

This gives a more bulky operation each day, but still
well short of what a Full takes.

During restoration to the third day, you'd use two files

    Full       Diff2

and that would return you to the state of Diff2.

*******

On Windows, both of those sequences can be done "hot"
during the backup. VSS (Volume Shadow Service) allows
tracking of the work to be done. This is unlike the
old "Ghost" program, where a reboot and an offline
backup was done. Certain files, like pagefile.sys
and Windows.edb are not captured, but this does not
affect system operation. Windows.edb cannot be captured,
because the subsystem that has it open (Search Indexer)
refuses to quiesce it on demand.

Incremental backup costs money. Differential backup
can be acquired for free.

On a Windows 10 system, an Incremental might be 1GB per
day, as there is some maintenance activity that creates
file differences. Your "notes.txt" file is a small
contributor to the size of the Incremental file collected.

*******

An example of cloning, where only new files are written,
would be something like Linux RSYNC. On Windows, Robocopy
or Richcopy might do it. The problem with some of these
solutions, is whether they handle the metadata properly
or not. The backup softwares, generally they seem to do a
good job. That's because the metadata on a volume, is also
a file, and so the metadata can be backed up as an incremental
file change.

But for programs which might choose to record files
individually, the onus would be on the developer to make
sure that during restoration or recovery, all aspects of
the file (ownership, permissions, ACLs) are restored properly.
And for crappy situations like Windows 10 with reparse points,
this requires the developer understand everything that
Microsoft has done. At one time, Retrospect did file by file
recording to tape, as an example, then later, it could
also use a disk drive for the copy it kept. The first time
I used Retrospect, it took me a solid two days (16 hours)
of work, scripting everything, and even then, a couple partitions
never seemed to back up properly and would throw some error.

That doesn't mean every product works like that.

For example, the version of Acronis a few years back,
I ran a trial of it, and the workflow was particularly
easy to get it set up. The current version seems to be
getting a bit bloated and silly (they're trying to
trick customers into a subscription model of sorts).

*******

All of the ideas above, have mounters, and allow recovery
of single files. You don't have to restore the capture
entirely, to get at a single file. Most of the imaging
.container formats have the file directory at the end
of the recording, during "mount" that is consulted for
a file map, and when you navigate through the virtual
partition, all the files show up in File Explorer for you.
Even Acronis would do that. So would Macrium.

(Full and Differential are Free, Incremental requires a license)

https://www.macrium.com/reflectfree

    Backup at Home

    Reflect 7 Free

    Download Free  <===

When you install that, it will prompt you to create
rescue media. The rescue media boots and it is necessary
for "bare metal restore", when the drive C: used to be
on is ruined or dead. You make the CD so you have something
to boot from. It's a good idea to make that right away,
as the files to make the media will not be on Microsoft
servers forever. Some of the early versions of the
WinPE files, like for WinPE 3, are no longer available.
The process of making the rescue media is automated,
but there are some options the user can select.

If you want to, you can also boot the CD and do the backup
if you want. The backup runs from C:\Program Files for
sure, but the emergency boot media is also fully functional.

By selecting an option on the left hand side, you can
have all disk drives on the computer recorded into
a single container file. I don't necessarily recommend
this, from a file reliability point of view however.
Macrium has a Verify function, and if you keep old
backups around, sometimes running a Verify indicates
the .container file has become corrupted. In one case, this
was a function of the computer having bad RAM, and
that's how I detected there was a RAM problem, when
the computer could no longer make "good" backups.

*******

Some versions of Windows have File History, which occasionally
dumps to an external disk for capture. But this is an
acquired taste, navigating the thing and tricking it into
coughing up the file you need. This would mainly be for your
user profile area, and not for capturing the entire C: partition.

*******

For more examples of software, try here. Some of the
software featured in the reviews here, these were the
"good" versions, and later versions available today
aren't quite as nice. For example, the two links underneath
item #2, are probably better versions than the current ones
for download from the supplier site.

https://www.raymond.cc/blog/10-commercial-disk-imaging-software-features-and-backuprestore-speed-comparison/

Page 2 has a table of test data, for the programs.
Easeus had the fastest incremental. Aomei had the fastest Full.
You can see for the slowest programs, they're sometimes
five times slower than the fast ones.

The programs have several jobs to do:

    Lightweight compression (like LZ4 maybe) - hard on CPU, not multithreaded.
    Compute checksum over image file - MD5 maybe, limits peak transfer rate.
    (Copy the data clusters sequentially...)

For things like computing a checksum, some processors are
faster at this than others. Then your backup runs faster.

The compressors used, do not save a lot of space. They are
not there to make the absolute tiniest file, instead they
are there to take compressible material (your "notes.txt")
and make that smaller. Only the files on the disk are
backed up, and the white space is not recorded. This means
if, after a restore, you want to run Recuva UnDelete, that's
not going to work on a freshly restored volume. No program
is supposed to "record deleted material", that's logically
a silly thing to do. If you want deleted material recorded
too, check the Preferences for "Forensic Copy", and select
that. Note that Forensic Copy is not tested by the developer
all that well, and I caught mine not making Forensic copies
when I told it to. Forensic just records everything and
is a "giant space waster" :-) But if you insist that
Recuva work for you, that's one way to do it.

*******

Windows File History, probably comes closest to your
carefree requirement. But the various OSes do it different
ways (Win10 one might be better than the Win7 one), and
thus this is an "uneven" solution and depends on what
you're running, as to whether this option is worth
entertaining. The good version, it prompts you to
"connect the backup drive now, so I can dump the
collection to your backup disk". The backup drive
does not have to be connected continuously. Windows
is probably using the VSS shadow capability, to record
file differences, and when it gets enough of them,
it'll request an opportunity to dump what has
been collected.

Robocopy can do mirroring, such that the state of
the clone drive matches the source drive. But it
does not do a good job on C:\Windows and is not
a replacement for a Full. It would do a good job
on C:\users\username\Downloads for example. And
you would want to be exited from Excel and Word,
when doing a run in that case. There are lots of
similar things, that will make a mess if you don't
do them properly. This is one reason I harp on the
backup programs above, first, because the automation
tends to handle the rough edges better. If you
craft a crappy solution like our "xcopy brethren"
might, the mess you make is yours to clean up :-)
While it is possible to manually make VSS snapshots,
to "freeze" file content and make the syncing safe,
it's above my pay scale to tell you how to do that.
And you're unlikely to get every detail and nuance
of the permissions right, if hand crafting a solution
using bailing wire and binder twine.

The above just scratches the surface. I'm not an
expert in this stuff, just a user like you. It takes
a lot of work, to prove the method you select, actually
works, and I do recommend you test it. It helps to
have two drives you can use for scratch purposes, to
test safely. Not a lot of people have drives just
sitting around for this sort of work.

    Paul