Re: backup manifests
Suraj Kharage <[email protected]>
| Newsgroups | gmane.comp.db.postgresql.devel.general |
|---|---|
| Message-ID | <CAF1DzPUAuY5HqwzBoyFssuN9eRYmTkAKYLPEika1bOTvV2W1YA@mail.gmail.com> |
Thank you for review comments. On Mon, Dec 30, 2019 at 11:53 PM Robert Haas <[email protected]> wrote: > On Tue, Dec 24, 2019 at 5:42 AM Suraj Kharage > <[email protected]> wrote: > > To examine the first word of each line, I am using below check: > > if (strncmp(line, "File", 4) == 0) > > { > > .. > > } > > else if (strncmp(line, "Manifest-Checksum", 17) == 0) > > { > > .. > > } > > else > > error > > > > strncmp might be not right here, but we can not put '\0' in between the > line (to find out first word) > > before we recognize the line type. > > All the lines expect line last one (where we have manifest checksum) are > feed to the checksum machinary to calculate manifest checksum. > > so update_checksum() should be called after recognizing the type, i.e: > if it is a File type record. Do you see any issues with this? > > I see the problem, but I don't think your solution is right, because > the first test would pass if the line said FiletMignon rather than > just File, which we certainly don't want. You've got to write the test > so that you're checking against the whole first word, not just some > prefix of it. There are several possible ways to accomplish that, but > this isn't one of them. > Yeah. Fixed in the attached patch. > > >> + pg_log_error("invalid record found in \"%s\"", manifest_path); > >> > >> Error message needs work. > > Looks better now, but you have a messages that say "invalid checksums > type \"%s\" found in \"%s\"". This is wrong because checksums would > need to be singular in this context (checksum). Also, I think it could > be better phrased as "manifest file \"%s\" specifies unknown checksum > algorithm \"%s\" at line %d". > Corrected. > > >> Your function names should be consistent with the surrounding style, > >> and with each other, as far as possible. Three different conventions > >> within the same patch and source file seems over the top. > > This appears to be fixed. > > >> Also keep in mind that you're not writing code in a vacuum. There's a > >> whole file of code here, and around that, a whole project. > >> scan_data_directory() is a good example of a function whose name is > >> clearly too generic. It's not a general-purpose function for scanning > >> the data directory; it's specifically a support function for verifying > >> a backup. Yet, the name gives no hint of this. > > But this appears not to be fixed. > I have changed this function name to "VerifyDir" likewise, we have sendDir and sendFile in basebackup.c > > >> if (strcmp(newpathsuffix, "/pg_wal") == 0 || strcmp(newpathsuffix, > >> "/backup_manifest") == 0) > >> continue; > > > > Thanks for the suggestion. Corrected as per the above inputs. > > You need a comment here, like "Ignore the possible presence of a > backup_manifest file and/or a pg_wal directory in the backup being > verified." and then maybe another sentence explaining why that's the > right thing to do. > Corrected. > > + * The forth parameter to VerifyFile() will pass the relative > path > + * of file to match exactly with the filename present in > manifest. > > I don't know what this comment is trying to tell me, which might be > something you want to try to fix. However, I'm pretty sure it's > supposed to say "fourth" not "forth". > I have changed the fourth parameter of VerifyFile(), so my comment over there is no more valid. > > >> and the result would be that everything inside that long if-block is > >> now at the top level of the function and indented one level less. And > >> I think if you look at this function you'll see a way that you can > >> save a *second* level of indentation for much of that code. Please > >> check the rest of the patch for similar cases, too. > > > > Make sense. corrected. > > I don't agree. A large chunk of VerifyFile() is still subject to a > quite unnecessary level of indentation. > Yeah, corrected. > > > I have added a check for EOF, but not sure whether that woule be right > here. > > Do we need to check the length of buffer as well? > > That's really, really not right. EOF is not a character that can > appear in the buffer. It's chosen on purpose to be a value that never > matches any actual character when both the character and the EOF value > are regarded as values of type 'int'. That guarantee doesn't apply > here though because you're dealing with values of type 'char'. So what > this code is doing is searching for an impossible value using > incorrect logic, which has very little to do with the actual need > here, which is to avoid running off the end of the buffer. To see what > the problem is, try creating a file with no terminating newline, like > this: > > echo -n this file has no terminating newline >> some-file > > I doubt it will be very hard to make this patch crash horribly. Even > if you can't, it seems pretty clear that the logic isn't right. > > I don't really know what the \0 tests in NextLine() and NextWord() > think they're doing either. If there's a \0 in the buffer before you > add one, it was in the original input data, and pretending like that > marks a word or line boundary seems like a fairly arbitrary choice. > > What I suggest is: > > (1) Allocate one byte more than the file size for the buffer that's > going to hold the file, so that if you write a \0 just after the last > byte of the file, you don't overrun the allocated buffer. > > (2) Compute char *endptr = buf + len. > > (3) Pass endptr to NextLine and NextWord and write the loop condition > something like while (*buf != '\n' && buf < endptr). > Thanks for the suggestion. Corrected as per above suggestion. > > Other notes: > > - The error handling in ReadFileIntoBuffer() does not seem to consider > the case of a short read. If you look through the source tree, you can > find examples of how we normally handle that. > yeah, corrected. > > - Putting string_hash_sdbm() into encode.c seems like a surprising > choice. What does this have to do with encoding anything? And why is > it going into src/common at all if it's only intended for frontend > use? > I thought this function can be used in backend as well, i.e: likewise we are using in simplehash, so kept that in src/common. After your comment, I have moved this to pg_basebackup.c. I think this can be kept in common place but not in "srs/common/encode.c" thoughts? > > - It seems like whether or not any problems were found while verifying > the manifest ought to affect the exit status of pg_basebackup. I'm not > exactly sure what exit codes ought to be used, but you could look for > similar precedents. Document this, too. > I might be not getting this completely correct, but as per my observation, if any error occurs, pg_basebackup terminated with exit(1). Whereas in normal case (without an error), main function returns 0. The "help" and "version" option terminate normally with exit(0). So in our case, exit(0) would be appropriate. Please correct me if I misunderstood anything. > > - As much as possible let's have errors in the manifest file report > the line number, and let's also try to make them more specific, e.g. > instead of "invalid manifest record found in \"%s\"", perhaps > "manifest file \"%s\" contains invalid keyword \"%s\" at line %d". > yeah, added line number at possible places. I have also fixed few comments given by Jeevan Chalke offlist. Please find attached v7 patches and let me know your comments. -- -- Thanks & Regards, Suraj kharage, EnterpriseDB Corporation, The Postgres Database Company.
v7-0001-Backup-manifest-with-file-names-sizes-timestamps-.patch
(application/octet-stream, 39.9 KB)
From 9776c21bd201d7549e1ec53a62065c342a399e01 Mon Sep 17 00:00:00 2001 From: Suraj Kharage <[email protected]> Date: Thu, 12 Dec 2019 16:13:20 +0530 Subject: [PATCH v7 1/3] Backup manifest with file names, sizes, timestamps, optional checksums. Original patch from Robert Haas to create the backup manifest file. Fix review comments, making checksum optional and provide user to choose the checksum algorithm by Rushabh Lathia. Reviewed by Jeevan Chalke and Rushabh Lathia. Further refactored by me. --- doc/src/sgml/protocol.sgml | 24 ++- doc/src/sgml/ref/pg_basebackup.sgml | 12 ++ src/backend/access/transam/xlog.c | 3 +- src/backend/replication/basebackup.c | 287 ++++++++++++++++++++++++++++++--- src/backend/replication/repl_gram.y | 6 + src/backend/replication/repl_scanner.l | 1 + src/backend/utils/adt/encode.c | 17 +- src/backend/utils/adt/varlena.c | 1 + src/bin/pg_basebackup/pg_basebackup.c | 130 ++++++++++++++- src/common/Makefile | 2 + src/common/checksum_utils.c | 109 +++++++++++++ src/common/encode.c | 38 +++++ src/include/common/checksum_utils.h | 46 ++++++ src/include/common/encode.h | 20 +++ src/include/replication/basebackup.h | 15 +- src/include/utils/builtins.h | 1 - src/tools/pgindent/typedefs.list | 3 + 17 files changed, 665 insertions(+), 50 deletions(-) create mode 100644 src/common/checksum_utils.c create mode 100644 src/common/encode.c create mode 100644 src/include/common/checksum_utils.h create mode 100644 src/include/common/encode.h diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index 8027521..31f108b 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -2466,15 +2466,19 @@ The commands accepted in replication mode are: </varlistentry> <varlistentry> - <term><literal>BASE_BACKUP</literal> [ <literal>LABEL</literal> <replaceable>'label'</replaceable> ] [ <literal>PROGRESS</literal> ] [ <literal>FAST</literal> ] [ <literal>WAL</literal> ] [ <literal>NOWAIT</literal> ] [ <literal>MAX_RATE</literal> <replaceable>rate</replaceable> ] [ <literal>TABLESPACE_MAP</literal> ] [ <literal>NOVERIFY_CHECKSUMS</literal> ] + <term><literal>BASE_BACKUP</literal> [ <literal>LABEL</literal> <replaceable>'label'</replaceable> ] [ <literal>PROGRESS</literal> ] [ <literal>FAST</literal> ] [ <literal>WAL</literal> ] [ <literal>NOWAIT</literal> ] [ <literal>MAX_RATE</literal> <replaceable>rate</replaceable> ] [ <literal>TABLESPACE_MAP</literal> ] [ <literal>NOVERIFY_CHECKSUMS</literal> ] [ <literal>MANIFEST_CHECKSUMS</literal> <replaceable>'algorithm'</replaceable>] <indexterm><primary>BASE_BACKUP</primary></indexterm> </term> <listitem> <para> Instructs the server to start streaming a base backup. The system will automatically be put in backup mode before the backup - is started, and taken out of it when the backup is complete. The - following options are accepted: + is started, and taken out of it when the backup is complete. This also + store a manifest as part of each backup under the backup folder in a + file named backup_manifest. This file contains the list of files, and + the lengths of those files, file modified time and optional checksum + for each file. It also has checksum for manifest file which is always + geneared by SHA256 algorithm. The following options are accepted: <variablelist> <varlistentry> <term><literal>LABEL</literal> <replaceable>'label'</replaceable></term> @@ -2576,6 +2580,20 @@ The commands accepted in replication mode are: </para> </listitem> </varlistentry> + + <varlistentry> + <term><literal>MANIFEST_CHECKSUMS</literal></term> + <listitem> + <para> + By default, checksum for each backup file in backup manifest file is + off. Specifying <literal>MANIFEST_CHECKSUMS</literal> when a checksum + algorithm name, enables the checksum for each backup file in backup + manifest file as well a for manifest file itself. Currently it supports + SHA256 and CRC32C as checksum algorithm. + </para> + </listitem> + </varlistentry> + </variablelist> </para> <para> diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml index fc9e222..af7c731 100644 --- a/doc/src/sgml/ref/pg_basebackup.sgml +++ b/doc/src/sgml/ref/pg_basebackup.sgml @@ -536,6 +536,18 @@ PostgreSQL documentation </para> </listitem> </varlistentry> + + <varlistentry> + <term><option>--manifest-checksum <replaceable class="parameter">algorithm</replaceable></option></term> + <listitem> + <para> + Enables a checksum for the each backup file in manifest file as well as + for manifest file itself and will choose the given algorithm to generate + the checksum value. + </para> + </listitem> + </varlistentry> + </variablelist> </para> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 7f4f784..265bdf9 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -10499,7 +10499,8 @@ do_pg_start_backup(const char *backupidstr, bool fast, TimeLineID *starttli_p, ti->oid = pstrdup(de->d_name); ti->path = pstrdup(buflinkpath.data); ti->rpath = relpath ? pstrdup(relpath) : NULL; - ti->size = infotbssize ? sendTablespace(fullpath, true) : -1; + ti->size = infotbssize ? + sendTablespace(fullpath, ti->oid, true, NULL) : -1; if (tablespaces) *tablespaces = lappend(*tablespaces, ti); diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c index 1423e6c..b1fac55 100644 --- a/src/backend/replication/basebackup.c +++ b/src/backend/replication/basebackup.c @@ -18,6 +18,7 @@ #include "access/xlog_internal.h" /* for pg_start/stop_backup */ #include "catalog/pg_type.h" +#include "common/encode.h" #include "common/file_perm.h" #include "lib/stringinfo.h" #include "libpq/libpq.h" @@ -51,20 +52,30 @@ typedef struct bool includewal; uint32 maxrate; bool sendtblspcmapfile; + ChecksumAlgorithm checksumAlgo; } basebackup_options; static int64 sendDir(const char *path, int basepathlen, bool sizeonly, - List *tablespaces, bool sendtblspclinks); + List *tablespaces, bool sendtblspclinks, + manifestinfo *manifestInfo, const char *tsoid); static bool sendFile(const char *readfilename, const char *tarfilename, - struct stat *statbuf, bool missing_ok, Oid dboid); -static void sendFileWithContent(const char *filename, const char *content); + struct stat *statbuf, bool missing_ok, Oid dboid, + manifestinfo *manifestInfo, const char *tsoid); +static void sendFileWithContent(const char *filename, const char *content, + manifestinfo *manifestInfo); static int64 _tarWriteHeader(const char *filename, const char *linktarget, struct stat *statbuf, bool sizeonly); static int64 _tarWriteDir(const char *pathbuf, int basepathlen, struct stat *statbuf, bool sizeonly); static void send_int8_string(StringInfoData *buf, int64 intval); static void SendBackupHeader(List *tablespaces); +static void InitializeManifest(manifestinfo *manifestInfo, + ChecksumAlgorithm checksumAlgo); +static void AddFileToManifest(manifestinfo *manifestInfo, const char *tsoid, + const char *filename, size_t size, time_t mtime); +static void SendBackupManifest(manifestinfo *manifestInfo); +static char *escape_field_for_manifest(const char *s); static void perform_base_backup(basebackup_options *opt); static void parse_basebackup_options(List *options, basebackup_options *opt); static void SendXlogRecPtrResult(XLogRecPtr ptr, TimeLineID tli); @@ -231,6 +242,7 @@ perform_base_backup(basebackup_options *opt) StringInfo tblspc_map_file = NULL; int datadirpathlen; List *tablespaces = NIL; + manifestinfo manifestInfo; datadirpathlen = strlen(DataDir); @@ -238,6 +250,7 @@ perform_base_backup(basebackup_options *opt) labelfile = makeStringInfo(); tblspc_map_file = makeStringInfo(); + InitializeManifest(&manifestInfo, opt->checksumAlgo); total_checksum_failures = 0; @@ -274,7 +287,10 @@ perform_base_backup(basebackup_options *opt) /* Add a node for the base directory at the end */ ti = palloc0(sizeof(tablespaceinfo)); - ti->size = opt->progress ? sendDir(".", 1, true, tablespaces, true) : -1; + if (opt->progress) + ti->size = sendDir(".", 1, true, tablespaces, true, NULL, NULL); + else + ti->size = -1; tablespaces = lappend(tablespaces, ti); /* Send tablespace header */ @@ -321,7 +337,8 @@ perform_base_backup(basebackup_options *opt) struct stat statbuf; /* In the main tar, include the backup_label first... */ - sendFileWithContent(BACKUP_LABEL_FILE, labelfile->data); + sendFileWithContent(BACKUP_LABEL_FILE, labelfile->data, + &manifestInfo); /* * Send tablespace_map file if required and then the bulk of @@ -329,11 +346,14 @@ perform_base_backup(basebackup_options *opt) */ if (tblspc_map_file && opt->sendtblspcmapfile) { - sendFileWithContent(TABLESPACE_MAP, tblspc_map_file->data); - sendDir(".", 1, false, tablespaces, false); + sendFileWithContent(TABLESPACE_MAP, tblspc_map_file->data, + &manifestInfo); + sendDir(".", 1, false, tablespaces, + false, &manifestInfo, NULL); } else - sendDir(".", 1, false, tablespaces, true); + sendDir(".", 1, false, tablespaces, + true, &manifestInfo, NULL); /* ... and pg_control after everything else. */ if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) @@ -341,10 +361,11 @@ perform_base_backup(basebackup_options *opt) (errcode_for_file_access(), errmsg("could not stat file \"%s\": %m", XLOG_CONTROL_FILE))); - sendFile(XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, false, InvalidOid); + sendFile(XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, + false, InvalidOid, &manifestInfo, NULL); } else - sendTablespace(ti->path, false); + sendTablespace(ti->path, ti->oid, false, &manifestInfo); /* * If we're including WAL, and this is the main data directory we @@ -563,7 +584,7 @@ perform_base_backup(basebackup_options *opt) * complete segment. */ StatusFilePath(pathbuf, walFileName, ".done"); - sendFileWithContent(pathbuf, ""); + sendFileWithContent(pathbuf, "", &manifestInfo); } /* @@ -586,16 +607,20 @@ perform_base_backup(basebackup_options *opt) (errcode_for_file_access(), errmsg("could not stat file \"%s\": %m", pathbuf))); - sendFile(pathbuf, pathbuf, &statbuf, false, InvalidOid); + sendFile(pathbuf, pathbuf, &statbuf, false, InvalidOid, + &manifestInfo, NULL); /* unconditionally mark file as archived */ StatusFilePath(pathbuf, fname, ".done"); - sendFileWithContent(pathbuf, ""); + sendFileWithContent(pathbuf, "", &manifestInfo); } /* Send CopyDone message for the last tar file */ pq_putemptymessage('c'); } + + SendBackupManifest(&manifestInfo); + SendXlogRecPtrResult(endptr, endtli); if (total_checksum_failures) @@ -639,6 +664,7 @@ parse_basebackup_options(List *options, basebackup_options *opt) bool o_maxrate = false; bool o_tablespace_map = false; bool o_noverify_checksums = false; + bool o_manifest_checksums = false; MemSet(opt, 0, sizeof(*opt)); foreach(lopt, options) @@ -727,6 +753,24 @@ parse_basebackup_options(List *options, basebackup_options *opt) noverify_checksums = true; o_noverify_checksums = true; } + else if (strcmp(defel->defname, "manifest_checksums") == 0) + { + char *manifest_checksum_algo = NULL; + + if (o_manifest_checksums) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("duplicate option \"%s\"", defel->defname))); + manifest_checksum_algo = strVal(defel->arg); + + if (!parse_checksum_algorithm(manifest_checksum_algo, + &opt->checksumAlgo)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("invalid manifest_checksums option \"%s\"", + manifest_checksum_algo))); + } + else elog(ERROR, "option \"%s\" not recognized", defel->defname); @@ -848,6 +892,184 @@ SendBackupHeader(List *tablespaces) pq_puttextmessage('C', "SELECT"); } +static void +InitializeManifest(manifestinfo *manifestInfo, ChecksumAlgorithm checksumAlgo) +{ + Assert(manifestInfo != NULL); + + MemSet(manifestInfo, 0, sizeof(*manifestInfo)); + + manifestInfo->checksumAlgo = checksumAlgo; + manifestInfo->manifest = makeStringInfo(); + appendStringInfoString(manifestInfo->manifest, "PostgreSQL-Backup-Manifest-Version 1\n"); + + switch (manifestInfo->checksumAlgo) + { + case MC_SHA256: + strcpy(manifestInfo->checksum_label, "SHA256:"); + break; + case MC_CRC32C: + strcpy(manifestInfo->checksum_label, "CRC32C:"); + break; + case MC_NONE: + break; + } +} + +/* + * Add an entry to the backup manifest for a file. + */ +static void +AddFileToManifest(manifestinfo *manifestInfo, const char *tsoid, + const char *filename, size_t size, time_t mtime) +{ + char pathbuf[MAXPGPATH]; + char *escaped_filename; + static char timebuf[128]; + static char checksumbuf[256]; + char encode_checksumbuf[256]; + struct pg_tm *tm; + char *checksumlabel = manifestInfo->checksum_label; + int checksumbuflen; + ChecksumCtx *cCtx = &manifestInfo->cCtx; + StringInfo manifest = manifestInfo->manifest; + + /* + * If this file is part of a tablespace, the filename passed to this + * function will be relative to the tar file that contains it. We want the + * pathname relative to the data directory (ignoring the intermediate + * symlink traversal). + */ + if (tsoid != NULL) + { + snprintf(pathbuf, sizeof(pathbuf), "pg_tblspc/%s/%s", tsoid, filename); + filename = pathbuf; + } + + /* Escape filename, if necessary. */ + escaped_filename = escape_field_for_manifest(filename); + + /* + * Convert time to a string. Since it's not clear what time zone to use + * and since time zone definitions can change, possibly causing confusion, + * use GMT always. + */ + tm = pg_gmtime(&mtime); + if (tm == NULL) + elog(ERROR, "could not convert epoch to timestamp: %m"); + pg_strftime(timebuf, sizeof(timebuf), "%Y-%m-%d %H:%M:%S %Z", tm); + + /* Generate final checksum and Convert it to hexadecimal. */ + if (manifestInfo->checksumAlgo != MC_NONE) + { + checksumbuflen = finalize_checksum(cCtx, manifestInfo->checksumAlgo, + checksumbuf); + checksumbuflen = hex_encode(checksumbuf, checksumbuflen, + encode_checksumbuf); + encode_checksumbuf[checksumbuflen] = '\0'; + } + + /* Add to manifest. */ + appendStringInfo(manifest, "File\t%s\t%zu\t%s\t%s%s\n", + escaped_filename == NULL ? filename : escaped_filename, + size, timebuf, checksumlabel ? checksumlabel : "", + manifestInfo->checksumAlgo != MC_NONE ? encode_checksumbuf : "-"); + + /* Avoid leaking memory. */ + if (escaped_filename != NULL) + pfree(escaped_filename); +} + +/* + * Generate the checksum for backup manifest, and send it to the client. + * The checksum should be always generated with SHA256 algorithm irrespective + * of which algorithm user has provided. + */ +static void +SendBackupManifest(manifestinfo *manifestInfo) +{ + char checksumbuf[256]; + StringInfoData protobuf; + int checksumbuflen; + ChecksumCtx *cCtx = &manifestInfo->cCtx; + StringInfo manifest = manifestInfo->manifest; + + /* Checksum the manifest. */ + initialize_checksum(cCtx, MC_SHA256); + update_checksum(cCtx, MC_SHA256, manifest->data, manifest->len); + checksumbuflen = finalize_checksum(cCtx, + MC_SHA256, + (char *) checksumbuf); + appendStringInfoString(manifest, "Manifest-Checksum\t"); + appendStringInfoString(manifest, "SHA256:"); + enlargeStringInfo(manifest, checksumbuflen * 2); + checksumbuflen = hex_encode(checksumbuf, checksumbuflen, + manifest->data + manifest->len); + manifest->len += checksumbuflen; + appendStringInfoChar(manifest, '\n'); + + /* Send CopyOutResponse message */ + pq_beginmessage(&protobuf, 'H'); + pq_sendbyte(&protobuf, 0); /* overall format */ + pq_sendint16(&protobuf, 0); /* natts */ + pq_endmessage(&protobuf); + + /* Send CopyData message */ + pq_putmessage('d', manifest->data, manifest->len); + + /* And finally CopyDone message */ + pq_putemptymessage('c'); +} + +/* + * Escape a field for inclusion in a manifest. + * + * We use the following escaping rule: If a field contains \t, \r, or \n, + * the field must be surrounded by double-quotes, and any internal double + * quotes must be doubled. Otherwise, no escaping is required. + * + * The return value is a new palloc'd string with escaping added, or NULL + * if no escaping is required. + */ +static char * +escape_field_for_manifest(const char *s) +{ + bool escaping_required = false; + int escaped_length = 2; + const char *t; + char *result; + char *r; + + for (t = s; *t != '\0'; ++t) + { + if (*t == '\t' || *t == '\r' || *t == '\n') + escaping_required = true; + if (*t == '"') + ++escaped_length; + ++escaped_length; + } + + if (!escaping_required) + return NULL; + + result = palloc(escaped_length + 1); + result[0] = '"'; + result[escaped_length - 1] = '"'; + result[escaped_length] = '\0'; + r = result + 1; + + for (t = s; *t != '\0'; ++t) + { + *(r++) = *t; + if (*t == '"') + *(r++) = *t; + } + + Assert(r == &result[escaped_length - 1]); + + return result; +} + /* * Send a single resultset containing just a single * XLogRecPtr record (in text format) @@ -908,7 +1130,8 @@ SendXlogRecPtrResult(XLogRecPtr ptr, TimeLineID tli) * Inject a file with given name and content in the output tar stream. */ static void -sendFileWithContent(const char *filename, const char *content) +sendFileWithContent(const char *filename, const char *content, + manifestinfo *manifestInfo) { struct stat statbuf; int pad, @@ -945,6 +1168,11 @@ sendFileWithContent(const char *filename, const char *content) MemSet(buf, 0, pad); pq_putmessage('d', buf, pad); } + + initialize_checksum(&manifestInfo->cCtx, manifestInfo->checksumAlgo); + update_checksum(&manifestInfo->cCtx, manifestInfo->checksumAlgo, + content, len); + AddFileToManifest(manifestInfo, NULL, filename, len, statbuf.st_mtime); } /* @@ -955,7 +1183,7 @@ sendFileWithContent(const char *filename, const char *content) * Only used to send auxiliary tablespaces, not PGDATA. */ int64 -sendTablespace(char *path, bool sizeonly) +sendTablespace(char *path, char *oid, bool sizeonly, manifestinfo *manifestInfo) { int64 size; char pathbuf[MAXPGPATH]; @@ -988,7 +1216,8 @@ sendTablespace(char *path, bool sizeonly) sizeonly); /* Send all the files in the tablespace version directory */ - size += sendDir(pathbuf, strlen(path), sizeonly, NIL, true); + size += sendDir(pathbuf, strlen(path), sizeonly, NIL, + true, manifestInfo, oid); return size; } @@ -1007,7 +1236,7 @@ sendTablespace(char *path, bool sizeonly) */ static int64 sendDir(const char *path, int basepathlen, bool sizeonly, List *tablespaces, - bool sendtblspclinks) + bool sendtblspclinks, manifestinfo *manifestInfo, const char *tsoid) { DIR *dir; struct dirent *de; @@ -1283,7 +1512,8 @@ sendDir(const char *path, int basepathlen, bool sizeonly, List *tablespaces, skip_this_dir = true; if (!skip_this_dir) - size += sendDir(pathbuf, basepathlen, sizeonly, tablespaces, sendtblspclinks); + size += sendDir(pathbuf, basepathlen, sizeonly, tablespaces, + sendtblspclinks, manifestInfo, tsoid); } else if (S_ISREG(statbuf.st_mode)) { @@ -1291,7 +1521,8 @@ sendDir(const char *path, int basepathlen, bool sizeonly, List *tablespaces, if (!sizeonly) sent = sendFile(pathbuf, pathbuf + basepathlen + 1, &statbuf, - true, isDbDir ? pg_atoi(lastDir + 1, sizeof(Oid), 0) : InvalidOid); + true, isDbDir ? pg_atoi(lastDir + 1, sizeof(Oid), 0) : InvalidOid, + manifestInfo, tsoid); if (sent || sizeonly) { @@ -1354,8 +1585,9 @@ is_checksummed_file(const char *fullpath, const char *filename) * and the file did not exist. */ static bool -sendFile(const char *readfilename, const char *tarfilename, struct stat *statbuf, - bool missing_ok, Oid dboid) +sendFile(const char *readfilename, const char *tarfilename, + struct stat *statbuf, bool missing_ok, Oid dboid, + manifestinfo *manifestInfo, const char *tsoid) { FILE *fp; BlockNumber blkno = 0; @@ -1373,6 +1605,8 @@ sendFile(const char *readfilename, const char *tarfilename, struct stat *statbuf char *segmentpath; bool verify_checksum = false; + initialize_checksum(&manifestInfo->cCtx, manifestInfo->checksumAlgo); + fp = AllocateFile(readfilename, "rb"); if (fp == NULL) { @@ -1541,6 +1775,10 @@ sendFile(const char *readfilename, const char *tarfilename, struct stat *statbuf ereport(ERROR, (errmsg("base backup could not send data, aborting backup"))); + /* Also feed it to the checksum machinery. */ + update_checksum(&manifestInfo->cCtx, manifestInfo->checksumAlgo, + buf, cnt); + len += cnt; throttle(cnt); @@ -1565,6 +1803,8 @@ sendFile(const char *readfilename, const char *tarfilename, struct stat *statbuf { cnt = Min(sizeof(buf), statbuf->st_size - len); pq_putmessage('d', buf, cnt); + update_checksum(&manifestInfo->cCtx, manifestInfo->checksumAlgo, + buf, cnt); len += cnt; throttle(cnt); } @@ -1572,7 +1812,8 @@ sendFile(const char *readfilename, const char *tarfilename, struct stat *statbuf /* * Pad to 512 byte boundary, per tar format requirements. (This small - * piece of data is probably not worth throttling.) + * piece of data is probably not worth throttling, and is not checksummed + * because it's not actually part of the file.) */ pad = ((len + 511) & ~511) - len; if (pad > 0) @@ -1595,6 +1836,8 @@ sendFile(const char *readfilename, const char *tarfilename, struct stat *statbuf } total_checksum_failures += checksum_failures; + AddFileToManifest(manifestInfo, tsoid, tarfilename, statbuf->st_size, + statbuf->st_mtime); return true; } diff --git a/src/backend/replication/repl_gram.y b/src/backend/replication/repl_gram.y index 2d96567..fc80202 100644 --- a/src/backend/replication/repl_gram.y +++ b/src/backend/replication/repl_gram.y @@ -87,6 +87,7 @@ static SQLCmd *make_sqlcmd(void); %token K_EXPORT_SNAPSHOT %token K_NOEXPORT_SNAPSHOT %token K_USE_SNAPSHOT +%token K_MANIFEST_CHECKSUMS %type <node> command %type <node> base_backup start_replication start_logical_replication @@ -214,6 +215,11 @@ base_backup_opt: $$ = makeDefElem("noverify_checksums", (Node *)makeInteger(true), -1); } + | K_MANIFEST_CHECKSUMS SCONST + { + $$ = makeDefElem("manifest_checksums", + (Node *)makeString($2), -1); + } ; create_replication_slot: diff --git a/src/backend/replication/repl_scanner.l b/src/backend/replication/repl_scanner.l index 14c9a1e..e56384e 100644 --- a/src/backend/replication/repl_scanner.l +++ b/src/backend/replication/repl_scanner.l @@ -107,6 +107,7 @@ EXPORT_SNAPSHOT { return K_EXPORT_SNAPSHOT; } NOEXPORT_SNAPSHOT { return K_NOEXPORT_SNAPSHOT; } USE_SNAPSHOT { return K_USE_SNAPSHOT; } WAIT { return K_WAIT; } +MANIFEST_CHECKSUMS { return K_MANIFEST_CHECKSUMS; } "," { return ','; } ";" { return ';'; } diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c index b8d9ec7..dae5735 100644 --- a/src/backend/utils/adt/encode.c +++ b/src/backend/utils/adt/encode.c @@ -15,6 +15,7 @@ #include <ctype.h> +#include "common/encode.h" #include "utils/builtins.h" @@ -109,8 +110,6 @@ binary_decode(PG_FUNCTION_ARGS) * HEX */ -static const char hextbl[] = "0123456789abcdef"; - static const int8 hexlookup[128] = { -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, @@ -122,20 +121,6 @@ static const int8 hexlookup[128] = { -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, }; -unsigned -hex_encode(const char *src, unsigned len, char *dst) -{ - const char *end = src + len; - - while (src < end) - { - *dst++ = hextbl[(*src >> 4) & 0xF]; - *dst++ = hextbl[*src & 0xF]; - src++; - } - return len * 2; -} - static inline char get_hex(char c) { diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 1b351cb..a8b987e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -20,6 +20,7 @@ #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "common/encode.h" #include "common/int.h" #include "lib/hyperloglog.h" #include "libpq/pqformat.h" diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c index 238b671f7..ca9ccb9 100644 --- a/src/bin/pg_basebackup/pg_basebackup.c +++ b/src/bin/pg_basebackup/pg_basebackup.c @@ -88,6 +88,12 @@ typedef struct UnpackTarState FILE *file; } UnpackTarState; +typedef struct WriteManifestState +{ + char filename[MAXPGPATH]; + FILE *file; +} WriteManifestState; + typedef void (*WriteDataCallback) (size_t nbytes, char *buf, void *callback_data); @@ -135,6 +141,7 @@ static bool temp_replication_slot = true; static bool create_slot = false; static bool no_slot = false; static bool verify_checksums = true; +static char *manifest_checksums = NULL; static bool success = false; static bool made_new_pgdata = false; @@ -180,6 +187,12 @@ static void ReceiveTarCopyChunk(size_t r, char *copybuf, void *callback_data); static void ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum); static void ReceiveTarAndUnpackCopyChunk(size_t r, char *copybuf, void *callback_data); +static void ReceiveBackupManifest(PGconn *conn); +static void ReceiveBackupManifestChunk(size_t r, char *copybuf, + void *callback_data); +static void ReceiveBackupManifestInMemory(PGconn *conn, PQExpBuffer buf); +static void ReceiveBackupManifestInMemoryChunk(size_t r, char *copybuf, + void *callback_data); static void BaseBackup(void); static bool reached_end_position(XLogRecPtr segendpos, uint32 timeline, @@ -386,6 +399,8 @@ usage(void) printf(_(" --no-slot prevent creation of temporary replication slot\n")); printf(_(" --no-verify-checksums\n" " do not verify checksums\n")); + printf(_(" --manifest-checksums=SHA256|CRC32C|NONE\n" + " calculate checksums for manifest files using provided algorithm\n")); printf(_(" -?, --help show this help, then exit\n")); printf(_("\nConnection options:\n")); printf(_(" -d, --dbname=CONNSTR connection string\n")); @@ -924,8 +939,8 @@ ReceiveCopyData(PGconn *conn, WriteDataCallback callback, res = PQgetResult(conn); if (PQresultStatus(res) != PGRES_COPY_OUT) { - pg_log_error("could not get COPY data stream: %s", - PQerrorMessage(conn)); + pg_log_error("could not get COPY data stream: %s [%s]", + PQerrorMessage(conn), PQresStatus(PQresultStatus(res))); exit(1); } PQclear(res); @@ -1170,6 +1185,31 @@ ReceiveTarFile(PGconn *conn, PGresult *res, int rownum) } } + /* + * Normally, we emit the backup manifest as a separate file, but when + * we're writing a tarfile to stdout, we don't have that option, so + * include it in the one tarfile we've got. + */ + if (strcmp(basedir, "-") == 0) + { + char header[512]; + PQExpBufferData buf; + + initPQExpBuffer(&buf); + ReceiveBackupManifestInMemory(conn, &buf); + if (PQExpBufferDataBroken(buf)) + { + pg_log_error("out of memory"); + exit(1); + } + tarCreateHeader(header, "backup_manifest", NULL, buf.len, + pg_file_create_mode, 04000, 02000, + time(NULL)); + writeTarData(&state, header, sizeof(header)); + writeTarData(&state, buf.data, buf.len); + termPQExpBuffer(&buf); + } + /* 2 * 512 bytes empty data at end of file */ writeTarData(&state, zerobuf, sizeof(zerobuf)); @@ -1417,6 +1457,64 @@ get_tablespace_mapping(const char *dir) /* + * Receive the backup manifest file and write it out to a file. + */ +static void +ReceiveBackupManifest(PGconn *conn) +{ + WriteManifestState state; + + snprintf(state.filename, sizeof(state.filename), + "%s/backup_manifest", basedir); + state.file = fopen(state.filename, "wb"); + if (state.file == NULL) + { + pg_log_error("could not create file \"%s\": %m", state.filename); + exit(1); + } + + ReceiveCopyData(conn, ReceiveBackupManifestChunk, &state); + + fclose(state.file); +} + +/* + * Receive one chunk of the backup manifest file and write it out to a file. + */ +static void +ReceiveBackupManifestChunk(size_t r, char *copybuf, void *callback_data) +{ + WriteManifestState *state = callback_data; + + if (fwrite(copybuf, r, 1, state->file) != 1) + { + pg_log_error("could not write to file \"%s\": %m", state->filename); + exit(1); + } +} + +/* + * Receive the backup manifest file and write it out to a file. + */ +static void +ReceiveBackupManifestInMemory(PGconn *conn, PQExpBuffer buf) +{ + ReceiveCopyData(conn, ReceiveBackupManifestInMemoryChunk, buf); +} + +/* + * Receive one chunk of the backup manifest file and write it out to a file. + */ +static void +ReceiveBackupManifestInMemoryChunk(size_t r, char *copybuf, + void *callback_data) +{ + PQExpBuffer buf = callback_data; + + appendPQExpBuffer(buf, copybuf, r); +} + +/* * Receive a tar format stream from the connection to the server, and unpack * the contents of it into a directory. Only files, directories and * symlinks are supported, no other kinds of special files. @@ -1658,6 +1756,7 @@ BaseBackup(void) maxServerMajor; int serverVersion, serverMajor; + int writing_to_stdout; Assert(conn != NULL); @@ -1725,7 +1824,7 @@ BaseBackup(void) } basebkp = - psprintf("BASE_BACKUP LABEL '%s' %s %s %s %s %s %s %s", + psprintf("BASE_BACKUP LABEL '%s' %s %s %s %s %s %s %s MANIFEST_CHECKSUMS '%s'", escaped_label, showprogress ? "PROGRESS" : "", includewal == FETCH_WAL ? "WAL" : "", @@ -1733,7 +1832,8 @@ BaseBackup(void) includewal == NO_WAL ? "" : "NOWAIT", maxrate_clause ? maxrate_clause : "", format == 't' ? "TABLESPACE_MAP" : "", - verify_checksums ? "" : "NOVERIFY_CHECKSUMS"); + verify_checksums ? "" : "NOVERIFY_CHECKSUMS", + manifest_checksums ? manifest_checksums : "NONE"); if (PQsendQuery(conn, basebkp) == 0) { @@ -1821,7 +1921,8 @@ BaseBackup(void) /* * When writing to stdout, require a single tablespace */ - if (format == 't' && strcmp(basedir, "-") == 0 && PQntuples(res) > 1) + writing_to_stdout = format == 't' && strcmp(basedir, "-") == 0; + if (writing_to_stdout && PQntuples(res) > 1) { pg_log_error("can only write single tablespace to stdout, database has %d", PQntuples(res)); @@ -1850,6 +1951,19 @@ BaseBackup(void) ReceiveAndUnpackTarFile(conn, res, i); } /* Loop over all tablespaces */ + /* + * Now receive backup manifest, if appropriate. + * + * If we're writing a tarfile to stdout, ReceiveTarFile will have already + * processed the backup manifest and included it in the output tarfile. + * Such a configuration doesn't allow for writing multiple files. + * + * If we're talking to an older server, it won't send a backup manifest, + * so don't try to receive one. + */ + if (!writing_to_stdout && serverMajor >= 1300) + ReceiveBackupManifest(conn); + if (showprogress) { progress_report(PQntuples(res), NULL, true); @@ -2052,6 +2166,7 @@ main(int argc, char **argv) {"waldir", required_argument, NULL, 1}, {"no-slot", no_argument, NULL, 2}, {"no-verify-checksums", no_argument, NULL, 3}, + {"manifest-checksums", required_argument, NULL, 'm'}, {NULL, 0, NULL, 0} }; int c; @@ -2079,7 +2194,7 @@ main(int argc, char **argv) atexit(cleanup_directories_atexit); - while ((c = getopt_long(argc, argv, "CD:F:r:RS:T:X:l:nNzZ:d:c:h:p:U:s:wWkvP", + while ((c = getopt_long(argc, argv, "CD:F:r:RS:T:X:l:nNzZ:d:c:h:p:U:s:wWkvPm:", long_options, &option_index)) != -1) { switch (c) @@ -2220,6 +2335,9 @@ main(int argc, char **argv) case 3: verify_checksums = false; break; + case 'm': + manifest_checksums = pg_strdup(optarg); + break; default: /* diff --git a/src/common/Makefile b/src/common/Makefile index ffb0f6e..5fa2e23 100644 --- a/src/common/Makefile +++ b/src/common/Makefile @@ -48,9 +48,11 @@ LIBS += $(PTHREAD_LIBS) OBJS_COMMON = \ base64.o \ + checksum_utils.o \ config_info.o \ controldata_utils.o \ d2s.o \ + encode.o \ exec.o \ f2s.o \ file_perm.o \ diff --git a/src/common/checksum_utils.c b/src/common/checksum_utils.c new file mode 100644 index 0000000..e10ec30 --- /dev/null +++ b/src/common/checksum_utils.c @@ -0,0 +1,109 @@ +/*------------------------------------------------------------------------- + * + * checksum_utils.c + * checksum handling helpers + * + * + * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/common/checksum_utils.c + * + *------------------------------------------------------------------------- + */ + + +#ifndef FRONTEND +#include "postgres.h" +#else +#include "postgres_fe.h" +#endif + +#include "common/checksum_utils.h" + +/* + * Initialize the checksum context according to the provided algorithm. + */ +void +initialize_checksum(ChecksumCtx * cCtx, ChecksumAlgorithm checksumAlgo) +{ + switch (checksumAlgo) + { + case MC_SHA256: + pg_sha256_init(&cCtx->sha256_ctx); + break; + case MC_CRC32C: + INIT_CRC32C(cCtx->crc_ctx); + break; + case MC_NONE: + break; + } +} + +void +update_checksum(ChecksumCtx * cCtx, ChecksumAlgorithm checksumAlgo, + const char *buf, off_t cnt) +{ + switch (checksumAlgo) + { + case MC_SHA256: + pg_sha256_update(&cCtx->sha256_ctx, (uint8 *) buf, cnt); + break; + case MC_CRC32C: + COMP_CRC32C(cCtx->crc_ctx, buf, cnt); + break; + case MC_NONE: + break; + } +} + +/* + * Function calculate the final checksum for the provided context and returns + * the length of checksum. + */ +int +finalize_checksum(ChecksumCtx * cCtx, ChecksumAlgorithm checksumAlgo, + char *checksumbuf) +{ + int checksumlen = 0; + + switch (checksumAlgo) + { + case MC_SHA256: + pg_sha256_final(&cCtx->sha256_ctx, (uint8 *) checksumbuf); + checksumlen = PG_SHA256_DIGEST_LENGTH; + break; + case MC_CRC32C: + FIN_CRC32C(cCtx->crc_ctx); + sprintf(checksumbuf, "%u", cCtx->crc_ctx); + checksumlen = strlen(checksumbuf); + break; + case MC_NONE: + break; + } + return checksumlen; +} + +bool +parse_checksum_algorithm(char *name, ChecksumAlgorithm *checksumAlgo) +{ + if (pg_strcasecmp(name, "SHA256") == 0) + { + *checksumAlgo = MC_SHA256; + return true; + } + else if (pg_strcasecmp(name, "CRC32C") == 0) + { + *checksumAlgo = MC_CRC32C; + return true; + } + else if (pg_strcasecmp(name, "NONE") == 0) + { + *checksumAlgo = MC_NONE; + return true; + } + + return false; +} diff --git a/src/common/encode.c b/src/common/encode.c new file mode 100644 index 0000000..a450c53 --- /dev/null +++ b/src/common/encode.c @@ -0,0 +1,38 @@ +/*------------------------------------------------------------------------- + * + * encode.c + * data encoding/decoding things. + * + * + * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/common/encode.c + * + *------------------------------------------------------------------------- + */ + + +#ifndef FRONTEND +#include "postgres.h" +#else +#include "postgres_fe.h" +#endif + +#include "common/encode.h" + +unsigned +hex_encode(const char *src, unsigned len, char *dst) +{ + const char *end = src + len; + + while (src < end) + { + *dst++ = hextbl[(*src >> 4) & 0xF]; + *dst++ = hextbl[*src & 0xF]; + src++; + } + return len * 2; +} diff --git a/src/include/common/checksum_utils.h b/src/include/common/checksum_utils.h new file mode 100644 index 0000000..c931070 --- /dev/null +++ b/src/include/common/checksum_utils.h @@ -0,0 +1,46 @@ +/*------------------------------------------------------------------------- + * checksum_utils.h + * checksum handling helpers + * + * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/include/common/checksum_utils.h + * + *------------------------------------------------------------------------- + */ +#ifndef COMMON_CHECKSUM_UTILS_H +#define COMMON_CHECKSUM_UTILS_H + +#include "common/sha2.h" +#include "port/pg_crc32c.h" +#include "lib/stringinfo.h" + +/* Checksum algorithm option for manifest */ +typedef enum ChecksumAlgorithm +{ + MC_NONE = 0, + MC_SHA256, + MC_CRC32C +} ChecksumAlgorithm; + +/* checksum algorithm context */ +typedef union checksumCtx +{ + pg_sha256_ctx sha256_ctx; + pg_crc32c crc_ctx; +} ChecksumCtx; + +extern void initialize_checksum(ChecksumCtx * cCtx, + ChecksumAlgorithm checksumAlgo); +extern void update_checksum(ChecksumCtx * cCtx, + ChecksumAlgorithm checksumAlgo, + const char *buf, off_t cnt); +extern int finalize_checksum(ChecksumCtx * cCtx, + ChecksumAlgorithm checksumAlgo, + char *checksumbuf); +extern bool parse_checksum_algorithm(char *name, + ChecksumAlgorithm * checksumAlgo); + +#endif /* COMMON_CHECKSUM_UTILS_H */ diff --git a/src/include/common/encode.h b/src/include/common/encode.h new file mode 100644 index 0000000..63328bc --- /dev/null +++ b/src/include/common/encode.h @@ -0,0 +1,20 @@ +/*------------------------------------------------------------------------- + * encode.h + * data encoding/decoding things. + * + * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/include/common/encode.h + * + *------------------------------------------------------------------------- + */ +#ifndef COMMON_encode_H +#define COMMON_encode_H + +static const char hextbl[] = "0123456789abcdef"; + +extern unsigned hex_encode(const char *src, unsigned len, char *dst); + +#endif /* COMMON_ENCODE_H */ diff --git a/src/include/replication/basebackup.h b/src/include/replication/basebackup.h index 07ed281..cbf15ad 100644 --- a/src/include/replication/basebackup.h +++ b/src/include/replication/basebackup.h @@ -12,6 +12,9 @@ #ifndef _BASEBACKUP_H #define _BASEBACKUP_H +#include "common/checksum_utils.h" +#include "common/sha2.h" +#include "lib/stringinfo.h" #include "nodes/replnodes.h" /* @@ -29,8 +32,18 @@ typedef struct int64 size; } tablespaceinfo; +/* Backup manifest info */ +typedef struct +{ + ChecksumAlgorithm checksumAlgo; + char checksum_label[10]; + ChecksumCtx cCtx; + StringInfo manifest; +} manifestinfo; + extern void SendBaseBackup(BaseBackupCmd *cmd); -extern int64 sendTablespace(char *path, bool sizeonly); +extern int64 sendTablespace(char *path, char *oid, bool sizeonly, + manifestinfo * manifestInfo); #endif /* _BASEBACKUP_H */ diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h index a241af4..b911899 100644 --- a/src/include/utils/builtins.h +++ b/src/include/utils/builtins.h @@ -30,7 +30,6 @@ extern int errdatatype(Oid datatypeOid); extern int errdomainconstraint(Oid datatypeOid, const char *conname); /* encode.c */ -extern unsigned hex_encode(const char *src, unsigned len, char *dst); extern unsigned hex_decode(const char *src, unsigned len, char *dst); /* int.c */ diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index caf6b86..87556f6 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -329,6 +329,8 @@ CheckPointStmt CheckpointStatsData CheckpointerRequest CheckpointerShmemStruct +ChecksumAlgorithm +ChecksumCtx Chromosome CkptSortItem CkptTsStatus @@ -1351,6 +1353,7 @@ MultiXactOffset MultiXactStateData MultiXactStatus MyData +manifestinfo NDBOX NODE NUMCacheEntry -- 1.8.3.1
v7-0002-Implementation-of-backup-validator.patch
(application/octet-stream, 17.9 KB)
From 9d285112eaa14a1226acd48a5f1e8ea0d51c2fc6 Mon Sep 17 00:00:00 2001 From: Suraj Kharage <[email protected]> Date: Fri, 20 Dec 2019 16:19:44 +0530 Subject: [PATCH v7 2/3] Implementation of backup validator Patch by Suraj Kharage, inputs from Robert Haas, review from Jeevan Chalke, and Robert Haas. --- doc/src/sgml/ref/pg_basebackup.sgml | 12 + src/bin/pg_basebackup/pg_basebackup.c | 516 ++++++++++++++++++++++++++++++++++ src/tools/pgindent/typedefs.list | 2 + 3 files changed, 530 insertions(+) diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml index af7c731..cb11f74 100644 --- a/doc/src/sgml/ref/pg_basebackup.sgml +++ b/doc/src/sgml/ref/pg_basebackup.sgml @@ -548,6 +548,18 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>--verify-backup</option></term> + <listitem> + <para> + Validate the given backup directory and detect the modification if any + without restarting the server. For plain backup, provide the backup + directory path with <option>--pgdata</option> option. Tar format + backups can be verified after untarring. + </para> + </listitem> + </varlistentry> + </variablelist> </para> diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c index ca9ccb9..e90e48e 100644 --- a/src/bin/pg_basebackup/pg_basebackup.c +++ b/src/bin/pg_basebackup/pg_basebackup.c @@ -27,9 +27,12 @@ #endif #include "access/xlog_internal.h" +#include "common/checksum_utils.h" +#include "common/encode.h" #include "common/file_perm.h" #include "common/file_utils.h" #include "common/logging.h" +#include "common/sha2.h" #include "common/string.h" #include "fe_utils/recovery_gen.h" #include "fe_utils/string_utils.h" @@ -97,6 +100,30 @@ typedef struct WriteManifestState typedef void (*WriteDataCallback) (size_t nbytes, char *buf, void *callback_data); +typedef struct DataDirectoryFileInfo +{ + char *filename; + int filesize; + char *checksum; + bool matched; + uint32 status; /* hash status */ +} DataDirectoryFileInfo; + +/* This is declared in advance because it is required for manifesthash */ +static uint32 string_hash_sdbm(const char *key); + +#define SH_PREFIX manifesthash +#define SH_ELEMENT_TYPE DataDirectoryFileInfo +#define SH_KEY_TYPE char* +#define SH_KEY filename +#define SH_HASH_KEY(tb, key) string_hash_sdbm(key) +#define SH_EQUAL(tb, a, b) (strcmp(a, b) == 0) +#define SH_SCOPE static inline +#define SH_RAW_ALLOCATOR pg_malloc +#define SH_DECLARE +#define SH_DEFINE +#include "lib/simplehash.h" + /* * pg_xlog has been renamed to pg_wal in version 10. This version number * should be compared with PQserverVersion(). @@ -201,6 +228,19 @@ static bool reached_end_position(XLogRecPtr segendpos, uint32 timeline, static const char *get_tablespace_mapping(const char *dir); static void tablespace_list_append(const char *arg); +static void VerifyBackup(void); +static manifesthash_hash *CreateManifestHash(char *manifest_path, + ChecksumAlgorithm *checksum_type); +static void VerifyDir(char *basepath, char *subdirpath, + manifesthash_hash *hashtab, + ChecksumAlgorithm checksum_type); +static void VerifyFile(char *filepath, struct stat st, + char *basepath, + manifesthash_hash *hashtab, + ChecksumAlgorithm checksum_type); +static char *NextLine(char *buf, char *endptr); +static char *NextWord(char *line, char ch, char *endlineptr); +static char *ReadFileIntoBuffer(char *filename, int *length); static void cleanup_directories_atexit(void) @@ -401,6 +441,7 @@ usage(void) " do not verify checksums\n")); printf(_(" --manifest-checksums=SHA256|CRC32C|NONE\n" " calculate checksums for manifest files using provided algorithm\n")); + printf(_(" --verify-backup validate the backup\n")); printf(_(" -?, --help show this help, then exit\n")); printf(_("\nConnection options:\n")); printf(_(" -d, --dbname=CONNSTR connection string\n")); @@ -2167,11 +2208,13 @@ main(int argc, char **argv) {"no-slot", no_argument, NULL, 2}, {"no-verify-checksums", no_argument, NULL, 3}, {"manifest-checksums", required_argument, NULL, 'm'}, + {"verify-backup", no_argument, NULL, 4}, {NULL, 0, NULL, 0} }; int c; int option_index; + bool verify_backup = false; pg_logging_init(argv[0]); progname = get_progname(argv[0]); @@ -2338,6 +2381,9 @@ main(int argc, char **argv) case 'm': manifest_checksums = pg_strdup(optarg); break; + case 4: + verify_backup = true; + break; default: /* @@ -2460,6 +2506,13 @@ main(int argc, char **argv) } #endif + /* --verify-backup option is specified, validate the backup */ + if (verify_backup) + { + VerifyBackup(); + exit(0); + } + /* connection in replication mode to server */ conn = GetConnection(); if (!conn) @@ -2524,3 +2577,466 @@ main(int argc, char **argv) success = true; return 0; } + +static void +VerifyBackup(void) +{ + char manifest_path[MAXPGPATH]; + manifesthash_hash *hashtab; + manifesthash_iterator i; + DataDirectoryFileInfo *entry; + ChecksumAlgorithm checksum_type; + + snprintf(manifest_path, sizeof(manifest_path), "%s/%s", basedir, + "backup_manifest"); + + /* create hash table */ + hashtab = CreateManifestHash(manifest_path, &checksum_type); + + VerifyDir(basedir, "", hashtab, checksum_type); + + manifesthash_start_iterate(hashtab, &i); + while ((entry = manifesthash_iterate(hashtab, &i)) != NULL) + { + if (!entry->matched) + pg_log_info("file \"%s\" is present in manifest but missing from the backup", + entry->filename); + } +} + +/* + * Given a file path, read that file into buffer, parse that buffer line by + * line and generate the hash table for each line. Also, generate the SHA256 + * checksum for the records that are read from buffer and compare that with + * manifest checksum written in backup_manifest file. If both checksums are + * identical then proceed, otherwise throw an error and abort. Set the + * checksum type of manifest record to out parameter checksum_type. + */ +static manifesthash_hash * +CreateManifestHash(char *manifest_path, ChecksumAlgorithm *checksum_type) +{ + manifesthash_hash *hashtab; + DataDirectoryFileInfo *entry; + char *buf; + ChecksumCtx cCtx; + int length; + char *endptr; + int numlines = 0; + + buf = ReadFileIntoBuffer(manifest_path, &length); + endptr = buf + length; + + hashtab = manifesthash_create(1024, NULL); + + /* Read the first line of buffer */ + if (buf < endptr) + { + char *headerline; + int headerlength; + + /* + * Read the header from file, here header_line is pointing to start of + * file. Advanced the buffer to next line and then buf - header_line + * will give us the header length. + */ + headerline = buf; + buf = NextLine(buf, endptr); + headerlength = buf - headerline; + + numlines++; + + /* + * Initialize the checksum for the first time. Since checksum for the + * manifest file is always generated with SHA256 so initializing with + * SHA256. + */ + initialize_checksum(&cCtx, MC_SHA256); + + /* feed the header to the checksum machinery */ + update_checksum(&cCtx, MC_SHA256, headerline, headerlength); + } + + /* + * Once we read the header, then read the buffer line by line and check + * whether it is a File record or Manifest-Checksum entry and parse + * accordingly. + */ + while (buf < endptr) + { + int length; + char *line; + char *word; + int wordlength; + char *endlineptr; + char *nextword; + + line = buf; + /* read the next line and calculate the length for the line */ + buf = NextLine(buf, endptr); + length = buf - line; + + /* calculate the end of line */ + endlineptr = line + length; + + numlines++; + + /* parse the first word of line */ + word = line; + nextword = NextWord(line, '\t', endlineptr); + wordlength = nextword - word - 1; + + /* + * If it is a File record, then parse it into fields. With this we + * will get the filename, checksum and size. + */ + if (strncmp(word, "File", wordlength) == 0) + { + char *size; + long filesize; + bool found; + long filelength, + sizelength; + char *filename; + char *checksumlabel; + char *checksum; + ChecksumAlgorithm currentchecksumtype; + + /* + * feed line to checksum machinery as it is a FILE type manifest + * record + */ + update_checksum(&cCtx, MC_SHA256, line, length); + + line[length - 1] = '\0'; + + /* parse the filename */ + filename = nextword; + size = NextWord(filename, '\t', endlineptr); + filelength = size - filename; + filename[filelength - 1] = '\0'; + + /* parse the filesize and convert it to long */ + word = NextWord(size, '\t', endlineptr); + sizelength = word - size; + size[sizelength - 1] = '\0'; + filesize = strtol(size, NULL, 10); + + /* skip mtime field */ + checksum = NextWord(word, '\t', endlineptr); + + /* + * parse the checksum field. If it is a "-" that means no + * checksum. Otherwise split this field by ":" character to + * identify the checksum type. + */ + if (strcmp(checksum, "-") == 0) + currentchecksumtype = MC_NONE; + else + { + checksumlabel = checksum; + checksum = NextWord(checksum, ':', endlineptr); + length = checksum - checksumlabel; + checksumlabel[length - 1] = '\0'; + + if (!parse_checksum_algorithm(checksumlabel, ¤tchecksumtype)) + { + pg_log_error("manifest file \"%s\" specifies unknown checksum algorithm \"%s\" at line %d", + manifest_path, checksumlabel, numlines); + exit(1); + } + } + + /* + * All the manifest records should have same checksum type. Error + * out if we find checksum mismatch between manifest records. + */ + if (numlines > 2 && *checksum_type != currentchecksumtype) + { + pg_log_error("manifest file \"%s\" specifies different checksum algorithm \"%s\" at line %d", + manifest_path, checksumlabel, numlines); + exit(1); + } + else + *checksum_type = currentchecksumtype; + + /* insert the hash record */ + entry = manifesthash_insert(hashtab, filename, &found); + entry->filesize = filesize; + entry->checksum = checksum; + } + else if (strncmp(word, "Manifest-Checksum", wordlength) == 0) + { + char checksumbuf[256]; + int checksumbuflen; + char encodedchecksum[256]; + char *checksumlabel; + char *checksum; + + line[length - 1] = '\0'; + + /* parse the checksum label for backup manifest checksum */ + checksumlabel = nextword; + checksum = NextWord(nextword, ':', endlineptr); + length = checksum - checksumlabel; + checksumlabel[length - 1] = '\0'; + + if (strcmp(checksumlabel, "SHA256") != 0) + { + pg_log_error("manifest file \"%s\" specifies unknown manifest checksum algorithm \"%s\" at line %d", + manifest_path, checksumlabel, numlines); + exit(1); + } + + /* finalize the checksum */ + checksumbuflen = finalize_checksum(&cCtx, MC_SHA256, + (char *) checksumbuf); + checksumbuflen = hex_encode(checksumbuf, checksumbuflen, + encodedchecksum); + encodedchecksum[checksumbuflen] = '\0'; + + if (strcmp(encodedchecksum, checksum) != 0) + { + pg_log_error("manifest file \"%s\" has manifest checksum \"%s\" but calculated manifest checksum is \"%s\"", + manifest_path, checksum, encodedchecksum); + exit(1); + } + } + else + { + word[wordlength] = '\0'; + pg_log_error("manifest file \"%s\" contains invalid keyword \"%s\" at line %d", + manifest_path, word, numlines); + exit(1); + } + } + return hashtab; +} + +/* + * Verify all files from the given directory. Scans the given directory + * and for each regular file within that directory, calls VerifyFile() + * for the verification. + */ +static void +VerifyDir(char *basepath, char *pathsuffix, manifesthash_hash *hashtab, + ChecksumAlgorithm checksum_type) +{ + char path[MAXPGPATH]; + DIR *dir; + struct dirent *de; + + snprintf(path, MAXPGPATH, "%s%s", basepath, pathsuffix); + + dir = opendir(path); + if (!dir) + { + pg_log_error("could not open directory \"%s\": %m", path); + exit(1); + } + + while ((de = readdir(dir)) != NULL) + { + char fn[MAXPGPATH]; + char newpathsuffix[MAXPGPATH]; + struct stat st; + + if (strcmp(de->d_name, ".") == 0 || strcmp(de->d_name, "..") == 0) + continue; + + snprintf(newpathsuffix, MAXPGPATH, "%s/%s", pathsuffix, de->d_name); + + /* + * Ignore the possible presence of a backup_manifest file and/or a + * pg_wal directory in the backup being verified. A backup_manifest + * file generated while backup, does not have entry for + * backup_manifest itself and files in pg_wal directory, so skipping + * those would be right here. + */ + if (strcmp(newpathsuffix, "/pg_wal") == 0 || + strcmp(newpathsuffix, "/backup_manifest") == 0) + continue; + + snprintf(fn, sizeof(fn), "%s/%s", path, de->d_name); + if (stat(fn, &st) < 0) + { + pg_log_error("could not stat file \"%s\": %m", fn); + exit(1); + } + if (S_ISREG(st.st_mode)) + VerifyFile(fn, st, basepath, hashtab, checksum_type); + else if (S_ISDIR(st.st_mode)) + VerifyDir(basepath, newpathsuffix, hashtab, checksum_type); + } + closedir(dir); +} + +/* + * Given the file and its details, check whether it is present in hash table + * and if yes, then compare its details with hash table entry. + */ +static void +VerifyFile(char *filepath, struct stat st, char *basepath, + manifesthash_hash *hashtab, ChecksumAlgorithm checksum_type) +{ + DataDirectoryFileInfo *record; + FILE *fp; + char buf[1048576]; /* 1MB chunk */ + pgoff_t len = 0; + off_t cnt; + char checksumbuf[256]; + char encodedchecksumbuf[256]; + int checksumbuflen; + ChecksumCtx cCtx; + char *relativefilepath; + + /* + * Since backup manifest contains the relative path of file as a filename, + * we need that relative path to lookup into hash table. + */ + relativefilepath = filepath + strlen(basepath) + 1; + + /* + * Lookup into hash table and if record found then we match the file size + * and checksum (if enabled). Modified time cannot be compared with the + * file in the backup directory and its entry in the manifest as manifest + * entry gives mtime from server file whereas the same file in the backup + * will have different mtime. + */ + record = manifesthash_lookup(hashtab, relativefilepath); + if (record == NULL) + { + pg_log_info("file \"%s\" is present in backup but not in manifest", + relativefilepath); + return; + } + + record->matched = true; + if (record->filesize != st.st_size) + pg_log_info("file \"%s\" has size %d in manifest but size %lu in backup", + relativefilepath, record->filesize, st.st_size); + + if (checksum_type == MC_NONE) + return; + + /* + * If checksum_type is other than MC_NONE then generate the checksum based + * on checksum_type. + */ + initialize_checksum(&cCtx, checksum_type); + + fp = fopen(filepath, "r"); + if (!fp) + { + pg_log_error("could not open file \"%s\": %m", filepath); + exit(1); + } + + /* Read file in chunks [1 MB each chunk] */ + while ((cnt = fread(buf, 1, Min(sizeof(buf), st.st_size - len), fp)) > 0) + { + update_checksum(&cCtx, checksum_type, buf, cnt); + len += cnt; + } + + checksumbuflen = finalize_checksum(&cCtx, checksum_type, checksumbuf); + + /* Convert checksum to hexadecimal. */ + checksumbuflen = hex_encode(checksumbuf, checksumbuflen, + encodedchecksumbuf); + encodedchecksumbuf[checksumbuflen] = '\0'; + + fclose(fp); + + /* compare the generated checksum with the checksum present in hash entry */ + if (strcmp(record->checksum, encodedchecksumbuf) != 0) + pg_log_info("file \"%s\" has checksum %s in manifest but checksum %s in backup", + relativefilepath, record->checksum, encodedchecksumbuf); +} + +/* + * Find out the next new line character from the provided string and return + * char pointer pointing to next character after that. + */ +static char * +NextLine(char *buf, char *endptr) +{ + while (*buf != '\n' && buf < endptr) + buf++; + + return ++buf; +} + +/* + * Advance the string until provided character is detected. Return the + * advanced string. + */ +static char * +NextWord(char *line, char ch, char *endlineptr) +{ + while (*line != ch && line < endlineptr) + line++; + + return ++line; +} + +/* Read the given file into buffer and return that buffer */ +static char * +ReadFileIntoBuffer(char *filename, int *length) +{ + int fd; + char *buf; + struct stat stat; + + fd = open(filename, O_RDONLY, 0); + if (fd < 0) + { + pg_log_error("could not open file \"%s\": %m", filename); + exit(1); + } + + if (fstat(fd, &stat)) + { + pg_log_error("could not stat file \"%s\": %m", filename); + close(fd); + exit(1); + } + + buf = pg_malloc(stat.st_size + 1); + + *length = read(fd, buf, stat.st_size); + if (*length != stat.st_size) + { + if (*length < 0) + pg_log_error("could not read file \"%s\": %m", filename); + else + pg_log_error("could not read file \"%s\": read %d of %lu", filename, + *length, (unsigned long) stat.st_size); + + close(fd); + exit(1); + } + + buf[*length] = '\0'; + + close(fd); + + return buf; +} + +/* + * Simple string hash function from http://www.cse.yorku.ca/~oz/hash.html + * + * The backend uses a more sophisticated function for hashing strings, + * but we don't really need that complexity here. + */ +uint32 +string_hash_sdbm(const char *key) +{ + uint32 hash = 0; + int c; + + while ((c = *key++)) + hash = c + (hash << 6) + (hash << 16) - hash; + + return hash; +} diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 87556f6..14e475e 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -487,6 +487,7 @@ DR_sqlfunction DR_transientrel DSA DWORD +DataDirectoryFileInfo DataDumperPtr DataPageDeleteStack DateADT @@ -1353,6 +1354,7 @@ MultiXactOffset MultiXactStateData MultiXactStatus MyData +manifesthash_hash manifestinfo NDBOX NODE -- 1.8.3.1
v7-0003-Tap-test-case-patch-to-verify-the-backup-using-ve.patch
(application/octet-stream, 7.5 KB)
From 390f865a2345a7aa9d6d153c643a062253f4da72 Mon Sep 17 00:00:00 2001 From: Suraj Kharage <[email protected]> Date: Tue, 24 Dec 2019 14:34:45 +0530 Subject: [PATCH v7 3/3] Tap test case patch to verify the backup using --verify-backup option Patch by Rajkumar Raghuwanshi --- src/bin/pg_basebackup/t/010_pg_basebackup.pl | 126 ++++++++++++++++++++++++++- 1 file changed, 125 insertions(+), 1 deletion(-) diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl index b7d36b6..04512a6 100644 --- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl +++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl @@ -6,7 +6,7 @@ use File::Basename qw(basename dirname); use File::Path qw(rmtree); use PostgresNode; use TestLib; -use Test::More tests => 106; +use Test::More tests => 137; program_help_ok('pg_basebackup'); program_version_ok('pg_basebackup'); @@ -329,6 +329,130 @@ SKIP: 'pg_replslot symlink copied as directory'); rmtree("$tempdir/backup1"); + # verify backup cluster using --verify-backup options. + # take backup with default "NONE" manifest checksum option + $node->command_ok( + [ + 'pg_basebackup', '-D', "$tempdir/backup1", + "-T$shorter_tempdir/tblspc1=$tempdir/tbackup/tblspc_verify" + ], + 'plain backup with default manifest checksum with tablespaces succeeds'); + ok(-f "$tempdir/backup1/PG_VERSION", 'backup1 created'); + ok(-f "$tempdir/backup1/backup_manifest", 'backup_manifest file is present'); + # verify plain backup with default NONE manifest-checksum. + $node->command_ok( + [ + 'pg_basebackup', '-D', "$tempdir/backup1", '--verify-backup' + ], + 'plain backup with default manifest checksum is verified'); + + # --verify-backup without checksums should detect removed file, new file or changed size of a file + # delete a file from mapped tablespace + rmtree("$tempdir/tbackup/tblspc_verify/${tblspc1UnloggedBackupPath}_init"); + # create a new file in backup cluster + open my $new_file_none, '>', "$tempdir/backup1/postgresql.new" or die "unable to create file postgresql.new"; + close $new_file_none; + # append text to file to change file size + open my $modify_file_none, '>>', "$tempdir/backup1/postgresql.conf" or die "unable to open file postgresql.conf"; + print $modify_file_none "port = 5555\n"; + close $modify_file_none; + $node->command_checks_all( + [ 'pg_basebackup', '-D', "$tempdir/backup1", '--verify-backup' ], + 0, + [qr{^$}], + [ + qr/\Qpg_basebackup: file "$tblspc1UnloggedPath\E_init" is present in manifest but missing from the backup/, + qr/\Qpg_basebackup: file "postgresql.new" is present in backup but not in manifest/, + qr/\Qpg_basebackup: file "postgresql.conf" has size\E/ + ], + 'backup verification without checksum detected removed file, new file and changed size of a file'); + rmtree("$tempdir/backup1"); + rmtree("$tempdir/tbackup/tblspc_verify"); + + # take backup with SHA256 manifest checksum + $node->command_ok( + [ + 'pg_basebackup', '-D', "$tempdir/backup1", '--manifest-checksums', 'SHA256', + "-T$shorter_tempdir/tblspc1=$tempdir/tbackup/tblspc_verify" + ], + 'plain backup with SHA256 manifest checksum with tablespaces succeeds'); + ok(-f "$tempdir/backup1/PG_VERSION", 'backup1 created'); + ok(-f "$tempdir/backup1/backup_manifest", 'backup_manifest file is present'); + # verify plain backup with SHA256 manifest-checksum. + $node->command_ok([ 'pg_basebackup', '-D', "$tempdir/backup1", '--verify-backup'], + 'plain backup with SHA256 manifest-checksum is verified'); + + # --verify-backup with SHA256 checksums should detect removed file, new file or modified file with and without size change + # delete a file from mapped tablespace + rmtree("$tempdir/tbackup/tblspc_verify/${tblspc1UnloggedBackupPath}_init"); + # create a new file in backup cluster + open my $new_file_sha256, '>', "$tempdir/backup1/postgresql.new" or die "unable to create file postgresql.new"; + close $new_file_sha256; + # append text to a file to change file size + open my $modify_file_sha256, '>>', "$tempdir/backup1/postgresql.conf" or die "unable to open file postgresql.conf"; + print $modify_file_sha256 "port = 5555\n"; + close $modify_file_sha256; + # replace text with same size from a file + open my $same_size_file_sha256, '>', "$tempdir/backup1/PG_VERSION" or die "unable to open file PG_VERSION"; + print $same_size_file_sha256 "00"; + close $same_size_file_sha256; + $node->command_checks_all( + [ 'pg_basebackup', '-D', "$tempdir/backup1", '--verify-backup' ], + 0, + [qr{^$}], + [ + qr/\Qpg_basebackup: file "$tblspc1UnloggedPath\E_init" is present in manifest but missing from the backup/, + qr/\Qpg_basebackup: file "postgresql.new" is present in backup but not in manifest/, + qr/\Qpg_basebackup: file "PG_VERSION" has checksum/, + qr/\Qpg_basebackup: file "postgresql.conf" has size/, + qr/\Qpg_basebackup: file "postgresql.conf" has checksum\E/ + ], + 'backup verification with SHA256 checksum detected removed file, new file and modified file with and without size change'); + rmtree("$tempdir/backup1"); + rmtree("$tempdir/tbackup/tblspc_verify"); + + # take backup with --manifest-checksums=CRC32C + $node->command_ok( + [ + 'pg_basebackup', '-D', "$tempdir/backup1", '--manifest-checksums', 'CRC32C', + "-T$shorter_tempdir/tblspc1=$tempdir/tbackup/tblspc_verify" + ], + 'plain backup with manifest-checksums=CRC32C with tablespaces succeeds'); + ok(-f "$tempdir/backup1/PG_VERSION", 'backup1 created'); + ok(-f "$tempdir/backup1/backup_manifest",'backup_manifest file is present'); + # verify plain backup with CRC32C manifest-checksum. + $node->command_ok([ 'pg_basebackup', '-D', "$tempdir/backup1", '--verify-backup'], + 'plain backup with CRC32C manifest-checksum is verified'); + + # --verify-backup with CRC32C checksums should detect removed file, new file or modified file with and without size change + # delete a file from mapped tablespace + rmtree("$tempdir/tbackup/tblspc_verify/${tblspc1UnloggedBackupPath}_init"); + # create a new file in backup cluster + open my $new_file_crc32c, '>', "$tempdir/backup1/postgresql.new" or die "unable to create file postgresql.new"; + close $new_file_crc32c; + # append text to a file to change file size + open my $modify_file_crc32c, '>>', "$tempdir/backup1/postgresql.conf" or die "unable to open file postgresql.conf"; + print $modify_file_crc32c "port = 5555\n"; + close $modify_file_crc32c; + # replace text with same size from a file + open my $same_size_file_crc32c, '>', "$tempdir/backup1/PG_VERSION" or die "unable to open file PG_VERSION"; + print $same_size_file_crc32c "00"; + close $same_size_file_crc32c; + $node->command_checks_all( + [ 'pg_basebackup', '-D', "$tempdir/backup1", '--verify-backup' ], + 0, + [qr{^$}], + [ + qr/\Qpg_basebackup: file "$tblspc1UnloggedPath\E_init" is present in manifest but missing from the backup/, + qr/\Qpg_basebackup: file "postgresql.new" is present in backup but not in manifest/, + qr/\Qpg_basebackup: file "PG_VERSION" has checksum/, + qr/\Qpg_basebackup: file "postgresql.conf" has size/, + qr/\Qpg_basebackup: file "postgresql.conf" has checksum\E/ + ], + 'backup verification with CRC32C checksum detected removed file, new file and modified file with and without size change'); + rmtree("$tempdir/backup1"); + rmtree("$tempdir/tbackup/tblspc_verify"); + mkdir "$tempdir/tbl=spc2"; $node->safe_psql('postgres', "DROP TABLE test1;"); $node->safe_psql('postgres', "DROP TABLE tblspc1_unlogged;"); -- 1.8.3.1