Re: [HACKERS] odd output in restore mode
Simon Riggs <[email protected]>
| Newsgroups | gmane.comp.db.postgresql.devel.patches |
|---|---|
| Message-ID | <[email protected]> |
On Mon, 2008-06-30 at 19:29 -0400, Andrew Dunstan wrote: > > Bruce Momjian wrote: > > Dave Page wrote: > > > >> On Sun, May 18, 2008 at 1:38 PM, Andrew Dunstan <[email protected]> wrote: > >> > >>> I don't know how you tested, > >>> > >> Copy a large file across a relatively slow network, and check the size > >> on the destination drive before it finishes. > >> > >> > >>> but could you please repeat the test with > >>> GnuWin32's cp.exe? If it doesn't preallocate the space then I think our way > >>> forward is reasonably clear: > >>> > >> It does not pre-allocate. > >> > >> > >>> . we recommend its use for Windows archive_command settings > >>> . we provide the delay kluge as switchable behaviour on Windows instead of > >>> having it always on. > >>> > >> Sounds reasonable to me. > >> > > > > Are there any changes we need to make here? > > > > > > Yes. Simon has promised a patch to do the above. Patch implements * recommendation to use GnuWin32 cp on Windows * provide "holdtime" delay, default 0 (on all platforms) * default stays same on Windows="copy" to ensure people upgrading don't get stung Patch should be backpatched to 8.3, plus to CVS HEAD. We should recommend in next 8.3 release notes that people use "-p" or "-l" rather than just letting it default. Will add permalink to Wiki when patch appears in archives. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-patches mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-patches
pg_standby_win32.v1.patch
(text/x-patch, 9.7 KB)
Index: contrib/pg_standby/pg_standby.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/contrib/pg_standby/pg_standby.c,v
retrieving revision 1.12
diff -c -r1.12 pg_standby.c
*** contrib/pg_standby/pg_standby.c 17 May 2008 01:28:21 -0000 1.12
--- contrib/pg_standby/pg_standby.c 1 Jul 2008 08:18:50 -0000
***************
*** 44,49 ****
--- 44,50 ----
/* Options and defaults */
int sleeptime = 5; /* amount of time to sleep between file checks */
+ int holdtime = 0; /* amount of time to wait once file appears full */
int waittime = -1; /* how long we have been waiting, -1 no wait
* yet */
int maxwaittime = 0; /* how long are we prepared to wait for? */
***************
*** 67,75 ****
char exclusiveCleanupFileName[MAXPGPATH]; /* the file we need to
* get from archive */
! #define RESTORE_COMMAND_COPY 0
! #define RESTORE_COMMAND_LINK 1
! int restoreCommandType;
#define XLOG_DATA 0
#define XLOG_HISTORY 1
--- 68,83 ----
char exclusiveCleanupFileName[MAXPGPATH]; /* the file we need to
* get from archive */
! #define RESTORE_COMMAND_CP 0
! #define RESTORE_COMMAND_LN 1
! #define RESTORE_COMMAND_WIN32_MKLINK 2
! #define RESTORE_COMMAND_WIN32_COPY 3
! #ifdef WIN32
! int restoreCommandType = RESTORE_COMMAND_WIN32_COPY;
! #else
! int restoreCommandType = RESTORE_COMMAND_CP;
! #endif
!
#define XLOG_DATA 0
#define XLOG_HISTORY 1
***************
*** 113,142 ****
{
#ifdef WIN32
snprintf(WALFilePath, MAXPGPATH, "%s\\%s", archiveLocation, nextWALFileName);
switch (restoreCommandType)
{
! case RESTORE_COMMAND_LINK:
SET_RESTORE_COMMAND("mklink", WALFilePath, xlogFilePath);
! case RESTORE_COMMAND_COPY:
! default:
SET_RESTORE_COMMAND("copy", WALFilePath, xlogFilePath);
break;
! }
! #else
! snprintf(WALFilePath, MAXPGPATH, "%s/%s", archiveLocation, nextWALFileName);
! switch (restoreCommandType)
! {
! case RESTORE_COMMAND_LINK:
! #if HAVE_WORKING_LINK
SET_RESTORE_COMMAND("ln -s -f", WALFilePath, xlogFilePath);
break;
! #endif
! case RESTORE_COMMAND_COPY:
! default:
SET_RESTORE_COMMAND("cp", WALFilePath, xlogFilePath);
break;
}
- #endif
/*
* This code assumes that archiveLocation is a directory You may wish to
--- 121,148 ----
{
#ifdef WIN32
snprintf(WALFilePath, MAXPGPATH, "%s\\%s", archiveLocation, nextWALFileName);
+ #else
+ snprintf(WALFilePath, MAXPGPATH, "%s/%s", archiveLocation, nextWALFileName);
+ #endif
switch (restoreCommandType)
{
! case RESTORE_COMMAND_WIN32_MKLINK:
SET_RESTORE_COMMAND("mklink", WALFilePath, xlogFilePath);
! case RESTORE_COMMAND_WIN32_COPY:
SET_RESTORE_COMMAND("copy", WALFilePath, xlogFilePath);
break;
! case RESTORE_COMMAND_LN:
SET_RESTORE_COMMAND("ln -s -f", WALFilePath, xlogFilePath);
break;
! case RESTORE_COMMAND_CP:
SET_RESTORE_COMMAND("cp", WALFilePath, xlogFilePath);
break;
+ default:
+ fprintf(stderr, "pg_standby: unknown restore command\n");
+ fflush(stderr);
+ exit(2);
+ break;
}
/*
* This code assumes that archiveLocation is a directory You may wish to
***************
*** 175,191 ****
}
else if (stat_buf.st_size == XLOG_SEG_SIZE)
{
- #ifdef WIN32
-
/*
! * Windows reports that the file has the right number of bytes
* even though the file is still being copied and cannot be opened
! * by pg_standby yet. So we wait for sleeptime secs before
* attempting to restore. If that is not enough, we will rely on
* the retry/holdoff mechanism.
*/
! pg_usleep(sleeptime * 1000000L);
! #endif
nextWALFileType = XLOG_DATA;
return true;
}
--- 181,196 ----
}
else if (stat_buf.st_size == XLOG_SEG_SIZE)
{
/*
! * Some utilities report that the file has the right number of bytes
* even though the file is still being copied and cannot be opened
! * by pg_standby yet. So we wait for holdtime secs before
* attempting to restore. If that is not enough, we will rely on
* the retry/holdoff mechanism.
*/
! if (holdtime > 0)
! pg_usleep(holdtime * 1000000L);
!
nextWALFileType = XLOG_DATA;
return true;
}
***************
*** 441,448 ****
--- 446,455 ----
fprintf(stderr, " -d generate lots of debugging output (testing only)\n");
fprintf(stderr, " -k NUMFILESTOKEEP if RESTARTWALFILE not used, removes files prior to limit (0 keeps all)\n");
fprintf(stderr, " -l links into archive (leaves file in archive)\n");
+ fprintf(stderr, " -p always uses GNU compatible 'cp' command on all platforms\n");
fprintf(stderr, " -r MAXRETRIES max number of times to retry, with progressive wait (default=3)\n");
fprintf(stderr, " -s SLEEPTIME seconds to wait between file checks (min=1, max=60, default=5)\n");
+ fprintf(stderr, " -h HOLDTIME seconds to wait once file appears full (min=0, max=60, default=0)\n");
fprintf(stderr, " -t TRIGGERFILE defines a trigger file to initiate failover (no default)\n");
fprintf(stderr, " -w MAXWAITTIME max seconds to wait for a file (0=no limit)(default=0)\n");
fflush(stderr);
***************
*** 463,474 ****
(void) signal(SIGINT, sighandler);
(void) signal(SIGQUIT, sighandler);
! while ((c = getopt(argc, argv, "cdk:lr:s:t:w:")) != -1)
{
switch (c)
{
case 'c': /* Use copy */
! restoreCommandType = RESTORE_COMMAND_COPY;
break;
case 'd': /* Debug mode */
debug = true;
--- 470,488 ----
(void) signal(SIGINT, sighandler);
(void) signal(SIGQUIT, sighandler);
! while ((c = getopt(argc, argv, "cdh:k:lpr:s:t:w:")) != -1)
{
switch (c)
{
case 'c': /* Use copy */
! #ifdef WIN32
! restoreCommandType = RESTORE_COMMAND_WIN32_COPY;
! #else
! restoreCommandType = RESTORE_COMMAND_CP;
! #endif
! break;
! case 'p': /* Use cp */
! restoreCommandType = RESTORE_COMMAND_CP;
break;
case 'd': /* Debug mode */
debug = true;
***************
*** 483,489 ****
}
break;
case 'l': /* Use link */
! restoreCommandType = RESTORE_COMMAND_LINK;
break;
case 'r': /* Retries */
maxretries = atoi(optarg);
--- 497,511 ----
}
break;
case 'l': /* Use link */
! #ifdef WIN32
! restoreCommandType = RESTORE_COMMAND_WIN32_MKLINK;
! #else
! #if HAVE_WORKING_LINK
! restoreCommandType = RESTORE_COMMAND_LN;
! #else
! restoreCommandType = RESTORE_COMMAND_CP;
! #endif
! #endif
break;
case 'r': /* Retries */
maxretries = atoi(optarg);
***************
*** 503,508 ****
--- 525,539 ----
exit(2);
}
break;
+ case 'h': /* Hold time */
+ holdtime = atoi(optarg);
+ if (holdtime < 0 || holdtime > 60)
+ {
+ fprintf(stderr, "usage: pg_standby -h holdtime incorrectly set\n");
+ usage();
+ exit(2);
+ }
+ break;
case 't': /* Trigger file */
triggerPath = optarg;
if (CheckForExternalTrigger())
Index: doc/src/sgml/standby.sgml
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/doc/src/sgml/standby.sgml,v
retrieving revision 1.1
diff -c -r1.1 standby.sgml
*** doc/src/sgml/standby.sgml 10 Nov 2007 23:30:46 -0000 1.1
--- doc/src/sgml/standby.sgml 1 Jul 2008 08:29:16 -0000
***************
*** 136,141 ****
--- 136,149 ----
</entry>
</row>
<row>
+ <entry>-p</entry>
+ <entry>
+ <para>
+ always use cp command on any platform
+ </para>
+ </entry>
+ </row>
+ <row>
<entry>-r maxretries</entry>
<entry>
<para>
***************
*** 160,165 ****
--- 168,182 ----
</entry>
</row>
<row>
+ <entry>-h holdtime</entry>
+ <entry>
+ the number of seconds to wait once a file appears full.
+ This is required with some utilities, such as Win32 Copy, which
+ sets the filesize before it has completed the copy process.
+ <literal>Default=0</literal>
+ </entry>
+ </row>
+ <row>
<entry>-t triggerfile</entry>
<entry>
the presence of the triggerfile will cause recovery to end
***************
*** 228,234 ****
*not* in the restore_command, in 8.2, 8.1, 8.0 on Windows.
</para>
<programlisting>
! restore_command = 'pg_standby -c -d -s 5 -w 0 -t C:\pgsql.trigger.5442
..\archive %f %p 2>> standby.log'
</programlisting>
<para>
--- 245,251 ----
*not* in the restore_command, in 8.2, 8.1, 8.0 on Windows.
</para>
<programlisting>
! restore_command = 'pg_standby -c -d -s 5 -h 5 -w 0 -t C:\pgsql.trigger.5442
..\archive %f %p 2>> standby.log'
</programlisting>
<para>
***************
*** 238,248 ****
--- 255,272 ----
<listitem><para>use a copy command to restore WAL files from archive</para></listitem>
<listitem><para>produce logfile output in standby.log</para></listitem>
<listitem><para>sleep for 5 seconds between checks for next WAL file is full</para></listitem>
+ <listitem><para>sleep for 5 second after the file appears full</para></listitem>
<listitem><para>never timeout if file not found</para></listitem>
<listitem><para>stop waiting when a trigger file called C:\pgsql.trigger.5442 appears</para></listitem>
</itemizedlist>
</listitem>
</itemizedlist>
+ <para>
+ On Windows you are advised to use GNUWin32's cp utility. The cp utility
+ does not set filesize until file copying is complete, so there is no
+ need to use the -h option to introduce additional delays to ensure
+ correctness. See <ulink url="http://gnuwin32.sourceforge.net/"></ulink>
+ </para>
</sect2>
</sect1>