Streaming Replication Error

<[email protected]> Fri, 20 Dec 2019 16:08:33 +0000
Newsgroups gmane.comp.db.postgresql.admin
Message-ID <2A91BEF8171A5349931391E0C721CC537D772A5F@CPEMS-KPN501.KPNCNL.LOCAL>
--_000_2A91BEF8171A5349931391E0C721CC537D772A5FCPEMSKPN501KPNC_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi Experts,

I have set up streaming replication on PostgreSQL 12.1

The master and slave are configured as below and WAL files are accumulating=
 on the master.

However, something is wrong as I get complaints that the WAL files are miss=
ing after a pg_restore on MASTER

MASTER

postgres@srvm:~$ 2019-12-20 16:35:07.910 CET [1334] replicator@[unknown] ER=
ROR:  requested WAL segment 000000010000000100000076 has already been remov=
ed
2019-12-20 16:35:12.920 CET [1338] replicator@[unknown] ERROR:  requested W=
AL segment 000000010000000100000076 has already been removed
2019-12-20 16:35:17.925 CET [1340] replicator@[unknown] ERROR:  requested W=
AL segment 000000010000000100000076 has already been removed
2019-12-20 16:35:22.932 CET [1362] replicator@[unknown] ERROR:  requested W=
AL segment 000000010000000100000076 has already been removed
2019-12-20 16:35:27.935 CET [1364] replicator@[unknown] ERROR:  requested W=
AL segment 000000010000000100000076 has already been removed
2019-12-20 16:35:32.942 CET [1365] replicator@[unknown] ERROR:  requested W=
AL segment 000000010000000100000076 has already been removed
2019-12-20 16:35:37.948 CET [1366] replicator@[unknown] ERROR:  requested W=
AL segment 000000010000000100000076 has already been removed
2019-12-20 16:35:42.954 CET [1367] replicator@[unknown] ERROR:  requested W=
AL segment 000000010000000100000076 has already been removed


SLAVE

postgres@srvs:~$ 2019-12-20 16:36:53.027 CET [21978] LOG:  started streamin=
g WAL from primary at 1/76000000 on timeline 1
2019-12-20 16:36:53.027 CET [21978] FATAL:  could not receive data from WAL=
 stream: ERROR:  requested WAL segment 000000010000000100000076 has already=
 been removed
2019-12-20 16:36:58.029 CET [21979] LOG:  started streaming WAL from primar=
y at 1/76000000 on timeline 1
2019-12-20 16:36:58.029 CET [21979] FATAL:  could not receive data from WAL=
 stream: ERROR:  requested WAL segment 000000010000000100000076 has already=
 been removed
2019-12-20 16:37:03.040 CET [21980] LOG:  started streaming WAL from primar=
y at 1/76000000 on timeline 1
2019-12-20 16:37:03.040 CET [21980] FATAL:  could not receive data from WAL=
 stream: ERROR:  requested WAL segment 000000010000000100000076 has already=
 been removed
2019-12-20 16:37:08.042 CET [21981] LOG:  started streaming WAL from primar=
y at 1/76000000 on timeline 1
2019-12-20 16:37:08.042 CET [21981] FATAL:  could not receive data from WAL=
 stream: ERROR:  requested WAL segment 000000010000000100000076 has already=
 been removed


Then the pg_basebackup is run and the slave started.

The slave has all the data as of the time of the backup, but no new data fr=
om the WAL files, and the error above.

max_wal_senders =3D 10
wal_keep_segments =3D 120

What have I mis-configured? Do we need to enabled archive_mode =3D on for s=
treaming replication?

--_000_2A91BEF8171A5349931391E0C721CC537D772A5FCPEMSKPN501KPNC_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0cm;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;
	mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:#0563C1;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:#954F72;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal-compose;
	font-family:"Calibri",sans-serif;
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-family:"Calibri",sans-serif;
	mso-fareast-language:EN-US;}
@page WordSection1
	{size:612.0pt 792.0pt;
	margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3D"NL" link=3D"#0563C1" vlink=3D"#954F72">
<div class=3D"WordSection1">
<p class=3D"MsoNormal"><span lang=3D"EN-GB">Hi Experts,<o:p></o:p></span></=
p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">I have set up streaming replica=
tion on PostgreSQL 12.1<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">The master and slave are config=
ured as below and WAL files are accumulating on the master.
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">However, something is wrong as =
I get complaints that the WAL files are missing after a pg_restore on MASTE=
R<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><b><u><span lang=3D"EN-GB">MASTER<o:p></o:p></span><=
/u></b></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">postgres@srvm:~$ 2019-12-20 16:=
35:07.910 CET [1334] replicator@[unknown] ERROR:&nbsp; requested WAL segmen=
t 000000010000000100000076 has already been removed<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">2019-12-20 16:35:12.920 CET [13=
38] replicator@[unknown] ERROR:&nbsp; requested WAL segment 000000010000000=
100000076 has already been removed<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">2019-12-20 16:35:17.925 CET [13=
40] replicator@[unknown] ERROR:&nbsp; requested WAL segment 000000010000000=
100000076 has already been removed<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">2019-12-20 16:35:22.932 CET [13=
62] replicator@[unknown] ERROR:&nbsp; requested WAL segment 000000010000000=
100000076 has already been removed<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">2019-12-20 16:35:27.935 CET [13=
64] replicator@[unknown] ERROR:&nbsp; requested WAL segment 000000010000000=
100000076 has already been removed<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">2019-12-20 16:35:32.942 CET [13=
65] replicator@[unknown] ERROR:&nbsp; requested WAL segment 000000010000000=
100000076 has already been removed<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">2019-12-20 16:35:37.948 CET [13=
66] replicator@[unknown] ERROR:&nbsp; requested WAL segment 000000010000000=
100000076 has already been removed<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">2019-12-20 16:35:42.954 CET [13=
67] replicator@[unknown] ERROR:&nbsp; requested WAL segment 000000010000000=
100000076 has already been removed<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><b><u><span lang=3D"EN-GB">SLAVE<o:p></o:p></span></=
u></b></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">postgres@srvs:~$ 2019-12-20 16:=
36:53.027 CET [21978] LOG:&nbsp; started streaming WAL from primary at 1/76=
000000 on timeline 1<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">2019-12-20 16:36:53.027 CET [21=
978] FATAL:&nbsp; could not receive data from WAL stream: ERROR:&nbsp; requ=
ested WAL segment 000000010000000100000076 has already been removed<o:p></o=
:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">2019-12-20 16:36:58.029 CET [21=
979] LOG:&nbsp; started streaming WAL from primary at 1/76000000 on timelin=
e 1<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">2019-12-20 16:36:58.029 CET [21=
979] FATAL:&nbsp; could not receive data from WAL stream: ERROR:&nbsp; requ=
ested WAL segment 000000010000000100000076 has already been removed<o:p></o=
:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">2019-12-20 16:37:03.040 CET [21=
980] LOG:&nbsp; started streaming WAL from primary at 1/76000000 on timelin=
e 1<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">2019-12-20 16:37:03.040 CET [21=
980] FATAL:&nbsp; could not receive data from WAL stream: ERROR:&nbsp; requ=
ested WAL segment 000000010000000100000076 has already been removed<o:p></o=
:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">2019-12-20 16:37:08.042 CET [21=
981] LOG:&nbsp; started streaming WAL from primary at 1/76000000 on timelin=
e 1<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">2019-12-20 16:37:08.042 CET [21=
981] FATAL:&nbsp; could not receive data from WAL stream: ERROR:&nbsp; requ=
ested WAL segment 000000010000000100000076 has already been removed<o:p></o=
:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">Then the pg_basebackup is run a=
nd the slave started.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">The slave has all the data as o=
f the time of the backup, but no new data from the WAL files, and the error=
 above.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><b><span lang=3D"EN-GB"><o:p>&nbsp;</o:p></span></b>=
</p>
<p class=3D"MsoNormal"><b><span lang=3D"EN-GB">max_wal_senders =3D 10&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <o:p></o:p></span></b></p>
<p class=3D"MsoNormal"><b><span lang=3D"EN-GB">wal_keep_segments =3D 120<o:=
p></o:p></span></b></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">What have I mis-configured? Do =
we need to enabled archive_mode =3D on for streaming replication?<o:p></o:p=
></span></p>
</div>
</body>
</html>

--_000_2A91BEF8171A5349931391E0C721CC537D772A5FCPEMSKPN501KPNC_--