logical replication: could not create file "state.tmp": File exists

Grigory Smolkin <[email protected]> Sat, 30 Nov 2019 15:09:39 +0300

Newsgroups	gmane.comp.db.postgresql.bugs
Message-ID	<[email protected]>

Hello!

One of my colleagues encountered an out of space condition, which broke 
his logical replication setup.
It`s manifested with the following errors:

ERROR:  could not receive data from WAL stream: ERROR:  could not create 
file "pg_replslot/some_sub/state.tmp": File exists

I`ve digged a bit into this problem, and it`s turned out that in 
SaveSlotToPath() temp file for replication slot is opened with 'O_CREAT 
| O_EXCL' flags, which makes this routine as not very reentrant.

Since an exclusive lock is taken before temp file creation, I think it 
should be safe to replace O_EXCL with O_TRUNC.
Script to reproduce and patch are attached.

-- 
Grigory Smolkin
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

replication_bug.sh (application/x-shellscript, 1.4 KB)

#!/bin/bash

set -exu
set -o errexit

PREFIX=/home/gsmol/task/13_devel
PGDATA_PUB=${PREFIX}/data_pub
PGDATA_SUB=${PREFIX}/data_sub
PORT_PUB=15431
PORT_SUB=15430

${PREFIX}/bin/pg_ctl stop -m immediate -D ${PGDATA_PUB} || echo "all is well"
${PREFIX}/bin/pg_ctl stop -m immediate -D ${PGDATA_SUB} || echo "all is well"
rm -rf ${PGDATA_PUB} ${PGDATA_SUB}

${PREFIX}/bin/initdb -k -D ${PGDATA_PUB}
echo "port = ${PORT_PUB}" >> ${PGDATA_PUB}/postgresql.auto.conf
echo "wal_level = logical" >> ${PGDATA_PUB}/postgresql.auto.conf

${PREFIX}/bin/initdb -k -D ${PGDATA_SUB}
echo "port = ${PORT_SUB}" >> ${PGDATA_SUB}/postgresql.auto.conf

# start publisher cluster
${PREFIX}/bin/pg_ctl start -D ${PGDATA_PUB}

# start subscriber cluster
${PREFIX}/bin/pg_ctl start -D ${PGDATA_SUB}

# generate some data
${PREFIX}/bin/pgbench -i -s 10 -p ${PORT_PUB} postgres

# PUBLICATION
${PREFIX}/bin/psql -p ${PORT_PUB} -d postgres -c "CREATE PUBLICATION mypub FOR table pgbench_accounts"

# SUBSCRIPTION
${PREFIX}/bin/psql -p ${PORT_SUB} -d postgres -c "CREATE TABLE pgbench_accounts(aid int not null PRIMARY KEY, bid int,abalance int,filler char(84))"
${PREFIX}/bin/psql -p ${PORT_SUB} -d postgres -c "CREATE SUBSCRIPTION mysub CONNECTION 'host=localhost port=${PORT_PUB} dbname=postgres' PUBLICATION mypub"

# Imitate out_of_space/write_operation_error
touch ${PGDATA_PUB}/pg_replslot/mysub/state.tmp

# generate some write load
${PREFIX}/bin/pgbench -T100 -P1 -p ${PORT_PUB} postgres

0001-logical_replication_fix.patch (text/x-patch, 543 B)

diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 21ae8531b3..ef415b44c8 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -1253,7 +1253,7 @@ SaveSlotToPath(ReplicationSlot *slot, const char *dir, int elevel)
 	sprintf(tmppath, "%s/state.tmp", dir);
 	sprintf(path, "%s/state", dir);
 
-	fd = OpenTransientFile(tmppath, O_CREAT | O_EXCL | O_WRONLY | PG_BINARY);
+	fd = OpenTransientFile(tmppath, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
 	if (fd < 0)
 	{
 		ereport(elevel,