iscsi target process not starting after upgrade to 14.4

Hi all,


I've got a strange problem with one of my systems after upgrading from=20
13.x to 14.4.


I have found this bug:=20
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D271460


which describes possibly the problem (maybe in a different context) and=20
a fix at least.


I don't have 2x ctld processes or the setup described but what I'm=20
finding is that the LUNs are inaccessible and showing as "connection=20
refused" on the Initiator node after a startup or reboot of the Target.


It's almost like the ctld process is hung or fails to start properly,=20
there is definitely a pid file and I can see output using the command:


ctladm portlist -v


Port Online Frontend Name=C2=A0 =C2=A0 =C2=A0pp vp
0=C2=A0 =C2=A0 NO=C2=A0 =C2=A0 =C2=A0camsim=C2=A0 =C2=A0camsim=C2=A0 =C2=A0=
0=C2=A0 0=C2=A0 naa.5000000c046ef301
 =C2=A0 Target: naa.5000000c046ef300
 =C2=A0 All LUNs mapped
 =C2=A0 =C2=A0 =C2=A0 port_type=3D8
1=C2=A0 =C2=A0 YES=C2=A0 =C2=A0 ioctl=C2=A0 =C2=A0 ioctl=C2=A0 =C2=A0 0=C2=
=A0 0
 =C2=A0 All LUNs mapped
 =C2=A0 =C2=A0 =C2=A0 port_type=3D4
2=C2=A0 =C2=A0 YES=C2=A0 =C2=A0 tpc=C2=A0 =C2=A0 =C2=A0 tpc=C2=A0 =C2=A0 =
=C2=A0 0=C2=A0 0
 =C2=A0 All LUNs mapped
 =C2=A0 =C2=A0 =C2=A0 port_type=3D8
3=C2=A0 =C2=A0 YES=C2=A0 =C2=A0 iscsi=C2=A0 =C2=A0 iscsi=C2=A0 =C2=A0 257=
 1 iqn.2008-10.com.xxx:target0,t,0x0101
 =C2=A0 Target: iqn.2008-10.com.xxx:target0
 =C2=A0 Initiator 0: iqn.1994-09.org.freebsd:xxx,i,0x8063a7ae15f0
 =C2=A0 LUN 0: 0
 =C2=A0 =C2=A0 =C2=A0 port_type=3D16
 =C2=A0 =C2=A0 =C2=A0 cfiscsi_state=3D1
 =C2=A0 =C2=A0 =C2=A0 cfiscsi_target=3Diqn.2008-10.com.xxx:target0
 =C2=A0 =C2=A0 =C2=A0 ctld_portal_group_name=3Dpg0
 =C2=A0 =C2=A0 =C2=A0 cfiscsi_portal_group_tag=3D257
4=C2=A0 =C2=A0 YES=C2=A0 =C2=A0 iscsi=C2=A0 =C2=A0 iscsi=C2=A0 =C2=A0 257=
 2 iqn.2008-10.com.xxx:target1,t,0x0101
 =C2=A0 Target: iqn.2008-10.com.xxx:target1
 =C2=A0 Initiator 0: iqn.1994-09.org.freebsd:xxx,i,0x80e4e737c577
 =C2=A0 LUN 1: 1
 =C2=A0 =C2=A0 =C2=A0 port_type=3D16
 =C2=A0 =C2=A0 =C2=A0 cfiscsi_state=3D1
 =C2=A0 =C2=A0 =C2=A0 cfiscsi_target=3Diqn.2008-10.com.xxx:target1
 =C2=A0 =C2=A0 =C2=A0 ctld_portal_group_name=3Dpg0
 =C2=A0 =C2=A0 =C2=A0 cfiscsi_portal_group_tag=3D257
5=C2=A0 =C2=A0 YES=C2=A0 =C2=A0 iscsi=C2=A0 =C2=A0 iscsi=C2=A0 =C2=A0 257=
 3 iqn.2008-10.com.xxx:target2,t,0x0101
 =C2=A0 Target: iqn.2008-10.com.xxx:target2
 =C2=A0 Initiator 0: iqn.1994-09.org.freebsd:.xxx.com,i,0x8047f6cf2784
 =C2=A0 LUN 3: 2
 =C2=A0 =C2=A0 =C2=A0 port_type=3D16
 =C2=A0 =C2=A0 =C2=A0 cfiscsi_state=3D1
 =C2=A0 =C2=A0 =C2=A0 cfiscsi_target=3Diqn.2008-10.com.xxx:target2
 =C2=A0 =C2=A0 =C2=A0 ctld_portal_group_name=3Dpg0
 =C2=A0 =C2=A0 =C2=A0 cfiscsi_portal_group_tag=3D257
6=C2=A0 =C2=A0 YES=C2=A0 =C2=A0 iscsi=C2=A0 =C2=A0 iscsi=C2=A0 =C2=A0 257=
 4 iqn.2008-10.com.xxx:target3,t,0x0101
 =C2=A0 Target: iqn.2008-10.com.xxx:target3
 =C2=A0 Initiator 0: iqn.1994-09.org.freebsd:.xxx.com,i,0x80109ffe486f
 =C2=A0 LUN 4: 3
 =C2=A0 =C2=A0 =C2=A0 port_type=3D16
 =C2=A0 =C2=A0 =C2=A0 cfiscsi_state=3D1
 =C2=A0 =C2=A0 =C2=A0 cfiscsi_target=3Diqn.2008-10.com.xxx:target3
 =C2=A0 =C2=A0 =C2=A0 ctld_portal_group_name=3Dpg0
 =C2=A0 =C2=A0 =C2=A0 cfiscsi_portal_group_tag=3D257


If I restart the ctld process, it fails to restart.


I end up having to use this command:


ctladm port -d iscsi -r -p DONTCARE -O cfiscsi_portal_group_tag=3DTAG -O=20
cfiscsi_target=3DTARGET


to remove each LUN then finally I can start the process and it works as=20
per usual.


One of the comments by Alan Somers was:


"I've confirmed that the cause of the problem is that ctld opens its=20
pidfile too late.=C2=A0 It reads the current list of targets from the ker=
nel,=20
then reads the config file, then opens its pidfile, and then applies=20
changes based on the differences between the kernel's state and the=20
config file.=C2=A0 But the kernel's state could've changed before the pid=
file=20
got opened.

I've hacked ctld to open the pidfile earlier and verified that this=20
fixes the problem.=C2=A0 However, doing it properly is hard, because the =
code=20
for opening the config file is intermingled with the code for=20
interacting with the kernel.=C2=A0 The biggest problem is the conf_pports=
=20
list, added in 057abcb00413010898f3046f7704444b8f537bab ."


....but I'm wondering if this is the case as ctld was working fine=20
before the upgrade and also works fine post upgrade on other systems=20
that are Targets and have been upgraded to 14.4, so it's an isolated=20
case on a single system.


I'm wondering if perhaps a certain file got corrupted upon merger during=20
the "upgrade" process or perhaps as the system had ZFS "snapshots"=20
installed something got rolled back somehow, even though I removed all=20
the snapshots post upgrade.


Can anyone suggest anything?


Many thanks.


Kaya