Re: Endless loop in FAULT state with two virtual_instances on same interface

The issue was observed and was fixed in 1.2.11. From the changelog:

* vrrp: Release and refresh properly fd hash index. Rehashing into the same loop as releasing is not really the best idea... Reworked a little previous patch to properly release hash entries related to the same instance and then hash it back on new fd.

The fix was to iterate through the vrrp list again and rehash the ones with the same fd_in. I would say that my fix is slightly more efficient, but it makes little to no difference.

Thanks and sorry for the mail spam!




On Wednesday, February 26, 2014 11:12 PM, Catalin Gheorghe <[email protected]> wrote:
 
Hello,

I am using keepalived-1.2.10 with lvs support disabled, so only for VRRP. I came upon the following situation:
- configure two distinct virtual instances on the same interface
- start keepalived. The two instances will transition to MASTER, as it is the only VR on the line
- unplug cable -> interface goes down -> both instances go to FAULT state
- plug-in cable -> interface goes up -> keepalived process goes into an endless loop, consuming CPU and becoming unresponsive


Due to the fact that it is quite a simple setup, I am a little surprised to encounter this, and not 100% confident that I didn't missconfigure something. The setup information is in the last part of the email; please let me know if I did something I wasn't suppose to.

I will continue with the inner details of the problem, as I found them.


From what I've seen, the endless loop is in set_vrrp_fd_bucket() ( called by new_vrrp_socket(), called by vrrp_fault()).
When the interface comes back up, the current/old fd_in and fd_out sockets of that vrrp instance are closed and the vrrp instance is removed from the fd bucket (vrrp_data->vrrp_index_fd). Then, the sockets are reopened (the file descriptors will be the same), and the instance re-added to the bucket (same fd_in -> same list in vrrp_data->vrrp_index_fd). When finally calling set_vrrp_fd_bucket(old_fd, vrrp), this does the following:

- goes to the list in the bucket of the old_fd (vrrp_data->vrrp_index_fd[old_fd%1024+1]

- iterates on the list starting with the head (the list hast two vrrp instances)
- removes the first one (vrrp1) from the list, updates its fd_in and fd_out to the newly opened ones
 (the same in this case), adds it to the bucket again (alloc_vrrp_fd_bucket(vrrp_ptr)) ---> ! because the fd's are the same, it will be added to the end of the same list!
- we now have vrrp2, vrrp1 in the current list
- iterates to vrrp2 and does the same thing -> vrrp1, vrrp2
- so on...

The solution that I found is to iterate the list starting form the tail. This way, you don't iterate over elements that will be added to the end of list during this process. The solution is in the git diff attached.

Please let me know what do you think. I admit that it is not tested thoroughly.

Thanks!
Catalin Gheorghe



Setup information:
0. Ubuntu 12.04, x86_64, 2 NIC -> eth0 and eth2

1. keepalived-1.2.10, clean, compiled from source


2. configured with

$ ./configure --prefix=/home/catalin/work/vrrp/keepalived/install-clean --disable-lvs --disable-lvs-syncd --enable-snmp
3. keepalived configuration file (VI_1 and VI_3 on same interface)
! Configuration File for keepalived
>
>
>vrrp_instance VI_1 {
>    state BACKUP
>    interface eth2
>    virtual_router_id 20
>    priority 100
>    advert_int 1
>    virtual_ipaddress {
>        192.168.0.200
>    }
>}
>
>
>vrrp_instance VI_2 {
>    state BACKUP
>    interface eth0
>    virtual_router_id 35
>    priority 200
>    advert_int 1
>    virtual_ipaddress {
>        10.10.10.10
>    }
>}
>
>
>vrrp_instance VI_3 {
>    state BACKUP
>    interface eth2
>    virtual_router_id 40
>    priority 200
>    advert_int 1
>    virtual_ipaddress {
>        20.20.20.20
>    }
>}
>
>
>4. keepalived cmd line arguments
>
>
>-P -D -d -n -l -f /home/catalin/work/vrrp/keepalived/install-clean/etc/keepalived/keepalived.conf.me -p /home/catalin/work/vrrp/keepalived/install-clean/sbin/keepalived.pid -r /home/catalin/work/vrrp/keepalived/install-clean/sbin/keepalived_vrrp.pid
>
>
>
>
>
>

------------------------------------------------------------------------------
Flow-based real-time traffic analytics software. Cisco certified tool.
Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer
Customize your own dashboards, set traffic alerts and generate reports.
Network behavioral analysis & security monitoring. All-in-one tool.
http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk
_______________________________________________
Keepalived-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/keepalived-devel

------------------------------------------------------------------------------
Flow-based real-time traffic analytics software. Cisco certified tool.
Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer
Customize your own dashboards, set traffic alerts and generate reports.
Network behavioral analysis & security monitoring. All-in-one tool.
http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk

_______________________________________________
Keepalived-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/keepalived-devel