Solaris10 IPF and return-rst

I am frustrated trying to get clients to drop TCP connections cleanly
using return-rst.

I am using a Solaris 10 host with its installed ipf package:

 	uname -a: SunOS 5.10 Generic_118833-36 sun4u sparc SUNW,Ultra-60
 	ipf -V:  v4.0.3 (592)

I'm also testing against a bunch of Solaris8 with the latest ipfilter
4.1.28.

These are the relevant rules

 	# Split into incoming and outgoing rule groups
 	block in quick on hme0 all head 100
 	pass out quick on hme0 all head 200
 	block in log level local1.info quick proto tcp from any to any head 110 group 100
 	...
 	# The rule I want to get working
 	block return-rst in log level mail.info quick proto tcp from any to any port = 25 group 110
 	...
 	# Outgoing rule: free and easy access.
 	pass out quick from any to any keep state keep frags group 200

It doesn't seems to work: client hangs and ipfilter host gets multiple
SYN packets as the client retries to connect.  A RST/ACK packet doesn't
get emitted (verified by "snoop" on client).   I know about the outgoing
rule workaround as described in Phil's FAQ, but since I split my incoming
and outgoing rules into separate rule groups, I though the last rule
fixed that.  Anyways, if you read on, it gets stranger.

If I modify the egress rule (the last rule) by removing the "keep
frags" modifier,

 	pass out quick from any to any keep state group 200

connections to Solaris8/ipfilter4.1.28 hosts then closed down properly.

For Solaris10/ipfilter4.0.3, a RST/ACK packets gets returned, but the
client still hangs.  Here is a snoop dump showing the SYN sent out,
and the RST/ACK response (client=xx.xx.xx.xx, ipfilter host=yy.yy.yy.yy).

The client doesn't seem to acknowledge the RST/ACK, and sends another
SYN and these packet pairs go back and forth until the client gives up.
Is it possible the RST/ACK response is somehow corrupted?  I've tested
using Linux/Solaris/Windows clients and they all hang.

Packet 1: xx.xx.xx.xx -> yy.yy.yy.yy (SYN to port 25/tcp):

 	ETHER:  ----- Ether Header -----
 	ETHER:
 	ETHER:  Packet 1 arrived at 20:45:25.14298
 	ETHER:  Packet size = 66 bytes
 	ETHER:  Destination = 8:0:20:cf:3d:3e, Sun
 	ETHER:  Source      = 8:0:20:8a:a:2f, Sun
 	ETHER:  Ethertype = 0800 (IP)
 	ETHER:
 	IP:   ----- IP Header -----
 	IP:
 	IP:   Version = 4
 	IP:   Header length = 20 bytes
 	IP:   Type of service = 0x00
 	IP:         xxx. .... = 0 (precedence)
 	IP:         ...0 .... = normal delay
 	IP:         .... 0... = normal throughput
 	IP:         .... .0.. = normal reliability
 	IP:         .... ..0. = not ECN capable transport
 	IP:         .... ...0 = no ECN congestion experienced
 	IP:   Total length = 52 bytes
 	IP:   Identification = 22482
 	IP:   Flags = 0x4
 	IP:         .1.. .... = do not fragment
 	IP:         ..0. .... = last fragment
 	IP:   Fragment offset = 0 bytes
 	IP:   Time to live = 64 seconds/hops
 	IP:   Protocol = 6 (TCP)
 	IP:   Header checksum = 8844
 	IP:   Source address = xx.xx.xx.xx, xx.xx.xx.xx
 	IP:   Destination address = yy.yy.yy.yy, yy.yy.yy.yy
 	IP:   No options
 	IP:
 	TCP:  ----- TCP Header -----
 	TCP:
 	TCP:  Source port = 38953
 	TCP:  Destination port = 25 (SMTP)
 	TCP:  Sequence number = 3749460709
 	TCP:  Acknowledgement number = 0
 	TCP:  Data offset = 32 bytes
 	TCP:  Flags = 0x02
 	TCP:        0... .... = No ECN congestion window reduced
 	TCP:        .0.. .... = No ECN echo
 	TCP:        ..0. .... = No urgent pointer
 	TCP:        ...0 .... = No acknowledgement
 	TCP:        .... 0... = No push
 	TCP:        .... .0.. = No reset
 	TCP:        .... ..1. = Syn
 	TCP:        .... ...0 = No Fin
 	TCP:  Window = 49640
 	TCP:  Checksum = 0x9fdd
 	TCP:  Urgent pointer = 0
 	TCP:  Options: (12 bytes)
 	TCP:    - Maximum segment size = 1460 bytes
 	TCP:    - No operation
 	TCP:    - Window scale = 0
 	TCP:    - No operation
 	TCP:    - No operation
 	TCP:    - SACK permitted option
 	TCP:
 	SMTP:  ----- SMTP:   -----
 	SMTP:
 	SMTP:  ""
 	SMTP:

Packet 2: yy.yy.yy.yy -> xx.xx.xx.xx (RST/ACK response):

 	ETHER:  ----- Ether Header -----
 	ETHER:
 	ETHER:  Packet 2 arrived at 20:45:25.14329
 	ETHER:  Packet size = 60 bytes
 	ETHER:  Destination = 8:0:20:8a:a:2f, Sun
 	ETHER:  Source      = 8:0:20:cf:3d:3e, Sun
 	ETHER:  Ethertype = 0800 (IP)
 	ETHER:
 	IP:   ----- IP Header -----
 	IP:
 	IP:   Version = 4
 	IP:   Header length = 20 bytes
 	IP:   Type of service = 0x00
 	IP:         xxx. .... = 0 (precedence)
 	IP:         ...0 .... = normal delay
 	IP:         .... 0... = normal throughput
 	IP:         .... .0.. = normal reliability
 	IP:         .... ..0. = not ECN capable transport
 	IP:         .... ...0 = no ECN congestion experienced
 	IP:   Total length = 40 bytes
 	IP:   Identification = 9010
 	IP:   Flags = 0x4
 	IP:         .1.. .... = do not fragment
 	IP:         ..0. .... = last fragment
 	IP:   Fragment offset = 0 bytes
 	IP:   Time to live = 255 seconds/hops
 	IP:   Protocol = 6 (TCP)
 	IP:   Header checksum = fdef
 	IP:   Source address = yy.yy.yy.yy, yy.yy.yy.yy
 	IP:   Destination address = xx.xx.xx.xx, xx.xx.xx.xx
 	IP:   No options
 	IP:
 	TCP:  ----- TCP Header -----
 	TCP:
 	TCP:  Source port = 25
 	TCP:  Destination port = 38953
 	TCP:  Sequence number = 0
 	TCP:  Acknowledgement number = 3749460710
 	TCP:  Data offset = 20 bytes
 	TCP:  Flags = 0x14
 	TCP:        0... .... = No ECN congestion window reduced
 	TCP:        .0.. .... = No ECN echo
 	TCP:        ..0. .... = No urgent pointer
 	TCP:        ...1 .... = Acknowledgement
 	TCP:        .... 0... = No push
 	TCP:        .... .1.. = Reset
 	TCP:        .... ..0. = No Syn
 	TCP:        .... ...0 = No Fin
 	TCP:  Window = 0
 	TCP:  Checksum = 0x7a37
 	TCP:  Urgent pointer = 0
 	TCP:  No options
 	TCP:
 	SMTP:  ----- SMTP:   -----
 	SMTP:
 	SMTP:  ""
 	SMTP:

Joseph Tam <[email protected]>