ipfilter NAT + opensolaris zones + amazon ec2

I have an OpenSolaris 2009.06 (snv_111b) AMI running on Amazon EC2.
I've set it up with a single non-global zone, much as described at
http://blogs.sun.com/seano/entry/opensolaris_ec2_and_zones_a.  But I'm
having a problem with (I think) the ipnat configuration that I can't
find an answer to, and I can't pin down whether the fault is in ec2,
opensolaris, the zone, ipnat, or my configuration.

To summarize that blog post, I have OpenSolaris virtual NICs passing
traffic between the global zone and the other zone.  (EC2 only permits
one exposed IP address per instance.)  IPv4 routing is enabled.  Here
are the commands that start the network configuration on the global
zone:

    dladm create-etherstub stub0
    dladm create-vnic -l stub0 vnic0
    dladm create-vnic -l stub0 vnic1
    ifconfig vnic0 plumb
    ifconfig vnic0 inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255 up
    routeadm -u -e ipv4-forwarding
    svcadm enable network/ipfilter

vnic1 is activated with IP 192.168.0.101 within the non-global zone.

ipnat.conf reads:
    map xnf0 192.168.0.0/24 -> 0/32 portmap tcp/udp auto
    map xnf0 192.168.0.0/24 -> 0/32

"ipnat -l" confirms these settings.


My test case is a mercurial (hg) clone of http://dev.mutt.org/hg/mutt.
This clone operation never completes -- it stalls very early on.  I
analyzed with "snoop".  Several HTTP queries during the clone succeed,
but the clone appears to fail as soon as the remote hg server sends an
HTTP response that is large enough to be broken into multiple TCP segments.
I only receive the first segment within my zone, but a comparison to
the same clone operation on another server looks like I should expect
a TCP stream continuation (sliding window).

Indeed, I see the same behavior when I try to download over HTTP any
file big enough to require multiple response packets.  I've used
ipfilter for many years, but I know little about ipnat, and I'm unable
to explain this.

The same clone operation succeeds when I run it from the global zone.
The HTTP response is Content-Transfer-Encoding: chunked; the first
packet contains one chunk of length 2 and no other chunks.  The other
chunks arrive in subsequent packets.

I have full packet dumps of this behavior, but since this is a large
mailing list I'll leave it off for now.  They look pretty normal, except
that when you to a large, segmented TCP packet only the first IP packet
comes through.

Host system details, per FAQ:

root@domU-12-31-39-03-02-48:/cm/meta# uname -a
SunOS domU-12-31-39-03-02-48 5.11 snv_111b i86pc i386 i86xpv Solaris
root@domU-12-31-39-03-02-48:/cm/meta# isainfo -vk
32-bit i386 kernel modules
root@domU-12-31-39-03-02-48:/cm/meta# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
xnf0: flags=1104843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,ROUTER,IPv4> mtu 1500 index 2
        inet 10.249.1.178 netmask ffffff00 broadcast 10.249.1.255
        ether 12:31:39:3:2:48
vnic0: flags=1100843<UP,BROADCAST,RUNNING,MULTICAST,ROUTER,IPv4> mtu 9000 index 3
        inet 192.168.0.1 netmask ffffff00 broadcast 192.168.0.255
        ether 2:8:20:66:9f:1

root@domU-12-31-39-03-02-48:/cm/meta# netstat -rn

Routing Table: IPv4
  Destination           Gateway           Flags  Ref     Use     Interface 
-------------------- -------------------- ----- ----- ---------- --------- 
default              10.249.1.1           UG        1         30 xnf0      
10.249.1.0           10.249.1.178         U         1          2 xnf0      
192.168.0.0          192.168.0.1          U         1          1 vnic0     
127.0.0.1            127.0.0.1            UH        1         28 lo0       

root@domU-12-31-39-03-02-48:/cm/meta# netstat -i
Name  Mtu  Net/Dest      Address        Ipkts  Ierrs Opkts  Oerrs Collis Queue 
lo0   8232 loopback      localhost      135    0     135    0     0      0     
xnf0  1500 domU-12-31-39-03-02-48.compute-1.internal domU-12-31-39-03-02-48.compute-1.int
ernal 1247   0     945    0     0      0     
vnic0 9000 ip-192-168-0-1.ec2.internal ip-192-168-0-1.ec2.internal 40     0     55     0 
    0      0     

Name  Mtu  Net/Dest                    Address                     Ipkts  Ierrs Opkts  Oe
rrs Collis
lo0   8252 localhost                   localhost                   135    0     135    0 
    0     

root@domU-12-31-39-03-02-48:/cm/meta# netstat -s -P ip

IPv4    ipForwarding        =     1     ipDefaultTTL        =   255
        ipInReceives        =  1269     ipInHdrErrors       =     0
        ipInAddrErrors      =     0     ipInCksumErrs       =     0
        ipForwDatagrams     =    40     ipForwProhibits     =     0
        ipInUnknownProtos   =    20     ipInDiscards        =     0
        ipInDelivers        =  1330     ipOutRequests       =   931
        ipOutDiscards       =     0     ipOutNoRoutes       =     0
        ipReasmTimeout      =    15     ipReasmReqds        =     0
        ipReasmOKs          =     0     ipReasmFails        =     0
        ipReasmDuplicates   =     0     ipReasmPartDups     =     0
        ipFragOKs           =     0     ipFragFails         =     0
        ipFragCreates       =     0     ipRoutingDiscards   =     0
        tcpInErrs           =     0     udpNoPorts          =     2
        udpInCksumErrs      =     0     udpInOverflows      =     0
        rawipInOverflows    =     0     ipsecInSucceeded    =     0
        ipsecInFailed       =     0     ipInIPv6            =     0
        ipOutIPv6           =     0     ipOutSwitchIPv6     =     1

root@domU-12-31-39-03-02-48:/cm/meta# ipf -V
ipf: IP Filter: v4.1.9 (500)
Kernel: IP Filter: v4.1.9               
Running: yes
Log Flags: 0 = none set
Default: pass all, Logging: available
Active list: 1
Feature mask: 0x107

root@domU-12-31-39-03-02-48:/cm/meta# ipfstat
bad packets:            in 0    out 0
 IPv6 packets:          in 0 out 0
 input packets:         blocked 0 passed 180 nomatch 180 counted 0 short 0
output packets:         blocked 0 passed 145 nomatch 145 counted 0 short 0
 input packets logged:  blocked 0 passed 0
output packets logged:  blocked 0 passed 0
 packets logged:        input 0 output 0
 log failures:          input 0 output 0
fragment state(in):     kept 0  lost 0  not fragmented 0
fragment state(out):    kept 0  lost 0  not fragmented 0
packet state(in):       kept 0  lost 0
packet state(out):      kept 0  lost 0
ICMP replies:   0       TCP RSTs sent:  0
Invalid source(in):     0
Result cache hits(in):  0       (out):  0
IN Pullups succeeded:   0       failed: 0
OUT Pullups succeeded:  0       failed: 0
Fastroute successes:    0       failures:       0
TCP cksum fails(in):    0       (out):  0
IPF Ticks:      227
Packet log flags set: (0)
        none

root@domU-12-31-39-03-02-48:/cm/meta# ipfstat -io
empty list for ipfilter(out)
empty list for ipfilter(in)

root@domU-12-31-39-03-02-48:/cm/meta# ipnat -slv
mapped  in      8       out     10
added   1       expired 0
no memory       0       bad nat 0
inuse   1
orphans 0
rules   2
wilds   0
table 8047c5c list e1cf5200
List of active MAP/Redirect filters:
map xnf0 192.168.0.0/24 -> 0.0.0.0/32 portmap tcp/udp auto
map xnf0 192.168.0.0/24 -> 0.0.0.0/32

List of active sessions:
MAP 192.168.0.101   39566 <- -> 10.249.1.178    26478 [198.162.52.58 80]
        age 14529 use 0 sumd 0x187d/0xb462 pr 6 bkt 2036/533 flags 1
        ifp @,@ bytes 1443/1543 pkts 8/10 ipsumd 4b9d

List of active host mappings:
192.168.0.101,198.162.52.58 -> 0.0.0.0 (use = 2 hv = 0)

-- 
 -D.    [email protected]    NSIT    University of Chicago