ipf 4.1.32RC4 crashes Solaris9

I spoke too soon when I reported this being stable for Solaris8/9.
It still crashes my Sol9 loaded DNS server.  The symptoms are the same:
it is stable just as long as you don't filter by UDP ports, but as soon
as you add UDP port filter rules, it will crash after a while.

I ran ACT on the core dump, and I've included the relevant portion of the
output.  Trap 0x31 is some sort of data access MMU miss in fr_derefrule().
What could cause that?

Joseph Tam <[email protected]>

Edited output of act:

act 8.5

Release: SunOS 5.9 Generic_122300-31

panic: BAD TRAP: type=%x rp=%p addr=%p mmu_fsr=%x
lbolt is 0x20fd0e2, 0t34590946

...

PANIC occurred on this thread

***
process id 363 is /var/djb/bin/dnscache, parent process is 358
uid is 0x1fb 0t507, gid is 0x1fb 0t507

thread addr 0x30003367340, proc addr 0x30003364078, lwp addr 0x30003359c30
Thread bound to cpu id 0x1

t_state is 0x4 - TS_ONPROC

Scheduling info:
t_pri is 0x0, t_epri is 0x0, t_cid is 0x1
scheduling class is: TS
t_disp_time: is 0x20fbfd2, 0t34586578
last switched: 43 secs ago on cpu 0x1

pc is 0x104d1ac, sp is 0x1452210, t_stk 0x2a100371af0

Stack trace is:

unix: setjmp ()
unix: panicsys+0x44 (0x105d2c0,0x2a1003704c8,0x1,0x1,0x8,0x8)
unix: vpanic ()
unix: panic+0x1c (0x105d2c0,0x31,0x2a100370720,0x30000c59b2c,0x0)
unix: die+0xa4 (0x31,0x2a100370720,0x30000c59b2c,0x0)
unix: trap+0x8e4 (0x2a100370720)
unix: priv_rtt
ipf: fr_derefrule+0xe8 (0x30000c59b2c,0x2a100370a5e,0x30,0x89522416,0xffffffffffffffe8,0x30)
SUNW,UltraSPARC-III+: bcopy ()
ce: ce_wput+0x264 (0x30001e21818,0x30001f4eb40)
unix: putnext+0x21c (0x300011112f0,0x30001f4eb40)
pfil: pfilmodwput+0x40 (0x300011112f0,0x30001f4eb40)
unix: putnext+0x21c (0x30001111060,0x30001f4eb40)
ip: ip_wput_frag+0xb30 (0x30009475048?,0x30001f4eb40?,0x30009475098,0x0,0x4000)
ip: ip_wput_ire_fragmentit ()
ip: ip_wput_ire+0x16ac (0x30003290438,0x30003c44b80,0x30009475048,0x300025e5540,0x0)
ip: ip_wput_pktoptions+0x56c (0x30003290438,0x30003c44b80,0x14fa230)
ip: ip_wput+0x78 (0x30003290438,0x30003c44b80)
unix: putnext+0x21c (0x300032901a8,0x30003c44b80)
udp: udp_wput+0x6bc (0x300032901a8,0x3000413f280)
unix: putnext+0x21c (0x300032906c8,0x3000413f280)
genunix: strput+0x270 (0x3000328b348,0x3000413f280,0x2a100371a00,0x2a100371698,0x0,0x0)
genunix: kstrputmsg+0x36c (0x30001f0ce50,0x30003d16000,0x2a100371a00,0x7c,0x0,0x4)
sockfs: sosend_dgram+0x25c (0x30001f0ce50,0x30009357c90,0x10,0x2a100371a00,0x8)
sockfs: sosendmsg+0x3f4 (0x30001f0ce50,0x2a1003719d0,0x2a100371a00)
sockfs: sendit+0x15c (0x3,0x2a1003719d0,0x2a100371a00,0x8)
sockfs: sendto+0x78 (0x3,0x69ed8,0x7c,0x0,0xffbffcb0,0x10)
sockfs: sendto32+0x3c (0x3,0x69ed8,0x7c,0x0,0xffbffcb0,0x10)
unix: syscall_trap32+0xa8 (0x3,0x69ed8,0x7c,0x0,0xffbffcb0,0x10)

...

List of modules:						### MODULES ###

  Id Loadaddr  Size Info Rev Module Filename
  ...
  80  13dd635   39be   -   1  pfil (pfil Streams module 2.1.13) - drv/pfil
  80  13dd635   39be   -   1  pfil (pfil Streams driver 2.1.13) - drv/pfil
  ...
114 780aa000  57311   -   1  ipf (IP Filter: v4.1.32_RC4) - /usr/kernel/drv/ipf

...

KERNEL MESSAGE BUFFER:						### MSGBUF ###

lbolt       msg
...
6a8         IP Filter: v4.1.32_RC4, running.
...
7e1         IP Filter: v4.1.32_RC4 detached.
7f5         IP Filter: v4.1.32_RC4, running.
...
20fc07a     panic[cpu1]/thread=30003367340: 
20fc07a     BAD TRAP: type=31 rp=2a100370720 addr=30000c59b2c mmu_fsr=0
20fc07a 
20fc07a     dnscache: 
20fc07a     trap type = 0x31
20fc07a     addr=0x30000c59b2c
20fc07a     pid=363, pc=0x118b8a8, sp=0x2a10036ffc1, tstate=0x80001600,
 	    context=0x8a4
20fc07a     g1-g7: 3000365e16c, 1, 1388, 14, 18, 10, 30003367340
20fc07a 
20fc07a     000002a100370440 unix:die+a4 (31, 2a100370720, 30000c59b2c, 0,
 	    3000413f280, 3000365e158)
20fc07a       %l0-3: 0000000000000000 0000030000c59b2c 000002a100370720
 	    000002a100370610
 	      %l4-7: 0000000000000031 0000000000000000 0000000000000000
 	    0000000000000008
20fc0de     000002a100370520 unix:trap+8e4 (2a100370720, 0, 10000, 10200, 300,
 	    5751428)
20fc0de       %l0-3: 0000000000000001 0000000000000000 0000030003364078
 	    0000000000000031
 	      %l4-7: 0000000000000006 0000000000000001 0000000000000000
 	    0000000000000000
20fc0de     000002a100370670 unix:ktl0+48 (30005751428, 30003367340, 14acc00,
 	    150, 1a100f29172, 780ef6c8)
20fc0de       %l0-3: 0000000000000001 0000000000001400 0000000080001600
 	    000000000102db30
 	      %l4-7: 000003000001bad8 0000000000000020 0000000000000000
 	    000002a100370720
20fc0de     000002a1003707c0 ipf:fr_derefrule+e8 (30000c59b2c, 2a100370a5e, 30,
 	    89522416, ffffffffffffffe8, 30)
20fc0de       %l0-3: 0000000000000000 0000000000000010 0000030000fe9a80
 	    0000000000000000
 	      %l4-7: 0000000000000000 0000000000000020 0000000000000000
 	    000002a100370750
20fc0de     000002a100370980 ce:ce_wput+264 (30001e21818, 30001f4eb40, 0,
 	    30001015510, 22, 30)
20fc0de       %l0-3: 0000030000c59b14 0000000000000022 0000000000000000
 	    0000000000000000
 	      %l4-7: 0000000000000800 0000030005c5d7c0 000002a100370a3c
 	    0000030000074788
20fc142     000002a100370a90 unix:putnext+21c (0, 30001f4eb40, 2, 30001e1bed8,
 	    14, 2a100370ae0)
20fc142       %l0-3: 0000000001390d14 0000030001e21908 0000030001e21818
 	    0000000000000000
 	      %l4-7: 0000000000000000 000000007805b3e8 00000300011112f0
 	    0000030001f4eb40
20fc142     000002a100370b40 pfil:pfilmodwput+40 (300011112f0, 30001e1bed8, 20,
 	    0, bad4f2, 10)
20fc142       %l0-3: 0000030001f4eb40 00000000ff000000 0000030005c5d7c0
 	    0000030000fe9a80
 	      %l4-7: 0000030003c44b80 0000030005c5d7c0 0000000000000000
 	    0000000001000000
20fc142     000002a100370c00 unix:putnext+21c (0, 30001f4eb40, 0, 8000000, 800,
 	    ba9617670800)
20fc142       %l0-3: 00000000013df918 00000300011113e0 00000300011112f0
 	    0000000000000000
 	      %l4-7: 0000000000000000 000000007805d928 0000030001111060
 	    0000030001f4eb40
20fc142     000002a100370cb0 ip:ip_wput_frag+b30 (e, 0, 10000, 30001111060,
 	    3000413f280, 3000365e158)
20fc142       %l0-3: 000003000408ea40 0000000000000000 0000030009475048
 	    0000000000000000
 	      %l4-7: 0000030001f4eb40 0000000000000000 0000000000000000
 	    0000000000000008
20fc1a6     000002a100370e10 ip:ip_wput_ire+16ac (10000, 300000742b0, 3, ffff,
 	    89522441, 30003c44b80)
20fc1a6       %l0-3: 0000000000015afb 00000300011d8000 0000030009475048
 	    0000030000fe9b18
 	      %l4-7: 0000000000000000 0000000000000000 0000030001111060
 	    0000000089522416
20fc1a6     000002a100371000 ip:ip_wput_pktoptions+56c (30009475048, 0,
 	    89522416, 30000fe9b18, 0, 0)
20fc1a6       %l0-3: 0000030003c44b80 0000000000000000 00000000014eb800
 	    00000300025e5540
 	      %l4-7: 0000030009475048 0000030003290438 0000000089522416
 	    00000300025e5550
20fc1a6     000002a1003710d0 ip:ip_wput+78 (30003290438, 30003c44b80, 20, 8, 1,
 	    0)
20fc1a6       %l0-3: 0000030009357c90 0000000000000000 0000000000000000
 	    0000000000000000
 	      %l4-7: 0000030003359c30 0000000000000003 000000000000007c
 	    000002a1003719d0
20fc1a6     000002a100371190 unix:putnext+21c (0, 30001199f80, 20, 0, 53, 10)
20fc1a6       %l0-3: 00000000012a4750 000003000119def8 0000030003290438
 	    0000000000000000
 	      %l4-7: 0000030001199f88 00000000014ebed8 00000300032901a8
 	    0000030003c44b80
20fc20a     000002a100371240 udp:udp_wput+6bc (3000194cf50, ffff, 84,
 	    30000fe9b2c, 0, 10)
20fc20a       %l0-3: 0000000000000084 0000030000fe9b18 0000000089522416
 	    0000030003c44b80
 	      %l4-7: 0000000000000014 000000000000096a 00000300032901a8
 	    000003000413f280
20fc20a     000002a100371320 unix:putnext+21c (0, 3000413f280, 20, 30003c44b80,
 	    0, 10)
20fc20a       %l0-3: 0000000001251118 0000030003290298 00000300032901a8
 	    0000000000000000
 	      %l4-7: 0000000000000000 00000000014e3f90 00000300032906c8
 	    000003000413f280
20fc20a     000002a1003713d0 genunix:strput+270 (3000328b348, 0, 0, 2a100371698,
 	     0, 0)
20fc20a       %l0-3: 0000000000000004 00000300032906c8 0000000000000000
 	    000003000328b348
 	      %l4-7: 000003000413f280 000002a100371a00 0000000000020400
 	    0000000000020400
20fc20a     000002a1003715c0 genunix:kstrputmsg+36c (30001f0ce50, 300032906c8,
 	    0, 0, 0, 0)
20fc20a       %l0-3: 000003000328b3c8 0000030003d16000 000003000328b348
 	    0000000000000004
 	      %l4-7: 0000000000000083 000002a100371a00 0000000000000000
 	    0000030003d16000
20fc26e     000002a1003716a0 sockfs:sosend_dgram+25c (0, 30009357c90, 10,
 	    2a100371a00, 8, 22)
20fc26e       %l0-3: 0000000000000010 0000000001469800 000000000000007c
 	    0000030001f0ce50
 	      %l4-7: 0000030003d16000 0000000000000078 0000000000000000
 	    0000000000000000
20fc26e     000002a100371790 sockfs:sosendmsg+3f4 (0, 30009357c90, 7, 20, 0, 0)
20fc26e       %l0-3: 0000000000000010 0000000000000008 0000030009357c90
 	    0000000000000000
 	      %l4-7: 0000030001f0ce50 000002a1003719d0 000002a100371a00
 	    0000000000000000
20fc26e     000002a100371850 sockfs:sendit+15c (2a100371a00, 8, 30001f0ce50, 8,
 	    1, 0)
20fc26e       %l0-3: 0000030009357c90 0000000000000000 0000000000000000
 	    0000000000000000
 	      %l4-7: 0000030003359c30 0000000000000003 000000000000007c
 	    000002a1003719d0
20fc26e     000002a100371920 sockfs:sendto+78 (3, 69ed8, 7c, 0, ffbffcb0, 10)
20fc26e       %l0-3: 000000000c3cac70 00000000ff2428e0 00000000ff2428e4
 	    0000000000000000
 	      %l4-7: 0000000000000000 0000000000000000 0000000000000000
 	    0000000000000000
20fc2d2     000002a100371a40 sockfs:sendto32+3c (3, 69ed8, 7c, 0, ffbffcb0, 10)
20fc2d2       %l0-3: 00000000ff2404e8 000000000116693c 00000000ffffc000
 	    000000000000000c
 	      %l4-7: 000000000000006e 000000000140f8b0 0000000000069ed8
 	    0000000000060c00