some new remote exploit?

Hi all,


I just had a weird expirience ... In a server system, running a free webmail 
service, i started getting weird oopsen.

On a quad p2/xeon box, 2gb ram, running rehdat 6.2 with sendmail 8.10 (patched 
with latest stuff), cyrus 2.0.16, openldap 1.2 and sasl 1.5, with a custom 
2.4.19-pre10 aa something kernel, it looked like this:

ksymoops 2.4.4 on i686 2.4.19-pre10.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.19-pre10/ (default)
     -m /boot/System.map-2.4.19-pre10 (default)

Warning: You did not tell me where to find symbol information.  I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc.  ksymoops -h explains the options.

Warning (compare_maps): ksyms_base symbol set_cpus_allowed_R__ver_set_cpus_allowed not found in System.map.
Ignoring ksyms_base entry
Apr  7 12:55:23 castor kernel: de08d7db 
Apr  7 12:55:23 castor kernel: *pde = 00000000 
Apr  7 12:55:23 castor kernel: Oops: 0000 
Apr  7 12:55:23 castor kernel: CPU:    0 
Apr  7 12:55:23 castor kernel: EIP:    0010:[<de08d7db>]    Not tainted 
Using defaults from ksymoops -t elf32-i386 -a i386
Apr  7 12:55:23 castor kernel: EFLAGS: 00010286 
Apr  7 12:55:23 castor kernel: eax: bfffff14   ebx: ec3f6000   ecx: 00000000   edx: 00000000 
Apr  7 12:55:23 castor kernel: esi: c0108efb   edi: 0000000b   ebp: ec3f7fb8   esp: ec3f7f80 
Apr  7 12:55:23 castor kernel: ds: 0018   es: 0018   ss: 0018 
Apr  7 12:55:23 castor kernel: Process mail2sms (pid: 13314, stackpage=ec3f7000) 
Apr  7 12:55:23 castor kernel: Stack: ec3f6000 c0108efb 0000000b 00000296 00000000 d8d23000 0000000b 00000296  
Apr  7 12:55:23 castor kernel:        d8d23000 bfffff14 c0107a5f 00000000 00000a3a 00000020 bffffd28 de08d9c4  
Apr  7 12:55:23 castor kernel:        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000  
Apr  7 12:55:23 castor kernel: Call Trace: [system_call+47/52] [sys_execve+79/92]  
Apr  7 12:55:23 castor kernel: Code: 8b 42 04 83 f8 ff 0f 84 69 01 00 00 83 f8 fc 77 07 c7 42 04 

>>EIP; de08d7db <END_OF_CODE+1abe12e0/????>   <=====
Code;  de08d7db <END_OF_CODE+1abe12e0/????>
00000000 <_EIP>:
Code;  de08d7db <END_OF_CODE+1abe12e0/????>   <=====
   0:   8b 42 04                  mov    0x4(%edx),%eax   <=====
Code;  de08d7de <END_OF_CODE+1abe12e3/????>
   3:   83 f8 ff                  cmp    $0xffffffff,%eax
Code;  de08d7e1 <END_OF_CODE+1abe12e6/????>
   6:   0f 84 69 01 00 00         je     175 <_EIP+0x175> de08d950 <END_OF_CODE+1abe1455/????>
Code;  de08d7e7 <END_OF_CODE+1abe12ec/????>
   c:   83 f8 fc                  cmp    $0xfffffffc,%eax
Code;  de08d7ea <END_OF_CODE+1abe12ef/????>
   f:   77 07                     ja     18 <_EIP+0x18> de08d7f3 <END_OF_CODE+1abe12f8/????>
Code;  de08d7ec <END_OF_CODE+1abe12f1/????>
  11:   c7 42 04 00 00 00 00      movl   $0x0,0x4(%edx)

Apr  7 12:55:23 castor kernel:  <1>Unable to handle kernel NULL pointer dereference at virtual address 00000004 
Apr  7 12:55:23 castor kernel: de08d7db 
Apr  7 12:55:23 castor kernel: *pde = 00000000 
Apr  7 12:55:23 castor kernel: Oops: 0000 
Apr  7 12:55:23 castor kernel: CPU:    2 
Apr  7 12:55:23 castor kernel: EIP:    0010:[<de08d7db>]    Not tainted 
Apr  7 12:55:23 castor kernel: EFLAGS: 00010286 
Apr  7 12:55:23 castor kernel: eax: bffffee4   ebx: c0846000   ecx: 00000000   edx: 00000000 
Apr  7 12:55:23 castor kernel: esi: c0108efb   edi: 0000000b   ebp: c0847fb8   esp: c0847f80 
Apr  7 12:55:23 castor kernel: ds: 0018   es: 0018   ss: 0018 
Apr  7 12:55:23 castor kernel: Process sendmail (pid: 13313, stackpage=c0847000) 
Apr  7 12:55:23 castor kernel: Stack: c0846000 c0108efb 0000000b 00000296 00000000 e9a6f000 0000000b 00000296  
Apr  7 12:55:23 castor kernel:        e9a6f000 bffffee4 c0107a5f 00000000 00000a3a 00000020 bffffb90 de08d9c4  
Apr  7 12:55:23 castor kernel:        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000  
Apr  7 12:55:23 castor kernel: Call Trace: [system_call+47/52] [sys_execve+79/92]  
Apr  7 12:55:23 castor kernel: Code: 8b 42 04 83 f8 ff 0f 84 69 01 00 00 83 f8 fc 77 07 c7 42 04  

>>EIP; de08d7db <END_OF_CODE+1abe12e0/????>   <=====
Code;  de08d7db <END_OF_CODE+1abe12e0/????>
00000000 <_EIP>:
Code;  de08d7db <END_OF_CODE+1abe12e0/????>   <=====
   0:   8b 42 04                  mov    0x4(%edx),%eax   <=====
Code;  de08d7de <END_OF_CODE+1abe12e3/????>
   3:   83 f8 ff                  cmp    $0xffffffff,%eax
Code;  de08d7e1 <END_OF_CODE+1abe12e6/????>
   6:   0f 84 69 01 00 00         je     175 <_EIP+0x175> de08d950 <END_OF_CODE+1abe1455/????>
Code;  de08d7e7 <END_OF_CODE+1abe12ec/????>
   c:   83 f8 fc                  cmp    $0xfffffffc,%eax
Code;  de08d7ea <END_OF_CODE+1abe12ef/????>
   f:   77 07                     ja     18 <_EIP+0x18> de08d7f3 <END_OF_CODE+1abe12f8/????>
Code;  de08d7ec <END_OF_CODE+1abe12f1/????>
  11:   c7 42 04 00 00 00 00      movl   $0x0,0x4(%edx)


2 warnings issued.  Results may not be reliable.


There were two oopsen at 12:55 and 41 from 15:00:01 to 15:01:32 localtime . 
The machine appears to be running fine.



The other box, dual xeon 2.0ghz (HT enabled), 4gb ram, running 7.3 with 
kernel 2.4.18-26 bigmem, started returning segmentation fault for every 
command i wanted to run at about 15:15. It is running apache 1.3.26, php 
4.3.1 and postfix 1.11. I was lucky to have two ssh sessions opened to it 
before this started. The oops looked like this:

ksymoops 2.4.4 on i686 2.4.18-26.7.xbigmem.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.18-26.7.xbigmem/ (default)
     -m /boot/System.map-2.4.18-26.7.xbigmem (default)

Warning: You did not tell me where to find symbol information.  I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc.  ksymoops -h explains the options.

Error (expand_objects): cannot stat(/lib/ext3.o) for ext3
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/jbd.o) for jbd
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/raid1.o) for raid1
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/aic7xxx.o) for aic7xxx
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/sd_mod.o) for sd_mod
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/scsi_mod.o) for scsi_mod
ksymoops: No such file or directory
/usr/bin/find: /lib/modules/2.4.18-26.7.xbigmem/build: No such file or directory
Error (pclose_local): find_objects pclose failed 0x100
Warning (map_ksym_to_module): cannot match loaded module ext3 to a unique module object.  Trace may not be reliable.
Apr  7 15:12:31 www kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000004
Apr  7 15:12:31 www kernel: d01597ae
Apr  7 15:12:31 www kernel: *pde = 1a8b6001
Apr  7 15:12:31 www kernel: Oops: 0000
Apr  7 15:12:31 www kernel: CPU:    2
Apr  7 15:12:31 www kernel: EIP:    0010:[<d01597ae>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
Apr  7 15:12:31 www kernel: EFLAGS: 00010286
Apr  7 15:12:31 www kernel: eax: bffff9d4   ebx: ce346000   ecx: 00000000   edx: 00000000
Apr  7 15:12:31 www kernel: esi: c0108c93   edi: 0000000b   ebp: ce347fb8   esp: ce347f80
Apr  7 15:12:31 www kernel: ds: 0018   es: 0018   ss: 0018
Apr  7 15:12:31 www kernel: Process sh (pid: 15725, stackpage=ce347000)
Apr  7 15:12:31 www kernel: Stack: 00000000 bffffff4 00000292 f441c000 c014e48e f441c000 bffff9d4 0000000b 
Apr  7 15:12:31 www kernel:        00000000 f441c000 0000000b ce346000 c0108c93 0000000b bffe6118 d01599a2 
Apr  7 15:12:31 www kernel:        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
Apr  7 15:12:31 www kernel: Call Trace: [<c014e48e>] getname [kernel] 0x5e (0xce347f90))
Apr  7 15:12:31 www kernel: [<c0108c93>] system_call [kernel] 0x33 (0xce347fb0))
Apr  7 15:12:31 www kernel: Code: 8b 42 04 83 f8 ff c7 45 f0 20 00 00 00 c7 45 ec 3a 0a 00 00 

>>EIP; d01597ae <_end+fd16d92/383ca5e4>   <=====
Trace; c014e48e <getname+5e/a0>
Trace; c0108c93 <system_call+33/38>
Code;  d01597ae <_end+fd16d92/383ca5e4>
00000000 <_EIP>:
Code;  d01597ae <_end+fd16d92/383ca5e4>   <=====
   0:   8b 42 04                  mov    0x4(%edx),%eax   <=====
Code;  d01597b1 <_end+fd16d95/383ca5e4>
   3:   83 f8 ff                  cmp    $0xffffffff,%eax
Code;  d01597b4 <_end+fd16d98/383ca5e4>
   6:   c7 45 f0 20 00 00 00      movl   $0x20,0xfffffff0(%ebp)
Code;  d01597bb <_end+fd16d9f/383ca5e4>
   d:   c7 45 ec 3a 0a 00 00      movl   $0xa3a,0xffffffec(%ebp)


2 warnings and 7 errors issued.  Results may not be reliable.

The box also segfaulted on a shutdown -r now, so i had no other choice but to
press the reset button. Then, it started appearing again seconds after the box 
came up. After another reboot at around 15:46, it stopped. There are 517 oopsen 
recorded in the log from 15:12:31 to 15:39:34.


Any ideas?

Both machines have good hardware and are running problemfree. This *looks* like 
it is remotely triggered, so i suspect it is some kind of attack. system_call
is the common point in oopsen on both machines. Any ideas how to mess with this 
remotely?


--

Jure Pecar