[wplug] bug, cosmic ray or security breach?
Alexandros Papadopoulos
apapadop at alumni.cmu.edu
Tue Jan 4 04:13:00 EST 2005
Dear all
I'm facing a weird situation with one of my (Debian) servers.
After a power outage and the necessary reboot, some important
system binaries started to segfault. These include /bin/grep, /bin/tar,
/usr/bin/find and /usr/bin/perl.
They all exhibit the same behavior. I'll demonstrate the behavior of
/usr/bin/perl to illustrate, but what I'll show is not limited to perl in any way.
helios:~# perl
Segmentation fault
Okay, let's see what it's trying to do:
helios:~# strace perl
execve("/usr/bin/perl", ["perl"], [/* 15 vars */]) = 0
uname({sys="Linux", node="helios", ...}) = 0
brk(0) = 0x814be80
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40017000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=13605, ...}) = 0
old_mmap(NULL, 13605, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40018000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/libdl.so.2", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0p\34\0\000"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=9872, ...}) = 0
old_mmap(NULL, 8632, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x4001c000
old_mmap(0x4001e000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x4001e000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/libm.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0 at 5\0\000"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=134464, ...}) = 0
old_mmap(NULL, 136944, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x4001f000
old_mmap(0x40040000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x20000) = 0x40040000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/libpthread.so.0", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\340A\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=81127, ...}) = 0
old_mmap(NULL, 331716, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x40041000
old_mmap(0x4004e000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0xc000) = 0x4004e000
old_mmap(0x40050000, 270276, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40050000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200^\1"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=1244080, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40092000
old_mmap(NULL, 1254244, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x40093000
old_mmap(0x401bb000, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x127000) = 0x401bb000
old_mmap(0x401c3000, 9060, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x401c3000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/libcrypt.so.1", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\0\n\0\000"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=18780, ...}) = 0
old_mmap(NULL, 181596, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x401c6000
old_mmap(0x401cb000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x4000) = 0x401cb000
old_mmap(0x401cc000, 157020, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x401cc000
close(3) = 0
munmap(0x40018000, 13605) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=RLIM_INFINITY, rlim_max=RLIM_INFINITY}) = 0
setrlimit(RLIMIT_STACK, {rlim_cur=2044*1024, rlim_max=RLIM_INFINITY}) = 0
getpid() = 1587
rt_sigaction(SIGRTMIN, {0x40049030, [], SA_RESTORER, 0x400bc5e8}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x40049080, [], SA_RESTORER, 0x400bc5e8}, NULL, 8) = 0
rt_sigaction(SIGRT_2, {0x40049150, [], SA_RESTORER, 0x400bc5e8}, NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [RTMIN], NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RT_1], NULL, 8) = 0
_sysctl({{CTL_KERN, KERN_VERSION}, 2, 0xbffff804, 30, (nil), 0}) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
That didn't help me much... Let's run it through a debugger:
helios:~# gdb perl
GNU gdb 6.1-debian
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-linux"...(no debugging symbols found)...Using host libthread_db library "/lib/libthread_db.so.1".
(gdb) run
Starting program: /usr/bin/perl
(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...[Thread debugging using libthread_db enabled]
[New Thread 16384 (LWP 1595)]
(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 16384 (LWP 1595)]
0xffffffdf in ?? ()
(gdb) continue
Continuing.
Cannot find user-level thread for LWP 1595: generic error
(gdb) quit
The program is running. Exit anyway? (y or n) y
Quitting: Cannot find thread 16384: generic error
helios:~#
Hm. I don't understand why, after it appears to segfault, GDB seems to
think it's still alive. It doesn't seem to try to spawn a different process
anyway (which I would expect if this were an "evil" binary).
So I scp over a /usr/bin/perl binary from another Debian sarge system, with
the same CPU architecture and Debian release (sarge i386):
I rename the buggy /usr/bin/perl as /usr/bin/perl_SUSPECT. I then install
the freshly-fetched binary (which doesn't segfault) as /usr/bin/perl.
Let's examine the two binaries to see if they're the same:
helios:~# ls -l /usr/bin/perl*
-rwxr-xr-x 1 root root 1057324 Jan 3 19:07 /usr/bin/perl
-rwxr-xr-x 2 root root 1057324 Oct 24 19:37 /usr/bin/perl5.8.4
-rwxr-xr-x 2 root root 1057324 Oct 24 19:37 /usr/bin/perl_SUSPECT
<snip>
Well, they're exactly the same size. One segfaults and the other doesn't.
helios:~# md5sum /usr/bin/perl /usr/bin/perl_SUSPECT
8685fe6a66ac50d2f281a70c0a2a6651 /usr/bin/perl
8527e3fbc47dc314e1feb2e45fe1917a /usr/bin/perl_SUSPECT
helios:~#
So, they ARE different. But in which way? Let's try to take a peek into the
binary. Existence of any "l33t" comments would signify that my server has
been cracked, but:
helios:~# strings /usr/bin/perl > perl_new
helios:~# strings /usr/bin/perl_SUSPECT > perl_old
helios:~# diff perl_new perl_old
helios:~#
No dice. Exactly the same strings readable in the clean and suspect binary.
If I restore the binary from the last backup of the machine, it's exactly the
same with the binary I fetched from the other machine.
So how about that? Remember that this is not only a perl thing, it happens
with grep, tar and find (and I'm sure others will come up too, I just haven't
noticed yet).
I have a (presumably) clean backup and I can restore easily, but this is a
very interesting situation that I wish to decipher before bulldozing over.
Oh, and I did run chrootkit and checked all logs under /var/log and all seems
in order. I should also mention that this machine is not directly exposed to the
Internet or any intrusted network for that matter - one would have to take
over a firewall, then a user account, and then elevate privileges to root to take
over the machine in question (helios). I wish I'd been running tripwire, but
I wasn't, as I've found it to be too much of a hassle.
Eagerly awaiting ideas for further investigation of the suspect binaries...
Cheers
-A
More information about the wplug
mailing list