It isnt
every day that a companys servers are compromised by a hacker, but when that
unfortunately inevitable day arrives, it might as well be everyday. I dont pretend
to pass myself off as a computer security expert, but I always believed that I could hold
my own when it came to protecting our Internet servers. When our Linux server fell into
the hands of a smart hacker, I felt shock, embarrassment, and lost a good part of my
confidence which I am still battling to regain.
This blow also had a second negative impact on me. In February, I wrote a piece for this
very column, titled
"The Little OS That Could", praising and lauding Linux as the ultimate
operating system. I mentioned how easily I had installed and deployed our Linux server and
how well it was performing in comparison to our Windows NT servers. I also mentioned a few
points about Linux, that in my opinion, made this OS a superior product in the market.
Guess which one of the servers was hacked? It was the Linux server, after all.
THE BREAK-IN
One fine morning in early April I settled in at my desk , running an experiment
with the HTTP protocol. I was trying to establish a theory regarding the HTTP header
fields returned from various servers. After completing the task on the NT servers, I
proceeded to the Linux server and ran the client program, expecting a quick response from
the Linux machine. But the server never answered. Not feeling any alarm, I calmly logged
onto the server to poke around. Web-service crashes on NT machines were something I was
used to. I would have to simply log onto the NT machine and restart the service.
But the Linux machine running the Apache Web server had
never crashed before. Upon examining the running services (known as daemons in the Unix
world), I noticed that indeed the httpd (Web) daemon was down. This was
peculiar, but I thought perhaps this was a rare case and a restart would fix the problem.
So I manually restarted the service, and it came back to life with a little informational
message informing me of a permission mismatch on a certain file. The Web service was now
up and running, and working fine, but the error message had started to bug me. Without
knowing what I was getting into, I started to dig into the file that was reported as
having a wrong permission. I didnt know what the original permission was supposed to
be, so this effort had no conclusive results. Normally, I would have just changed the
permissions on the file and passed this incident off as a fluke, but this time I was
feeling very uneasy.
Next, I checked the contents of the startup files for
telltale signs of break-in or sabotage. But that didnt lead me anywhere either.
Checking the passwd file that holds the accounts information didnt point
to a suspicious activity either. Just before giving up the search, I thought maybe I would
take a look at the log files for logins and other activities and thats when
things begun to unravel. The last command, which shows the login activity on
the system, failed to execute and produced a message about a missing wtmp
file. Indeed, the file that the last command relies on to get its information
from was missing. I decided to create the file manually and see what would happen. The
file was now there, but nothing was getting logged into it. At this point, I rebooted the
system to possibly give it a chance to re-initialize itself and hopefully get back to
normal. No chance. Although the log file was in place now, no information was being
written to it and therefore, there was no way to find out what kind of logging activity
was going on in the system. Having misinterpreted system faults (and my own mistakes) as
hacker attacks in the past, I was reluctant to jump to a conclusion and go to red alert.
Besides, hacker attacks are not supposed to happen in my domain. Its just something
you read about in trade magazines.
So I continued digging deeper, but the more I dug, the
more the evidence pointed to the fact that the server was in fact under siege. The
w command, which shows the current logged on users, was producing no results
proving the fact the its data file, utmp, was tampered with. The
top command, which displays processor and memory utilization, showed unusual
activity on the system but surprisingly the ps command, which displays running
processes on the system, showed no unusual processes running. Confounded, I finally
decided to contact our service provider for assistance. They could not say with certainty
whether our system was hacked, but advised that I reformat the servers hard drive
and reinstall the OS from scratch, selecting new passwords for the accounts. I found their
response totally inadequate. After all, if the intruder was able to get in once, he/she
would do it again. I wanted to find who had attacked our system, how he breached it, and
what he was doing. I was not about to rest until I could find some answers to this puzzle.
THE FACELESS HACKER
Not having come face to face (in a virtual sense) with a hacker in the past, I
tried to put myself in his position. Where would my point attack be, if I were to hack
into a system on the Internet? At the simplest level, it would be the network interface.
There had to be a trace of activity within the network files and resources. Bingo. An
execution of the netstat command revealed socket connections to several
Internet Relay Chat (IRC) servers around the world, but the first and the permanent
connection in the list was to a server in Slovenia. After rebooting the system a few times
and observing the same connection to the Slovenian server, I had established with good
certainty that the offending machine was in Slovenia, and the activity had something to do
with chat servers. I traced one of the other connected chat servers to New York City, and
called the company to inquire about the ongoing activity. The tech person confirmed that
indeed there was a connection from our server to theirs, and there was chatting activity
going on as we spoke, but he had no authority to break the connection nor could he divulge
the nature of the chat activity to me. Now I started to get nervous. There was a good
chance the hacker was performing illegal activities disguised as us, and we would have had
to take the blame had he gotten out of hand.
Now that I almost knew who I was dealing with and what he
was doing, I wanted to figure out how he was doing this and what he was ultimately
accomplishing with his chatting activity. I scoured the system for unknown files and
directories hoping to come across something unfamiliar. It paid off. Tucked under the
/dev/char subdirectory were a number of files that didnt seem to belong
there. There was a program file containing code to attack a machine over TCP/IP sockets; a
script file to add a new user, carry out some activity, and remove the user; and a file
containing a number of suspicious program names, among which the name eggdrop
stood out. I had never heard of this program, so I started to pore over the newsgroups
searching for it, and I found lots of posts regarding eggdrop, which apparently has been
around for years and involves IRC chat. Not being a chat fan, I had to learn a little
about IRC along the way.
THE CHAT INTRUSION
I found out that most chat servers allow newcomers to create and own a channel,
and invite others to chat on that particular channel. However, once the person logs out of
the server, the channel is closed, and another person may create and take ownership of a
channel with the same name therefore, preventing the ex-owner from establishing and
taking control of the channel. In order to protect ones ownership of a channel, some
people use bots, which are programs that remain in constant connection with
the chat server, preventing the channel from shutting down. Eggdrop is one such program,
and I came to find out that it is a favorite of hackers. I also discovered that our hacker
had deployed a rootkit on our server, which allowed him and his processes to
remain just below my radar screen.
Rootkit replaces some of the system files (such as ps) with the hackers
version, filtering out his processes from the output, and thereby tricking the system
manager into believing that no foreign processes are running on the system. There may be
thousands of servers out there with hackers operating on them right under the system
managers noses.
I had the final pieces of the puzzle delivered to me when
an ISP from Arizona called me as a result of my newsgroup posting regarding the attack.
The hackers footprint was almost identical on both of our servers. The stricken
ISPs tech person told me the hacker ran a program testing batches of IP addresses on
the Internet for possible attack. Once the potential victim was identified, he would enter
the system through a procedure known as buffer overrun on the NFS
or SMB daemons, which are used to connect to other UNIX and Windows NT
machines resources. Upon entry, the hacker would deploy rootkit and activate
eggdrop, allowing him to use the victims faster machine to freely control the IRC
channels rather than using his own slow connection. The ISP suspected that the hacker was
using the IRC channels to trade pirated software. Once the ISP had tried to kick him out
of their system by flooding his connection, the hacker had apparently gotten angry and
reformatted their hard drive.
Strangely enough, we were attacked one hour after the
hacker was ejected from the ISPs machines on a Saturday. I found out about the hack
on Monday morning. And why had the Web server daemon died, which brought me on this
fact-finding journey? After the hacker deleted the log files, the httpd daemon
(the Web service) had no log files to write to, and had simply crashed. Perhaps this was
something the hacker overlooked in his quest to remain stealth. Without this tiny clue, I
might have never suspected anything for a long time. The damage to us was minimal, but the
said ISP suffered irreparable damages, which included loss of some of their irate Web
hosting customers.
This incident has not shaken my faith in Linux, but it has
opened my eyes to the fact that computers will be hacked as long as there are determined
hackers out there and they can find the smallest security hole. As many of our daily
activities are carried out on computers, and as Internet telephony gateways and
gatekeepers carry more of our voices on data networks, we must seriously consider
deploying strong security measures to combat such invasions. Our Linux server is still
crippled and off the network, a testament to the exploits of one hacker. The same calamity
could befall tomorrows telephony gateways and gatekeepers. Without proper and
stringent security measures, its not a question of if, its a question of when.
Stay vigilant. You could be next.
Robert Vahid Hashemian provides us with a healthy dose
of reality each month in his Reality Check column. Robert currently holds the position of
Webmaster for TMCnet.com your online resource
for CTI, Internet telephony, and call center solutions. He can be reached at rhashemian@tmcnet.com.
|