Event Analysis Training – “Could you look at some odd IRC Connections?”

At one of the research sites that we monitor, an analyst noted that a few servers were consistently making a large number of IRC connections. These connections occurred in a periodic manner and appeared to be automated. This blog entry describes the various steps taken in analyzing the connections and historical data. We used Tenable’s log analysis, network monitoring and passive profiling solutions to perform this analysis, but the principals could be applied to various SIMs, NBADs and analytical tools.

IRC Connections to an External IP

The sanitized IP address that the servers connected to via IRC was in the 217.0.0.0/8 network range. Using the Security Center and Log Correlation Engine, we filtered the last 24 hours of all normalized log activity, as shown below:

Except for a small break in the early evening, the “intrusion” events are almost continuous. If you notice the flatness of the graph, this corresponds to continuous activity at a consistent level. We were curious if the activity was longer in duration than just the 24 hour window we were looking at. Below is a trend of the same query for seven days and another one for 25 days:

With this type of analysis, it is easy to see that the activity levels started sometime at the end of June with scanning and other types of activity occurring in early July.

These graphs are high-level “type” summaries. It would be interesting to note what types of specific normalized events made up the “intrusion” type graph. Shown below is the graph of just normalized event types for the “intrusion” category.

In this case, it is a generic Snort-TCP_Event. This event is a catchall normalization for Snort IDS logs from both the Sourcefire VRT and Emerging Threats library of rules. If an event cannot be normalized to something like an “Administrator Attempt” or “Denial of Service” category, it is classified as a generic Snort-TCP_Event.

Before we dig into the event further, let’s consider the direction of these Snort TCP events. Below is a graph of all events matching this event type, which is either inbound, outbound or internal to our network.

In this case, all of the graphed events are indeed heading to sites that are external to the network we are monitoring. Now let’s consider which ports are in use:

Port 6667 is a common port used for Internet Relay Chat. At this point, we have a bunch of events that indicate outbound connections on port 6667 to one particular host in the 217.0.0.0/8 address block. Let’s summarize all of the IP addresses that were participating in these events:

In this case, there was only one internal host. Since we are scanning this network with Nessus and also monitoring traffic with the Passive Vulnerability Scanner, it is interesting to note what we know about this host. The Security Center can quickly show what is known about the host in question as shown below:

For this particular site, we’ve configured the Security Center to create a variety of dynamic asset lists based on results from active and passive vulnerability scanning. This process produces one or more lists of assets and in this case, the IP address in question belongs to the following asset groups:

Based on all of the information so far, all we really know is that we have a single internal Linux system making a lot of continuous IRC connections to a single external system. We were still concerned that the consistent IRC connections indicate some sort of automation, and possibly a botnet control channel.

Digging a bit further, we looked at the types of vulnerabilities for the internal host in question as shown below:

Sorted by severity, it can be seen that many of the “high” vulnerabilities were related to SSH and HTTP. Knowing that there have been many SSH worms and botnets, we decided to list out all port 22 connections and events for the past 25 days:

We noticed a few instances of interest. The “detected-change” events pointed to new internal trust relationships within our network on port 22. In addition, the vulnerability category was from real-time detections with the PVS. As it learned about new SSH issues, it logged these in real-time. What I really wanted to look at was the specific network sessions that we knew about this host. Filtering on the “network” category, the following normalized events showed all port 22 traffic on this host for the last month:

The LCE network events are described with some analysis here:

• PVS Accepts External Connections – Any time the PVS sees an open port, it will also  mark if that port is open to the Internet. In this case, the PVS discovered that port 22 was open and there were indeed connections externally from the Internet to it.
• TNM Long TCP Sessions – These events mark when a TCP session has finished and was of a certain length. In this case, we had three port 22 sessions that lasted about five minutes during this time and one that lasted as long as 60 minutes.
• TNM TCP Sessions Short and Time Out – The TNM also logs when sessions have very short times and if the session never completed. The 230 “short” sessions indicate that there were very quick connections usually lasting only a few seconds. The 573 “timedout” sessions indicate a session started, but never completed. I’ve chosen to describe these events together because in the graph, it can be seen that they occur at the same time. This sort of pairing usually indicates network scanning on port 22.

The Tenable Network Monitor also logs the amount of client side and server side bandwidth that has been transferred between these hosts. I wanted to see how much data was being sent in these sessions. Below are a few sanitized samples:

Wed Jun 10 23:47:56 - TNM-Long_TCP_Session_5_Minutes[4021]:149.X.X.X:22 -> 79.X.X.X:2455 [u:2388 d:1633]|1244692007|1244692076|69

Sat Jun 27 21:18:31 - TNM-Long_TCP_Session_5_Minutes[90810]:149.X.X.X:22 -> 66.X.X.X:3675 [u:56340 d:34470]|1246151306|1246151911|605

Sat Jun 27 22:41:17 - TNM-Long_TCP_Session_5_Minutes[852]:149.X.X.X:22 -> 210.X.X.X:56612 [u:172 d:680]|1246156564|1246156877|313

For the three “5 minute” sessions, you can see that the amount of data transferred was 4021 bytes, 90810 bytes and 852 bytes.

Additionally, there are three unique IP addresses connecting to our IP in question. One is from the 79 Class A, another from 66 and another from 210. I consider it odd that there are this many “random” successful connections.

Let’s take a step away from port 22 activity and take a look at all traffic:

Port 7000 is also on our list and this too is a port associated with IRC. Let’s compare the port 6667 activity with that on port 7000:

Port 6667:

Port 7000:

It can be seen that there was some port 7000 activity prior to the 6667 IRC traffic. We thought this could be significant, but in the end, whichever IRC client software was being used was simply making IRC connections to the same remote host on a different port.

At this point, we still were not ready to call this box infected or owned. The amount of IRC connections was still very odd, but there was no smoking gun.

We started to go back through all of the connection events and reported PVS vulnerabilities. PVS had reported a web client, but there was very little port 80 traffic and it didn’t even show up on the summary of top active ports. Graphing out all port 80 traffic for the past month showed the following activity pattern:

This host had made several port 80 connections to a single remote host. Typically, when we see a connection like this at a research site, we will attempt to gather packets to see if the site is hosting malware or some other type of hostile code. However, we saw no such connections and instead decided to visit the site directly with a secured web client. Upon doing so, the IP address returned the following text:

The bshellz site offers Debian shells to its customers.

After doing some further analysis, it was discovered that the IP address (and an account) of our internal server had been posted online. The continuous IRC connection had been to advertise the existence and availability of the server.

Lessons Learned

#1 - Don’t assume IRC activity means a botnet

Almost any time someone mentions IRC, I tend to immediately think of a system infected or being controlled as part of a botnet. However, in this case a password for the Linux server was easily guessable and was being used for some light IRC activity.

#2 – Access Control Monitoring was more important than detecting anomalies

We were asked to look at the server in question because of some IRC connections that had shown up in some firewall logs that we did not have direct access to. Seeing the IRC traffic led us down the wrong path.

The few SSH connections were much more interesting in this case. It would have been easier to see that connections to port 22 were coming from unauthorized sources if we had known who was using this server and why.

Additionally, if actual SSH log files and system monitoring tools from the LCE were in effect, we’d have full logs from the SSH server, as well as potentially process accounting so we would not have to guess what was occurring on the host after seeing some network traces.

#3 – System configuration auditing would also have been helpful

Being able to see how the Linux system was configured would have been very helpful. It may have been possible to know the:

• Poor password policy (Nessus can test this)
• If there were any users added (the LCE can log on this)
• If there were any applications added (LCE can log this in real-time and Nessus can list all installed software)

For More Information

This blog is part of an in-depth series of entries named “Event Analysis Training”. These provide examples about how events can be analyzed using a combination of log analysis, system and network traffic events.