Making Penetration Testers Lives Awful

Awful, awful, awful.....Magic!

It was my wife’s turn to choose a movie the other night, which means there were no kung fu fight scenes, sword fights or car chases. Instead, there was a scene that depicted a father-to-be talking to a father of three children. The father with three children was explaining to the father-to-be what parenthood was really like and stated: "Parenthood is awful... awful… awful... but then there is this magical moment that makes it all worth it… then awful... awful... awful and repeat". Parents reading this, especially ones with small children, are probably laughing. However, I thought that the "awful, awful, awful, magic!" analogy also very accurately described penetration testing.

While many penetration testers may not openly admit it, most of the penetration testing process is pretty awful. It was Vinnie Liu who first described a penetration test to me this way in his presentation "The Good, The Bad, and the Ridiculous". He presented a hand-drawn diagram of how a penetration tester feels during an actual test. A penetration tester will start off strong, then struggle to find vulnerabilities and exploit systems, and then like magic, exploit a system and gain a foothold into the organization. This is how most penetration tests go for me as well. It starts off with the magic part; there is lots of excitement and plenty of systems, services and vulnerabilities to attack. You get giddy like a 5-year-old on Christmas morning. Then reality sets in and you realize that an open FTP server or XSS vulnerability doesn't really help you all that much. This is when the "awful" starts to set in and you try really hard to make something of what you've found so far, but it's an uphill battle. I believe that unless you get lucky, this is where the majority of penetration testing time is spent. In this "awful" stage, you are looking for that "aha!" moment. Whether it is through reverse engineering code or guessing a password, you need a way out of this awful stage. Good penetration testers will rise above this stage and make the "magic" happen. They will compromise a system, find and exploit a weakness and gain control of sensitive information. This magic is short-lived however, because then the report writing phase begins, and more "awfulness" ensues.

Keeping the Penetration Tester in the "Awful" Stage

While I have spent a significant amount of time on the offense, I want to take the opportunity to look at this from a defensive perspective. Many defenders ask, "What can we do as defenders to make the attackers' lives more difficult?” This is an excellent question! I came up with the following list based on some of my experiences interviewing people in the industry on various podcasts and recent presentations by John Strand and ("Ron Gula"). Below are some of most interesting, and what I think are effective, methods to foil penetration testers and attackers alike:

1. Password auditing - One of the easiest and most reliable methods for gaining access to a host is to guess a username and password. Systems will often be connected to the network and the default username and password will not be changed! It's no secret that this is the first thing that many attackers will look for. As a defender you must audit your entire network on a regular basis for these default values. For example, Nessus comes with several plugins that test default login credentials, and several other tools exist that can be automated to detect default passwords. The most important thing is to have a process in place to change usernames and passwords once they have been guessed or cracked, depending on which method you choose.
2. Patching - Most organizations have some type of patch management process in place to apply the latest security patches. However, on a large percentage of tests there are systems that are missing patches and have fallen out of the regular patch cycle. Network scanning will quickly identify these systems, and when found, they must be patched or pulled off the network immediately. Even if it's a test system, or a system that should be "end of life" very soon, it should be remediated. Attackers will compromise these systems and use them to gain a foothold in your network.
3. Monitoring - I believe there to be two areas of monitoring that go overlooked that are highly effective at catching attackers. The first is outgoing network traffic. There is no secret, and it's not a new concept, but looking at outgoing connections can yield a high rate of detection of compromised hosts. I've spent time monitoring university networks for compromises, and simply using netflow analysis tools, I have been able to pick out hosts exhibiting suspicious behavior. Not only that, but correlating this data with other network traffic or other system logs can find multiple hosts that have been compromised. For example, if one host is connecting to a known botnet command and control server (or penetration testing framework "phone home" server), you can query the network logs for other systems making connections to that host. Another great source of correlation is antivirus logs. If hosts are making suspicious outgoing connections and triggering antivirus alerts, this is a telltale sign that the host has been compromised. While antivirus software can be quite noisy, a great tip is to look at alerts that are happening outside of normal business hours on end-user workstations, which indicates a remote attacker may be trying to upload a malicious payload.
4. Different Defensive Countermeasures - I really like the methods in this category as they represent some "out-of-the-box" thinking when it comes to defense. I firmly believe that the following techniques need to be in more widespread use if we are to protect our networks and information from attackers:
• Fake DNS records - To use an analogy, this is the canary in the mine. It is simple to implement, just create DNS records using common names and point them to IP addresses that are not in use or lead to a honeypot system. For example, DNS subdomain brute forcing or guessing is common practice. Attackers are betting that your email server is called "imap.yourdomain.com", and test to see if that server exists. If you have a known set of these records and tell your users to use different names such as "mail.yourdomain.com", you can potentially catch nefarious activity. The one downside is that your users may get confused, see "imap.yourdomain.com" and trip your honeypot.
• Bogus robots.txt - This is one of my favorite tricks, and I've been using it on all my public web servers. It is similar to the fake DNS records in that you create fake entries in your robots.txt file that go to known "interesting" directories. For example, I like to create an entry for "/secret" that goes to a page that logs the attacker's IP address. Most penetration testers will look at robots.txt and trigger this alert.
• Dark IP space - I've implemented monitoring for so-called "dark" IP address space for quite some time with excellent results. All you have to do is route IP address space that is not in use. Since there is no reason for anyone to go to the IP addresses that you know are not in use, it's a good way to catch people port scanning the network from the outside or from the inside. If a host on the internal network starts contacting IP addresses that have never been in use, there is a good chance the host has been compromised. Dark IP space is a good place to point your fake DNS records, or even host honeypot systems.
• Honeypots - Honeypots are a great way to observe attackers in action, however, they are akin to playing with fire. If you do set up a honeypot, be certain that it's not easily identifiable as a honeypot and that the attackers can do very little once the honeypot has been compromised. Typically, firewall rules and bandwidth limiting are good enough to protect the rest of the network. I've always thought that building a system and configuring it with your own hardening guidelines, then using it as a honeypot, is a good way to test your systems and procedures.
• Outbound ACLs - One of the techniques used most sucessfully by penetration testers and attackers is to use a reverse connecting HTTPS backdoor. This is difficult to defend against, as it is encrypted and must be let out in order for your network to function. However, it's beneficial to your defenses to make sure this is the only way out of your network. Make the penetration tester use this method. If anything else is used (such as SSH) to leave the network then this will be logged and is indicative of an attack.
• Respond on every port to a scanner - Most penetration testers, and sometimes attackers, will execute a port scan against a host or network. There are ways that you can configure the network to respond to a scanner as if every port is open. This makes it difficult for attackers to figure out what ports are really open, and therefore more difficult to attack the services.

Conclusion

The methods outlined above will give you more information about penetration testers and others attacking your network. This information by itself is useful; however, correlating the data with other sources such as intrusion detection logs, web server logs and general system logs will help you put together the pieces of how attackers are gaining access to your systems. Offensive countermeasures will cause attackers to leave a trail that will allow you to more easily detect their activity and respond accordingly.