20/20 Hindsight – Walmart Lessons Learned for Tenable Customers

Wired magazine recently ran an excellent story detailing how Walmart suffered a deep intrusion. The story provides many examples of cliché security lapses such as not disabling a remote VPN account for a former Walmart worker. This blog entry describes how customers using Tenable Unified Security Monitoring solutions can learn from these mistakes and get more value out of their investment with Tenable.

Crash Analysis

Walmart IT staff was first alerted to the compromise when they responded to a report of a crashed server. The server crashed when the intruder ran the L0phtcrack password-cracking tool.

Logs that indicate crashes or reboots can be valuable for analysis and trending. Although servers and applications can crash because of high usage, resource starvation, poor configuration or design,  most of the time they do not crash without external help. Malicious users can cause systems to inadvertently crash because they are trying to exploit a technical attack such as a buffer overflow or  because they are specifically attempting a denial of service attack or unintentionally exhausting resources such as memory or disk space.

Performing crash analysis with Tenable’s Log Correlation Engine is very straightforward.

• Tenable’s log normalization rules recognize crashes, critical errors and restarts in a wide variety of applications and operating systems.
• The first time a server crash or reboot is observed, it will be flagged as a “first seen” type of event by the “Never Before Seen” correlation rule.
• The amount of crashes for a server will be statistically profiled and an alert will be generated for any sequences of crashes outside of the normal variation.
• Any set of crashes occurring on multiple servers in a short period of time will also generate and alert as a potential network worm or DoS event.
• If a network intrusion detection system is present, any crash that was preceded by a detected attack will also be highlighted.

Program Installation

When the Walmart IT administrators detected the L0phtcrack program installation, they realized that the system was likely compromised. Consider what could have happened if the crash never occurred. If L0phtcrack had run without any issue, the IT administrators may have never known the system was compromised. This is where program usage auditing can be of great value.

For both Unix and Windows systems, the Log Correlation Engine (LCE) will automatically summarize all unique programs run during a given hour or day as well as all of the programs run by a certain user on a daily basis. This can facilitate auditing what is actually running on a system. Perhaps more useful is that when the correlation rule for the Log Correlation Engine sees a new program for the first time, it will generate an alert. In Walmart’s case, they would have seen an alert that said L0phtcrack was being used for the first time.

In general, I feel that any time a program is installed or a system is changed, this is worth correlating and tracking. The Log Correlation Engine can highlight any event that indicates  a system change. In an enterprise environment, change is something that needs to be tracked and all changes must be authorized. If it cannot be explained, it must be investigated.

Lastly, although the article did not mention it specifically, I’m sure Walmart did an audit to determine if L0phtcrack was installed on its servers. If these servers were regularly scanned with Nessus and the Security Center with credentials, a list of all installed programs would be readily available for searching.

Login and User Activity

The Wired article further explains that the remote hacker had leveraged the VPN login of a former Walmart employee, and then leveraged a login to a server using a generic network administrator account. This type of transitive identity can be very difficult to track and analyze unless some automation is used to simplify the process.

A key feature of the Log Correlation Engine for insider threat monitoring is the ability to dynamically associate a user ID to an IP address. In the case of the Walmart incident, if the logs from the VPN device were present, these could have been used to track the user’s activity by filtering on the user’s IP address automatically and over time.

For example, if the hacker logged into the VPN account on June 1^st with an account of “joesmith” and received an IP address of 172.10.10.10, any other log that was sent to the Log Correlation Engine involving this IP would get tagged as belonging to “joesmith”. This includes firewall logs, intrusion detection logs, web logs and even logins to other servers such as the generic network administrator account. If “joesmith” logs in again and gets a new IP address such as 172.20.20.50, then all logs associated with this IP will be tagged as “joesmith” as well.

In a situation such as Walmart’s, this type of user tracking solves two unique problems. First, it allows for simple analysis of what the account “joesmith” did. The article mentioned that Walmart was very concerned with finding out what the account was used for. If logs were present from firewalls, servers and other sources, the Log Correlation Engine could have presented a simple picture of what had occurred. Second, this account was being abused for some time. Each time the VPN was used to log in, it is possible that a different IP address was given to the connection. Trying to map these changes over the weeks or months of access is difficult to do manually. If the logs are immediately tagged with the user name, it is much easier to track.

Conclusion

Please keep in mind that any type of analysis of an attack that occurred several years ago is leveraging “20/20” hindsight. An attacker who breaks into your network today will likely try new tactics and not repeat the methods that have been exposed. However, as information security practitioners, we have the obligation to defend against all threats and we can learn from the information shared by the Walmart intrusion.