Auditing 100,000 Hosts or More with Nessus

Recently, the State Department Deputy CIO and CISO John Streufert participated in a podcast where he talked about moving past the Federal Information Security Management Act (FISMA) to a metrics based security program. Performing routine vulnerability scans is a key metric to his strategy and he referenced the State Department’s Tenable solution for accomplishing this. After this podcast, Tenable received several inbound requests for more information on very large-scale network scanning from a variety of federal and commercial organizations. This blog entry summarizes some of the political and deployment strategies our customers use to scan hundreds of thousands of hosts on an ongoing basis with multiple Nessus scanners and the Security Center.

The Politics of Scanning

The larger your organization is, the more issues you are likely to face when monitoring its security. Tenable customers often cite the following issues when performing rollouts of large numbers of Nessus scanners and performing the actual scans. Consider these issues before doing any rollout of a large scanner deployment.

Scanners must be placed behind firewalls

To be the most effective, a Nessus scanner needs to have unimpeded access to the hosts it is scanning. This often means that the scanners are deployed behind an organization’s firewall. Any scanner deployments that require a firewall rule change can take longer than desired.

In larger organizations, firewall rule changes need to be reviewed in a committee or authorized at the executive level. Firewalls are also used within different parts of an organization and not just on “perimeter” Internet connections.

Scans must be given credentials to perform a full audit

Nessus can perform very detailed and low impact configuration (such as FDCC scans) and patch audits if given the proper credentials to log into a host or server. In larger organizations, these credentials need to be managed in accordance with a password policy that mandates requirements such as use of complex passwords, limiting who has access to them and changing them on a timely basis.

In larger organizations, it can become politically difficult to give the required access to an IT auditing group. To compensate for this, the IT auditing group must partner with IT and work within the security policy to ensure that patch audits and configuration testing are performed within the organization’s requirements.

Production systems are off limits for ad hoc scanning

For any given network of sufficient size, certain portions of it will be placed off limits to active or ad-hoc scanning. This is a fact that is very common across all of our customers with network nodes numbering more than 50,000.

When I first encountered this, I felt it was a reaction to the perception that vulnerability scanning could have a negative impact on the systems being scanned. I offered some insights to our customers about how Tenable mitigates the impact of a Nessus scan. However, I was usually assured it was not scanning that was in question; it was the fact that the network, routers, firewalls, servers, databases or whatever were “frozen”: nothing was to be done to them. No new agents. No new configurations. No change at all.

From a political point of view, the ability to scan a very large network is of extreme value, but the ability to exclude a portion of this network from the scanning is even better. To compensate for monitoring this type of excluded network, consider using the Tenable Passive Vulnerability Scanner as described in this example blog that describes how it can be used to monitor networks during end of year freezes.

Deploying the Scanners

Once the political issues have been resolved, what does an actual deployment of multiple Nessus scanners and a Security Center look like?

Deploying Security Center

The Security Center application is available as software, as a virtual appliance or as a hardware appliance. It does not run a database, but it does run many different types of Unix processes. For very large organizations, Tenable recommends having the fastest possible disks available, 8 GB of memory and at least 4 CPU cores to take advantage of parallel task processing. By today’s standards, most of our customers feel this is a lower end server or something that can run effectively in a virtual environment.

Deploying Nessus Scanners

Like the Security Center, Nessus is available as software, a virtual appliance or on a hardware appliance. Our hardware appliance comes with four physical network interfaces, facilitating direct access to switching VLANs.

Scanner Topology

Nessus scanners can be placed into scanning “zones” - a set of scanners and scan targets. Any scan of an IP or network within a zone is load balanced between the scanners assigned to that zone. This allows many different types of topologies to be deployed based on the underlying network. Some examples include:

• Adding one zone per business unit and one scanner within each zone.
• Using one large zone and placing multiple scanners in this zone.
• Placing zones “on the other side” of slow links that are limited in bandwidth or latency.

There is no right answer on how to deploy scanners. Tenable has been able to support many large enterprise customers over the past years and we rarely see two organizations perform scanner deployment the same way.

For example, a regional bank with 30 physical locations may deploy five scanners internally at their data center and just perform scans over the network links. A similar bank may choose to put a scanner at each physical location.

To illustrate this concept of differing topologies a bit further, Tenable has operational customers that use 40 Nessus scanners to audit 300,000 nodes as well as one that deployed 300 Nessus scanners to scan 37,000 nodes. The use of 300 scanners was driven by a requirement to deploy a scanner inside 300 different locations.

Getting Operational Results

I have the opportunity to speak with a lot of our customers and have asked the administrators who run our products what they do to get the most out of their efforts. Some of their ideas are listed below:

Assume the Scanner Will Be Running at 100% CPU

At Tenable, we put a lot of effort into increasing the performance of Nessus. Many of our very large customers find it easier to just schedule more scans than add new scanners. If Nessus is scanning all of the time against a large number of targets, the CPU usage will be very high.

This makes management of the Nessus scanner an issue that needs to be tracked. Many of our customers use the Security Center to control how often new plugins are pushed to each scanner and indexed. Having applications at 100% CPU all of the time can make network monitoring and applying patches more difficult.

IT Changes Can Ruin Your Scanning

Our more experienced customers with very large scanning deployments are in sync with their IT changes. If they were not, it is possible that a simple router, firewall rule, host configuration or other issue can make their scan fail. For example, what should your corporate policy be to conduct a re-scan if half of your vulnerability scanners go “off line” during a scan because of a firewall change which prevents communication to them?

Another basic issue is management of system credentials. If you are scanning a very large domain and there is a password change for the audit account, it is important that this new password be given to your scan policy so the scan will be able to log into the hosts being audited.

Tenable has also seen network changes affect scan results. Performing a credentialed scan during a time when DHCP leases are set to expire can ensure that your targets change IP addresses during the scan. Similarly, scheduling scans at the same time the “deep” anti-virus scan is being run can also make both processes take much longer than they should.

There May Still be a Need for “Manual” Scanners

Enterprise scanning may require a manual mobile solution for certain areas of the network, such as the DMZ. Tenable has some customers with very strict firewall polices that necessitate connecting directly into the protected network and performing a direct scan. Nessus on a laptop fits this bill very well. Results can be later uploaded manually into the Security Center. In addition, it is not unusual for Tenable's larger customers to acquire another company, requiring an audit before the networks of each organization can be connected. A mobile solution is perfect for this situation. Later on when the networks are joined, Enterprise Vulnerability Assessment (VA) can be rolled out to incorporate the acquired organization.

Scanning Over “Slow” WAN Links

In a large distributed environment, WAN connections usually prove a challenging obstruction to designing a centralized enterprise VA solution. The nature of leasing bandwidth on WAN links or having limited bandwidth to begin with results in organizations generally only paying for the bandwidth they absolutely need. WAN connections usually have a bad reputation for being slow but this can  be a side effect of a deliberate network design to get the most “bang for the buck” out of a leased line.

If credentials are not available, scanning through WAN connections with Nessus limits advanced techniques such as port scanning with netstat. This can result in false negatives since extreme network latency can cause connection timeouts and dropped packets. One solution to this situation is to perform automated scheduled scans during defined scan windows - periods of time when a WAN connection is not heavily utilized such as off-peak business hours.

The optimum solution, however, is to place Nessus at the other end of the WAN connection near the scan targets as the bandwidth usage between Nessus and the Security Center is minimal. When firewalls are involved, it is also not unusual for a remote location at the other end of a WAN connection to have perimeter defenses at the WAN entry point, which will obstruct an active vulnerability auditing tool such as Nessus.

You May Need to Move Your Scanners

Many customers have told me that they appreciate that they can move their Nessus scanners around or add new ones at any time. Nessus is available to run on many different software platforms and our customers also have access to a hardened virtual appliance. Up to 500 Nessus scanner can be managed by a single Security Center under our licensing. If your network changes over time, you can move the Nessus scanners around as needed.

Focus on Your Business, not the Raw Vulnerability Counts

I’ve had the chance to see a lot of customers use the vulnerability data from hundreds of thousands of hosts both effectively and ineffectively. Below are some techniques I’ve seen work well for large scale vulnerability management.

• Divide your network up into the business units, groups or political organizations that comprise your organization. This allows you to treat each of these groups as a unique entity or as part of the big picture.
• Determine what sort of access control each group is supposed to have with the Intranet, Internet and with other organizations before deploying scans specifically designed to monitor for changes in that access.
• Determine what sort of patch life cycle each group needs to have and then develop reports and alerts that focus on vulnerabilities that indicate a failed or late patch cycle.
• Identify hosts that are authorized or unauthorized as a separate effort. I have seen organizations look for a lack of a DNS entry for a given IP address, the presence of a registry key, unauthorized operating systems or applications and many other types of technical tests to look for hosts that are not supposed to be there. The hard part is coming up with the test and then getting the buy-in from executive management to perform these scans.

For More Information

To learn more about using the Security Center and Nessus to perform scans of large numbers of network nodes, please contact our sales staff. Tenable has also written a whitepaper about using active scans, credentialed scans and passive network monitoring titled “Blended Vulnerability Assessments”. Another useful Tenable whitepaper is “Maximizing ROI in Vulnerability Management Programs”.

Last, Tenable customers can communicate with each other and exchange tips for scanner deployment and large scale scans on the Tenable Discussion Forums.