Nmap Development: Re: Nmap's memory use (original) (raw)

nmap-dev logo

Nmap Developmentmailing list archives

From: David Fifield <david () bamsoftware com>
Date: Sat, 19 Dec 2009 15:19:26 -0700

On Thu, Nov 19, 2009 at 06:04:58PM -0700, David Fifield wrote:

We've had some report recently about Nmap using a lot of memory.

"Port memory bloat" http://seclists.org/nmap-dev/2009/q3/926 "nmap 5 memory usage" http://seclists.org/nmap-dev/2009/q4/300

In the first link above, Pavel Kankovsky observed that it is the Port class that is using up most of the memory in some scans. My preliminary tests agree with this. I used the Massif memory profiling too (http://valgrind.org/docs/manual/ms-manual.html) from the Valgrind suite as follows:

Pavel provided a patch that reduces the memory usage of all the Ports in the default state at the end. I think that is the right idea. I am going to try to see if there's a way to reduce their memory use even further, perhaps by changing the programming interface so that the creation of discrete objects isn't necessary.

I've finished working on memory reduction for now. Apart from the earlier nmap-os-db parsing memory improvement described in http://seclists.org/nmap-dev/2009/q4/479, I just merged two more important improvements from the nmap-mem branch in r16308.

For a bit of context, first look at this benchmark scan, all UDP ports on four hosts. It uses a maximum of 33.63 MB, almost all of it coming from addPort at the end of the scan. http://www.bamsoftware.com/wiki/Nmap/Memory#a20091118

The first new improvement was a reduction in size of Port objects. I used Pavel's idea of dynamically allocating parts of Port that aren't need for every port, like service and RPC information. The size of a bare Port object dropped from 92 to 40 bytes on my machine. Now the peak memory usage is 17.66 MB. http://www.bamsoftware.com/wiki/Nmap/Memory#r16276

The second improvement is to avoid allocating some Port objects at all. In many scans, the majority of probes don't receive a response. The big spike in memory at the end of the previous two graphs is thousands of Ports being created for the open|filtered ports that didn't respond. Now the code has a notion of a "default port state" for ports that don't get a response. All the ports in the default state share a single Port object. This further reduced memory use to 5.94 MB. http://www.bamsoftware.com/wiki/Nmap/Memory#r16290

The second improvement won't help, for example, when TCP ports are closed because of RSTs rather than being filtered, but you will still get the benefit of smaller Port objects.

David Fifield

Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://seclists.org/nmap-dev/

Current thread: