[NSE] HTTP Library w/ Caching (original) (raw)

nmap-dev logo

Nmap Developmentmailing list archives

From: Patrick Donnelly <batrick () batbytes com>
Date: Sun, 23 Aug 2009 21:15:57 -0400

The HTTP Library now caches responses generated from http.get or http.head (in revision 15243). Because many scripts tend to request the same object, this helps to prevent sending duplicate requests that consume network resources needlessly.

The cache is transparent to the script writer in that it will return a cached result if present when http.get/http.head is called. How the cache is used may be controlled by options table (described in [1]) passed to the http functions. Three new boolean values are present: o bypass_cache -- The contents of the cache is ignored for the request (method == "GET" or "HEAD") o no_cache -- The result of the request is not saved in the cache (method == "GET" or "HEAD") o no_cache_body -- The body of the request is not saved in the cache (method == "GET" or "HEAD")

The size of the cache is by default 1 MB. The size can be changed using the script arg (--script-args) http-max-cache-size (e.g. --script-args http-max-cache-size=1e8).

In some recent simple benchmarks (over a number of different hostgroup sizes) the cache had a large improvement on bytes saved over the network. For a cache size of 1e5 (100 Kilobytes), the cache would generally save about 50% the traffic that would normally be sent during the scan. For larger host groups this number decreases. For a cache size of 1e8 (100 Megabytes), the cache would generally save about 200% as much traffic over the network (that is, the cache saved twice as much traffic as was sent in the scan).

[1] http://nmap.org/nsedoc/lib/http.html#request

-- -Patrick Donnelly

"Let all men know thee, but no man know thee thoroughly: Men freely ford that see the shallows."

Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://SecLists.Org

Current thread: