Nmap Development: Re: [Yokoso-devel] Updates to http-enum.nse (original) (raw)

nmap-dev logo

Nmap Developmentmailing list archives

From: Kevin Johnson <kevin () inguardians com>
Date: Fri, 21 Aug 2009 12:38:27 -0400

On Aug 21, 2009, at 10:09 AM, Ron wrote:

Hi all,

Hi All,

I am one of the lead developers for Yokoso! so I thought I should respond to some of these points here.

I'm cross-posting this to nmap-dev and yokoso-devel, since it sort of affects both the products. I don't really know what etiquette is regarding cross-posting, so hopefully that isn't a no-no. :)

I don't mind it at all. And am also cross-posting. ;-)

On 08/21/2009 02:49 AM, Fyodor wrote:

On Thu, Aug 20, 2009 at 11:57:43AM -0500, Ron wrote:

I agree that the script should only show 200 by default, but give an option to show others. That makes sense. You're going to miss hidden folders, by by default Apache hides certain things so that kinda makes sense.

For the average person, just giving 200 makes sense, and for the

non-average who understand the extra errors, they can enable a script-arg.

This idea makes sense to me.

The format of the fingerprint file is a bit questionable. Comments lines starting with '#' are parsed and then printed in the script output when paths given later in the file are discovered. I realize you didn't invent this format, but it is so simple that it could easily be improved. For example, each path could include the

description on the same line. Or there could be a keyword introducing

the description on one line, followed by the paths on their own lines as they are now. Then we could use comments for notes which we don't want parsed and printed by http-enum. In deciding on the format, it may be worth thinking about how it could be extended if we want to include more information later (just as an idea off the top of my head, we might want to later indicate what status code we look for to address issues such as http-enum reporting that scanme.nmap.org has "TeraStation PRO RAID 0/1/5 Network Attached Storage" just because /cgi-bin/image/shikaku2.png shows forbidden).

The format is a bit odd, and I'd talked to Kevin a bit about changing itto have the ability to AND things together (so if a host contains these

4 files that have common names, but are unique as a combo, it'll check for all of them). At the same time, maybe we can consider other format

changes. Maybe I'll think about it and propose something, see where that

goes.

Regarding including the status code, that is sort of an issue with the different purposes of Yokoso vs Nmap. Yokoso doesn't care what the status code is, but rather a binary: the user has been to the page, or

the user hasn't. Nmap, on the other hand, cares of the page exists. I'm

sure there must be a way to get the best of both worlds, of course -- maybe optional fields or something?

As the guy who created the "format" of the fingerprint file, I think its important to realize that it was just a file. I like the idea of something similar to the nikto DB where we include the status code we expect and a description on the same line. Comments could then be used for notes to end users and ignored by scripts.

Another serious issue involves inclusion of the Yokoso DB. You say:

That last point is the interesting one, to me -- we use the same fileformat as the Yokoso project (by Kevin Johnson and others, from Intel

Guardians).

BTW its InGuardians. Not relevant to this discussion but important to me. :-)

This lets us leverage their fingerprints as well (and they've given me permission to include a copy of their fingerprints file, too,

That was nice of them, but it is important to get more clarification and more explicit permission whenever we include 3rd party code or data into Nmap. I hate dealing with copyright stuff as much as the next guy, but we really need to be very careful about this sort of thing. When they say we can include the DB with Nmap, what does that really mean. Remember that Nmap is open source, so people can incorporate parts into other projects or fork Nmap under a different name. A strict reading of "you may include this file within Nmap"

would not allow such things, which would mean that part of Nmap is not

open source. Also, a strict reading might mean that we can only include the file and not modify it (create derivative works). In general, we can put third party code/data in with Nmap if it is given to us under one of the following licenses (either via special permission or because the code is already under such a license):

o Public domain -- that means people can do whatever they want with it.

o BSD-style (includes MIT license, Lua license, etc.) - preferably 2-clause variant. If it has the advertising clause, we need to mention it explicitly in the man page (http://nmap.org/book/man-legal.html#third-party-soft) and potentially other places.

o Nmap license - if they're OK with us distributing it under the terms

of the Nmap license (http://nmap.org/data/COPYING), that is OK too.

So if they let us use the data under one of these licenses, inclusion

with Nmap is OK. In any case, the license permission granted needs to

be included (described) at the top of the file. We only need license rights to the list of URL paths and descriptions, and not the rest of Yokoso.

Note that even when a data file isn't licensed appropriately for distribution with Nmap, we can generally point to it in the documentation (e.g. if it is a URL somewhere) so users can download and use it.

I'll leave that question for the Yokoso guys to answer.

We currently use the GPL for Yokoso! but have no problems with the NMAP license and would love to see it included with that in NMAP.

One thought I had -- http-enum.nse and Yokoso sort of have different points. http-enum.nse is designed for finding common locations, like

/icons, /scripts, /test, etc, and Yokoso is designed for fingerprintingcommon web apps. So, for that reason, it might make sense to put it in a

different script that the user can run separately. Or maybe not. I'm happy with going either way.

I'm not sure, but my gut reaction is that with a good file format (which doesn't have to be too complex), we could probably combine the Yokoso DB with the existing enum DB. There is also a DB by HD Moore (a NASL script he wrote) which I hope to request permission for us to use.

If we end up with more URLs than we want to scan by default, we could look at either splitting it up into multiple scripts or introducing NSE arguments to select categories of paths to try.

You're probably right about combining the databases. My concern is thatit'll make it more difficult to update when Yokoso is updated -- Yokoso

promises to be an active project, and if they're updating the fingerprints and we've combined the fingerprints, we might run into versioning problems.

I think the best bet is to have multiple fingerprint files of the same format. Yokoso, defaults, extended, etc. Then the script can load whatever it sees fit.

I'd definitely like to use HD's database if it's for the same purpose!

I can take responsibility to keep the yokoso fingerprints updated in the nmap file if you would like!

Thanks for including us, and if there are more questions feel free to ask!

Kevin

Kevin Johnson Senior Security Analyst InGuardians, Inc. office: 202.448.8958 cell: 904.403.8024

Attachment:PGP.sig
Description: This is a digitally signed message part

Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://SecLists.Org

Current thread: