Can you send the Nmap Project some scan data?

Daniel Miller <[email protected]> Tue, 9 Oct 2018 13:21:57 -0500

Newsgroups	gmane.comp.security.nmap.devel
Message-ID	<CABmvJnPMgBygoXpwzHm=jsiEhj2UWue2fG+8A98X80bARh3CHA@mail.gmail.com>

Nmap community members and developers,

The Nmap Project is in need of current port scans in order to update the
open-port frequency data that Nmap uses to determine which ports to scan.
This data is used every time a user scans the default 1000 ports, and it is
based on scan data that is over 10 years old. A lot has changed in that
time, and Nmap's port data is no longer up-to-date.

We are already doing scans of the public Internet for this purpose. What we
desperately need is port scans of *internal* networks, behind the firewalls
and NAT, from as many different sources as possible. Ideal data would be
all-port scans (-p 1-65535), but everything is helpful, especially scans
that include more than the current default 1000 ports or that include UDP
or SCTP port scan results.

In order to make sharing scans safer and easier, I've attached a couple of
scripts that can be used to strip the output files of any identifying
information, including service and OS fingerprinting results, hostnames, IP
addresses, and traceroute data. Our preference is for XML output files,
which can be processed with nmap-sanitize.py. If you have gnmap scan output
instead, gnmap-convert.pl will convert it into a stripped-down XML format.

I really appreciate any help you can provide in this regard. Accurate port
frequency data will result in faster and more complete scan results for
everyone.

Dan

_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/

nmap-sanitize.py (text/x-python, 2.3 KB)

#!/usr/bin/env python2

import struct
import xml
xml.__path__ = [x for x in xml.__path__ if "_xmlplus" not in x]

import xml.sax
import xml.sax.saxutils

class NmapSanitizer(xml.sax.saxutils.XMLGenerator, object):
    """The xml.sax ContentHandler for the XML parser. Ensures only sanitized
    elements are written back out and addresses are rewritten."""
    def __init__(self, f):
        self.f = f
        self.block = 0
        self.block_elements = (
                u"output",
                u"script",
                u"hostscript",
                u"prescript",
                u"postscript",
                u"hostnames",
                u"service",
                u"os",
                u"uptime",
                u"tcpsequence",
                u"ipidsequence",
                u"tcptssequence",
                u"trace",
                )
        self.targetnum = 10 << 24
        super(NmapSanitizer, self).__init__(f)

    def startElement(self, name, attrs):
        if name in self.block_elements:
            self.block += 1
        if self.block > 0:
            return
        if name == u"address":
            if attrs[u"addrtype"] == u"mac":
                return
            else:
                attrs = dict(attrs)
                self.targetnum += 1
                addr = u".".join(map(unicode, struct.unpack("BBBB", struct.pack(">I", self.targetnum))))
                if attrs[u"addrtype"] == u"ipv6":
                    addr = u"::ffff:" + addr
                attrs[u"addr"] = addr
        super(NmapSanitizer, self).startElement(name, attrs)

    def endElement(self, name):
        if name in self.block_elements:
            self.block -= 1
            return
        if self.block > 0:
            return
        super(NmapSanitizer, self).endElement(name)

    def characters(self, content):
        if self.block > 0:
            return
        super(NmapSanitizer, self).characters(content)

if __name__ == "__main__":
    import sys
    parser = xml.sax.make_parser()
    handler = NmapSanitizer(sys.stdout)
    parser.setContentHandler(handler)
    try:
        parser.parse(sys.argv[1])
    except xml.sax.SAXParseException, e:
        # We expect to be processing half-completed XML files, so ignore parsing
        # errors.
        print >>sys.stderr, "Ignored SAXParseException: %s" % e.getMessage()

gnmap-convert.pl (application/x-perl, 1.9 KB)

#!/usr/bin/perl

use strict;
use warnings;

print q{
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE nmaprun>
};

sub escapechar {
    my $x = shift;
    return sprintf("&#%d;", ord $x)
}
sub xmlescape {
    my $x = shift;
    $x =~ s/([\x00-\x1f]<>&'"[\x7f-\xff]|(?<=-)-)/escapechar $1/eg;
    return $x;
}

do {
    my $counter = 0x0a000000; # 10.0.0.0
    sub get_next_addr {
        my $addr = shift;
        my $genaddr = join(".", unpack("C4", pack("N", ++$counter)));
        my $af = "ipv4";
        if (index($addr, ":") != -1) {
            $genaddr = "::ffff:$genaddr";
            $af = "ipv6"
        }
        return qq{<address addr="$genaddr" addrtype="$af"/>\n};
    }
};

while (<>) {
    if (/^# Nmap (.*?) scan initiated (.*?) as: (.*)$/) {
        print q{<nmaprun scanner="nmap" args="}, xmlescape($3),
        q{" startstr="}, xmlescape($2), q{" version="}, xmlescape($1),
        qq{" xmloutputversion="1.04">\n};
    }
    elsif (/^Host: (\S+) [^\t]*\tPorts: ([^\t]*)\t.*Ignored State: (\S+) \((\d+)\)/) {
        my $ip = $1;
        my $portstring = $2;
        my $ignoredstate = $3;
        my $ignoredcount = $4;
        print q{<host><status state="up"/>}, get_next_addr($1),
        q{<ports><extraports state="}, xmlescape($ignoredstate), qq{" count="$ignoredcount"/>\n};
        while ($portstring =~ m|(\d+)/([^/]*)/([tcuds]+p)/[^,]*|g) {
            print qq{<port protocol="$3" portid="$1"><state state="}, xmlescape($2), qq{"/></port>\n};
        }
        print qq{</ports></host>\n};
    }
    elsif (/^# Nmap done at (.*?) -- (\d+) IP addresses \((\d+) hosts up\) scanned in ([\d.]+) seconds/) {
        my $time = xmlescape($1);
        my $total = $2;
        my $up = $3;
        my $elap = xmlescape($4);
        chomp;
        s/^# //;
        print qq{<runstats><finished timestr="$time" elapsed="$elap" summary="}, xmlescape($_),
        qq{" exit="success"/><hosts up="$up" total="$total"/>\n};
    }
}

print "</nmaprun>\n";