Re: Feature: per-target port specification (with patch!)
Jan GocnÃk <[email protected]> Fri, 3 May 2019 13:46:14 +0200
| Newsgroups | gmane.comp.security.nmap.devel |
|---|---|
| Message-ID | <OFFE7CE571.F58699D2-ONC12583EF.00403CE7-C12583EF.0040A89E@dcit.cz> |
This is a multipart message in MIME format. --===============8778059700860649237== Content-Type: multipart/alternative; boundary="=_alternative 0040A89CC12583EF_=" This is a multipart message in MIME format. --=_alternative 0040A89CC12583EF_= Content-Type: text/plain; charset="ISO-8859-2" Content-Transfer-Encoding: quoted-printable Hey, it's been a month since I submitted this patch. Is there anything else I=20 can do to help it eventually get through? If it's just that you had no time to check it so far, no worries, I get=20 that maintaining OSS projects must be hard :) Jan From: Jan Gocn=EDk/Dcit To: "Daniel Miller" <[email protected]> Cc: "Nmap-dev" <[email protected]> Date: 08.04.2019 15:15 Subject: Re: Feature: per-target port specification (with patch!) Hey, I worked on this some more. Fixed all of the memory leaks that were my fault, and the crash that was a = result of a bug in mergeHostSpecificPorts. I took this opportunity to=20 rewrite the mergeHostSpecificPorts algorithm, so now the results are=20 properly sorted. The disrepancy between "Scanning X [max N ports]" and "Completed Connect=20 Scan at X (0 ports max)" should be fixed as well, the totalprobes value is = now properly initialized. About the features you mentioned: > The "Not shown: X ports" output for Normal output. Well, from what I know, this doesn't always allow you to infer information = about all ports anyway. For example, if you get "Not shown: 3995 closed=20 ports, 514 filtered ports". Therefore, as long as the counts are=20 calculated correctly, I don't think it's necessary to output information=20 about additional scanned ports, as you can get these from other more=20 verbose outputs. > Properly formed XML output, with changes to the DTD and a=20 "xmloutputversion" number increase. I simply output another scaninfo tag into each host tag. Updated the DTD=20 and xmloutputversion to 1.05. For greppable, I output it in a format similiar to the ports listing. > Combination of this feature with existing --top-ports/port-ratio and -p=20 options From my testing, it merges with them properly. > Combination of this feature with CIDR subnetting and IPv4 octet ranges Should work just fine. > Use of this feature along with advanced features like -O --traceroute=20 and -sV I tested it with these options and everything seemed good to me. On the topic of memory usage: I wrote it so that if you do regular scans, the additional memory usage=20 should be very small - only a scan=5Flists per target group, which is <64=20 bytes, and a few pointers here and there. I can't get reproducible heap=20 reports form valgrind unfortunately, but in one case the new version even=20 allocated less memory than latest SVN trunk for "nmap scanme.nmap.org". The biggest item of all is port=5Fmap=5Frev, which is 65536*sizeof(u16) =3D= 128=20 kB. This is allocated for each target that has specific ports (but only=20 for those). Only way I can think of making this better is sharing those=20 for the whole target group, but I dunno if it's worth the effort. Now, on the rationale. A very specific (real) example when this feature is useful: We are=20 scanning a network, which has some kind of network appliance that replies=20 to SYNs on port 80 on every unleased IP, but doesn't reply to most other=20 ports at all (not even with a RST). That means that nmap considers every=20 single IP live, but then waits for a long time when attempting to scan all the other=20 ports. I was scanning 521 IPs, of which 11 were really occupied and active = and the 511 others were the "fakes". I was testing three options: Scan all hosts for all ports that appeared at = least once (marked all=5Fports), write a simple script that calls nmap for = each target (script), and then used the per-target ports (per=5Ftarget). With ping scan, the results were the following: * all=5Fports: 102 seconds * per=5Ftarget: 80 seconds * script: 60 seconds Without ping scan: * script: 45 seconds * all=5Fports: 38 seconds * per=5Ftarget: 4 seconds My conclusion is that this feature is useful when you get filtered ports,=20 as these take a lot of time. Basically, if you only have open and closed=20 ports, scanning some additional ones is fast and the "union all ports"=20 strategy is alright. But once you have filtered ports, the waiting quickly = gets bad. The script could be upgraded to be parallel, but then you're reinventing=20 nmap's parallel engine. Now I know the "proper" solution on this network would be to use a brain=20 and try to discern real and fake machines, but this already gets much=20 faster results without a need for thinking, which is always nice. I understand your concern about losing "new" services, but I see the main=20 usability of this in manual scans, for example running additional nse=20 scripts against your last scan results etc. When you are doing automatic=20 scans (for exmple as a network admin scanning network for new devices each = week), you probably don't need to care about speed anyway... Updated patch (to svn revision 37611): [attachment "nmap=5Fpertarget=5F37611.patch" deleted by Jan Gocn=EDk/Dcit] = Looking forward to further comments and hopefully we can make this work :) Jan From: Jan Gocn=EDk/Dcit To: "Daniel Miller" <[email protected]> Cc: "Nmap-dev" <[email protected]> Date: 02.04.2019 23:53 Subject: Re: Feature: per-target port specification (with patch!) Hey Dan, thanks for the reply! It's a shame that I didn't find the GitHub issue you = link to before implementing this, as it does raise a lot of valid=20 concerns. First, let me say that the company I work for wants this feature, so even=20 if it doesn't end up in upstream, I will try to keep it at least as a fork = - as I'll have to maintain it internally anyway, I wanted to share with=20 the community, in case others have a need for it as well. I will go through all the things you mentioned (most of the compatibility=20 with other options should be taken care of, but memory leaks are a=20 problem), fix up the code, look at maybe getting the memory footprint=20 lower, and try to come up with some stronger numbers and rationale. Jan From: "Daniel Miller" <[email protected]> To: "Jan Gocn=EDk" <[email protected]> Cc: "Nmap-dev" <[email protected]> Date: 02.04.2019 21:21 Subject: Re: Feature: per-target port specification (with patch!) Sent by: "dev" <[email protected]> Some initial notes from building and testing this: ./nmap scanme.nmap.org^22-80 -d Starting Nmap 7.70SVN ( https://nmap.org ) at 2019-04-02 18:37 UTC PORTS: Using top 1000 ports found open (TCP:1000, UDP:0, SCTP:0) --------------- Timing report --------------- hostgroups: min 1, max 100000 rtt-timeouts: init 1000, min 100, max 10000 max-scan-delay: TCP 1000, UDP 1000, SCTP 1000 parallelism: min 0, max 0 max-retries: 10, host-timeout: 0 min-rate: 0, max-rate: 0 --------------------------------------------- Initiating Ping Scan at 18:37 Scanning scanme.nmap.org (45.33.32.156) [max 2 ports] Completed Ping Scan at 18:37, 0.09s elapsed (1 total hosts) Overall sending rates: 24.44 packets / s. mass=5Frdns: Using DNS server X.X.X.X Initiating Parallel DNS resolution of 1 host. at 18:37 mass=5Frdns: 0.12s 0/1 [#: 3, OK: 0, NX: 0, DR: 0, SF: 0, TR: 1] Completed Parallel DNS resolution of 1 host. at 18:37, 0.06s elapsed DNS resolution of 1 IPs took 0.15s. Mode: Async [#: 3, OK: 1, NX: 0, DR:=20 0, SF: 0, TR: 1, CN: 0] Initiating Connect Scan at 18:37 Scanning scanme.nmap.org (45.33.32.156) [max 1059 ports] Discovered open port 80/tcp Discovered open port 22/tcp Discovered open port 9929/tcp Discovered open port 31337/tcp nmap: portlist.cc:688: void PortList::mapPort(u16*, u8*) const: Assertion=20 `mapped=5Fportno < port=5Flist=5Fcount[mapped=5Fprotocol]' failed. Aborted (core dumped) Valgrind identified some memory leaks. These were the ones that are=20 definitely from this patch: Portlist::setIdStr(), idstr;=20 PortList::mergeHostSpecificPorts(), new=5Fport=5Fmap and new=5Fport=5Fmap= =5Frev; There was also a discrepancy between the "Scanning X [max N ports]" and=20 "Completed Connect Scan at X (0 ports max)", which valgrind says is due to = an uninitialized value, probably GroupScanStats::totalprobes. Of course, as you noted before, the results are not sorted by port number, = which we would have to do to ensure predictable output that can be=20 compared with previous scans. IPv6 addresses, CIDR, and IPv4 octet ranges seem to work fine with this. With just my one test scan of localhost and scanme.nmap.org with one=20 unique port each and one port in common, the patched code allocated an=20 additional 383KB of heap memory. I do not know how this would scale up or=20 down, and I haven't compared it with a scan that should behave the same in = both cases (for instance, "nmap 192.168.1.0/24". My primary concern remains that repeated use of this feature will result=20 in missing new services and dropping existing ones if they are missed in=20 just one scan. This is more about use case than about the code, though, so = I will defer to users on that. Dan On Tue, Apr 2, 2019 at 1:30 PM Daniel Miller <[email protected]>=20 wrote: Jan, Thanks for this contribution. We've had many requests for this type of=20 feature in the past, but have elected not to include it for a variety of=20 reasons. There is an open discussion on our issue tracker that lays out=20 some of the challenges in correctly implementing such a feature:=20 http://issues.nmap.org/1217 It looks like your patch has tried to handle some of these situations, for = example the "Ports scanned" output for Grepable output (and maybe XML, but = it didn't look complete at first glance). If we are to do an actual code=20 review and include this new feature, we would have to look for a complete=20 solution that can handle the following situations: * The "Not shown: X ports" output for Normal output. * Properly formed XML output, with changes to the DTD and a=20 "xmloutputversion" number increase. * Combination of this feature with existing --top-ports/port-ratio and -p=20 options * Combination of this feature with CIDR subnetting and IPv4 octet ranges * Use of this feature along with advanced features like -O --traceroute=20 and -sV Have you done any measurement of scans before and after adding this=20 feature to determine the actual impact on scan times and bandwidth? Do you = have a bandwidth target for your scans that Nmap is exceeding right now,=20 and by how much? What does a typical nmap command line look like, and what = performance options have you already tried? I look forward to hearing more about this from you and our other devs and=20 users. Dan On Tue, Apr 2, 2019 at 8:07 AM Jan Gocn=EDk <[email protected]> wrote: Hey,=20 I would like to propose a feature enabling specifying ports for each=20 target separately.=20 Rationale:=20 It often happens that we already have an nmap scan of 200 machines, and we = want to do a service scan on those same machines. Usually that forces us=20 to scan the whole network for all the ports that appeared at least once.=20 That is a big waste of time and bandwidth. What we want to have is=20 essentially a rescan-like feature, that would rescan just ports that were=20 found to be open before.=20 User experience:=20 Everywhere where you could specify a target (-iL file, command line) you=20 can supply a "target^ports". It works with all the nmap magic ranges, so=20 "192.168.1.1-255^22-60" works. The common ports (supplied with -p) are=20 scanned on all targets.=20 Implementation details:=20 I tried to keep it so that if you don't use any "^" in the targets, the=20 code path should remain largely the same, so there should be no=20 regressions. However, I had to do some tuning in functions that expected=20 they can just get the number of probes by multiplying common ports by=20 targets.=20 There's a small issue, in that the results of the scan are not sorted=20 properly, as the target-specific ports get scanned last.=20 Usage example:=20 =3D=3D=3Dpaste start=3D=3D=3D=20 $ nmap -v -Pn -n -p22 "165.227.141.119^80,443" "40.113.73.59^8080"=20 Starting Nmap 7.70SVN ( https://nmap.org ) at 2019-04-01 19:46 CEST=20 Initiating SYN Stealth Scan at 19:46=20 Scanning 2 hosts [max 3 ports/host]=20 Discovered open port 22/tcp=20 Discovered open port 80/tcp=20 Discovered open port 443/tcp=20 Discovered open port 22/tcp=20 Completed SYN Stealth Scan at 19:46, 1.45s elapsed (1626388576 total ports = max)=20 Nmap scan report for 165.227.141.119=20 Host is up (0.0090s latency).=20 PORT STATE SERVICE=20 22/tcp open ssh=20 80/tcp open http=20 443/tcp open https=20 Nmap scan report for 40.113.73.59=20 Host is up (0.038s latency).=20 PORT STATE SERVICE=20 22/tcp open ssh=20 8080/tcp filtered http-proxy=20 Read data files from: /home/gocnik/nmap=20 Nmap done: 2 IP addresses (2 hosts up) scanned in 1.52 seconds=20 Raw packets sent: 6 (264B) | Rcvd: 4 (176B)=20 =3D=3D=3Dpaste end=3D=3D=3D=20 If done the usual way:=20 $ nmap -v -Pn -n -p22,80,443,8080 165.227.141.119 40.113.73.59=20 [...]=20 Raw packets sent: 10 (440B) | Rcvd: 6 (260B)=20 The patch is against svn trunk at this moment (revision 37608).=20 Looking forward to all comments!=20 JaGoTu=20 P.S.: Sorry if you recieve this e-mail twice, but the previous one=20 apparently got caught in a moderation queue or something, as it doesn't=20 show on seclists.org=20 =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F Sent through the dev mailing list https://nmap.org/mailman/listinfo/dev Archived at http://seclists.org/nmap-dev/ =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F Sent through the dev mailing list https://nmap.org/mailman/listinfo/dev Archived at http://seclists.org/nmap-dev/ --=_alternative 0040A89CC12583EF_= Content-Type: text/html; charset="ISO-8859-2" Content-Transfer-Encoding: quoted-printable <span style=3D" font-size:10pt;font-family:sans-serif">Hey,</span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">it's been a month since I submitted this patch. Is there anything else I can do to help it eventually get through?</span> <br><span style=3D" font-size:10pt;font-family:sans-serif">If it's just that you had no time to check it so far, no worries, I get that maintaining OSS projects must be hard :)</span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">Jan</span> <br> <br> <br> <br><span style=3D" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Fro= m: </span><span style=3D" font-size:9pt;font-family= :sans-serif">Jan Gocn=EDk/Dcit</span> <br><span style=3D" font-size:9pt;color:#5f5f5f;font-family:sans-serif">To: </span><span style=3D" font-size:9pt;font-family= :sans-serif">"Daniel Miller" <[email protected]></span> <br><span style=3D" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Cc: </span><span style=3D" font-size:9pt;font-family= :sans-serif">"Nmap-dev" <[email protected]></span> <br><span style=3D" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Dat= e: </span><span style=3D" font-size:9pt;font-family= :sans-serif">08.04.2019 15:15</span> <br><span style=3D" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Sub= ject: </span><span style=3D" font-size:9pt;font-family= :sans-serif">Re: Feature: per-target port specification (with patch!)</span> <br> <hr noshade> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">Hey,</span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">I worked on this some more.</span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">Fixed all of the <b>memory leaks</b> that were my fault, and the crash that was a result of a bug in mergeHostSpecificPorts. I took this opportunity to rewrite the mergeHostSpecificPorts algorithm, so now <b>the results are properly sorted</b>.</span> <br><span style=3D" font-size:10pt;font-family:sans-serif">The disrepancy between "Scanning X [max N ports]" and "Completed Connect Scan at X (0 ports max)" should be fixed as well, the totalprobes value is now properly initialized.</span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">About the featur= es you mentioned:</span> <br><span style=3D" font-size:10pt;font-family:sans-serif">> The "N= ot shown: X ports" output for Normal output.</span> <br><span style=3D" font-size:10pt;font-family:sans-serif">Well, from what I know, this doesn't always allow you to infer information about all ports anyway. For example, if you get "Not shown: 3995 closed ports, 514 filtered ports". Therefore, as long as the counts are calculated corre= ctly, I don't think it's necessary to output information about additional scanned ports, as you can get these from other more verbose outputs.</span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">> Properly formed XML output, with changes to the DTD and a "xmloutputversion&quo= t; number increase.</span> <br><span style=3D" font-size:10pt;font-family:sans-serif">I simply output another scaninfo tag into each host tag. Updated the DTD and xmloutputversi= on to 1.05.</span> <br><span style=3D" font-size:10pt;font-family:sans-serif">For greppable, I output it in a format similiar to the ports listing.</span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">> Combination of this feature with existing --top-ports/port-ratio and -p options</span> <br><span style=3D" font-size:10pt;font-family:sans-serif">From my testing, it merges with them properly.</span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">> Combination of this feature with CIDR subnetting and IPv4 octet ranges</span> <br><span style=3D" font-size:10pt;font-family:sans-serif">Should work just fine.</span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">> Use of this feature along with advanced features like -O --traceroute and -sV</span> <br><span style=3D" font-size:10pt;font-family:sans-serif">I tested it with these options and everything seemed good to me.</span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">On the topic of <b>memory usage:</b></span> <br><span style=3D" font-size:10pt;font-family:sans-serif">I wrote it so that if you do regular scans, the additional memory usage should be very small - only a scan=5Flists per target group, which is <64 bytes, and a few pointers here and there. I can't get reproducible heap reports form valgrind unfortunately, but in one case the new version even allocated less memory than latest SVN trunk for "nmap scanme.nmap.org".</sp= an> <br><span style=3D" font-size:10pt;font-family:sans-serif">The biggest item of all is port=5Fmap=5Frev, which is 65536*sizeof(u16) =3D 128 kB. This is = allocated for each target that has specific ports (but only for those). Only way I can think of making this better is sharing those for the whole target group, but I dunno if it's worth the effort.</span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">Now, on the <b>r= ationale.</b></span> <br><span style=3D" font-size:10pt;font-family:sans-serif">A very specific (real) example when this feature is useful: We are scanning a network, which has some kind of network appliance that replies to SYNs on port 80 on every unleased IP, but doesn't reply to most other ports at all (not even with a RST). That means that nmap considers every single IP live,</spa= n> <br><span style=3D" font-size:10pt;font-family:sans-serif">but then waits for a long time when attempting to scan all the other ports. I was scanning 521 IPs, of which 11 were really occupied and active and the 511 others were the "fakes".</span> <br><span style=3D" font-size:10pt;font-family:sans-serif">I was testing three options: Scan all hosts for all ports that appeared at least once (marked all=5Fports), write a simple script that calls nmap for each target (script), and then used the per-target ports (per=5Ftarget).</span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">With ping scan, the results were the following:</span> <br><span style=3D" font-size:10pt;font-family:sans-serif">* all=5Fports: 1= 02 seconds</span> <br><span style=3D" font-size:10pt;font-family:sans-serif">* per=5Ftarget: 80 seconds</span> <br><span style=3D" font-size:10pt;font-family:sans-serif">* script: 60 sec= onds</span> <br><span style=3D" font-size:10pt;font-family:sans-serif">Without ping sca= n:</span> <br><span style=3D" font-size:10pt;font-family:sans-serif">* script: 45 sec= onds</span> <br><span style=3D" font-size:10pt;font-family:sans-serif">* all=5Fports: 38 seconds</span> <br><span style=3D" font-size:10pt;font-family:sans-serif">* per=5Ftarget: 4 seconds</span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">My conclusion is that this feature is useful when you get filtered ports, as these take a lot of time. Basically, if you only have open and closed ports, scanning some additional ones is fast and the "union all ports" strategy is alright. But once you have filtered ports, the waiting quickly gets bad.</span> <br><span style=3D" font-size:10pt;font-family:sans-serif">The script could be upgraded to be parallel, but then you're reinventing nmap's parallel engine.</span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">Now I know the "proper" solution on this network would be to use a brain and try to discern real and fake machines, but this already gets much faster results without a need for thinking, which is always nice.</span> <br><span style=3D" font-size:10pt;font-family:sans-serif">I understand your concern about losing "new" services, but I see the main usability of this in manual scans, for example running additional nse scripts against your last scan results etc. When you are doing automatic scans (for exmple as a network admin scanning network for new devices each week), you probably don't need to care about speed anyway...</span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">Updated patch (to svn revision 37611):</span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">[attachment &quo= t;nmap=5Fpertarget=5F37611.patch" deleted by Jan Gocn=EDk/Dcit] </span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">Looking forward to further comments and hopefully we can make this work :)</span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">Jan</span> <br> <br> <br> <br> <br><span style=3D" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Fro= m: </span><span style=3D" font-size:9pt;font-family= :sans-serif">Jan Gocn=EDk/Dcit</span> <br><span style=3D" font-size:9pt;color:#5f5f5f;font-family:sans-serif">To: </span><span style=3D" font-size:9pt;font-family= :sans-serif">"Daniel Miller" <[email protected]></span> <br><span style=3D" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Cc: </span><span style=3D" font-size:9pt;font-family= :sans-serif">"Nmap-dev" <[email protected]></span> <br><span style=3D" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Dat= e: </span><span style=3D" font-size:9pt;font-family= :sans-serif">02.04.2019 23:53</span> <br><span style=3D" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Sub= ject: </span><span style=3D" font-size:9pt;font-family= :sans-serif">Re: Feature: per-target port specification (with patch!)</span> <br> <hr noshade> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">Hey Dan,</span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">thanks for the reply! It's a shame that I didn't find the GitHub issue you link to before implementing this, as it does raise a lot of valid concerns.</span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">First, let me say that the company I work for wants this feature, so even if it doesn't end up in upstream, I will try to keep it at least as a fork - as I'll have to maintain it internally anyway, I wanted to share with the community, in case others have a need for it as well.</span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">I will go through all the things you mentioned (most of the compatibility with other options should be taken care of, but memory leaks are a problem), fix up the code, look at maybe getting the memory footprint lower, and try to come up with some stronger numbers and rationale.</span> <br> <br><span style=3D" font-size:10pt;font-family:sans-serif">Jan</span> <br> <br> <br> <br> <br> <br><span style=3D" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Fro= m: </span><span style=3D" font-size:9pt;font-family= :sans-serif">"Daniel Miller" <[email protected]></span> <br><span style=3D" font-size:9pt;color:#5f5f5f;font-family:sans-serif">To: </span><span style=3D" font-size:9pt;font-family= :sans-serif">"Jan Gocn=EDk" <[email protected]></span> <br><span style=3D" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Cc: </span><span style=3D" font-size:9pt;font-family= :sans-serif">"Nmap-dev" <[email protected]></span> <br><span style=3D" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Dat= e: </span><span style=3D" font-size:9pt;font-family= :sans-serif">02.04.2019 21:21</span> <br><span style=3D" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Sub= ject: </span><span style=3D" font-size:9pt;font-family= :sans-serif">Re: Feature: per-target port specification (with patch!)</span> <br><span style=3D" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Sent by: </span><span style=3D" font-size:9pt;font-fa= mily:sans-serif">"dev" <[email protected]></span> <br> <hr noshade> <br> <br> <br><span style=3D" font-size:12pt">Some initial notes from building and testing this:</span> <br> <br><span style=3D" font-size:12pt">./nmap </span><a href=3Dhttp://scanme.n= map.org/><span style=3D" font-size:12pt;color:blue"><u>scanme.nmap.org</u><= /span></a><span style=3D" font-size:12pt">^22-80 -d<br> Starting Nmap 7.70SVN ( </span><a href=3Dhttps://nmap.org/><span style=3D" = font-size:12pt;color:blue"><u>https://nmap.org</u></span></a><span style=3D= " font-size:12pt"> ) at 2019-04-02 18:37 UTC<br> PORTS: Using top 1000 ports found open (TCP:1000, UDP:0, SCTP:0)<br> --------------- Timing report ---------------<br> hostgroups: min 1, max 100000<br> rtt-timeouts: init 1000, min 100, max 10000<br> max-scan-delay: TCP 1000, UDP 1000, SCTP 1000<br> parallelism: min 0, max 0<br> max-retries: 10, host-timeout: 0<br> min-rate: 0, max-rate: 0<br> ---------------------------------------------<br> Initiating Ping Scan at 18:37<br> Scanning </span><a href=3Dhttp://scanme.nmap.org/><span style=3D" font-size= :12pt;color:blue"><u>scanme.nmap.org</u></span></a><span style=3D" font-siz= e:12pt"> (45.33.32.156) [max 2 ports]<br> Completed Ping Scan at 18:37, 0.09s elapsed (1 total hosts)<br> Overall sending rates: 24.44 packets / s.<br> mass=5Frdns: Using DNS server X.X.X.X<br> Initiating Parallel DNS resolution of 1 host. at 18:37<br> mass=5Frdns: 0.12s 0/1 [#: 3, OK: 0, NX: 0, DR: 0, SF: 0, TR: 1]<br> Completed Parallel DNS resolution of 1 host. at 18:37, 0.06s elapsed<br> DNS resolution of 1 IPs took 0.15s. Mode: Async [#: 3, OK: 1, NX: 0, DR: 0, SF: 0, TR: 1, CN: 0]<br> Initiating Connect Scan at 18:37<br> Scanning </span><a href=3Dhttp://scanme.nmap.org/><span style=3D" font-size= :12pt;color:blue"><u>scanme.nmap.org</u></span></a><span style=3D" font-siz= e:12pt"> (45.33.32.156) [max 1059 ports]<br> Discovered open port 80/tcp<br> Discovered open port 22/tcp<br> Discovered open port 9929/tcp<br> Discovered open port 31337/tcp<br> nmap: portlist.cc:688: void PortList::mapPort(u16*, u8*) const: Assertion `mapped=5Fportno < port=5Flist=5Fcount[mapped=5Fprotocol]' failed.<br> Aborted (core dumped)</span> <br> <br> <br><span style=3D" font-size:12pt">Valgrind identified some memory leaks. These were the ones that are definitely from this patch: Portlist::setIdStr= (), idstr; PortList::mergeHostSpecificPorts(), new=5Fport=5Fmap and new=5Fport= =5Fmap=5Frev;</span> <br> <br><span style=3D" font-size:12pt">There was also a discrepancy between the "Scanning X [max N ports]" and "Completed Connect Scan at X (0 ports max)", which valgrind says is due to an uninitialized value, probably GroupScanStats::totalprobes.</span> <br> <br><span style=3D" font-size:12pt">Of course, as you noted before, the res= ults are not sorted by port number, which we would have to do to ensure predicta= ble output that can be compared with previous scans.</span> <br> <br><span style=3D" font-size:12pt">IPv6 addresses, CIDR, and IPv4 octet ranges seem to work fine with this.</span> <br> <br><span style=3D" font-size:12pt">With just my one test scan of localhost and </span><a href=3Dhttp://scanme.nmap.org/><span style=3D" font-size:12pt= ;color:blue"><u>scanme.nmap.org</u></span></a><span style=3D" font-size:12p= t"> with one unique port each and one port in common, the patched code allocated an additional 383KB of heap memory. I do not know how this would scale up or down, and I haven't compared it with a scan that should behave the same in both cases (for instance, "nmap </span><a href=3Dhttp://192.16= 8.1.0/24><span style=3D" font-size:12pt;color:blue"><u>192.168.1.0/24</u></= span></a><span style=3D" font-size:12pt">".</span> <br> <br><span style=3D" font-size:12pt">My primary concern remains that repeated use of this feature will result in missing new services and dropping existi= ng ones if they are missed in just one scan. This is more about use case than about the code, though, so I will defer to users on that.</span> <br> <br><span style=3D" font-size:12pt">Dan</span> <br> <br><span style=3D" font-size:12pt">On Tue, Apr 2, 2019 at 1:30 PM Daniel Miller <</span><a href=3Dmailto:[email protected]><span style=3D" f= ont-size:12pt;color:blue"><u>[email protected]</u></span></a><span sty= le=3D" font-size:12pt">> wrote:</span> <br><span style=3D" font-size:12pt">Jan,</span> <br> <br><span style=3D" font-size:12pt">Thanks for this contribution. We've had many requests for this type of feature in the past, but have elected not to include it for a variety of reasons. There is an open discussion on our issue tracker that lays out some of the challenges in correctly impleme= nting such a feature: </span><a href=3Dhttp://issues.nmap.org/1217 target=3D=5Fbl= ank><span style=3D" font-size:12pt;color:blue"><u>http://issues.nmap.org/12= 17</u></span></a> <br> <br><span style=3D" font-size:12pt">It looks like your patch has tried to handle some of these situations, for example the "Ports scanned" output for Grepable output (and maybe XML, but it didn't look complete at first glance). If we are to do an actual code review and include this new feature, we would have to look for a complete solution that can handle the following situations:</span> <br> <br><span style=3D" font-size:12pt">* The "Not shown: X ports" output for Normal output.</span> <br><span style=3D" font-size:12pt">* Properly formed XML output, with chan= ges to the DTD and a "xmloutputversion" number increase.</span> <br><span style=3D" font-size:12pt">* Combination of this feature with exis= ting --top-ports/port-ratio and -p options</span> <br><span style=3D" font-size:12pt">* Combination of this feature with CIDR subnetting and IPv4 octet ranges</span> <br><span style=3D" font-size:12pt">* Use of this feature along with advanc= ed features like -O --traceroute and -sV</span> <br> <br><span style=3D" font-size:12pt">Have you done any measurement of scans before and after adding this feature to determine the actual impact on scan times and bandwidth? Do you have a bandwidth target for your scans that Nmap is exceeding right now, and by how much? What does a typical nmap command line look like, and what performance options have you already tried?</span> <br> <br><span style=3D" font-size:12pt">I look forward to hearing more about this from you and our other devs and users.</span> <br> <br><span style=3D" font-size:12pt">Dan</span> <br> <br><span style=3D" font-size:12pt">On Tue, Apr 2, 2019 at 8:07 AM Jan Gocn= =EDk <</span><a href=3Dmailto:[email protected] target=3D=5Fblank><span style=3D= " font-size:12pt;color:blue"><u>[email protected]</u></span></a><span style=3D= " font-size:12pt">> wrote:</span> <br><span style=3D" font-size:10pt;font-family:sans-serif">Hey,</span><span= style=3D" font-size:12pt"> <br> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> I would like to propose a feature enabling specifying ports for each target separately.</span><span style=3D" font-size:12pt"> <br> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> Rationale:</span><span style=3D" font-size:12pt"> </span><span style=3D" fo= nt-size:10pt;font-family:sans-serif"><br> It often happens that we already have an nmap scan of 200 machines, and we want to do a service scan on those same machines. Usually that forces us to scan the whole network for all the ports that appeared at least once.= </span><span style=3D" font-size:12pt"> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> That is a big waste of time and bandwidth. What we want to have is essentia= lly a rescan-like feature, that would rescan just ports that were found to be open before.</span><span style=3D" font-size:12pt"> <br> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> User experience:</span><span style=3D" font-size:12pt"> </span><span style= =3D" font-size:10pt;font-family:sans-serif"><br> Everywhere where you could specify a target (-iL file, command line) you can supply a "target^ports". It works with all the nmap magic ranges, so "192.168.1.1-255^22-60" works. The common ports (suppl= ied with -p) are scanned on all targets.</span><span style=3D" font-size:12pt"> <br> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> Implementation details:</span><span style=3D" font-size:12pt"> </span><span= style=3D" font-size:10pt;font-family:sans-serif"><br> I tried to keep it so that if you don't use any "^" in the target= s, the code path should remain largely the same, so there should be no regress= ions. However, I had to do some tuning in functions that expected they can just get the number of probes by multiplying common ports by targets.</span><spa= n style=3D" font-size:12pt"> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> There's a small issue, in that the results of the scan are not sorted prope= rly, as the target-specific ports get scanned last.</span><span style=3D" font-s= ize:12pt"> <br> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> Usage example:</span><span style=3D" font-size:12pt"> </span><span style=3D= " font-size:10pt;font-family:sans-serif"><br> =3D=3D=3Dpaste start=3D=3D=3D</span><span style=3D" font-size:12pt"> </span= ><span style=3D" font-size:10pt;font-family:sans-serif"><br> $ nmap -v -Pn -n -p22 "165.227.141.119^80,443" "40.113.73.59= ^8080"</span><span style=3D" font-size:12pt"> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> Starting Nmap 7.70SVN ( </span><a href=3Dhttps://nmap.org/ target=3D=5Fblan= k><span style=3D" font-size:10pt;color:blue;font-family:sans-serif"><u>http= s://nmap.org</u></span></a><span style=3D" font-size:10pt;font-family:sans-= serif"> ) at 2019-04-01 19:46 CEST</span><span style=3D" font-size:12pt"> </span><s= pan style=3D" font-size:10pt;font-family:sans-serif"><br> Initiating SYN Stealth Scan at 19:46</span><span style=3D" font-size:12pt"> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> Scanning 2 hosts [max 3 ports/host]</span><span style=3D" font-size:12pt"> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> Discovered open port 22/tcp</span><span style=3D" font-size:12pt"> </span><= span style=3D" font-size:10pt;font-family:sans-serif"><br> Discovered open port 80/tcp</span><span style=3D" font-size:12pt"> </span><= span style=3D" font-size:10pt;font-family:sans-serif"><br> Discovered open port 443/tcp</span><span style=3D" font-size:12pt"> </span>= <span style=3D" font-size:10pt;font-family:sans-serif"><br> Discovered open port 22/tcp</span><span style=3D" font-size:12pt"> </span><= span style=3D" font-size:10pt;font-family:sans-serif"><br> Completed SYN Stealth Scan at 19:46, 1.45s elapsed (1626388576 total ports max)</span><span style=3D" font-size:12pt"> </span><span style=3D" font-siz= e:10pt;font-family:sans-serif"><br> Nmap scan report for 165.227.141.119</span><span style=3D" font-size:12pt"> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> Host is up (0.0090s latency).</span><span style=3D" font-size:12pt"> <br> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> PORT STATE SERVICE</span><span style=3D" font-size:12pt"> </sp= an><span style=3D" font-size:10pt;font-family:sans-serif"><br> 22/tcp open ssh</span><span style=3D" font-size:12pt"> </span><= span style=3D" font-size:10pt;font-family:sans-serif"><br> 80/tcp open http</span><span style=3D" font-size:12pt"> </span>= <span style=3D" font-size:10pt;font-family:sans-serif"><br> 443/tcp open https</span><span style=3D" font-size:12pt"> <br> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> Nmap scan report for 40.113.73.59</span><span style=3D" font-size:12pt"> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> Host is up (0.038s latency).</span><span style=3D" font-size:12pt"> <br> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> PORT STATE SERVICE</span><span style=3D" font-si= ze:12pt"> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> 22/tcp open ssh</span><span style=3D" font-size:12pt"> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> 8080/tcp filtered http-proxy</span><span style=3D" font-size:12pt"> <br> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> Read data files from: /home/gocnik/nmap</span><span style=3D" font-size:12p= t"> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> Nmap done: 2 IP addresses (2 hosts up) scanned in 1.52 seconds</span><span = style=3D" font-size:12pt"> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> Raw packets sent: 6 (264B) | Rcvd: 4 (176B)</span><span style=3D" font-size:12pt"> </span><span style=3D" font= -size:10pt;font-family:sans-serif"><br> =3D=3D=3Dpaste end=3D=3D=3D</span><span style=3D" font-size:12pt"> <br> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> If done the usual way:</span><span style=3D" font-size:12pt"> </span><span = style=3D" font-size:10pt;font-family:sans-serif"><br> $ nmap -v -Pn -n -p22,80,443,8080 165.227.141.119 40.113.73.59</span><span = style=3D" font-size:12pt"> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> [...]</span><span style=3D" font-size:12pt"> </span><span style=3D" font-si= ze:10pt;font-family:sans-serif"><br> Raw packets sent: 10 (440B) | Rcvd: 6 (260B)</span><span style=3D" font-siz= e:12pt"> <br> <br> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> The patch is against svn trunk at this moment (revision 37608).</span><span= style=3D" font-size:12pt"> <br> <br> <br> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> Looking forward to all comments!</span><span style=3D" font-size:12pt"> </s= pan><span style=3D" font-size:10pt;font-family:sans-serif"><br> JaGoTu</span><span style=3D" font-size:12pt"> <br> </span><span style=3D" font-size:10pt;font-family:sans-serif"><br> P.S.: Sorry if you recieve this e-mail twice, but the previous one apparent= ly got caught in a moderation queue or something, as it doesn't show on </span= ><a href=3Dhttp://seclists.org/ target=3D=5Fblank><span style=3D" font-size= :10pt;color:blue;font-family:sans-serif"><u>seclists.org</u></span></a><spa= n style=3D" font-size:12pt"> <br> =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F<br> Sent through the dev mailing list</span><span style=3D" font-size:12pt;colo= r:blue"><u><br> </u></span><a href=3Dhttps://nmap.org/mailman/listinfo/dev target=3D=5Fblan= k><span style=3D" font-size:12pt;color:blue"><u>https://nmap.org/mailman/li= stinfo/dev</u></span></a><span style=3D" font-size:12pt"><br> Archived at </span><a href=3D"http://seclists.org/nmap-dev/" target=3D=5Fbl= ank><span style=3D" font-size:12pt;color:blue"><u>http://seclists.org/nmap-= dev/</u></span></a><tt><span style=3D" font-size:10pt">=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F<br> Sent through the dev mailing list<br> </span></tt><a href=3Dhttps://nmap.org/mailman/listinfo/dev><tt><span style= =3D" font-size:10pt">https://nmap.org/mailman/listinfo/dev</span></tt></a><= tt><span style=3D" font-size:10pt"><br> Archived at </span></tt><a href=3D"http://seclists.org/nmap-dev/"><tt><span= style=3D" font-size:10pt">http://seclists.org/nmap-dev/</span></tt></a> <br> <br> <br> <br> --=_alternative 0040A89CC12583EF_=-- --===============8778059700860649237== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Sent through the dev mailing list https://nmap.org/mailman/listinfo/dev Archived at http://seclists.org/nmap-dev/ --===============8778059700860649237==--