Re: Feature: per-target port specification (with patch!)
Daniel Miller <[email protected]> Tue, 2 Apr 2019 14:21:16 -0500
| Newsgroups | gmane.comp.security.nmap.devel |
|---|---|
| Message-ID | <CABmvJnN=x2nutzTbRCjZp=r3FtHvQqqrahyYh0Dv3ApTXdaRqg@mail.gmail.com> |
--===============3520502230170464335== Content-Type: multipart/alternative; boundary="0000000000002aec220585910b08" --0000000000002aec220585910b08 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Some initial notes from building and testing this: ./nmap scanme.nmap.org^22-80 -d Starting Nmap 7.70SVN ( https://nmap.org ) at 2019-04-02 18:37 UTC PORTS: Using top 1000 ports found open (TCP:1000, UDP:0, SCTP:0) --------------- Timing report --------------- hostgroups: min 1, max 100000 rtt-timeouts: init 1000, min 100, max 10000 max-scan-delay: TCP 1000, UDP 1000, SCTP 1000 parallelism: min 0, max 0 max-retries: 10, host-timeout: 0 min-rate: 0, max-rate: 0 --------------------------------------------- Initiating Ping Scan at 18:37 Scanning scanme.nmap.org (45.33.32.156) [max 2 ports] Completed Ping Scan at 18:37, 0.09s elapsed (1 total hosts) Overall sending rates: 24.44 packets / s. mass_rdns: Using DNS server X.X.X.X Initiating Parallel DNS resolution of 1 host. at 18:37 mass_rdns: 0.12s 0/1 [#: 3, OK: 0, NX: 0, DR: 0, SF: 0, TR: 1] Completed Parallel DNS resolution of 1 host. at 18:37, 0.06s elapsed DNS resolution of 1 IPs took 0.15s. Mode: Async [#: 3, OK: 1, NX: 0, DR: 0, SF: 0, TR: 1, CN: 0] Initiating Connect Scan at 18:37 Scanning scanme.nmap.org (45.33.32.156) [max 1059 ports] Discovered open port 80/tcp Discovered open port 22/tcp Discovered open port 9929/tcp Discovered open port 31337/tcp nmap: portlist.cc:688: void PortList::mapPort(u16*, u8*) const: Assertion `mapped_portno < port_list_count[mapped_protocol]' failed. Aborted (core dumped) Valgrind identified some memory leaks. These were the ones that are definitely from this patch: Portlist::setIdStr(), idstr; PortList::mergeHostSpecificPorts(), new_port_map and new_port_map_rev; There was also a discrepancy between the "Scanning X [max N ports]" and "Completed Connect Scan at X (0 ports max)", which valgrind says is due to an uninitialized value, probably GroupScanStats::totalprobes. Of course, as you noted before, the results are not sorted by port number, which we would have to do to ensure predictable output that can be compared with previous scans. IPv6 addresses, CIDR, and IPv4 octet ranges seem to work fine with this. With just my one test scan of localhost and scanme.nmap.org with one unique port each and one port in common, the patched code allocated an additional 383KB of heap memory. I do not know how this would scale up or down, and I haven't compared it with a scan that should behave the same in both cases (for instance, "nmap 192.168.1.0/24". My primary concern remains that repeated use of this feature will result in missing new services and dropping existing ones if they are missed in just one scan. This is more about use case than about the code, though, so I will defer to users on that. Dan On Tue, Apr 2, 2019 at 1:30 PM Daniel Miller <[email protected]> wrote= : > Jan, > > Thanks for this contribution. We've had many requests for this type of > feature in the past, but have elected not to include it for a variety of > reasons. There is an open discussion on our issue tracker that lays out > some of the challenges in correctly implementing such a feature: > http://issues.nmap.org/1217 > > It looks like your patch has tried to handle some of these situations, fo= r > example the "Ports scanned" output for Grepable output (and maybe XML, bu= t > it didn't look complete at first glance). If we are to do an actual code > review and include this new feature, we would have to look for a complete > solution that can handle the following situations: > > * The "Not shown: X ports" output for Normal output. > * Properly formed XML output, with changes to the DTD and a > "xmloutputversion" number increase. > * Combination of this feature with existing --top-ports/port-ratio and -p > options > * Combination of this feature with CIDR subnetting and IPv4 octet ranges > * Use of this feature along with advanced features like -O --traceroute > and -sV > > Have you done any measurement of scans before and after adding this > feature to determine the actual impact on scan times and bandwidth? Do yo= u > have a bandwidth target for your scans that Nmap is exceeding right now, > and by how much? What does a typical nmap command line look like, and wha= t > performance options have you already tried? > > I look forward to hearing more about this from you and our other devs and > users. > > Dan > > On Tue, Apr 2, 2019 at 8:07 AM Jan Gocn=C3=ADk <[email protected]> wrote: > >> Hey, >> >> I would like to propose a feature enabling specifying ports for each >> target separately. >> >> Rationale: >> It often happens that we already have an nmap scan of 200 machines, and >> we want to do a service scan on those same machines. Usually that forces= us >> to scan the whole network for all the ports that appeared at least once. >> That is a big waste of time and bandwidth. What we want to have is >> essentially a rescan-like feature, that would rescan just ports that wer= e >> found to be open before. >> >> User experience: >> Everywhere where you could specify a target (-iL file, command line) you >> can supply a "target^ports". It works with all the nmap magic ranges, so >> "192.168.1.1-255^22-60" works. The common ports (supplied with -p) are >> scanned on all targets. >> >> Implementation details: >> I tried to keep it so that if you don't use any "^" in the targets, the >> code path should remain largely the same, so there should be no >> regressions. However, I had to do some tuning in functions that expected >> they can just get the number of probes by multiplying common ports by >> targets. >> There's a small issue, in that the results of the scan are not sorted >> properly, as the target-specific ports get scanned last. >> >> Usage example: >> =3D=3D=3Dpaste start=3D=3D=3D >> $ nmap -v -Pn -n -p22 "165.227.141.119^80,443" "40.113.73.59^8080" >> Starting Nmap 7.70SVN ( https://nmap.org ) at 2019-04-01 19:46 CEST >> Initiating SYN Stealth Scan at 19:46 >> Scanning 2 hosts [max 3 ports/host] >> Discovered open port 22/tcp >> Discovered open port 80/tcp >> Discovered open port 443/tcp >> Discovered open port 22/tcp >> Completed SYN Stealth Scan at 19:46, 1.45s elapsed (1626388576 total >> ports max) >> Nmap scan report for 165.227.141.119 >> Host is up (0.0090s latency). >> >> PORT STATE SERVICE >> 22/tcp open ssh >> 80/tcp open http >> 443/tcp open https >> >> Nmap scan report for 40.113.73.59 >> Host is up (0.038s latency). >> >> PORT STATE SERVICE >> 22/tcp open ssh >> 8080/tcp filtered http-proxy >> >> Read data files from: /home/gocnik/nmap >> Nmap done: 2 IP addresses (2 hosts up) scanned in 1.52 seconds >> Raw packets sent: 6 (264B) | Rcvd: 4 (176B) >> =3D=3D=3Dpaste end=3D=3D=3D >> >> If done the usual way: >> $ nmap -v -Pn -n -p22,80,443,8080 165.227.141.119 40.113.73.59 >> [...] >> Raw packets sent: 10 (440B) | Rcvd: 6 (260B) >> >> >> The patch is against svn trunk at this moment (revision 37608). >> >> >> >> Looking forward to all comments! >> JaGoTu >> >> P.S.: Sorry if you recieve this e-mail twice, but the previous one >> apparently got caught in a moderation queue or something, as it doesn't >> show on seclists.org >> _______________________________________________ >> Sent through the dev mailing list >> https://nmap.org/mailman/listinfo/dev >> Archived at http://seclists.org/nmap-dev/ > > --0000000000002aec220585910b08 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><div>Some initial notes from building and= testing this:</div><div><br></div><div>./nmap <a href=3D"http://scanme.nma= p.org">scanme.nmap.org</a>^22-80 -d<br>Starting Nmap 7.70SVN ( <a href=3D"h= ttps://nmap.org">https://nmap.org</a> ) at 2019-04-02 18:37 UTC<br>PORTS: U= sing top 1000 ports found open (TCP:1000, UDP:0, SCTP:0)<br>---------------= Timing report ---------------<br>=C2=A0 hostgroups: min 1, max 100000<br>= =C2=A0 rtt-timeouts: init 1000, min 100, max 10000<br>=C2=A0 max-scan-delay= : TCP 1000, UDP 1000, SCTP 1000<br>=C2=A0 parallelism: min 0, max 0<br>=C2= =A0 max-retries: 10, host-timeout: 0<br>=C2=A0 min-rate: 0, max-rate: 0<br>= ---------------------------------------------<br>Initiating Ping Scan at 18= :37<br>Scanning <a href=3D"http://scanme.nmap.org">scanme.nmap.org</a> (45.= 33.32.156) [max 2 ports]<br>Completed Ping Scan at 18:37, 0.09s elapsed (1 = total hosts)<br>Overall sending rates: 24.44 packets / s.<br>mass_rdns: Usi= ng DNS server X.X.X.X<br>Initiating Parallel DNS resolution of 1 host. at 1= 8:37<br>mass_rdns: 0.12s 0/1 [#: 3, OK: 0, NX: 0, DR: 0, SF: 0, TR: 1]<br>C= ompleted Parallel DNS resolution of 1 host. at 18:37, 0.06s elapsed<br>DNS = resolution of 1 IPs took 0.15s. Mode: Async [#: 3, OK: 1, NX: 0, DR: 0, SF:= 0, TR: 1, CN: 0]<br>Initiating Connect Scan at 18:37<br>Scanning <a href= =3D"http://scanme.nmap.org">scanme.nmap.org</a> (45.33.32.156) [max 1059 po= rts]<br>Discovered open port 80/tcp<br>Discovered open port 22/tcp<br>Disco= vered open port 9929/tcp<br>Discovered open port 31337/tcp<br>nmap: portlis= t.cc:688: void PortList::mapPort(u16*, u8*) const: Assertion `mapped_portno= < port_list_count[mapped_protocol]' failed.<br>Aborted (core dumped= )</div><div><br></div><div><br></div><div>Valgrind identified some memory l= eaks. These were the ones that are definitely from this patch: Portlist::se= tIdStr(), idstr; PortList::mergeHostSpecificPorts(), new_port_map and new_p= ort_map_rev;</div><div><br></div><div>There was also a discrepancy between = the "Scanning X [max N ports]" and "Completed Connect Scan a= t X (0 ports max)", which valgrind says is due to an uninitialized val= ue, probably GroupScanStats::totalprobes.</div><div><br></div><div>Of cours= e, as you noted before, the results are not sorted by port number, which we= would have to do to ensure predictable output that can be compared with pr= evious scans.</div><div><br></div><div>IPv6 addresses, CIDR, and IPv4 octet= ranges seem to work fine with this.</div><div><br></div><div>With just my = one test scan of localhost and <a href=3D"http://scanme.nmap.org">scanme.nm= ap.org</a> with one unique port each and one port in common, the patched co= de allocated an additional 383KB of heap memory. I do not know how this wou= ld scale up or down, and I haven't compared it with a scan that should = behave the same in both cases (for instance, "nmap <a href=3D"http://1= 92.168.1.0/24">192.168.1.0/24</a>".</div><div><br></div><div>My primar= y concern remains that repeated use of this feature will result in missing = new services and dropping existing ones if they are missed in just one scan= . This is more about use case than about the code, though, so I will defer = to users on that.</div><div><br></div><div>Dan<br></div></div></div><br><di= v class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Tue, Apr 2= , 2019 at 1:30 PM Daniel Miller <<a href=3D"mailto:[email protected]= m">[email protected]</a>> wrote:<br></div><blockquote class=3D"gmai= l_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,20= 4,204);padding-left:1ex"><div dir=3D"ltr"><div dir=3D"ltr"><div>Jan,</div><= div><br></div><div>Thanks for this contribution. We've had many request= s for this type of feature in the past, but have elected not to include it = for a variety of reasons. There is an open discussion on our issue tracker = that lays out some of the challenges in correctly implementing such a featu= re: <a href=3D"http://issues.nmap.org/1217" target=3D"_blank">http://issues= .nmap.org/1217</a></div><div><br></div><div>It looks like your patch has tr= ied to handle some of these situations, for example the "Ports scanned= " output for Grepable output (and maybe XML, but it didn't look co= mplete at first glance). If we are to do an actual code review and include = this new feature, we would have to look for a complete solution that can ha= ndle the following situations:</div><div><br></div><div>* The "Not sho= wn: X ports" output for Normal output.</div><div>* Properly formed XML= output, with changes to the DTD and a "xmloutputversion" number = increase.</div><div>* Combination of this feature with existing --top-ports= /port-ratio and -p options</div><div>* Combination of this feature with CID= R subnetting and IPv4 octet ranges</div><div>* Use of this feature along wi= th advanced features like -O --traceroute and -sV</div><div><br></div><div>= Have you done any measurement of scans before and after adding this feature= to determine the actual impact on scan times and bandwidth? Do you have a = bandwidth target for your scans that Nmap is exceeding right now, and by ho= w much? What does a typical nmap command line look like, and what performan= ce options have you already tried?</div><div><br></div><div>I look forward = to hearing more about this from you and our other devs and users.</div><div= ><br></div><div>Dan<br></div></div></div><br><div class=3D"gmail_quote"><di= v dir=3D"ltr" class=3D"gmail_attr">On Tue, Apr 2, 2019 at 8:07 AM Jan Gocn= =C3=ADk <<a href=3D"mailto:[email protected]" target=3D"_blank">gocnik@dcit= .cz</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"marg= in:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1e= x"><span style=3D"font-size:10pt;font-family:sans-serif">Hey,</span> <br> <br><span style=3D"font-size:10pt;font-family:sans-serif">I would like to propose a feature enabling specifying ports for each target separately.</sp= an> <br> <br><span style=3D"font-size:10pt;font-family:sans-serif">Rationale:</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">It often happens that we already have an nmap scan of 200 machines, and we want to do a service scan on those same machines. Usually that forces us to scan the whole network for all the ports that appeared at least once.</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">That is a big waste of time and bandwidth. What we want to have is essentially a rescan-l= ike feature, that would rescan just ports that were found to be open before.</s= pan> <br> <br><span style=3D"font-size:10pt;font-family:sans-serif">User experience:<= /span> <br><span style=3D"font-size:10pt;font-family:sans-serif">Everywhere where you could specify a target (-iL file, command line) you can supply a "= target^ports". It works with all the nmap magic ranges, so "192.168.1.1-255^22-60&quo= t; works. The common ports (supplied with -p) are scanned on all targets.</spa= n> <br> <br><span style=3D"font-size:10pt;font-family:sans-serif">Implementation details:</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">I tried to keep it so that if you don't use any "^" in the targets, the code path should remain largely the same, so there should be no regressions. However, I had to do some tuning in functions that expected they can just get the number of probes by multiplying common ports by targets.</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">There's a sma= ll issue, in that the results of the scan are not sorted properly, as the target-specific ports get scanned last.</span> <br> <br><span style=3D"font-size:10pt;font-family:sans-serif">Usage example:</s= pan> <br><span style=3D"font-size:10pt;font-family:sans-serif">=3D=3D=3Dpaste st= art=3D=3D=3D</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">$ nmap -v -Pn -n -p22 "165.227.141.119^80,443" "40.113.73.59^8080"</s= pan> <br><span style=3D"font-size:10pt;font-family:sans-serif">Starting Nmap 7.70SVN ( </span><a href=3D"https://nmap.org/" target=3D"_blank"><span styl= e=3D"font-size:10pt;color:blue;font-family:sans-serif">https://nmap.org</sp= an></a><span style=3D"font-size:10pt;font-family:sans-serif"> ) at 2019-04-01 19:46 CEST</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">Initiating SYN Stealth Scan at 19:46</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">Scanning 2 hosts [max 3 ports/host]</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">Discovered open port 22/tcp</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">Discovered open port 80/tcp</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">Discovered open port 443/tcp</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">Discovered open port 22/tcp</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">Completed SYN Stealth Scan at 19:46, 1.45s elapsed (1626388576 total ports max)</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">Nmap scan report for 165.227.141.119</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">Host is up (0.009= 0s latency).</span> <br> <br><span style=3D"font-size:10pt;font-family:sans-serif">PORT =C2=A0 =C2= =A0STATE SERVICE</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">22/tcp =C2=A0open =C2=A0ssh</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">80/tcp =C2=A0open =C2=A0http</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">443/tcp open =C2= =A0https</span> <br> <br><span style=3D"font-size:10pt;font-family:sans-serif">Nmap scan report for 40.113.73.59</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">Host is up (0.038= s latency).</span> <br> <br><span style=3D"font-size:10pt;font-family:sans-serif">PORT =C2=A0 =C2= =A0 STATE =C2=A0 =C2=A0SERVICE</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">22/tcp =C2=A0 open =C2=A0 =C2=A0 ssh</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">8080/tcp filtered http-proxy</span> <br> <br><span style=3D"font-size:10pt;font-family:sans-serif">Read data files from: /home/gocnik/nmap</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">Nmap done: 2 IP addresses (2 hosts up) scanned in 1.52 seconds</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Raw packets sent: 6 (264B) | Rcvd: 4 (176B)</spa= n> <br><span style=3D"font-size:10pt;font-family:sans-serif">=3D=3D=3Dpaste en= d=3D=3D=3D</span> <br> <br><span style=3D"font-size:10pt;font-family:sans-serif">If done the usual way:</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">$ nmap -v -Pn -n -p22,80,443,8080 165.227.141.119 40.113.73.59</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">[...]</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">Raw packets sent: 10 (440B) | Rcvd: 6 (260B)</span> <br> <br> <br><span style=3D"font-size:10pt;font-family:sans-serif">The patch is agai= nst svn trunk at this moment (revision 37608).</span> <br> <br> <br> <br><span style=3D"font-size:10pt;font-family:sans-serif">Looking forward to all comments!</span> <br><span style=3D"font-size:10pt;font-family:sans-serif">JaGoTu</span> <br> <br><span style=3D"font-size:10pt;font-family:sans-serif">P.S.: Sorry if you recieve this e-mail twice, but the previous one apparently got caught in a moderation queue or something, as it doesn't show on <a href=3D"ht= tp://seclists.org" target=3D"_blank">seclists.org</a></span> <br>_______________________________________________<br> Sent through the dev mailing list<br> <a href=3D"https://nmap.org/mailman/listinfo/dev" rel=3D"noreferrer" target= =3D"_blank">https://nmap.org/mailman/listinfo/dev</a><br> Archived at <a href=3D"http://seclists.org/nmap-dev/" rel=3D"noreferrer" ta= rget=3D"_blank">http://seclists.org/nmap-dev/</a></blockquote></div> </blockquote></div> --0000000000002aec220585910b08-- --===============3520502230170464335== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Sent through the dev mailing list https://nmap.org/mailman/listinfo/dev Archived at http://seclists.org/nmap-dev/ --===============3520502230170464335==--