Re: monitoring proxied web traffic
Konrad Weglowski <[email protected]> Tue, 12 Nov 2019 10:54:24 -0500
| Newsgroups | gmane.comp.security.detection.bro |
|---|---|
| Message-ID | <CA+fGMJSWrgWGiy71bi5tik_reYpuHGU=e5WPFu29p97qHweCcA@mail.gmail.com> |
Thanks again Michał for the breakdown. I have the full picture now :) Take care Konrad On Fri, Nov 8, 2019 at 3:21 AM Michał Purzyński <[email protected]> wrote: > You're welcome!! > > If the traffic from the client to the proxy is not encrypted and the proxy > is not "transparent" (meaning you configured it on the client) then you > have the same case as I do here. > > So there's this host > source IP 10.48.75.81 > > trying to talk to clr.telemetry.intel.com > > via our proxy at 10.48.122.10 / 10.48.74.16 / 10.48.74.17 > > *(for anyone working at a place where disclosing an internal IP or an FQDN > equal incident - you know a lot of great places are hiring, right?)* > > I used the "uid" to correlate this connection across three types of logs - > conn.log, http.log and ssl.log. You're gonna get them due to the way > proxies operate. > > Please ignore tons of metadata from our SIEM (MozDef = custom Python + > pure ES), I'm going to C&P three logs as they are without any redactions. > This post is HUGE!! > > *conn.log* - host -> proxy > > { > "_index": "events-20191108", > "_type": "_doc", > "_id": "JmLRSW4BiWsbe3JQtC8S", > "_version": 1, > "_score": null, > "_source": { > "type": "nsm", > "details": { > "ts": 1573196540.178897, > * "uid": "C9PFt711Odfwpbr0Ti",* > * "proto": "tcp",* > * "service": "http,ssl",* > "duration": 0.183212, > "orig_bytes": 1604, > "resp_bytes": 7180, > "conn_state": "SF", > "local_orig": true, > "local_resp": true, > "missed_bytes": 0, > * "history": "ShADadFf",* > * "orig_pkts": 14,* > * "resp_pkts": 13,* > "peer": "nsm2-af_packet-enp18s0f0-8", > * "orig_l2_addr": "b4:0c:25:e0:40:10",* > * "resp_l2_addr": "00:50:56:a1:48:64",* > * "sourceipaddress": "10.48.75.81",* > * "sourceport": 55330,* > * "destinationipaddress": "10.48.122.10",* > * "destinationport": 3128,* > * "originipbytes": 2340,* > * "responseipbytes": 7864,* > "sourceipv4address": "10.48.75.81", > "destinationipv4address": "10.48.122.10" > }, > "customendpoint": "bro", > "hostname": "nsm2.private.mdc1.xxxxxxxxxx.com", > "tags": "bro", > "category": "bro", > * "source": "conn",* > * "utctimestamp": "2019-11-08T07:02:20.178897+00:00",* > "timestamp": "2019-11-08T07:02:20.178897+00:00", > "receivedtimestamp": "2019-11-08T07:02:27.686061+00:00", > "eventsource": "nsm", > "severity": "INFO", > "mozdefhostname": "mozdef5.private.mdc1.xxxxxxxxxx.com", > "summary": "10.48.75.81:55330 -> 10.48.122.10:3128 ShADadFf 2340 > bytes / 7864 bytes", > "plugins": [ > "customDocType", > "broFixup", > "ipFixup", > "geoip" > ], > "processid": "UNKNOWN", > "processname": "UNKNOWN" > }, > "fields": { > "receivedtimestamp": [ > "2019-11-08T07:02:27.686Z" > ], > "timestamp": [ > "2019-11-08T07:02:20.178Z" > ], > "utctimestamp": [ > "2019-11-08T07:02:20.178Z" > ] > }, > "highlight": { > "summary": [ > "@[email protected]@/kibana-highlighted-field@:55330 > -> 10.48.122.10:3128 ShADadFf 2340 bytes / 7864 bytes" > ], > "details.sourceipv4address": [ > "@[email protected]@/kibana-highlighted-field@" > ], > "details.uid": [ > "@kibana-highlighted-field@C9PFt711Odfwpbr0Ti > @/kibana-highlighted-field@" > ], > "category": [ > "@kibana-highlighted-field@bro@/kibana-highlighted-field@" > ] > }, > "sort": [ > 1573196540178 > ] > } > > *http.log* (because the connection from the client to the proxy was > technically a http connection) > > { > "_index": "events-20191108", > "_type": "_doc", > "_id": "jPDRSW4BC2j90LU9UdgL", > "_version": 1, > "_score": null, > "_source": { > "type": "nsm", > "details": { > "ts": 1573196540.202888, > * "uid": "C9PFt711Odfwpbr0Ti",* > "trans_depth": 1, > "method": "CONNECT", > > *"host": "clr.telemetry.intel.com <http://clr.telemetry.intel.com>", > "uri": "clr.telemetry.intel.com:443 <http://clr.telemetry.intel.com:443>",* > "version": "1.1", > "request_body_len": 0, > "response_body_len": 0, > * "status_code": 200,* > * "status_msg": "Connection established",* > "tags": [], > "proxied": [ > "PROXY-CONNECTION -> Keep-Alive" > ], > > > > *"sourceipaddress": "10.48.75.81", "sourceport": 55330, > "destinationipaddress": "10.48.122.10", "destinationport": 3128,* > "sourceipv4address": "10.48.75.81", > "destinationipv4address": "10.48.122.10" > }, > "customendpoint": "bro", > "hostname": "nsm2.private.mdc1.xxxxxxxxxx.com", > "tags": "bro", > "category": "bro", > * "source": "http",* > * "utctimestamp": "2019-11-08T07:02:20.202888+00:00",* > "timestamp": "2019-11-08T07:02:20.202888+00:00", > "receivedtimestamp": "2019-11-08T07:02:21.427537+00:00", > "eventsource": "nsm", > "severity": "INFO", > "mozdefhostname": "mozdef7.private.mdc1.xxxxxxxxxx.com", > "summary": "HTTP CONNECT 10.48.75.81 -> 10.48.122.10:3128", > "plugins": [ > "customDocType", > "broFixup", > "ipFixup", > "geoip" > ], > "processid": "UNKNOWN", > "processname": "UNKNOWN" > }, > "fields": { > "receivedtimestamp": [ > "2019-11-08T07:02:21.427Z" > ], > "timestamp": [ > "2019-11-08T07:02:20.202Z" > ], > "utctimestamp": [ > "2019-11-08T07:02:20.202Z" > ] > }, > "highlight": { > "summary": [ > "HTTP CONNECT @[email protected] > @/kibana-highlighted-field@ -> 10.48.122.10:3128" > ], > "details.sourceipv4address": [ > "@[email protected]@/kibana-highlighted-field@" > ], > "details.uid": [ > "@kibana-highlighted-field@C9PFt711Odfwpbr0Ti > @/kibana-highlighted-field@" > ], > "category": [ > "@kibana-highlighted-field@bro@/kibana-highlighted-field@" > ] > }, > "sort": [ > 1573196540202 > ] > } > > And the *ssl.log* - over the previously established http connection the > client told proxy to establish a tunnel to the destination host - and Zeek > parsed that as TLS (find me another IDS/NSM that can do that) > > { > "_index": "events-20191108", > "_type": "_doc", > "_id": "sfDRSW4BC2j90LU9S8_Q", > "_version": 1, > "_score": null, > "_source": { > "type": "nsm", > "details": { > "ts": 1573196540.253057, > > > > > > > > > > > > > > * "uid": "C9PFt711Odfwpbr0Ti", "version": "TLSv12", "cipher": > "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "curve": "secp256r1", > "server_name": "clr.telemetry.intel.com <http://clr.telemetry.intel.com>", > "resumed": false, "established": true, "cert_chain_fuids": [ > "FknnGH2L2UIKGERW3b", "FGtbhK1HoQMWdgyKW", > "F8JbGM30JvqFVi41A6", "Fsc2fRRTyTDQrT1Gc" ], > "client_cert_chain_fuids": [],* > > > > > > > > > > > > > *"subject": "CN=c-all.telemetry.intel.com > <http://c-all.telemetry.intel.com>,OU=Multi-Domain SSL,OU=Hosted by Intel > Corporation,OU=SSG,O=Intel Corporation,street=2200 Mission College > Blvd,street=FM1-110,L=Santa > Clara,ST=California,postalCode=95054-1537,C=US", "issuer": "CN=COMODO > RSA Organization Validation Secure Server CA,O=COMODO CA > Limited,L=Salford,ST=Greater Manchester,C=GB", "validation_status": > "ok", "server_cert_md5": "27ba7ede8ceab7c896ee4b6bcb509f63", > "server_cert_sha1": "9077d0ce724c0f7bf1866d7cf8c4ec1303dc6c21", "pfs": > true, "ja3": "f436b9416f37d134cadd04886327d3e8", "ja3s": > "303951d4c50efb2e991652225a6f02b1", "sourceipaddress": "10.48.75.81", > "sourceport": 55330, "destinationipaddress": "10.48.122.10", > "destinationport": 3128,* "sourceipv4address": "10.48.75.81", > "destinationipv4address": "10.48.122.10" > }, > "customendpoint": "bro", > "hostname": "nsm2.private.mdc1.xxxxxxxxxx.com", > "tags": "bro", > "category": "bro", > "source": "ssl", > "utctimestamp": "2019-11-08T07:02:20.253057+00:00", > "timestamp": "2019-11-08T07:02:20.253057+00:00", > "receivedtimestamp": "2019-11-08T07:02:21.464921+00:00", > "eventsource": "nsm", > "severity": "INFO", > "mozdefhostname": "mozdef6.private.mdc1.xxxxxxxxxx.com", > "summary": "SSL: 10.48.75.81 -> 10.48.122.10:3128", > "plugins": [ > "customDocType", > "broFixup", > "ipFixup", > "geoip" > ], > "processid": "UNKNOWN", > "processname": "UNKNOWN" > }, > "fields": { > "receivedtimestamp": [ > "2019-11-08T07:02:21.464Z" > ], > "timestamp": [ > "2019-11-08T07:02:20.253Z" > ], > "utctimestamp": [ > "2019-11-08T07:02:20.253Z" > ] > }, > "highlight": { > "summary": [ > "SSL: @[email protected] > @/kibana-highlighted-field@ -> 10.48.122.10:3128" > ], > "details.sourceipv4address": [ > "@[email protected]@/kibana-highlighted-field@" > ], > "details.uid": [ > "@kibana-highlighted-field@C9PFt711Odfwpbr0Ti > @/kibana-highlighted-field@" > ], > "category": [ > "@kibana-highlighted-field@bro@/kibana-highlighted-field@" > ] > }, > "sort": [ > 1573196540253 > ] > } > > > Not enough? Let's look for the certificate's MD5 > > { > "_index": "events-20191108", > "_type": "_doc", > "_id": "_CwBSm4BC2j90LU9vfNt", > "_version": 1, > "_score": null, > "_source": { > "type": "nsm", > "details": { > "ts": 1573199701.992707, > "id": "FdE5qt1b93z9QOJnz1", > "san.dns": [ > "c-all.telemetry.intel.com", > "c.telemetry.intel.com", > "c1.telemetry.intel.com", > "c2.telemetry.intel.com", > "c3.telemetry.intel.com", > "c4.telemetry.intel.com", > "c5.telemetry.intel.com", > "c6.telemetry.intel.com", > "c7.telemetry.intel.com", > "c8.telemetry.intel.com", > "c9.telemetry.intel.com", > "clr.telemetry.intel.com", > "sur.telemetry.intel.com" > ], > "md5": "27ba7ede8ceab7c896ee4b6bcb509f63", > "sha1": "9077d0ce724c0f7bf1866d7cf8c4ec1303dc6c21", > "certificate": { > "basic_constraints_ca": false, > "exponent": "65537", > "issuer": "CN=COMODO RSA Organization Validation Secure Server > CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB", > "key_alg": "rsaEncryption", > "key_length": 2048, > "key_type": "rsa", > "not_valid_after": "2020-08-02T23:59:59+00:00", > "not_valid_before": "2018-08-03T00:00:00+00:00", > "sig_alg": "sha256WithRSAEncryption", > "subject": "CN=c-all.telemetry.intel.com,OU=Multi-Domain > SSL,OU=Hosted by Intel Corporation,OU=SSG,O=Intel Corporation,street=2200 > Mission College Blvd,street=FM1-110,L=Santa > Clara,ST=California,postalCode=95054-1537,C=US", > "version": 3, > "serial": "1406C0A906762390B7252C9598F2DB3C" > } > }, > "customendpoint": "bro", > "hostname": "nsm1-ber3.private.ber3.xxxxxxxxxx.com", > "tags": "bro", > "category": "bro", > "source": "x509", > "utctimestamp": "2019-11-08T07:55:01.992707+00:00", > "timestamp": "2019-11-08T07:55:01.992707+00:00", > "receivedtimestamp": "2019-11-08T07:55:17.040196+00:00", > "eventsource": "nsm", > "severity": "INFO", > "mozdefhostname": "mozdef1.private.mdc1.xxxxxxxxxx.com", > "summary": "X509 certificate seen", > "plugins": [ > "customDocType", > "broFixup" > ], > "processid": "UNKNOWN", > "processname": "UNKNOWN" > }, > "fields": { > "details.certificate.not_valid_after": [ > "2020-08-02T23:59:59.000Z" > ], > "receivedtimestamp": [ > "2019-11-08T07:55:17.040Z" > ], > "details.certificate.not_valid_before": [ > "2018-08-03T00:00:00.000Z" > ], > "timestamp": [ > "2019-11-08T07:55:01.992Z" > ], > "utctimestamp": [ > "2019-11-08T07:55:01.992Z" > ] > }, > "highlight": { > "details.md5": [ > "@kibana-highlighted-field@27ba7ede8ceab7c896ee4b6bcb509f63 > @/kibana-highlighted-field@" > ], > "source": [ > "@kibana-highlighted-field@x509@/kibana-highlighted-field@" > ], > "category": [ > "@kibana-highlighted-field@bro@/kibana-highlighted-field@" > ] > }, > "sort": [ > 1573199701992 > ] > } > > > Let's try to splice the first and the second half of the connection > > category:bro AND source:dns AND details.query:clr.telemetry.intel.com > > clr.ats.infra-host.com, 52.37.21.110, 34.208.200.27 > > Back to looking at conn.log with destination IP like above and source = > proxies (from there you can again do the crazy dance with ssl.log but > there's no need to) > > { > "_index": "events-20191108", > "_type": "_doc", > "_id": "jvDRSW4BC2j90LU9UdgL", > "_version": 1, > "_score": null, > "_source": { > "type": "nsm", > "details": { > "ts": 1573196540.253057, > "uid": "CrcIhj4lIsSLpPwPe5", > "version": "TLSv12", > "cipher": "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", > "curve": "secp256r1", > "server_name": "clr.telemetry.intel.com", > "resumed": false, > "established": true, > "cert_chain_fuids": [ > "FpfU1u4yMxOBNKQJk2", > "FYiimM3T6A5AKcTeze", > "FfAQxKi177b5fl95a", > "FWHqmB2emMx082lxO1" > ], > "client_cert_chain_fuids": [], > "subject": "CN=c-all.telemetry.intel.com,OU=Multi-Domain > SSL,OU=Hosted by Intel Corporation,OU=SSG,O=Intel Corporation,street=2200 > Mission College Blvd,street=FM1-110,L=Santa > Clara,ST=California,postalCode=95054-1537,C=US", > "issuer": "CN=COMODO RSA Organization Validation Secure Server > CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB", > "validation_status": "ok", > "server_cert_md5": "27ba7ede8ceab7c896ee4b6bcb509f63", > "server_cert_sha1": "9077d0ce724c0f7bf1866d7cf8c4ec1303dc6c21", > "pfs": true, > "ja3": "f436b9416f37d134cadd04886327d3e8", > "ja3s": "303951d4c50efb2e991652225a6f02b1", > "sourceipaddress": "10.48.74.16", > "sourceport": 58860, > "destinationipaddress": "52.37.21.110", > "destinationport": 443, > "sourceipv4address": "10.48.74.16", > "destinationipv4address": "52.37.21.110", > "destinationipgeolocation": { > "city": "Boardman", > "continent": "NA", > "country_code": "US", > "country_name": "United States", > "dma_code": 810, > "latitude": 45.8491, > "longitude": -119.7143, > "metro_code": "Boardman, OR", > "postal_code": "97818", > "region_code": "OR", > "time_zone": "America/Los_Angeles" > }, > "destinationipgeopoint": "45.8491,-119.7143" > }, > "customendpoint": "bro", > "hostname": "nsm2.private.mdc1.xxxxxxxxxx.com", > "tags": "bro", > "category": "bro", > "source": "ssl", > "utctimestamp": "2019-11-08T07:02:20.253057+00:00", > "timestamp": "2019-11-08T07:02:20.253057+00:00", > "receivedtimestamp": "2019-11-08T07:02:21.465325+00:00", > "eventsource": "nsm", > "severity": "INFO", > "mozdefhostname": "mozdef7.private.mdc1.xxxxxxxxxxxxxxxxxxxx.com", > "summary": "SSL: 10.48.74.16 -> 52.37.21.110:443", > "plugins": [ > "customDocType", > "broFixup", > "ipFixup", > "geoip" > ], > "processid": "UNKNOWN", > "processname": "UNKNOWN" > }, > "fields": { > "receivedtimestamp": [ > "2019-11-08T07:02:21.465Z" > ], > "timestamp": [ > "2019-11-08T07:02:20.253Z" > ], > "utctimestamp": [ > "2019-11-08T07:02:20.253Z" > ] > }, > "highlight": { > "category": [ > "@kibana-highlighted-field@bro@/kibana-highlighted-field@" > ] > }, > "sort": [ > 1573196540253 > ] > } > > I could now paste the third (?!) leg of this connection - after SNAT, but > you get the idea. > > Unfortunately, this sort of visibility partially goes away soon, due to > some pseudo-privacy mechanisms AKA a classic shoot in the foot, while > trying to kill a mosquito. From a shotgun. <- my private opinion BTW. I'm > well > after hours ;) > > On Thu, Nov 7, 2019 at 10:00 AM Konrad Weglowski < > [email protected]> wrote: > >> Thank You Michał >> >> Client to Proxy would not be encrypted. So based on that I will have all >> the logs, however my destination IP for proxied connections would be proxy >> IP (that was my understanding). I guess ingesting proxy logs would then >> provide full visibility as you mentioned. I am thinking maybe if for some >> reason I cannot get the proxy logs ingested and also tap after the proxy, I >> could possibly correlate both ends of the connections based on URI/SNI/etc >> and same timestamps in my analytics platform. Anyways I will give that a >> try as well. Thanks for your help again. >> >> Konrad >> >> >> >> On Wed, Nov 6, 2019 at 6:59 PM Michał Purzyński < >> [email protected]> wrote: >> >>> There is no way for Zeek to really correlate if the proxy does >>> everything at the application level, like Squid. The proxy >>> terminates the connection and then it's free to start the second leg, >>> to the destination host, as it wishes to. >>> >>> Is your traffic from clients to proxy encrypted as well? >>> >>> If the traffic between clients are the proxy is not encrypted (as it's >>> usually the case) then even if the client will establish a TLS session >>> _through_ the proxy you will see a destination with the "service" field >>> http,ssl and all the conn + ssl + http logs. >>> >>> We have that + proxy logs and that's pretty much a full visibility. >>> >>> >>> >>> On Wed, Nov 6, 2019 at 1:11 PM Konrad Weglowski < >>> [email protected]> wrote: >>> >>>> Hello, >>>> >>>> I need to monitor web traffic with Zeek that is going through an >>>> implicit web proxy. I would like to be able to see real client IPs in which >>>> case tap would be placed before the proxy, however I will not be able to >>>> see the real destination IP as it will always be proxy IP. I can also tap >>>> on the other side of the proxy where source will always be proxy IP at the >>>> same time. >>>> >>>> Is there a way with Zeek to correlate the two sessions (before and >>>> after proxy) somehow? >>>> >>>> I realize that UID or Community ID are probably not going do it since >>>> different source/destinations pairs for each for each session, but maybe >>>> based on the timestamp/URI/SNI/byte or packet counts/etc? >>>> >>>> I was also thinking "X-Forwarded-For" header but I guess that would >>>> only work for HTTP and not HTTPS connections. >>>> >>>> Thank You, >>>> >>>> Konrad >>>> _______________________________________________ >>>> Zeek mailing list >>>> [email protected] >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek >>> >>> _______________________________________________ Zeek mailing list [email protected] http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek