Re: Affordable TICSA LI Solution and Support
[email protected] Sat, 30 Jun 2018 17:28:27 +1200
| Newsgroups | gmane.org.operators.nznog |
|---|---|
| Message-ID | <[email protected]> |
55MPS will give max ~ 28G << 100G thruput @prema+sdf+64B+ifg. In the other direction, at 100G, with ave 512B your max is 20-25Mpps, in any case 40G is fine and you save your firms $ on the 100G. Actually line capture at these speeds and thruputs have been doable for a good while now, mutli-Tbps is the challenge now. Capture/Filtering and processing are actually two different but interdependent things, as you know speed/thruput are normally qualified in terms of packet sizes and app, nic profile:sustained/peak, For capture, L3 ACLs segment work etc its just fixed and static filtering; different to triggers which need higher state protocol/flow processing, db lookups, markov etc, alot more smarts and CPU CS this lowers the upper limit. CPU stats will be very different. regards Brian ---------------------------- Original Message ---------------------------- Subject: Re: [nznog] Affordable TICSA LI Solution and Support From: Joel Wirāmu Pauling <[email protected]> Date: Sat, June 30, 2018 12:40 pm To: [email protected] Cc: "Shane Alcock" <shane.alcock-r/[email protected]> "nznog" <[email protected]> -------------------------------------------------------------------------- > You can achieve 55MPS from the latest generation of Smartnics - enough to > do 100GBits ports, with Layer3 ACLs ; with almost no CPU overhead. This is > running pretty much Mainline Linux (with Vendor Specific Smartnic offloads > i.e ASAP2 (mellanox), or Netronome). You then obviously need to dump it > somewhere but with some NVMEE Cards on x8 Bus you can do that too - before > shuffling it to slow and cheap archival. > > So ; no you don't need any real hardware smarts to do this today, there is > off the shelf bits you can add to off the shelf x86 hardware. Just a little > planning of your Capture box hardware and knowing what NIC's to use. > > -Joel > > On 30 June 2018 at 11:39, <[email protected]> wrote: > >> Hi Shane, >> >> I appreciate your response and it sounds like you and the OpenLI team >> making great progress, i wish the project and deployments all the best. >> Writing packet capture isnt hard and anyone with a compiler can do it. Of >> course a little coding b/gnd helps and depending on what your trying to do, >> the level of hardware knowhow you need. >> >> Take 100% packet capture - 'shouldnt' be a problem at a constant 1 pps, >> but it will be if you take 2 s to process. Lets say you optimize hw down .5 >> s with a DAG* (i worked their native's ~2004)* your still not going to >> make it. Timing values are too ilustrate my point and my take away from you >> response is OpenLI is basically targeting the general usecase here in NZ >> where say GE or 10G ~50/60 pk. % link utilization is still pretty OK and >> fit-for-purpose. I assume here your OpenLI app (NWO->IAF/MF >> ASN.1Enc/MD->HI) is well optimized from your excellent experience too. One >> suggestion maybe to bench it with XIA/Spirent STC for thruput vs >> line-rate/packet sizes vs expected from NOC. I dont use cap libraries or >> anything like that myself, my arch/interface is very different in exo/no >> net_dev, zero intr,dis flow handling depending on the ICs i got on hand (i >> have a french foreign legion policy for hardware - survive the bench and >> your in ). LI started here in NZ but was taken to another level in the US >> with massive linerates and volumes - vetting starts with corner case >> burntests, sinking IPGs <9.7ns ..<.97ns (100G+) with 1 Target frame - and >> if you can catchit send try to send that 1 frame, only after that you get >> to do something usefull and process it. Sounds easy as you can appreciate >> back in the day with north/south PCIE this was a massive feat, no SoCs >> root/pcie qpi we have today. >> >> Obviously this level of performance is bit OTT for our general LI small >> ISP market, but data volumes aint gonna go down - cheers for the heads up. >> Again wish you and the OpenLI project team much success. >> >> regards >> >> Brian Parsons BE(E&EE) MIEE | CEO TFE | LI TICSA SIGINT >> >> ---------------------------- Original Message ---------------------------- >> Subject: Re: [nznog] Affordable TICSA LI Solution and Support >> From: "Shane Alcock" <shane.alcock-r/[email protected]> >> Date: Fri, June 29, 2018 2:38 pm >> To: [email protected] >> Cc: [email protected] >> -------------------------------------------------------------------------- >> >> > Hi Brian, all, >> > >> > As the primary developer on the OpenLI project, I've been asked to >> address >> > some points that you've raised and correct any misconceptions people >> might >> > have about OpenLI as a result of this thread. >> > >> > For anyone concerned about a possible lack of experience in writing >> packet >> > capture software, most of my 14 years working here at the WAND network >> > research group has been dedicated to both writing and running software >> for >> > high speed passive capture and analysis projects. I am one of the primary >> > authors and maintainers of libtrace [1] [2], which is widely used both in >> > academia and industry (as well as the OpenLI software itself). Libtrace >> > supports a variety of capture methods and hardware, including DAG and >> DPDK, >> > and has a parallel API for writing multi-threaded packet capture >> > applications, so I have a reasonable understanding of what is required of >> > both hardware and software to do high-rate packet capture and encoding. >> > >> > OpenLI is not offering a "complete" solution to your TICSA problems. We >> > provide software that can capture packets and convert them into ETSI >> > records, wrapped in a centralised provisioning system for managing >> > intercepts, along with a mediation function that will push the resulting >> > records to the appropriate agency. Integration of OpenLI into your >> network >> > is a matter for the operator; however, we have been working closely with >> > the organisations that responded to our initial requests for funding to >> get >> > their deployments up and running. >> > >> > Longer term, we are starting to have conversations about what support >> > models we want to provide in the future, especially after the feedback I >> > received at the NZIX AGM last night, so watch this space. WAND has a long >> > history of maintaining and continually improving any open-source software >> > that it has released to the public [3] even when it is no longer >> > specifically funded by our research income (libtrace being a prime >> > example), so if OpenLI is being used then you can anticipate that WAND >> will >> > continue to look after it. >> > >> > Finally, the core software components of OpenLI are getting close to >> > complete. We have provisioning, mediation, multi-threaded and >> > multi-input-interface distributable collectors. We have support for voice >> > intercept (SIP + RTP + RTCP) and support for standard IP intercept >> (RADIUS >> > + IPv4), with IPv6 and static IP range intercepts not too far away. Most >> of >> > the upcoming effort is focused on getting test deployments working within >> > our partner networks, throwing "real" traffic at them and squashing any >> > issues that come up. At this stage, it's looking like another month or >> two >> > of work to finish the original set of requirements. >> > >> > Any other questions about the OpenLI project, feel free to get in touch >> > with me directly. >> > >> > [1] S.Alcock, P. Lorier and R. Nelson, "Libtrace: a packet capture and >> > analysis library", https://dl.acm.org/citation.cfm?id=2185382 >> > [2] Libtrace Team, "libtrace: C library for working with network packet >> > traces", https://github.com/LibtraceTeam/libtrace >> > [3] WAND network research group on GitHub, https://github.com/wanduow >> > >> > Thanks, >> > >> > Shane >> > >> > On Wed, Jun 27, 2018 at 11:49 PM, <[email protected]> wrote: >> > >> >> Hi Nathan, >> >> >> >> We dont work with the NSA, and you dont need them to decrypt mobile data >> >> there are many keys used in a heirachy depending on the pairing mobile >> >> nodes eNB/UE/MME/HSS etc, you just need to be on the right interface to >> >> catch and process the exchange Control/User Plane (keys vary) is trivial >> >> for LI vendors like myself who carry key derivation/enc/dec logic and >> >> protocol sequencing to handle such an exchange. eg UE/HSS:CK,IK - a well >> >> known mapping is your USIM/AuC:K. As far as knocking on doors, yes >> >> absolutely probably me, i sell and support what I build and proud of >> it, but >> >> NSA lol, but we sell more off shore - why is that? >> >> >> >> My post was only intended to inform of a local option that could make a >> >> material difference to operators trying to meet their obligations under >> the >> >> ACT. >> >> >> >> As an LI provider i have met a couple of your members stung badly after >> >> signing offshore and we couldve helped them, also i was encouraged by >> the >> >> LEAs that speak at your conference - they obviously cant directly >> endorse >> >> but the spirit and intention of my post was not to offend just inform. >> TFE >> >> like most LI vendors do not advertise or market, and there are only a >> >> handfull of us OEMs that build LI solutions the rest are reseller middle >> >> men. >> >> >> >> As far as OpenLI free it is your developing, thats great but till then >> >> compliance is real, but we wish you all the best. 2050 is when you'l >> have >> >> enough field experience in CU/FIbre/hardware with ASN.1 + ETSI/CALEA at >> >> working at single thread code level. Lets say you do cobble something >> >> together before then - once you can multithread your kernel (tuplet >> flows) >> >> linerate 200g (64B) stream + LI let me know, sorry lets make it easy - >> >> start with GE @ 1.44 Mpps + LI and ASN enc over HI => garauntee you a >> DoS >> >> all day and dont even think about SLAs. But good luck with that, until >> then >> >> TICSA 2017 >> >> >> >> The point is LI is not just about about Software. Hardware, firmware and >> >> real world field experience is required - how do you handle FEXT/NEXT on >> >> upstream active/passive TAPs can you run in rx only given the >> requirements >> >> and restrictions on spurious noise and traffic, there are many tech and >> >> capability requirements of the probe/mf which ETSI Parts are you going >> to >> >> do first?. HA/asymmetric network deployments (non-collo paths), how will >> >> you handle this situ? and TIER 4 support on commercial fabrics (Open >> >> generally means zip support - so i geuss nothing is realtime). If you do >> >> all this how will you be able to demonstrate at any point in time the >> two >> >> core aspects of TICSA 'format usability' and 'nw readiness'. >> >> >> >> We do LI not 'network security', ie. we build the 'device' and MF as >> >> descirbed in the NZ Search and Surveillance Act 2012 and NZ TICSA 2017. >> >> >> >> regards >> >> >> >> Brian Parsons BE(E&EE) MIEE | CEO TFE | LI TICSA SIGINT >> >> >> >> >> >> >> >> cheers >> >> >> >> >> >> ---------------------------- Original Message >> ---------------------------- >> >> Subject: Re: [nznog] Affordable TICSA LI Solution and Support >> >> >> From: "Nathan Ward" <[email protected]> >> >> Date: Wed, June 27, 2018 10:06 pm >> >> To: [email protected] >> >> Cc: [email protected] >> >> ------------------------------------------------------------ >> -------------- >> >> >> >> > >> >> >> On 27/06/2018, at 11:45 AM, [email protected] wrote: >> >> >> >> >> >> We are Telecom Forensics, an OEM of LI solutions based on the Kapiti >> >> Coast. >> >> >> >> >> >> Spam spam spam >> >> > >> >> > Are you the chap from Kapiti who a few years ago was knocking about >> >> claiming that the NSA had given him some sort of secret codes to >> magically >> >> decrypt all mobile data, or was that someone else? >> >> > >> >> > Not too sure why you’d be pushing this here, when there’s a community >> >> effort to develop something open source and free. >> >> >> >> I’m looking forward to where the OpenLI project goes, and I’d suggest >> that >> >> anyone looking for LI tools look towards the OpenLI project, which has >> been >> >> discussed on this before, and at the NZNOG conference, and I believe >> will >> >> be discussed tomorrow at the NZIX AGM. >> >> > ps. It’s free, but, go support it with some $, it’ll still be cheaper >> >> than commercial solutions. >> >> > >> >> > NZ: Highest per capita ETSI compliant LI implementors in the world. >> >> > >> >> > How long before Fincham pops up saying “oh also there’s an OpenLI >> >> project you should look at”? Place your bets. >> >> > >> >> > Looking forward to laughing about this one over drinks tomorrow night. >> >> > >> >> > -- >> >> > Nathan Ward >> >> > >> >> > >> >> >> >> _______________________________________________ >> >> NZNOG mailing list >> >> [email protected] >> >> https://list.waikato.ac.nz/mailman/listinfo/nznog >> >> >> >> >> > >> >> >> _______________________________________________ >> NZNOG mailing list >> [email protected] >> https://list.waikato.ac.nz/mailman/listinfo/nznog >> >> > _______________________________________________ NZNOG mailing list [email protected] https://list.waikato.ac.nz/mailman/listinfo/nznog