Re: Deprecation of &persistent

I'm not a core developer so I could be totally incorrect here, but I've
always been under the impression that &persistent was nearly as broken as
&sychronized was, at least in a cluster environment.  If I'm correct in my
assessment, it's not really about retaining an existing feature as much as
correctly implementing a new feature which I imagine is quite complicated.

I agree the learning curve with Broker and persistent stores is a little
steep, but there's actually a fair bit of good methods for debugging to
make sure it's doing what you think it should be.  You also have full
control over when it updates the store so your persistence doesn't rely on
a clean shutdown of the nodes.

I imagine you could write some boilerplate code that just fires off a
scheduled task via bro_init to update the store, all you'd need to do is
paste it in and update the variable name for each policy.

-Dop



On Sat, Apr 20, 2019 at 4:47 PM Michał Purzyński <[email protected]>
wrote:

> I second David's opinion that some form of a quick-path stores or a new
> implementation of &persistent should be implemented.
>
> The solution we offer right now makes people write tens of lines of
> cluster code, learning all details of cluster communication and dealing
> with a product that's pretty much impossible to debug (broker). Rinse,
> repeat, guess why it does not work.
>
> I communicated internally we are not upgrading to 2.6 for now. I scoped
> the upgrade project to take me half a year (or we would have to take a
> detection impact).
>
> While it is understood the old &synchronized attribute was an enormous
> hack, broker gives us the ability to do it right.
> An easy to use, transparent attribute or some kind of wrapper is something
> we should consider to offer.
>
>
>
> On Sat, Apr 20, 2019 at 7:05 PM David Hoelzer <
> [email protected]> wrote:
>
>> Hello all!
>>
>> TLDR:
>> I'd like to ask that there be some thought given to the deprecation and
>> eventual removal of the &persistent option in favor of Broker data
>> stores.  IMHO, there are uses cases where the &persistent attribute is
>> much more attractive and lower overhead than the data store approach.
>>
>> Longer:
>> As you are likely aware, &persistent is now marked deprecated and we
>> expect it to disappear in the next version or two.  The recommendation
>> for replacement is the much more robust, SQLite backed, Broker data store.
>>
>> The data store solution is very elegant, though it does seem to require
>> more fiddling than it ought to to get a data store set up.  In the long
>> term and when dealing with large amounts of data that must be persistent
>>
>> and synchronized across nodes, this really is a wonderful solution.
>>
>> That said, there seem to me to be some use cases where that is a massive
>>
>> hammer to swing at some very small problems.  For example, we have one
>> analysis script that is tracking successful external DNS resolutions.
>> Specifically, it is keeping track of all IPv4 and IPv6 addresses
>> resolved in the last 7 days (&read_expire 7 days) in a set.  For all
>> outbound connection attempts, this script generates a notice when the
>> connection involves an external host that never appeared in a DNS answer
>>
>> record.  This is quite handy when it comes to locating unauthorized
>>
>> outbound scanning, some C2 behaviors that do not rely on DNS/fast flux
>> sorts of things, fragile configurations of enterprise services, etc.
>> This has been performing quite well for several years now in more than
>> one relatively decent sized networks (100,000+ hosts).
>>
>> For this problem (and others that I can imagine that would take a
>> similar tack - i.e., only storing a set, vector, or other single
>> primitive, rather than a massive record in a table or a table of
>> tables), the &persistent is perfectly "sized."
>>
>> Am I alone in thinking that this feature should be retained *along side
>> of* Broker data stores and potentially documented as recommended for
>> simple primitive data persistence?
>>
>> Thanks!
>>
>> --
>> ----
>> David Hoelzer
>> Chief of Operations
>> Enclave Forensics, Inc.
>>
>> _______________________________________________
>> Zeek mailing list
>> [email protected]
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
> _______________________________________________
> Zeek mailing list
> [email protected]
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

_______________________________________________
Zeek mailing list
[email protected]
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek