Re: Dynamically managed groups, etc

Thank you Howard (and openldap core team) for this synthesis of the 
current and future state of dynamic membership devs.

Indeed, some of my customers were suffering from dynlist performances, 
and the decision was taken to go back to more static features (like 
memberOf).

For now I did not test nestgroup overlay, but I'll give a look for next 
installations. Do you have any idea in which version of OpenLDAP 
nestgroup will be officially released?

Regards,


Le 26/02/2024 à 16:54, Howard Chu a écrit :
> The recent work on expanding dynamic group functionality in the dynlist overlay seems to have been
> a bad idea. It makes an already fairly complex overlay even more complicated, and it puts a lot more
> work into the read side of operations, which adds up to quite noticeable slowdowns in search performance
> on deployments that make heavy use of dynamic groups with large memberships.
>
> It appears that in these situations, the approach used in the autogroup overlay (which has been
> in contrib since 2007) is better. It moves all of the membership management into the write side
> of operations, updating memberships whenever dynamic group definitions are modified, or when
> member entries are written/updated/etc. As such, it allows dynamically defined group memberships
> to be read/searched at full speed, as if they were static groups. The search performance difference
> between autogroup and the dynlist approach is pretty drastic when large groups and large numbers
> of groups are in use. As such I believe autogroup should be recommended, going forward, and we
> should move it from contrib into the core code.
>
> For similar reasons, dynamically managing memberOf attributes in dynlist also has a major
> impact on search performance. And as alluded to before, adding that functionality to dynlist has
> made the code quite a bit more complex. Again for performance reasons, it's better to manage
> this attribute on the write side, updating a real attribute in the DB when groups and memberships
> are modified, instead of doing the lookup work on the read/search side. As such, we should
> reverse the decision to deprecate the memberof overlay. There were a couple problems that the
> overlay presented in replication environments before, that prompted us to deprecate it, but I
> believe those problems have now been resolved. The first one, referenced in ITS#7400, had to do
> with the actual memberof attributes getting replicated even though they weren't meant to be.
> The solution for that should have been to specify that memberof should be excluded from replication
> using "exattr" in a syncrepl consumer config. The exattr option wasn't behaving as intended in
> the past, and using it would cause data desyncs, but that problem was fixed long ago.
>
> The other problem was simply due to the lack of ordering guarantees in syncrepl refreshes,
> which prevented the memberof overlay from updating an member's memberof attribute if the group
> entry got replicated before the member entry. That problem has been solved in ITS#10167, simply
> adding a check whenever new entries are added, to see if they're already members of any existing groups.
>
> As such, the memberof overlay should be perfectly fine for use in replication scenarios now.
> More testing of those scenarios is welcome.
>
> With statically managed member and memberof attributes, the major hits to search performance have
> been removed. We're still trying dynamically managed nesting of groups though, which is what the
> new nestgroup overlay (ITS#10161) is for. Again, we need testing to see how much performance impact
> there is from this much-reduced overhead in the read/search side of things. The config is also a
> lot easier to understand in its own overlay, instead of shoe-horned into the dynlist config.
>
> Testing and feedback appreciated.
>
-- 
David Coutadeur | IAM integrator

[email protected]
+33 7 88 46 85 34
16 avenue Hoche, Paris 75008

Worteks | https://www.worteks.com