Re: classfiltercsv: filtering only works on bundle level
Xander Cage <[email protected]>
| Newsgroups | gmane.comp.sysutils.cfengine.general |
|---|---|
| Message-ID | <[email protected]> |
simplified version...
csv data used:
ClassExpression,uid,name,group,ssh_key
weirdos,mscott,Michael
Scott,pathetic,ssh-someSupERsecretKey287483438dsdjhdsdsdjahhsgdsg
weirdos,jkras,John
Krasinski,pathetic,ssh-someSupERsecretKey287483438dsdjhdsdsdjahhsgdsg
twats,tflen,Toby
Flenderson,tedious,ssh-someSupERsecretKey287483438dsdjhdsdsdjahhsgdsg
zombie,wimm,,dbaldap,ssh-someSupERsecretKey287483438dsdjhdsdsdjahhsgdsg
bundle agent parent_bundle
{
vars:
"data_file" string => "/root/cfe_testbed/users.csv";
"my_data" data => classfiltercsv($(data_file), "true", 0);
"keys_unsorted" slist => getindices(my_data);
"keys" slist => sort(keys_unsorted, "lex");
"user" string => "$(my_data[$(keys)][uid])";
methods:
"call_check_user" usebundle => check_user( @(my_data) ),
handle => "remove_it";
"call_create_user" usebundle => create_users_new( @(my_data) ),
depends_on => { "remove_it" };
reports:
"parent bundle: $(user)";
}
bundle agent check_user (info) {
vars:
"keys_unsorted" slist => getindices(info);
"keys" slist => sort(keys_unsorted, "lex");
"user" string => "$(info[$(keys)][uid])";
classes:
"zombie";
reports:
"check_user reached: $(user)";
}
bundle agent create_users_new(info) {
vars:
"keys_unsorted" slist => getindices(info);
"keys" slist => sort(keys_unsorted, "lex");
"user" string => "$(info[$(keys)][uid])";
"group" string => "$(info[$(keys)][group])";
"ssh_key" string => "$(info[$(keys)][ssh_key])";
"name" string => "$(info[$(keys)][name])";
classes:
"weirdos";
"twats";
reports:
"create_users reached: $(user)";
}
bundle agent __main__
{
methods:
"parent_bundle";
}
output:
root@aixtest01: /root/cfe_testbed # /var/cfengine/bin/cf-agent -KI -f
./data_container_test_simplified.cf
R: check_user reached: $(user)
R: create_users reached: $(user)
R: parent bundle: $(user)
verbose:
verbose: Using bundlesequence => {"main"}
verbose: B:
*****************************************************************
verbose: B: BEGIN bundle main
verbose: B:
*****************************************************************
verbose: P: .........................................................
verbose: P: BEGIN promise 'promise_data_container_test_simplified_cf_88'
of type "methods" (pass 1)
verbose: P: Promiser/affected object: 'parent_bundle'
verbose: P: Part of bundle: main
verbose: P: Base context class: any
verbose: P: Stack path: /default/main/methods/'parent_bundle'[1]
verbose: B:
*****************************************************************
verbose: B: BEGIN bundle parent_bundle
verbose: B:
*****************************************************************
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: V: .........................................................
verbose: V: BEGIN variables (pass 1)
verbose: V: Computing value of 'data_file'
verbose: V: Computing value of 'my_data'
verbose: V: Computing value of 'keys_unsorted'
verbose: V: Computing value of 'keys'
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: P: .........................................................
verbose: P: BEGIN promise 'remove_it' of type "methods" (pass 1)
verbose: P: Promiser/affected object: 'call_check_user'
verbose: P: Part of bundle: parent_bundle
verbose: P: Base context class: any
verbose: P: Stack path:
/default/main/methods/'parent_bundle'/default/parent_bundle/methods/'call_check_user'[1]
verbose: B:
*****************************************************************
verbose: B: BEGIN bundle check_user( {"@(my_data)"})
verbose: B:
*****************************************************************
verbose: V: + Private parameter: 'info' in scope 'check_user' (type:
s) in pass 1
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: V: .........................................................
verbose: V: BEGIN variables (pass 1)
verbose: V: Computing value of 'keys_unsorted'
verbose: V: Computing value of 'keys'
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: C: .........................................................
verbose: C: BEGIN classes / conditions (pass 1)
verbose: C: + Private class: zombie
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: P: .........................................................
verbose: P: BEGIN promise 'promise_data_container_test_simplified_cf_54'
of type "reports" (pass 1)
verbose: P: Promiser/affected object: 'check_user reached: $(user)'
verbose: P: From parameterized bundle: check_user( {"@(my_data)"})
verbose: P: Base context class: any
verbose: P: Stack path:
/default/main/methods/'parent_bundle'/default/parent_bundle/methods/'call_check_user'/default/check_user/reports/'check_user
reached: $(user)'[1]
R: check_user reached: $(user)
verbose: A: Promise was KEPT
verbose: P: END reports promise (check_user reached: $(user))
verbose: V: .........................................................
verbose: V: BEGIN variables (pass 2)
verbose: V: Computing value of 'keys_unsorted'
verbose: V: Computing value of 'keys'
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: C: .........................................................
verbose: C: BEGIN classes / conditions (pass 2)
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: V: .........................................................
verbose: V: BEGIN variables (pass 3)
verbose: V: Computing value of 'keys_unsorted'
verbose: V: Computing value of 'keys'
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: C: .........................................................
verbose: C: BEGIN classes / conditions (pass 3)
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: A: ...................................................
verbose: A: Bundle Accounting Summary for 'check_user' in namespace default
verbose: A: Promises kept in 'check_user' = 1
verbose: A: Promises not kept in 'check_user' = 0
verbose: A: Promises repaired in 'check_user' = 0
verbose: A: Aggregate compliance (promises kept/repaired) for bundle
'check_user' = 100.0%
verbose: A: ...................................................
verbose: Additional promise info: handle 'remove_it' source path
'./data_container_test_simplified.cf' at line 26
verbose: Method 'check_user' verified
verbose: B:
*****************************************************************
verbose: B: END bundle check_user
verbose: B:
*****************************************************************
verbose: A: Promise was KEPT
verbose: P: END methods promise (call_check_user)
verbose: P: .........................................................
verbose: P: BEGIN promise 'promise_data_container_test_simplified_cf_29'
of type "methods" (pass 1)
verbose: P: Promiser/affected object: 'call_create_user'
verbose: P: Part of bundle: parent_bundle
verbose: P: Base context class: any
verbose: P: Stack path:
/default/main/methods/'parent_bundle'/default/parent_bundle/methods/'call_create_user'[1]
verbose: B:
*****************************************************************
verbose: B: BEGIN bundle create_users_new( {"@(my_data)"})
verbose: B:
*****************************************************************
verbose: V: + Private parameter: 'info' in scope 'create_users_new'
(type: s) in pass 1
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: V: .........................................................
verbose: V: BEGIN variables (pass 1)
verbose: V: Computing value of 'keys_unsorted'
verbose: V: Computing value of 'keys'
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: C: .........................................................
verbose: C: BEGIN classes / conditions (pass 1)
verbose: C: + Private class: weirdos
verbose: C: + Private class: twats
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: P: .........................................................
verbose: P: BEGIN promise 'promise_data_container_test_simplified_cf_79'
of type "reports" (pass 1)
verbose: P: Promiser/affected object: 'create_users reached: $(user)'
verbose: P: From parameterized bundle: create_users_new( {"@(my_data)"})
verbose: P: Base context class: any
verbose: P: Stack path:
/default/main/methods/'parent_bundle'/default/parent_bundle/methods/'call_create_user'/default/create_users_new/reports/'create_users
reached: $(user)'[1]
R: create_users reached: $(user)
verbose: A: Promise was KEPT
verbose: P: END reports promise (create_users reached: $(user))
verbose: V: .........................................................
verbose: V: BEGIN variables (pass 2)
verbose: V: Computing value of 'keys_unsorted'
verbose: V: Computing value of 'keys'
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: C: .........................................................
verbose: C: BEGIN classes / conditions (pass 2)
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: V: .........................................................
verbose: V: BEGIN variables (pass 3)
verbose: V: Computing value of 'keys_unsorted'
verbose: V: Computing value of 'keys'
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: C: .........................................................
verbose: C: BEGIN classes / conditions (pass 3)
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: A: ...................................................
verbose: A: Bundle Accounting Summary for 'create_users_new' in namespace
default
verbose: A: Promises kept in 'create_users_new' = 1
verbose: A: Promises not kept in 'create_users_new' = 0
verbose: A: Promises repaired in 'create_users_new' = 0
verbose: A: Aggregate compliance (promises kept/repaired) for bundle
'create_users_new' = 100.0%
verbose: A: ...................................................
verbose: Additional promise info: source path
'./data_container_test_simplified.cf' at line 29
verbose: Method 'create_users_new' verified
verbose: B:
*****************************************************************
verbose: B: END bundle create_users_new
verbose: B:
*****************************************************************
verbose: A: Promise was KEPT
verbose: P: END methods promise (call_create_user)
verbose: P: .........................................................
verbose: P: BEGIN promise 'promise_data_container_test_simplified_cf_34'
of type "reports" (pass 1)
verbose: P: Promiser/affected object: 'parent bundle: $(user)'
verbose: P: Part of bundle: parent_bundle
verbose: P: Base context class: any
verbose: P: Stack path:
/default/main/methods/'parent_bundle'/default/parent_bundle/reports/'parent
bundle: $(user)'[1]
R: parent bundle: $(user)
verbose: A: Promise was KEPT
verbose: P: END reports promise (parent bundle: $(user))
verbose: V: .........................................................
verbose: V: BEGIN variables (pass 2)
verbose: V: Computing value of 'data_file'
verbose: V: Computing value of 'my_data'
verbose: V: Computing value of 'keys_unsorted'
verbose: V: Computing value of 'keys'
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: V: .........................................................
verbose: V: BEGIN variables (pass 3)
verbose: V: Computing value of 'data_file'
verbose: V: Computing value of 'my_data'
verbose: V: Computing value of 'keys_unsorted'
verbose: V: Computing value of 'keys'
verbose: Skipping iteration since variable 'keys' resolves to an empty list
verbose: A: ...................................................
verbose: A: Bundle Accounting Summary for 'parent_bundle' in namespace
default
verbose: A: Promises kept in 'parent_bundle' = 5
verbose: A: Promises not kept in 'parent_bundle' = 0
verbose: A: Promises repaired in 'parent_bundle' = 0
verbose: A: Aggregate compliance (promises kept/repaired) for bundle
'parent_bundle' = 100.0%
verbose: A: ...................................................
verbose: Additional promise info: source path
'./data_container_test_simplified.cf' at line 88
verbose: Method 'parent_bundle' verified
verbose: B:
*****************************************************************
verbose: B: END bundle parent_bundle
verbose: B:
*****************************************************************
verbose: A: Promise was KEPT
verbose: P: END methods promise (parent_bundle)
verbose: A: ...................................................
verbose: A: Bundle Accounting Summary for 'main' in namespace default
verbose: A: Promises kept in 'main' = 6
verbose: A: Promises not kept in 'main' = 0
verbose: A: Promises repaired in 'main' = 0
verbose: A: Aggregate compliance (promises kept/repaired) for bundle
'main' = 100.0%
verbose: A: ...................................................
verbose: B:
*****************************************************************
verbose: B: END bundle main
verbose: B:
*****************************************************************
verbose: Waiting for background processes
verbose: No more background processes to wait for
verbose: No lock purging scheduled
verbose: Logging total compliance, total 'Outcome of version (not
specified) (agent-0): Promises observed to be kept 100.00%, Promises
repaired 0.00%, Promises not repaired 0.00%'
On Monday, October 18, 2021 at 10:15:06 AM UTC+2 Xander Cage wrote:
> ah good to know, so there is no hurry in eliminating arrays.
>
> btw...i tried you suggestion and passed the data container to the child
> function...no fun. variable undefined and a strange error thrown.
>
> /var/cfengine/bin/cf-agent -KI -f ./data_container_test.cf
> error: The value of variable 'default:remove_in_files.user' contains a
> reference to itself, '$(user)'
> R: $(user)
> R: Time: Mon Oct 18 10:07:14 2021 - Bundle: check_user - Message: uid
> $(user) completely removed.
>
>
> bundle agent parent_bundle
> {
>
>
> vars:
> "data_file" string => "/root/cfe_testbed/users.csv";
> "d" data => classfiltercsv($(data_file), "true", 0);
>
>
> methods:
>
> "call_check_user" usebundle => check_user( @(d) ),
> handle => "remove_it";
>
>
> }
>
> bundle agent check_user (info) {
>
> vars:
>
> "keys_unsorted" slist => getindices("info");
>
> "keys" slist => sort(keys_unsorted, "lex");
> "user" string => "$(info[$(keys)][uid])";
>
> classes:
>
> classes:
>
> "zombie";
>
>
> "EXISTS_PASSWD"
> comment => "check if user exists as local user (in
> /etc/passwd)",
> expression => regline("^$(user):.*", "/etc/passwd");
>
> "EXISTS_LDAP"
> comment => "check if user has an entry in the
> /etc/security/user file",
> expression => regline("^$(user):",
> "/etc/security/user");
>
> "EXISTS_HOMEDIR"
> comment => "user has a homedir..",
> expression => fileexists("/home/$(user)/.");
>
> methods:
>
> "remove_by_aix" usebundle => remove_by_aix($(user)),
> comment => "if its an OS-user let AIX do
> a rmuser",
> ifvarclass => "EXISTS_PASSWD";
>
> "remove_from_files" usebundle => remove_in_files($(user)),
> comment => "remove in stanza-files and
> cronjob-dir";
>
> "remove_home_dir" usebundle => remove_home_dir($(user)),
> comment => "will assume /home/$USER is
> the homedir",
> ifvarclass => "EXISTS_HOMEDIR";
>
> reports:
>
> "$(user)";
>
> EXISTS_PASSWD.!EXISTS_LDAP::
>
> "Time: $(sys.date) - Bundle: $(this.bundle) - Message: uid
> $(user) exists in /etc/passwd only (OS user).";
>
> !EXISTS_PASSWD.EXISTS_LDAP::
>
> "Time: $(sys.date) - Bundle: $(this.bundle) - Message: uid
> $(user) exists only in security-file (LDAP user).";
>
> EXISTS_HOMEDIR.!EXISTS_LDAP.!EXISTS_PASSWD::
>
> "Time: $(sys.date) - Bundle: $(this.bundle) - Message: uid
> $(user) leftover /home/dir.";
>
> !EXISTS_HOMEDIR.!EXISTS_LDAP.!EXISTS_PASSWD::
>
> "Time: $(sys.date) - Bundle: $(this.bundle) - Message: uid
> $(user) completely removed.";
>
> }
>
> bundle agent remove_by_aix(user) {
>
>
> commands:
>
> "/usr/sbin/rmuser -p "
> args => "$(user)",
> comment => "remove with OS means ..";
>
> }
>
> bundle agent remove_in_files(user) {
>
>
> vars:
>
> "stanza_files" slist => { "/etc/security/limits",
> "/etc/security/user",
> "/etc/security/lastlog",
> "/etc/security/passwd",
> "/etc/security/environ",
> "/etc/security/user.roles",
> };
>
> files:
>
> "/var/spool/cron/crontabs/$(user)"
> comment => "delete leftover cronjob file",
> delete => tidy;
>
> methods:
>
> "call_remove_stanza" usebundle => delete_stanza($(stanza_files),
> $(user)),
> comment => "delete user in each file";
> }
>
> bundle agent remove_home_dir(user) {
>
>
> methods:
>
> "remove_home_dir" usebundle => rm_rf("/home/$(user)"),
> comment => "delete leftover home-dir";
> }
>
>
> On Friday, October 15, 2021 at 5:02:53 PM UTC+2 Nick Anderson wrote:
>
>> *Deprecated* is probably too strong of a word there. They aren't going
>> anywhere, at least not in CFEngine 3. But there is more focus on data type
>> variables and generally I would tend to use functions that return data vs
>> returning a classic array.
>>
>> So, I probably would not spend time hunting down classic arrays unless I
>> had nothing better to do.
>>
>> On Friday, October 15, 2021 at 3:03:09 AM UTC-5 Xander Cage wrote:
>>
>>> ah...yes, just a matter of reorganization, thank you. helped a lot.
>>>
>>> while we are at it...i am just doing all this because of this phrase in
>>> ->
>>> https://docs.cfengine.com/docs/3.18/reference-language-concepts-variables.html#associative-arrays
>>>
>>> Note that associative arrays are being deprecated in favor of the data
>>> <https://docs.cfengine.com/docs/3.18/reference-promise-types-vars.html#data>
>>> variable type.
>>>
>>> this sent me on the hunt for arrays in my policy jungle. or is this
>>> statement just informational?
>>> On Thursday, October 14, 2021 at 9:53:49 PM UTC+2 Nick Anderson wrote:
>>>
>>>> Hi,
>>>>
>>>> Hopefully this attaches itself to the correct thread, no email in my
>>>> inbox to respond to for this one. ¯\_(ツ)_/¯
>>>>
>>>> Can you provide a completely stand-alone policy so that we can easily
>>>> run and re-produce the problem you have?
>>>>
>>>> Proceeding only with my internal parser :)….
>>>>
>>>> Regarding "only works if the data and filter classes are in the same
>>>> bundle", I suspect that you are just seeing class scope at work. In
>>>> the documentation, Classes and Decisions in Language Concepts of the
>>>> Reference manual
>>>> <https://docs.cfengine.com/docs/3.18/reference-language-concepts-classes.html>
>>>> has a section on Class scope
>>>> <https://docs.cfengine.com/docs/3.18/reference-language-concepts-classes.html#class-scope>
>>>> which talks about this a bit. It looks like you are using classes that are
>>>> defined with a *bundle* scope (only visible within that bundle) and
>>>> when you call your child_bundle you did not have it inherit the classes, so
>>>> inside of child_bundle, any *bundle* scoped classes that were defined
>>>> in parent_bundle won't be visible.
>>>>
>>>> bundle agent __main__{
>>>> methods:
>>>> "parent";
>>>> }bundle agent parent{
>>>> classes:
>>>> "my_bundle_scoped_class_defined_in_parent";
>>>>
>>>> methods:
>>>> "Write our child into our will"
>>>> usebundle => child,
>>>> inherit => "true";
>>>>
>>>> "Step children get the shaft by default"
>>>> usebundle => step_child;
>>>> }bundle agent child{
>>>> reports:
>>>> "$(this.bundle): I can see my_bundle_scoped_class_defined_in_parent since my parent wrote me into the will."
>>>> if => "my_bundle_scoped_class_defined_in_parent";
>>>> "$(this.bundle): I wasn't written into the will, at least I am not red-headed."
>>>> unless => "my_bundle_scoped_class_defined_in_parent";}bundle agent step_child{
>>>> reports:
>>>> "$(this.bundle): I can see my_bundle_scoped_class_defined_in_parent since my parent wrote me into the will."
>>>> if => "my_bundle_scoped_class_defined_in_parent";
>>>> "$(this.bundle): I wasn't written into the will, at least I am not red-headed."
>>>> unless => "my_bundle_scoped_class_defined_in_parent";}
>>>>
>>>>
>>>> R: child: I can see my_bundle_scoped_class_defined_in_parent since my parent wrote me into the will.
>>>> R: step_child: I wasn't written into the will, at least I am not red-headed.
>>>>
>>>> Regarding "i want to use the data container as a somewhat "global"
>>>> variable without repeating it" , I don't think that you have to repeat
>>>> the data container, you can pass it whole like you do, or pass it's name.
>>>> Either way, you don't get a variable registered in the state (see
>>>> --show-evaluated-vars).
>>>> Listing 1: Example Policy
>>>>
>>>> bundle agent __main__{
>>>> vars:
>>>> "my_data" data => '{ "data": "here" }';
>>>>
>>>> methods:
>>>> "Pass data container"
>>>> usebundle => example_1( @(my_data) );
>>>>
>>>> "Pass NAME of data container"
>>>> usebundle => example_2( "$(this.namespace):$(this.bundle).my_data" );}
>>>> bundle agent example_1(my_data){
>>>> vars: "my_data_$(this.bundle)" data => '{ "bundle": "$(this.bundle)" }';
>>>> reports: "$(this.bundle):$(with)" with => storejson( @(my_data) );}bundle agent example_2(my_data){
>>>> vars: "my_data_$(this.bundle)" data => '{ "bundle": "$(this.bundle)" }';
>>>> reports: "$(this.bundle):$(with)" with => storejson( $(my_data) );}
>>>>
>>>>
>>>> # cf-agent --no-lock --log-level info --show-evaluated-vars=my_data --file /tmp/example.cf
>>>> R: example_1:{
>>>> "data": "here"
>>>> }
>>>> R: example_2:{
>>>> "data": "here"
>>>> }
>>>> Variable name Variable value Meta tags Comment
>>>> default:example_1.my_data_example_1 {"bundle":"example_1"} source=promise
>>>> default:example_2.my_data_example_2 {"bundle":"example_2"} source=promise
>>>> default:main.my_data {"data":"here"} source=promise
>>>>
>>>> Hope this helps …
>>>>
>>>
--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/7cf77001-6f14-478f-95d7-5940a153c2b9n%40googlegroups.com.