Re: [Fuego] Eliminating the usage of jenkins-cli.jar in ftc script

> -----Original Message-----
> From: Srivatsan S <[email protected]>
> 
> Tim,
> 
> I hadn't tested with the change, but this was a work in progress as it created the job with empty XML config. I had the job_xml stuff in
> progress to pass the configs to Jenkins. Noticed that you've committed this change. Shall test with your latest changes.

Sorry.  I didn't realize this was a work in progress.  I would have given you feedback and more time
to complete it if I had known.

Please do test the changes I committed.  Thanks.

 -- Tim

 
> Thanks,
> Srivatsan
> 
> On Thu, Nov 19, 2020 at 2:24 AM Bird, Tim <[email protected] <mailto:[email protected]> > wrote:
> 
> 
> 
> 
> 	> -----Original Message-----
> 	> From: Srivatsan S <[email protected] <mailto:[email protected]> >
> 	>
> 	> Tim,
> 	>
> 	> While we were looking at eliminating the usage of jenkins-cli.jar in the ftc script, we came across create_job() API in the Jenkins
> Python
> 	> module which can be used to create jobs in Jenkins, thus eliminating the usage of jenkins-cli.jar in the ftc script.
> 	>
> 	> The below patch works for job creation in Jenkins through ftc script. We shall get back to you on ways to programmatically
> update other
> 	> job properties.
> 
> 	Thanks for sending this.  It helped me work the issues with this change.
> 	There were a few problems with the implementation.  Did you test this?
> 	It doesn’t seem like it would have worked, since it would have created
> 	jobs with no shell command data.
> 
> 	I ended up using the basic concept, but handling other aspects required for
> 	this change that you-all missed.
> 
> 	>
> 	> git diff ftc
> 	>
> 	> diff --git a/scripts/ftc b/scripts/ftc
> 	> index 30195fb..a193c4d 100755
> 	> --- a/scripts/ftc
> 	> +++ b/scripts/ftc
> 	> @@ -1559,9 +1559,7 @@ ftc run-test -b $NODE_NAME -t {testdir} -s {testspec} \\
> 	>      job_name = board + "." + test.spec + "." + test.name <http://test.name>  <http://test.name>
> 	>      print("Creating job " + job_name)
> 	>      try:
> 	> -        rcode = subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+' create-job ' +
> 	> -            job_name + ' < ' + tmp, shell=True)
> 
> 	In this case, the tmp file contains the data for config.xml for the new job.  The code above this needed
> 	to be refactored to submit the xml data as a string rather than piping in to jenkins-cli.jar from a file.
> 
> 	> -        os.unlink(tmp)
> 	> +        rcode = server.create_job(job_name, jenkins.EMPTY_CONFIG_XML)
> 	This EMPTY_CONFIG_XML will end up creating a stub job, with no shell command
> 	to invoke the appropriate Fuego test.
> 
> 	>          return rcode
> 	server.create_job returns "None" on success.  The caller can only handle a numeric
> 	return result (it does a bitwise |, in the case of creating multiple jobs).
> 
> 	>      except Exception as e:
> 	>          print("Job already exists")
> 	> @@ -1609,9 +1607,7 @@ def create_batch_job(conf, board, testplan, plan_tests):
> 	>      print("Creating batch job ")
> 	>      try:
> 	>          job_name = board+'.'+testplan+'.batch'
> 	> -        rcode = subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+' create-job ' +
> 	> -             job_name + '< ' + tmp, shell=True)
> 	> -        os.unlink(tmp)
> 	> +        rcode = server.create_job(job_name, jenkins.EMPTY_CONFIG_XML)
> 	Same issues here as above.
> 
> 	>          return rcode
> 	>      except Exception as e:
> 	>          print("Job already exists")
> 	>
> 	>
> 	> Thanks,
> 	> Srivatsan
> 	>
> 
> 	FYI - Below is the patch I ended up with, that seems to work.  I tested it with
> 	both existing and non-existing regular jobs and batch jobs.  I also tested creating
> 	jobs for multiple boards on the same command line (some already existing
> 	and some not).
> 
> 	Thanks for the inspiration for the change.  I think this is a nice improvement
> 	to Fuego.
> 	  -- Tim
> 
> 
> 	commit b019ac2a2e7d16b03d24083c2fce868d736a61ec
> 	Author: Tim Bird <[email protected] <mailto:[email protected]> >
> 	Date:   Wed Nov 18 13:26:39 2020 -0700
> 
> 	    ftc: eliminate call to jenkins-cli.jar for creating jobs
> 
> 	    Use the python jenkins module to use the Jenkins REST API
> 	    to perform job creation, instead of using jenkins-cli.jar.
> 	    This eliminates all references to jenkins-cli.jar from ftc.
> 	    Theoretically, we could remove the installation of jenkins-cli.jar
> 	    from the Docker container, but let's hold off and make sure it's
> 	    not needed for anything else.
> 
> 	    The call to create jobs via the jenkins module is much simpler.
> 	    We don't need to create a temp file to hold the config xml for the
> 	    job.  We do have to convert a non-numeric return code into a
> 	    number since callers do bitwise or on it.
> 
> 	    Signed-off-by: Tim Bird <[email protected] <mailto:[email protected]> >
> 
> 	diff --git a/scripts/ftc b/scripts/ftc
> 	index 30195fb..d6c9033 100755
> 	--- a/scripts/ftc
> 	+++ b/scripts/ftc
> 	@@ -1438,6 +1438,7 @@ def link_key(item):
> 	         return "zzz+"+key
> 
> 
> 	+# return 0 on success
> 	 def create_job(conf, board, test):
> 	     flot_link = '<flotile.FlotPublisher plugin="[email protected]"/>'
> 
> 	@@ -1505,9 +1506,7 @@ def create_job(conf, board, test):
> 	         for cover, file_ref in link_tuples:
> 	             fail_links += ' ' + template_link % (str(file_ref), str(cover))
> 
> 	-    tmp = "/tmp/fuego_tmp_job"
> 	-    fd = open(tmp, "w+")
> 	-    fd.write("""<?xml version='1.0' encoding='UTF-8'?>
> 	+    job_xml = """<?xml version='1.0' encoding='UTF-8'?>
> 	 <project>
> 	     <actions/>
> 	     <description></description>
> 	@@ -1553,28 +1552,28 @@ ftc run-test -b $NODE_NAME -t {testdir} -s {testspec} \\
> 	 """.format(board=board, reboot=test.reboot, rebuild=test.rebuild,
> 	         precleanup=test.precleanup, postcleanup=test.postcleanup,
> 	         testdir=test.name <http://test.name> , testspec=test.spec, timeout=test.timeout,
> 	-        flot_link=flot_link, success_links=success_links, fail_links=fail_links))
> 	-    fd.close()
> 	+        flot_link=flot_link, success_links=success_links,
> 	+        fail_links=fail_links)
> 
> 	     job_name = board + "." + test.spec + "." + test.name <http://test.name>
> 	     print("Creating job " + job_name)
> 	     try:
> 	-        rcode = subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+' create-job ' +
> 	-            job_name + ' < ' + tmp, shell=True)
> 	-        os.unlink(tmp)
> 	+        rcode = server.create_job(job_name, job_xml)
> 	+        if not rcode:
> 	+            return 0
> 	+        else:
> 	+            return 1
> 	         return rcode
> 	     except Exception as e:
> 	         print("Job already exists")
> 	         print(e)
> 	-        sys.exit(1)
> 	+        return 1
> 
> 
> 	 def create_batch_job(conf, board, testplan, plan_tests):
> 	-    tmp = "/tmp/fuego_tmp_batch"
> 	     job_list = [board+'.'+test.spec+'.'+test.name <http://test.name>  for test in plan_tests]
> 
> 	-    fd = open(tmp, "w+")
> 	-    fd.write("""<?xml version='1.0' encoding='UTF-8'?>
> 	+    job_xml = """<?xml version='1.0' encoding='UTF-8'?>
> 	 <project>
> 	   <actions/>
> 	   <description></description>
> 	@@ -1602,21 +1601,20 @@ def create_batch_job(conf, board, testplan, plan_tests):
> 	   </publishers>
> 	   <buildWrappers/>
> 	 </project>
> 	-""".format(board=board, alljobs=','.join(job_list)))
> 	-
> 	-    fd.close()
> 	+""".format(board=board, alljobs=','.join(job_list))
> 
> 	     print("Creating batch job ")
> 	     try:
> 	         job_name = board+'.'+testplan+'.batch'
> 	-        rcode = subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+' create-job ' +
> 	-             job_name + '< ' + tmp, shell=True)
> 	-        os.unlink(tmp)
> 	-        return rcode
> 	+        rcode = server.create_job(job_name, job_xml)
> 	+        if not rcode:
> 	+            return 0
> 	+        else:
> 	+            return 1
> 	     except Exception as e:
> 	         print("Job already exists")
> 	         print(e)
> 	-        sys.exit(1)
> 	+        return 1
> 
> 
> 	 # parse the testplan and returns a list of test_class instances.
> 
> 
> 
> This message contains confidential information and is intended only for the individual(s) named. If you are not the intended recipient, you
> are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and attached file/s is strictly
> prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be
> secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The
> sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail
> transmission.