Re: [Fuego] Eliminating the usage of jenkins-cli.jar in ftc script

Tim,

I hadn't tested with the change, but this was a work in progress as it
created the job with empty XML config. I had the job_xml stuff in progress
to pass the configs to Jenkins. Noticed that you've committed this change.
Shall test with your latest changes.

Thanks,
Srivatsan

On Thu, Nov 19, 2020 at 2:24 AM Bird, Tim <[email protected]> wrote:

>
>
> > -----Original Message-----
> > From: Srivatsan S <[email protected]>
> >
> > Tim,
> >
> > While we were looking at eliminating the usage of jenkins-cli.jar in the
> ftc script, we came across create_job() API in the Jenkins Python
> > module which can be used to create jobs in Jenkins, thus eliminating the
> usage of jenkins-cli.jar in the ftc script.
> >
> > The below patch works for job creation in Jenkins through ftc script. We
> shall get back to you on ways to programmatically update other
> > job properties.
>
> Thanks for sending this.  It helped me work the issues with this change.
> There were a few problems with the implementation.  Did you test this?
> It doesn’t seem like it would have worked, since it would have created
> jobs with no shell command data.
>
> I ended up using the basic concept, but handling other aspects required for
> this change that you-all missed.
>
> >
> > git diff ftc
> >
> > diff --git a/scripts/ftc b/scripts/ftc
> > index 30195fb..a193c4d 100755
> > --- a/scripts/ftc
> > +++ b/scripts/ftc
> > @@ -1559,9 +1559,7 @@ ftc run-test -b $NODE_NAME -t {testdir} -s
> {testspec} \\
> >      job_name = board + "." + test.spec + "." + test.name <
> http://test.name>
> >      print("Creating job " + job_name)
> >      try:
> > -        rcode = subprocess.call('java -jar
> /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+'
> create-job ' +
> > -            job_name + ' < ' + tmp, shell=True)
>
> In this case, the tmp file contains the data for config.xml for the new
> job.  The code above this needed
> to be refactored to submit the xml data as a string rather than piping in
> to jenkins-cli.jar from a file.
>
> > -        os.unlink(tmp)
> > +        rcode = server.create_job(job_name, jenkins.EMPTY_CONFIG_XML)
> This EMPTY_CONFIG_XML will end up creating a stub job, with no shell
> command
> to invoke the appropriate Fuego test.
>
> >          return rcode
> server.create_job returns "None" on success.  The caller can only handle a
> numeric
> return result (it does a bitwise |, in the case of creating multiple jobs).
>
> >      except Exception as e:
> >          print("Job already exists")
> > @@ -1609,9 +1607,7 @@ def create_batch_job(conf, board, testplan,
> plan_tests):
> >      print("Creating batch job ")
> >      try:
> >          job_name = board+'.'+testplan+'.batch'
> > -        rcode = subprocess.call('java -jar
> /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+'
> create-job ' +
> > -             job_name + '< ' + tmp, shell=True)
> > -        os.unlink(tmp)
> > +        rcode = server.create_job(job_name, jenkins.EMPTY_CONFIG_XML)
> Same issues here as above.
>
> >          return rcode
> >      except Exception as e:
> >          print("Job already exists")
> >
> >
> > Thanks,
> > Srivatsan
> >
>
> FYI - Below is the patch I ended up with, that seems to work.  I tested it
> with
> both existing and non-existing regular jobs and batch jobs.  I also tested
> creating
> jobs for multiple boards on the same command line (some already existing
> and some not).
>
> Thanks for the inspiration for the change.  I think this is a nice
> improvement
> to Fuego.
>   -- Tim
>
>
> commit b019ac2a2e7d16b03d24083c2fce868d736a61ec
> Author: Tim Bird <[email protected]>
> Date:   Wed Nov 18 13:26:39 2020 -0700
>
>     ftc: eliminate call to jenkins-cli.jar for creating jobs
>
>     Use the python jenkins module to use the Jenkins REST API
>     to perform job creation, instead of using jenkins-cli.jar.
>     This eliminates all references to jenkins-cli.jar from ftc.
>     Theoretically, we could remove the installation of jenkins-cli.jar
>     from the Docker container, but let's hold off and make sure it's
>     not needed for anything else.
>
>     The call to create jobs via the jenkins module is much simpler.
>     We don't need to create a temp file to hold the config xml for the
>     job.  We do have to convert a non-numeric return code into a
>     number since callers do bitwise or on it.
>
>     Signed-off-by: Tim Bird <[email protected]>
>
> diff --git a/scripts/ftc b/scripts/ftc
> index 30195fb..d6c9033 100755
> --- a/scripts/ftc
> +++ b/scripts/ftc
> @@ -1438,6 +1438,7 @@ def link_key(item):
>          return "zzz+"+key
>
>
> +# return 0 on success
>  def create_job(conf, board, test):
>      flot_link = '<flotile.FlotPublisher plugin="[email protected]"/>'
>
> @@ -1505,9 +1506,7 @@ def create_job(conf, board, test):
>          for cover, file_ref in link_tuples:
>              fail_links += ' ' + template_link % (str(file_ref),
> str(cover))
>
> -    tmp = "/tmp/fuego_tmp_job"
> -    fd = open(tmp, "w+")
> -    fd.write("""<?xml version='1.0' encoding='UTF-8'?>
> +    job_xml = """<?xml version='1.0' encoding='UTF-8'?>
>  <project>
>      <actions/>
>      <description></description>
> @@ -1553,28 +1552,28 @@ ftc run-test -b $NODE_NAME -t {testdir} -s
> {testspec} \\
>  """.format(board=board, reboot=test.reboot, rebuild=test.rebuild,
>          precleanup=test.precleanup, postcleanup=test.postcleanup,
>          testdir=test.name, testspec=test.spec, timeout=test.timeout,
> -        flot_link=flot_link, success_links=success_links,
> fail_links=fail_links))
> -    fd.close()
> +        flot_link=flot_link, success_links=success_links,
> +        fail_links=fail_links)
>
>      job_name = board + "." + test.spec + "." + test.name
>      print("Creating job " + job_name)
>      try:
> -        rcode = subprocess.call('java -jar
> /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+'
> create-job ' +
> -            job_name + ' < ' + tmp, shell=True)
> -        os.unlink(tmp)
> +        rcode = server.create_job(job_name, job_xml)
> +        if not rcode:
> +            return 0
> +        else:
> +            return 1
>          return rcode
>      except Exception as e:
>          print("Job already exists")
>          print(e)
> -        sys.exit(1)
> +        return 1
>
>
>  def create_batch_job(conf, board, testplan, plan_tests):
> -    tmp = "/tmp/fuego_tmp_batch"
>      job_list = [board+'.'+test.spec+'.'+test.name for test in plan_tests]
>
> -    fd = open(tmp, "w+")
> -    fd.write("""<?xml version='1.0' encoding='UTF-8'?>
> +    job_xml = """<?xml version='1.0' encoding='UTF-8'?>
>  <project>
>    <actions/>
>    <description></description>
> @@ -1602,21 +1601,20 @@ def create_batch_job(conf, board, testplan,
> plan_tests):
>    </publishers>
>    <buildWrappers/>
>  </project>
> -""".format(board=board, alljobs=','.join(job_list)))
> -
> -    fd.close()
> +""".format(board=board, alljobs=','.join(job_list))
>
>      print("Creating batch job ")
>      try:
>          job_name = board+'.'+testplan+'.batch'
> -        rcode = subprocess.call('java -jar
> /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+'
> create-job ' +
> -             job_name + '< ' + tmp, shell=True)
> -        os.unlink(tmp)
> -        return rcode
> +        rcode = server.create_job(job_name, job_xml)
> +        if not rcode:
> +            return 0
> +        else:
> +            return 1
>      except Exception as e:
>          print("Job already exists")
>          print(e)
> -        sys.exit(1)
> +        return 1
>
>
>  # parse the testplan and returns a list of test_class instances.
>

-- 






This
message contains confidential information and is intended only 
for the
individual(s) named. If you are not the intended
recipient, you are 
notified that disclosing, copying, distributing or taking any
action in 
reliance on the contents of this mail and attached file/s is strictly
prohibited. Please notify the
sender immediately and delete this e-mail 
from your system. E-mail transmission
cannot be guaranteed to be secured or 
error-free as information could be
intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain
viruses. The sender therefore does 
not accept liability for any errors or
omissions in the contents of this 
message, which arise as a result of e-mail
transmission.