Re: [Fuego] ftc add-nodes fails

Wow - this is super helpful.

The version of Jenkins we are currently using in Fuego has a number of issues with plugins
(which have gotten stale and out-of-date).  We tried to address that by using a fixed list
of plugins and versions, but the dependencies eventually catch up with you and things
start to break.  (See the Dockerfile - the section where we use install-plugins.sh)

I had already started working on upgrading to a more recent version of Jenkins,
to address this problem, but ran into some compatibility issues that needed to
be addressed.  It looks like I should focus back on that work.  I think upgrading
the Jenkins version might help with a lot of these plugin problems.

Thanks.
 -- Tim


> -----Original Message-----
> From: Fuego <[email protected]> On Behalf Of Laszlo Sitzer
> Sent: Wednesday, July 1, 2020 10:46 AM
> Cc: [email protected]
> Subject: Re: [Fuego] ftc add-nodes fails
> 
> By now I found that if I use a different "launcher" I can create an executor.
> 
> ```
> import jenkins
> s = jenkins.Jenkins("http://localhost:8090/fuego")
> s.get_nodes()
> s.create_node('bbb', launcher='hudson.slaves.JNLPLauncher')
> ```
> 
> This in turn makes me think I should look deeper into hudson.slaves.CommandLauncher and the command-launcher plugin that is
> automatically installed using the Dockerfile.
> 
> So I check /var/log/jenkins/jenkins.log and find the following:
> 
> ```
> SEVERE: Failed Loading plugin Script Security Plugin v1.74 (script-security)
> java.io.IOException: Script Security Plugin version 1.74 failed to load.
>  - You must update Jenkins from version 2.164.2 to version 2.176.4 or later to run this plugin.
> Jul 01, 2020 4:30:08 PM jenkins.InitReactorRunner$1 onTaskFailed
> 
> SEVERE: Failed Loading plugin Command Agent Launcher Plugin v1.3 (command-launcher)
> java.io.IOException: Command Agent Launcher Plugin version 1.3 failed to load.
>  - Script Security Plugin version 1.74 failed to load. Fix this plugin first.
> 
> ```
> 
> I uploaded the full log here: https://filebin.net/zu5xadhi0mdwfrpl/jenkins.log?t=44eroook
> 
> I would be super curious to hear from others what "Script Security Plugin" version they have. As this plugin is not explicitly installed in
> the Dockerfile but either comes with Jenkins or is a dependency of another plugin.
> Interestingly v1.74 was just released 20hours ago. See https://github.com/jenkinsci/script-security-
> plugin/blob/master/CHANGELOG.md
> 
> Next I will try if I can somehow try older versions of the script security plugin.
> 
> 
> 
> On Wed, Jul 1, 2020 at 6:07 PM Bird, Tim <[email protected] <mailto:[email protected]> > wrote:
> 
> 
> 	Thanks for digging into this deeper.  The extra diagnostic information you provide
> 	should be helpful to narrow it down.
> 
> 	Unfortunately, I'm busy today with a virtual conference (ELC), so I can't look at
> 	this today, but I'll try to see what I can figure out tomorrow.
> 	 -- Tim
> 
> 
> 	> -----Original Message-----
> 	> From: Fuego <[email protected] <mailto:[email protected]> > On Behalf Of
> Laszlo Sitzer
> 	>
> 	> So I manually created an executor using the Jenkins webui. That part worked.
> 	> Next I tried to run the python-jenkins code from ftc in the container and could reproduce the error without using ftc.
> 	>
> 	>
> 	> ```
> 	> import jenkins
> 	> s = jenkins.Jenkins("http://localhost:8090/fuego")
> 	> s.get_nodes()
> 	> s.create_node('bbb')
> 	>
> 	> ```
> 	> The resulting HTTP POST request that is sent then causes the exception on the Jenkins side.
> 	> From what I understand about the Java part is that the class loader fails to locate a class mentioned in the JSON of the
> request.
> 	>
> 	> I started to look into the HTTP POST request that is created when I manually create a executor in Jenkins webui and compare
> that to
> 	> what python-jenkins creates.
> 	> By now I also found two more people having this issue with python-jenkins
> 	>
> 	> https://bugs.launchpad.net/python-jenkins/+bug/1845320
> 	> https://bugs.launchpad.net/python-jenkins/+bug/1853147
> 	>
> 	> So it is not a fuego issue ... but I have a hard time understanding who the docker container works for others but fails to work
> for me. I
> 	> know various factors like UID/GID play into the docker image. But I doubt this is relevant to this specific problem : (
> 	>
> 	>
> 	> On Wed, Jul 1, 2020 at 3:13 PM Laszlo Sitzer <[email protected] <mailto:laszlo.sitzer@native-
> instruments.de>  <mailto:[email protected] <mailto:[email protected]> > >
> 	> wrote:
> 	>
> 	>
> 	>       I managed to get the Jenkins error message that was masked by the python exception.
> 	>       I have uploaded it (it is a HTML page from Jenkins) to https://filebin.net/7ihevnus0vzwuoo6/err.html?t=h64q47ri
> 	>
> 	>       Aside from the Java exception on the HTML page, Jenkins mentions some that the following plugins could not be loaded:
> 	>
> 	>       - Script Security Plugin
> 	>       - Pipeline: Step API
> 	>       - JUnit Plugin
> 	>       - Command Agent Launcher Plugin
> 	>       - Matrix Project Plugin
> 	>
> 	>       Could this explain why adding an executor/node fails via ftc? I know this might be more an issue for the jenkins mailing
> list.
> 	>
> 	>       I will try to manually add a Jenkins executor to see if that part should work (without ftc).
> 	>       I will also try to use Jenkins 2.176.4 as suggested by the error message to see if the above mentioned plugins will start to
> work
> 	> again.
> 	>
> 	>
> 	>       On Tue, Jun 30, 2020 at 10:09 PM Bird, Tim <[email protected] <mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]> > > wrote:
> 	>
> 	>
> 	>
> 	>
> 	>               > -----Original Message-----
> 	>               > From: Fuego <[email protected] <mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]> > > On
> 	> Behalf Of Laszlo Sitzer
> 	>               >
> 	>               > Hello!
> 	>               >
> 	>               > I ran into this problem that "ftc add-nodes -b bbb" (or any other board) fails with a cryptic python unicode/ascii
> 	> encoding/decoding
> 	>               > exception. I think this is just masking the actual problem with Jenkins.
> 	>               >
> 	>               > ```
> 	>               > # ftc add-nodes -b bbb
> 	>               > Traceback (most recent call last):
> 	>               >   File "/usr/local/bin/ftc", line 5625, in <module>
> 	>               >     main()
> 	>               >   File "/usr/local/bin/ftc", line 5553, in main
> 	>               >     sys.exit(str(e) + '\n' + command_help['add-nodes'][1])
> 	>               > UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 4199: ordinal not in range(128)
> 	>
> 	>               First - thanks very much for the bug report!
> 	>
> 	>               This looks to be an issue with the str(e).  It would probably be helpful to see what that
> 	>               exception is from the server.  You might try moving the do_add_nodes() outside the try/except block
> 	>               (or just changing the generic 'except Exception as e:' to something specific that is unlikely for this
> 	>               case like: 'except ArithmeticError as e:'
> 	>
> 	>               Then the exception should bubble up and maybe print something useful.
> 	>
> 	>               (Independent of what the real error is, that str(e) should be changed to something
> 	>               that handles Unicode.  This is likely from some message from the server with a non-ascii
> 	>               char in it.
> 	>               >
> 	>               > ```
> 	>               >
> 	>               >
> 	>               > I am executing this inside of the fuego-container that I have built from current master branches of fuego.git and
> 	> fuego-core.git repos
> 	>               > and started using ./start.sh.
> 	>               >
> 	>               > I did some printf debugging on the python side and noticed the HTTP request is answered with HTTP 405.
> 	>               > When looking at /var/log/jenkins/jenkins.log I can see warnings/errors (ambiguous) and java exceptions.
> 	>               >
> 	>               > ```
> 	>               > WARNING: Error while serving http://localhost:8090/fuego/computer/doCreateItem
> 	>               > java.lang.reflect.InvocationTargetException
> 	>               > ```
> 	>               >
> 	>               > Anyone ran into this before?
> 	>
> 	>               I haven't seen it before, but it does look like the Unicode error is masking whatever the
> 	>               real issue is.
> 	>                -- Tim
> 	>
> 	>
> 
>