Re: [Fuego] ftc add-nodes fails

Welcome!

If you find the time could you try if you could reproduce the issue I
was having?
Running "./install.sh && ./start.sh" (without a proxy) and then "ftc
add-nodes -b bbb" in the container should trigger it.
I'd really like to see I am not the only one having this problem.

Maybe then the fix could also go onto the next or even master branch.

Regarding what you said about Jenkins ... I hear you! I actually started
looking into updating Jenkins
and its plugins but I think I need to understand Fuego and its use of the
plugins better.
But in the long run maintaining a stable Jenkins is a job of its own. Which
makes me think about ways to
have a clean "seam" between Fuego and Jenkins. Maybe by having a
scheduling/test execution API
that is implemented in ftc and in another daemon that would then talk to
Jenkins or other CI systems.

But for now ... I think I have to go back to actually using Fuego before I
start thinking about how to refactor it.

Best,

Lazlo

On Wed, Jul 1, 2020 at 7:25 PM Bird, Tim <[email protected]> wrote:

> Wow - this is super helpful.
>
> The version of Jenkins we are currently using in Fuego has a number of
> issues with plugins
> (which have gotten stale and out-of-date).  We tried to address that by
> using a fixed list
> of plugins and versions, but the dependencies eventually catch up with you
> and things
> start to break.  (See the Dockerfile - the section where we use
> install-plugins.sh)
>
> I had already started working on upgrading to a more recent version of
> Jenkins,
> to address this problem, but ran into some compatibility issues that
> needed to
> be addressed.  It looks like I should focus back on that work.  I think
> upgrading
> the Jenkins version might help with a lot of these plugin problems.
>
> Thanks.
>  -- Tim
>
>
> > -----Original Message-----
> > From: Fuego <[email protected]> On Behalf Of
> Laszlo Sitzer
> > Sent: Wednesday, July 1, 2020 10:46 AM
> > Cc: [email protected]
> > Subject: Re: [Fuego] ftc add-nodes fails
> >
> > By now I found that if I use a different "launcher" I can create an
> executor.
> >
> > ```
> > import jenkins
> > s = jenkins.Jenkins("http://localhost:8090/fuego")
> > s.get_nodes()
> > s.create_node('bbb', launcher='hudson.slaves.JNLPLauncher')
> > ```
> >
> > This in turn makes me think I should look deeper into
> hudson.slaves.CommandLauncher and the command-launcher plugin that is
> > automatically installed using the Dockerfile.
> >
> > So I check /var/log/jenkins/jenkins.log and find the following:
> >
> > ```
> > SEVERE: Failed Loading plugin Script Security Plugin v1.74
> (script-security)
> > java.io.IOException: Script Security Plugin version 1.74 failed to load.
> >  - You must update Jenkins from version 2.164.2 to version 2.176.4 or
> later to run this plugin.
> > Jul 01, 2020 4:30:08 PM jenkins.InitReactorRunner$1 onTaskFailed
> >
> > SEVERE: Failed Loading plugin Command Agent Launcher Plugin v1.3
> (command-launcher)
> > java.io.IOException: Command Agent Launcher Plugin version 1.3 failed to
> load.
> >  - Script Security Plugin version 1.74 failed to load. Fix this plugin
> first.
> >
> > ```
> >
> > I uploaded the full log here:
> https://filebin.net/zu5xadhi0mdwfrpl/jenkins.log?t=44eroook
> >
> > I would be super curious to hear from others what "Script Security
> Plugin" version they have. As this plugin is not explicitly installed in
> > the Dockerfile but either comes with Jenkins or is a dependency of
> another plugin.
> > Interestingly v1.74 was just released 20hours ago. See
> https://github.com/jenkinsci/script-security-
> > plugin/blob/master/CHANGELOG.md
> >
> > Next I will try if I can somehow try older versions of the script
> security plugin.
> >
> >
> >
> > On Wed, Jul 1, 2020 at 6:07 PM Bird, Tim <[email protected] <mailto:
> [email protected]> > wrote:
> >
> >
> >       Thanks for digging into this deeper.  The extra diagnostic
> information you provide
> >       should be helpful to narrow it down.
> >
> >       Unfortunately, I'm busy today with a virtual conference (ELC), so
> I can't look at
> >       this today, but I'll try to see what I can figure out tomorrow.
> >        -- Tim
> >
> >
> >       > -----Original Message-----
> >       > From: Fuego <[email protected] <mailto:
> [email protected]> > On Behalf Of
> > Laszlo Sitzer
> >       >
> >       > So I manually created an executor using the Jenkins webui. That
> part worked.
> >       > Next I tried to run the python-jenkins code from ftc in the
> container and could reproduce the error without using ftc.
> >       >
> >       >
> >       > ```
> >       > import jenkins
> >       > s = jenkins.Jenkins("http://localhost:8090/fuego")
> >       > s.get_nodes()
> >       > s.create_node('bbb')
> >       >
> >       > ```
> >       > The resulting HTTP POST request that is sent then causes the
> exception on the Jenkins side.
> >       > From what I understand about the Java part is that the class
> loader fails to locate a class mentioned in the JSON of the
> > request.
> >       >
> >       > I started to look into the HTTP POST request that is created
> when I manually create a executor in Jenkins webui and compare
> > that to
> >       > what python-jenkins creates.
> >       > By now I also found two more people having this issue with
> python-jenkins
> >       >
> >       > https://bugs.launchpad.net/python-jenkins/+bug/1845320
> >       > https://bugs.launchpad.net/python-jenkins/+bug/1853147
> >       >
> >       > So it is not a fuego issue ... but I have a hard time
> understanding who the docker container works for others but fails to work
> > for me. I
> >       > know various factors like UID/GID play into the docker image.
> But I doubt this is relevant to this specific problem : (
> >       >
> >       >
> >       > On Wed, Jul 1, 2020 at 3:13 PM Laszlo Sitzer <
> [email protected] <mailto:laszlo.sitzer@native-
> > instruments.de>  <mailto:[email protected] <mailto:
> [email protected]> > >
> >       > wrote:
> >       >
> >       >
> >       >       I managed to get the Jenkins error message that was masked
> by the python exception.
> >       >       I have uploaded it (it is a HTML page from Jenkins) to
> https://filebin.net/7ihevnus0vzwuoo6/err.html?t=h64q47ri
> >       >
> >       >       Aside from the Java exception on the HTML page, Jenkins
> mentions some that the following plugins could not be loaded:
> >       >
> >       >       - Script Security Plugin
> >       >       - Pipeline: Step API
> >       >       - JUnit Plugin
> >       >       - Command Agent Launcher Plugin
> >       >       - Matrix Project Plugin
> >       >
> >       >       Could this explain why adding an executor/node fails via
> ftc? I know this might be more an issue for the jenkins mailing
> > list.
> >       >
> >       >       I will try to manually add a Jenkins executor to see if
> that part should work (without ftc).
> >       >       I will also try to use Jenkins 2.176.4 as suggested by the
> error message to see if the above mentioned plugins will start to
> > work
> >       > again.
> >       >
> >       >
> >       >       On Tue, Jun 30, 2020 at 10:09 PM Bird, Tim <
> [email protected] <mailto:[email protected]>
> > <mailto:[email protected] <mailto:[email protected]> > > wrote:
> >       >
> >       >
> >       >
> >       >
> >       >               > -----Original Message-----
> >       >               > From: Fuego <
> [email protected] <mailto:
> [email protected]>
> > <mailto:[email protected] <mailto:
> [email protected]> > > On
> >       > Behalf Of Laszlo Sitzer
> >       >               >
> >       >               > Hello!
> >       >               >
> >       >               > I ran into this problem that "ftc add-nodes -b
> bbb" (or any other board) fails with a cryptic python unicode/ascii
> >       > encoding/decoding
> >       >               > exception. I think this is just masking the
> actual problem with Jenkins.
> >       >               >
> >       >               > ```
> >       >               > # ftc add-nodes -b bbb
> >       >               > Traceback (most recent call last):
> >       >               >   File "/usr/local/bin/ftc", line 5625, in
> <module>
> >       >               >     main()
> >       >               >   File "/usr/local/bin/ftc", line 5553, in main
> >       >               >     sys.exit(str(e) + '\n' +
> command_help['add-nodes'][1])
> >       >               > UnicodeEncodeError: 'ascii' codec can't encode
> character u'\xa0' in position 4199: ordinal not in range(128)
> >       >
> >       >               First - thanks very much for the bug report!
> >       >
> >       >               This looks to be an issue with the str(e).  It
> would probably be helpful to see what that
> >       >               exception is from the server.  You might try
> moving the do_add_nodes() outside the try/except block
> >       >               (or just changing the generic 'except Exception as
> e:' to something specific that is unlikely for this
> >       >               case like: 'except ArithmeticError as e:'
> >       >
> >       >               Then the exception should bubble up and maybe
> print something useful.
> >       >
> >       >               (Independent of what the real error is, that
> str(e) should be changed to something
> >       >               that handles Unicode.  This is likely from some
> message from the server with a non-ascii
> >       >               char in it.
> >       >               >
> >       >               > ```
> >       >               >
> >       >               >
> >       >               > I am executing this inside of the
> fuego-container that I have built from current master branches of fuego.git
> and
> >       > fuego-core.git repos
> >       >               > and started using ./start.sh.
> >       >               >
> >       >               > I did some printf debugging on the python side
> and noticed the HTTP request is answered with HTTP 405.
> >       >               > When looking at /var/log/jenkins/jenkins.log I
> can see warnings/errors (ambiguous) and java exceptions.
> >       >               >
> >       >               > ```
> >       >               > WARNING: Error while serving
> http://localhost:8090/fuego/computer/doCreateItem
> >       >               > java.lang.reflect.InvocationTargetException
> >       >               > ```
> >       >               >
> >       >               > Anyone ran into this before?
> >       >
> >       >               I haven't seen it before, but it does look like
> the Unicode error is masking whatever the
> >       >               real issue is.
> >       >                -- Tim
> >       >
> >       >
> >
> >
>
>