Re: [Fuego] ftc add-nodes fails
Laszlo Sitzer <[email protected]>
| Newsgroups | dev.linux.lists.fuego |
|---|---|
| Message-ID | <CACBqSLOyod7FWw6MGUWxEALVGmAjmEj8T-+qgJ-WEa02CB-UfQ@mail.gmail.com> |
That is so good to hear you could reproduce! Here is my patch (see attachment). On Fri, Jul 3, 2020 at 2:06 AM Bird, Tim <[email protected]> wrote: > Thanks. I was able to reproduce the bug - actually both bugs. > I tried your fix for the plugin version issue, and it works. I also tried > a fix > for the Unicode exception thrown by str(e) on an exception. I got ftc to > not > throw the Unicode exception, but I'm still researching whether to use my > fix > as the long-term solution. > > In any event, I think I'd like to incorporate your fix, and I plan to put > it in the > master branch. I'd like to give you credit. Can you possibly send me > (and CC: the list) a patch? > (with your own Signed-off-by line)? > > Thanks. > -- Tim > > > > > -----Original Message----- > > From: Fuego <[email protected]> On Behalf Of > Laszlo Sitzer > > Sent: Thursday, July 2, 2020 3:05 AM > > To: [email protected] > > Subject: Re: [Fuego] ftc add-nodes fails > > > > Welcome! > > > > If you find the time could you try if you could reproduce the issue I > was having? > > Running "./install.sh && ./start.sh" (without a proxy) and then "ftc > add-nodes -b bbb" in the container should trigger it. > > I'd really like to see I am not the only one having this problem. > > > > Maybe then the fix could also go onto the next or even master branch. > > > > Regarding what you said about Jenkins ... I hear you! I actually started > looking into updating Jenkins > > and its plugins but I think I need to understand Fuego and its use of > the plugins better. > > But in the long run maintaining a stable Jenkins is a job of its own. > Which makes me think about ways to > > have a clean "seam" between Fuego and Jenkins. Maybe by having a > scheduling/test execution API > > that is implemented in ftc and in another daemon that would then talk to > Jenkins or other CI systems. > > > > But for now ... I think I have to go back to actually using Fuego before > I start thinking about how to refactor it. > > > > Best, > > > > Lazlo > > > > On Wed, Jul 1, 2020 at 7:25 PM Bird, Tim <[email protected] <mailto: > [email protected]> > wrote: > > > > > > Wow - this is super helpful. > > > > The version of Jenkins we are currently using in Fuego has a > number of issues with plugins > > (which have gotten stale and out-of-date). We tried to address > that by using a fixed list > > of plugins and versions, but the dependencies eventually catch up > with you and things > > start to break. (See the Dockerfile - the section where we use > install-plugins.sh) > > > > I had already started working on upgrading to a more recent > version of Jenkins, > > to address this problem, but ran into some compatibility issues > that needed to > > be addressed. It looks like I should focus back on that work. I > think upgrading > > the Jenkins version might help with a lot of these plugin problems. > > > > Thanks. > > -- Tim > > > > > > > -----Original Message----- > > > From: Fuego <[email protected] <mailto: > [email protected]> > On Behalf Of > > Laszlo Sitzer > > > Sent: Wednesday, July 1, 2020 10:46 AM > > > Cc: [email protected] <mailto: > [email protected]> > > > Subject: Re: [Fuego] ftc add-nodes fails > > > > > > By now I found that if I use a different "launcher" I can create > an executor. > > > > > > ``` > > > import jenkins > > > s = jenkins.Jenkins("http://localhost:8090/fuego") > > > s.get_nodes() > > > s.create_node('bbb', launcher='hudson.slaves.JNLPLauncher') > > > ``` > > > > > > This in turn makes me think I should look deeper into > hudson.slaves.CommandLauncher and the command-launcher plugin > > that is > > > automatically installed using the Dockerfile. > > > > > > So I check /var/log/jenkins/jenkins.log and find the following: > > > > > > ``` > > > SEVERE: Failed Loading plugin Script Security Plugin v1.74 > (script-security) > > > java.io.IOException: Script Security Plugin version 1.74 failed > to load. > > > - You must update Jenkins from version 2.164.2 to version > 2.176.4 or later to run this plugin. > > > Jul 01, 2020 4:30:08 PM jenkins.InitReactorRunner$1 onTaskFailed > > > > > > SEVERE: Failed Loading plugin Command Agent Launcher Plugin v1.3 > (command-launcher) > > > java.io.IOException: Command Agent Launcher Plugin version 1.3 > failed to load. > > > - Script Security Plugin version 1.74 failed to load. Fix this > plugin first. > > > > > > ``` > > > > > > I uploaded the full log here: > https://filebin.net/zu5xadhi0mdwfrpl/jenkins.log?t=44eroook > > > > > > I would be super curious to hear from others what "Script > Security Plugin" version they have. As this plugin is not explicitly > > installed in > > > the Dockerfile but either comes with Jenkins or is a dependency > of another plugin. > > > Interestingly v1.74 was just released 20hours ago. See > https://github.com/jenkinsci/script-security- > > > plugin/blob/master/CHANGELOG.md > > > > > > Next I will try if I can somehow try older versions of the > script security plugin. > > > > > > > > > > > > On Wed, Jul 1, 2020 at 6:07 PM Bird, Tim <[email protected] > <mailto:[email protected]> <mailto:[email protected] > > <mailto:[email protected]> > > wrote: > > > > > > > > > Thanks for digging into this deeper. The extra diagnostic > information you provide > > > should be helpful to narrow it down. > > > > > > Unfortunately, I'm busy today with a virtual conference > (ELC), so I can't look at > > > this today, but I'll try to see what I can figure out > tomorrow. > > > -- Tim > > > > > > > > > > -----Original Message----- > > > > From: Fuego <[email protected] > <mailto:[email protected]> > > <mailto:[email protected] <mailto: > [email protected]> > > On Behalf Of > > > Laszlo Sitzer > > > > > > > > So I manually created an executor using the Jenkins > webui. That part worked. > > > > Next I tried to run the python-jenkins code from ftc in > the container and could reproduce the error without using ftc. > > > > > > > > > > > > ``` > > > > import jenkins > > > > s = jenkins.Jenkins("http://localhost:8090/fuego") > > > > s.get_nodes() > > > > s.create_node('bbb') > > > > > > > > ``` > > > > The resulting HTTP POST request that is sent then causes > the exception on the Jenkins side. > > > > From what I understand about the Java part is that the > class loader fails to locate a class mentioned in the JSON of the > > > request. > > > > > > > > I started to look into the HTTP POST request that is > created when I manually create a executor in Jenkins webui and > > compare > > > that to > > > > what python-jenkins creates. > > > > By now I also found two more people having this issue > with python-jenkins > > > > > > > > https://bugs.launchpad.net/python-jenkins/+bug/1845320 > > > > https://bugs.launchpad.net/python-jenkins/+bug/1853147 > > > > > > > > So it is not a fuego issue ... but I have a hard time > understanding who the docker container works for others but fails to > > work > > > for me. I > > > > know various factors like UID/GID play into the docker > image. But I doubt this is relevant to this specific problem : ( > > > > > > > > > > > > On Wed, Jul 1, 2020 at 3:13 PM Laszlo Sitzer < > [email protected] <mailto:laszlo.sitzer@native- > > instruments.de> <mailto:laszlo.sitzer@native- <mailto: > laszlo.sitzer@native-> > > > instruments.de <http://instruments.de> > <mailto: > [email protected] <mailto:laszlo.sitzer@native- > > instruments.de> <mailto:[email protected] <mailto: > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > I managed to get the Jenkins error message that > was masked by the python exception. > > > > I have uploaded it (it is a HTML page from > Jenkins) to https://filebin.net/7ihevnus0vzwuoo6/err.html?t=h64q47ri > > > > > > > > Aside from the Java exception on the HTML page, > Jenkins mentions some that the following plugins could not be > > loaded: > > > > > > > > - Script Security Plugin > > > > - Pipeline: Step API > > > > - JUnit Plugin > > > > - Command Agent Launcher Plugin > > > > - Matrix Project Plugin > > > > > > > > Could this explain why adding an executor/node > fails via ftc? I know this might be more an issue for the jenkins > > mailing > > > list. > > > > > > > > I will try to manually add a Jenkins executor to > see if that part should work (without ftc). > > > > I will also try to use Jenkins 2.176.4 as > suggested by the error message to see if the above mentioned plugins will > > start to > > > work > > > > again. > > > > > > > > > > > > On Tue, Jun 30, 2020 at 10:09 PM Bird, Tim < > [email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]> > > > > <mailto:[email protected] <mailto:[email protected]> <mailto: > [email protected] <mailto:[email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: Fuego < > [email protected] <mailto: > [email protected]> > > <mailto:[email protected] <mailto: > [email protected]> > > > > <mailto:[email protected] <mailto: > [email protected]> <mailto:fuego- > > [email protected] <mailto: > [email protected]> > > > On > > > > Behalf Of Laszlo Sitzer > > > > > > > > > > Hello! > > > > > > > > > > I ran into this problem that "ftc > add-nodes -b bbb" (or any other board) fails with a cryptic python > > unicode/ascii > > > > encoding/decoding > > > > > exception. I think this is just masking > the actual problem with Jenkins. > > > > > > > > > > ``` > > > > > # ftc add-nodes -b bbb > > > > > Traceback (most recent call last): > > > > > File "/usr/local/bin/ftc", line 5625, > in <module> > > > > > main() > > > > > File "/usr/local/bin/ftc", line 5553, > in main > > > > > sys.exit(str(e) + '\n' + > command_help['add-nodes'][1]) > > > > > UnicodeEncodeError: 'ascii' codec can't > encode character u'\xa0' in position 4199: ordinal not in range(128) > > > > > > > > First - thanks very much for the bug > report! > > > > > > > > This looks to be an issue with the > str(e). It would probably be helpful to see what that > > > > exception is from the server. You might > try moving the do_add_nodes() outside the try/except block > > > > (or just changing the generic 'except > Exception as e:' to something specific that is unlikely for this > > > > case like: 'except ArithmeticError as e:' > > > > > > > > Then the exception should bubble up and > maybe print something useful. > > > > > > > > (Independent of what the real error is, > that str(e) should be changed to something > > > > that handles Unicode. This is likely from > some message from the server with a non-ascii > > > > char in it. > > > > > > > > > > ``` > > > > > > > > > > > > > > > I am executing this inside of the > fuego-container that I have built from current master branches of fuego.git > > and > > > > fuego-core.git repos > > > > > and started using ./start.sh. > > > > > > > > > > I did some printf debugging on the > python side and noticed the HTTP request is answered with HTTP 405. > > > > > When looking at > /var/log/jenkins/jenkins.log I can see warnings/errors (ambiguous) and java > exceptions. > > > > > > > > > > ``` > > > > > WARNING: Error while serving > http://localhost:8090/fuego/computer/doCreateItem > > > > > > java.lang.reflect.InvocationTargetException > > > > > ``` > > > > > > > > > > Anyone ran into this before? > > > > > > > > I haven't seen it before, but it does look > like the Unicode error is masking whatever the > > > > real issue is. > > > > -- Tim > > > > > > > > > > > > > > > > > > > >
0001-Dockerfile-Install-script-security-plugin-explicitly.patch
(text/x-patch, 976 B)
From 9e23a75fc81bb4fbd91c5fe3e5dead2179e35fc3 Mon Sep 17 00:00:00 2001 From: Laszlo Sitzer <[email protected]> Date: Wed, 1 Jul 2020 19:15:42 +0200 Subject: [PATCH] Dockerfile: Install script-security plugin explicitly. Signed-off-by: Laszlo Sitzer <[email protected]> --- Dockerfile | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Dockerfile b/Dockerfile index 8688faf..f42b6c7 100644 --- a/Dockerfile +++ b/Dockerfile @@ -187,6 +187,12 @@ RUN service jenkins start && \ sleep 10 && \ service jenkins stop +# Explicitly install script-security v1.68, otherwise +# v1.74 will automatically be installed as a dependency of +# the junit plugin. Make sure to install before junit plugin. +RUN /usr/local/bin/install-plugins.sh \ + script-security:1.68 + # install other plugins from Jenkins update center # NOTE: not sure all of these are needed, but keep list # compatible with 1.2.1 release for now -- 2.27.0