Re: [Fuego] Fuego v1.4 not able to add nodes

Hi Tim,




To answer your below query, yes I have multiple Jenkins instances but they all are under different folders /home/dhinakar/proj1/fuego, /home/dhinakar/proj2/fuego etc.


Hope that won't have any impact. In any case I also tried stopping them and checked but the issue still remains.




'Just out of curiosity - do you have more than one instance of Jenkins running?
Is it possible that ftc is trying to contact the wrong jenkins instance?'



I looked at the options suggested by you, thank you for taking the time to write those suggestions.


Instead of doing a post mortem, I thought I will do a fresh 'clone, install and start' and see if everything is fine.


I tried 'next' branch as well as 'master' branch.


Please see my observation below.




Fuego v1.4 cloning & installation (master branch)


*****************************************************


I am following http://fuegotest.org/fuego-1.5/Fuego_Quickstart_Guide link.


I used git clone https://bitbucket.org/fuegotest/fuego.git to clone 'master' branch which I assume will get me 'fuego v1.4.0'.




I did a 'cat VERSION' after cloning 'fuego' directory, I only see the correct version as shown below:


# This project adheres to Semantic Versioning. See http://semver.org/
# The codename for this release is "Ember"
VERSION=1.4.0




Then I did './install.sh'.


But 'fuego-core' is not getting cloned during the './install.sh' step. I need to do it manually as per the message below.


'You need to clone fuego-core at /home/dhinakar/fuego/fuego-host-scripts/../../fuego-core' at the end of cloning.


So I went into 'fuego' director and did --> git clone https://bitbucket.org/fuegotest/fuego-core.git




But 'jenkins' folder itself is not getting created under '/home/dhinakar/fuego'.




Hence I moved onto clone 'fuego v1.5.0'.




Fuego v1.5 cloning & installation (next branch)


***************************************************


Is the below URL the right one to get fuego v1.5?


git clone -b next https://bitbucket.org/fuegotest/fuego.git




because if I do 'cat VERSION' after cloning 'fuego' directory, I only see the below:




# This project adheres to Semantic Versioning. See http://semver.org/
# The codename for this release is "Ember"
VERSION=1.4.0




And 'fuego-core' gets cloned automatically if I do './install.sh' after cloning 'fuego' and getting into 'fuego' directory.


So probably it must be fuego v1.5. Probably the version number needs to be changed to 1.5.0.


And default 'install.sh' will not use '--priv' option right?






Also 'jenkins' folder gets created under '/home/dhinakar/fuego' if I clone with 'next' branch.


Jenkins port here is '8090'.




I logged onto dashboard http://107.108.73.106:8090/fuego/ , Jenkins version was 2.164.2


As mentioned earlier, 'master' node is not seen initially but after I add one node e.g. 'fuego-test', 'master' node is also visible in the dashboard.




Able to list boards, add nodes, remove nodes as shown below.




root@linux-2030047407:/ # ftc list-boards
Boards in this system:
bbb
beaglebone-serial
docker
ftc-test
fuego-test
lager
local
localhost
minnow
qemu-arm
raspberrypi3
template-dev




root@linux-2030047407:/ # ftc add-nodes -b fuego-test

root@linux-2030047407:/ # ftc list-nodes
Jenkins nodes in this system:
fuego-test

root@linux-2030047407:/ # ftc rm-nodes fuego-test



root@linux-2030047407:/ # ftc list-nodes
Jenkins nodes in this system:



One more doubt.


Does the below work during install step?




I removed the default container (fuego-container) and image (fuego) and tried the below:


./install.sh --priv fuego-base-image-12thaug2019




CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ce71c72cf9bf fuego-base-image-12thaug2019 "/bin/sh -c 'service…" 6 minutes ago Exited (0) 38 seconds ago fuego-base-image-12thaug2019-container




So 'fuego-base-image-12thaug2019' and the corresponding 'fuego-base-image-12thaug2019-container' also created successfully.


ftc list-boards, add-nodes, list-nodes etc. also works fine.




One question.


What if I want to give my own name to the container? Rename 'fuego-base-image-12thaug2019-container' is the only option?




So to conclude, please check why 'jenkins' folder not created while cloning 'master' branch (v1.4.0).




All looks fine the v1.5.0 so far. Only thing is please change the version to 1.5.0 if 'next' branch is cloned.


I will report any other issues I face in future in fuego v1.5.0


Thanks a lot.




Regards,


Dhinakar








--------- Original Message ---------


Sender : [email protected] <[email protected]>


Date : 2019-08-10 04:55 (GMT+5:30)


Title : RE: RE: Fuego v1.4 not able to add nodes


> -----Original Message-----
> From: Dhinakar Kalyanasundaram 

> 
> Hi Tim,
> 
> 
> 
> Thanks for your prompt response.
> 
> Please find my response below.
> 
> 
> 
> Here are some things you can try:
> 1) in ftc, remove the try:/except block around the call to do_add_nodes,
> try the 'ftc add-nodes' command, and see what the traceback looks like.
> I'm guessing it's going to look a lot like the traceback for the list-nodes call.
> The first call to the server inside the do_add_nodes() function is to
> server.get_nodes(), to check for a duplicate. I'm guessing this is throwing
> the same exception.
> 
> 
> 
> Yes you are right, after removing the try:/except block, I get the same
> exception as shown below:
> 
> root@linux-2030047407:/fuego-rw <mailto:root@linux-2030047407:/fuego-
> rw> # ftc list-nodes
> Traceback (most recent call last):
> File "/usr/local/bin/ftc", line 5275, in <module>
> main()
> File "/usr/local/bin/ftc", line 5231, in main
> do_list_nodes(conf)
> File "/usr/local/bin/ftc", line 2062, in do_list_nodes
> node_list = [node['name'] for node in server.get_nodes()]
> File "/usr/local/lib/python2.7/dist-packages/jenkins/__init__.py", line 1447,
> in get_nodes
> requests.Request('GET', self._build_url(NODE_LIST, locals()))))
> File "/usr/local/lib/python2.7/dist-packages/jenkins/__init__.py", line 564,
> in jenkins_open
> return self.jenkins_request(req, add_crumb, resolve_auth).text
> File "/usr/local/lib/python2.7/dist-packages/jenkins/__init__.py", line 596,
> in jenkins_request
> raise NotFoundException('Requested item could not be found')
> jenkins.NotFoundException: Requested item could not be found
> 
> 
> 2) try manually adding a node, just to see if that clears the error.
> ftc list-nodes should show your manually created node.
> 
> 
> Actually I tried this yesterday itself, I get the same exception as mentioned
> above even after adding a 'Test' node (as shown in the picture below).
> 
> 
> 
> root@linux-2030047407:/fuego-rw <mailto:root@linux-2030047407:/fuego-
> rw> # ftc list-nodes -> exception is same as above.

Just out of curiosity - do you have more than one instance of Jenkins running?
Is it possible that ftc is trying to contact the wrong jenkins instance?

> 
> If it is installation of fuego or Jenkins issue then it is fine I will redo it.

I looked to see if there was maybe a file you could replace in /var/lib/jenkins
(like under the 'nodes' directory, or something), that might have
gotten corrupted.

I couldn't see anything obvious. The master node doesn't even show
up as a directory under /var/lib/jenkins/nodes, so it must be handled
differently from the other nodes.

> 
> Only thing is I have updated jenkins and installed a lot of plugins related to
> git, gerrit, pipeline etc.
> 
> Is there is a way I can retain those plugin changes while I reinstall/overwrite
> the existing installation? or I need to do a 'clean install' only.
> 
> If it is Jenkins installation issue, is it possible to reinstall Jenkins alone?


Well, this might be tedious, but you might try a new installation of Jenkins,
and then grabbing the contents of /var/lib/jenkins (the xml files at the root directory)
in the new container, tar-ring them up and putting them in the old container,
and doing a diff, to see what's different about the new xml files.

You might be able to figure out the problem if you see something different 
in the files for the new jenkins instance (the broken one).

In particular, I'd look at /var/lib/jenkins/config.xml, and examine the
content of the <nodeProperties> entity.

If you find a difference in config.xml, or some other top-level xml file,
you may be able to repair just that file. My experience is that when
Jenkins is restarted, if you haven't touched the plugins directory, none
of your plugins will be affected.

By the way - you have restarted jenkins in the container haven't you?

If not, try this inside the container:
# service jenkins stop
# service jenkins start

That's about it for ideas from me, I'm afraid.
-- Tim


> 
> 
> Regards,
> 
> Dhinakar
> 
> 
> 
> 
> 
> --------- Original Message ---------
> 
> Sender : [email protected] <[email protected]>
> 
> Date : 2019-08-09 07:02 (GMT+5:30)
> 
> Title : RE: Fuego v1.4 not able to add nodes
> 
> 
> 
> > -----Original Message-----
> > From: Dhinakar Kalyanasundaram
> >
> > Dear Fuego Users,
> >
> > I am setting up test environment with fuego v1.4
> >
> > dhinakar@linux-2030047407:~/fuego$ <mailto:dhinakar@linux-
> > 2030047407:~/fuego$> cat VERSION
> > # This project adheres to Semantic Versioning. See http://semver.org/
> > # The codename for this release is "Ember"
> > VERSION=1.4.0
> >
> >
> >
> > I am able to list the boards but not able to add nodes (please check error
> > below).
> >
> >
> >
> > root@linux-2030047407:/fuego-rw <mailto:root@linux-2030047407:/fuego-
> > rw> # ftc list-boards
> > Boards in this system:
> > ARM64-SSH-Test
> > ARM64-SSH-Test1
> > ARM64-Serial-Test
> > X86-SSH-Test
> > X86-Serial-Test
> > bbb
> > beaglebone-serial
> > docker
> > ftc-test
> > fuego-test
> > lager
> > local
> > localhost
> > minnow
> > qemu-arm
> > raspberrypi3
> > template-dev
> >
> >
> > root@linux-2030047407:/fuego-rw <mailto:root@linux-2030047407:/fuego-
> > rw> # ftc add-nodes -b ARM64-SSH-Test
> > Requested item could not be found
> > Usage: ftc add-nodes [-f] -b <board1>[,<board2>...]
> > By convention the name of the node and board is the same. Also,
> > the corresponding board file (<board_name>.board) must exist.
> > Use list-boards to see available boards.
> >
> > Use -f for "force" mode. This tries to remove the node before adding it
> >
> > One strange thing I noticed is that, I am not able to see 'master' node in
> > fuego dashboard.
> >
> > Please check the image below:
> >
> >
> >
> > But it is listed when I navigate to
> > http://101.102.103.104:8090/fuego/computer/
> >
> >
> >
> > The output of 'ftc list-nodes' command is given below.
> >
> > Should it list 'master node' atleast or it is the correct output since no nodes
> > are added yet via 'ftc add-nodes' command?
> >
> > root@linux-2030047407:/fuego-rw <mailto:root@linux-2030047407:/fuego-
> > rw> # ftc list-nodes
> > Traceback (most recent call last):
> > File "/usr/local/bin/ftc", line 5275, in <module>
> > main()
> > File "/usr/local/bin/ftc", line 5231, in main
> > do_list_nodes(conf)
> > File "/usr/local/bin/ftc", line 2062, in do_list_nodes
> > node_list = [node['name'] for node in server.get_nodes()]
> > File "/usr/local/lib/python2.7/dist-packages/jenkins/__init__.py", line
> 1447,
> > in get_nodes
> > requests.Request('GET', self._build_url(NODE_LIST, locals()))))
> > File "/usr/local/lib/python2.7/dist-packages/jenkins/__init__.py", line
> 564,
> > in jenkins_open
> > return self.jenkins_request(req, add_crumb, resolve_auth).text
> > File "/usr/local/lib/python2.7/dist-packages/jenkins/__init__.py", line
> 596,
> > in jenkins_request
> > raise NotFoundException('Requested item could not be found')
> > jenkins.NotFoundException: Requested item could not be found
> >
> >
> > Is it a problem with my installation of fuego or any missing plugin?
> 
> I think this might be a Fuego or Jenkins installation problem.
> There is code in 'ftc' to skip the 'master' node when it is returned
> from a call to server.get_nodes() internally.
> 
> It looks like this call is having problems.
> 
> The photo you supplied looks normal. There should be no problem with the
> python jenkins module retrieving the list of nodes from your system.
> 
> Here are some things you can try:
> 1) in ftc, remove the try:/except block around the call to do_add_nodes,
> try the 'ftc add-nodes' command, and see what the traceback looks like.
> I'm guessing it's going to look a lot like the traceback for the list-nodes call.
> The first call to the server inside the do_add_nodes() function is to
> server.get_nodes(), to check for a duplicate. I'm guessing this is throwing
> the same exception.
> 2) try manually adding a node, just to see if that clears the error.
> ftc list-nodes should show your manually created node.
> 
> I don't think this will resolve the issue, but maybe some information
> will surface that will help us track down the problem.
> -- Tim
> 
> >
> > If anyone experienced similar issue please help me out.
> >
> >
> >
> > Regards,
> >
> > Dhinakar
> >
> >
> >
> >
> >
> <http://ext.samsung.net/mail/ext/v1/external/status/update?userid=dhina
> > kar.k&do=bWFpbElEPTIwMTkwODA4MTU1NTQ0ZXBjbXM1cDdkY2E3YThlM
> >
> WIzZmRkMTc2NDNiYTVhNjkwMDFkMmFmMyZyZWNpcGllbnRBZGRyZXNzP
> > VRpbS5CaXJkQHNvbnkuY29t>
> 
> 
> 
> 
> 
> 
> 
> <http://ext.samsung.net/mail/ext/v1/external/status/update?userid=dhina
> kar.k&do=bWFpbElEPTIwMTkwODA5MTIxMjI5ZXBjbXM1cDIzNWFmMzE0M2
> FiNWMxYmYyYjQyNzEzY2NhNTYwNGM2MiZyZWNpcGllbnRBZGRyZXNzPVRp
> bS5CaXJkQHNvbnkuY29t>