Re: [Fuego] Installed plugins not listed under ‘Plu gin Manager’ -> ‘Installed’ page

Tim Bird,

One issue I noticed when I took a running container and save its current state as an image (docker commit command).
If I create a container from that image I see the Jenkins is still showing the old version (2.164.2 rather than 2.263.1) and the plugin manager page loads with errors. 
I actually wanted to take backup of the current container with upgraded Jenkins & plugins, any suggestions?

Regards,
Dhinakar

> -----Original Message-----
> From: Bird, Tim <[email protected]>
> Sent: Saturday, February 19, 2022 6:09 AM
> To: Dhinakar Kalyanasundaram/FDS SW /SSIR/Staff Engineer/삼성전자
> <[email protected]>; [email protected]
> Subject: RE: Installed plugins not listed under ‘Plugin Manager’ -> ‘Installed’
> page
> 
> Dhinakar,
> 
> I'm sorry that you are having problems with the Jenkins plugins.
> This has been a perennial problem for Fuego, trying to keep up with the
> constant churn as the versions of Jenkins and its plugins keeps changing.
> 
> Here is some information and some tips, in the hopes that it will be useful for
> you.
> 
> When Fuego installs, it uses a hardcoded Jenkins version.
> In previous versions of Fuego, we hardcoded frozen versions of the Jenkins
> plugins, so that when the docker container was created, it didn't pick up
> arbitrary plugin versions, that might not be compatible with the version of
> Jenkins used in the Fuego docker image.
> 
> However, the last time we did a Jenkins version upgrade was in Dec 1, 2020.
> At that time, for reasons unknown, we changed the Dockerfile so that the
> plugin versions could float (that is, the latest version of each Jenkins plugin
> would be used).  This code has since been refactored to make native installs
> easier, but the use of non-frozen versions of plugins still remains.
> 
> In any event, I have an old docker container from December of 2020, that has
> the original versions of plugins that were compatible with Jenkins 2.249.3
> (which was the version of Jenkins used in the 1.5.0 release of Fuego).
> 
> Here are the version of plugins  in that container:
> ant:1.11
> antisamy-markup-formatter:2.1
> bootstrap4-api:4.5.3-1
> bouncycastle-api:2.18
> checks-api:1.1.1
> command-launcher:1.5
> description-setter:1.10
> display-url-api:2.3.4
> echarts-api:4.9.0-2
> external-monitor-job:1.7
> flot:1.0-SNAPSHOT (private-07/21/2017 10:45-dsl)
> font-awesome-api:5.15.1-1
> greenballs:1.15
> icon-shim:2.0.3
> jackson2-api:2.12.0
> javadoc:1.6
> jdk-tool:1.4
> jquery3-api:3.5.1-2
> junit:1.45
> ldap:1.11
> mailer:1.32.1
> matrix-auth:2.6.4
> matrix-project:1.18
> pam-auth:1.6
> pegdown-formatter:1.3
> plugin-util-api:1.5.0
> popper-api:1.16.0-7
> scm-api:2.6.4
> script-security:1.75
> snakeyaml-api:1.27.0
> structs:1.20
> trilead-api:1.0.4
> windows-slaves:1.7
> workflow-api:2.40
> workflow-step-api:2.23
> 
> the Fuego install system uses an external script to install plugins (rather than
> doing so inside Jenkins itself).  This script is found inside the fuego docker
> container at: /usr/local/bin/install-plugins.sh
> 
> To use it, you specify a list of plugins to install, on the command line.  You can
> include a version number for a package by including the version number
> following the package name and a colon (e.g. structs:1.20)
> 
> It sounds like some of your plugin operations have upgraded your existing
> plugins, so that now there are dependency problems with the collection of
> plugins on your system.
> 
> There are 2 ways to proceed:  1) Stay with version 2.249.3 of Jenkins, and try
> to revert your plugins to ones that will work with that version of Jenkins, or 2)
> migrate to a newer version of Jenkins.
> 
> Option 2 is, IMHO, fraught with peril. But, depending on the extra modules
> you want to install and your sensitivity to security issues, it may be the best
> route forward.  If the jira plugins you want to install rely on newer modules
> which themselves rely on a newer Jenkins version, then in order to install
> those you may have no choice but to update the version of Jenkins.
> 
> Option 1 also entails some danger (as I have not tried the instructions I am
> about to provide), but it *should* work.  Try uninstalling the plugins that
> have versions that don't match the ones in the above list (that are known to
> be compatible with Jenkins 2.249.3).  Then use install-plugins.sh to re-install
> specific versions of the modules that match the list above.
> 
> In order to run install-plugins.sh, you need to define two environment
> variables, to indicate the Jenkins upload center and the Jenkins plugin
> directory.  These are names JENKINS_UC and REF, respectively.
> 
> $ export JENKINS_UC=https://updates.jenkins.io
> $ export REF=/var/lib/jenkins/plugins
> $ ./install-plugins.sh <plugin1> <plugin2> ...
> 
> To test the script, and get a list of currently installed plugins, you can specify
> to install a non-existent plugin.  Here's what I did, and the output I got back
> from it:
> Note that I get some curl certificate errors, but I think those can be ignored
> for the purposes of this operation.
> 
> --------
> 
> root@timdesk:/usr/local/bin# export
> JENKINS_UC=https://updates.jenkins.io
> root@timdesk:/usr/local/bin# export REF=/var/lib/jenkins/plugins
> root@timdesk:/usr/local/bin# ./install-plugins.sh foo Creating initial locks...
> Analyzing war...
> Registering preinstalled plugins...
> Downloading plugins...
> Downloading plugin: foo from
> https://updates.jenkins.io/download/plugins/foo/latest/foo.hpi
> curl: (60) SSL certificate problem: certificate has expired More details here:
> https://protect2.fireeye.com/v1/url?k=9e437f5d-c1d84794-9e42f412-
> 0cc47a336fae-52a4eafdb6f541c5&q=1&e=9a036cb4-fc47-47cb-a812-
> 14354795baa0&u=https%3A%2F%2Fcurl.haxx.se%2Fdocs%2Fsslcerts.html
> 
> curl performs SSL certificate verification by default, using a "bundle"
>  of Certificate Authority (CA) public keys (CA certs). If the default  bundle file
> isn't adequate, you can specify an alternate file  using the --cacert option.
> If this HTTPS server uses a certificate signed by a CA represented in  the
> bundle, the certificate verification probably failed due to a  problem with the
> certificate (it might be expired, or the name might  not match the domain
> name in the URL).
> If you'd like to turn off curl's verification of the certificate, use  the -k (or --
> insecure) option.
> 23:57:40 Failure (60) Retrying in 1 seconds...
> curl: (60) SSL certificate problem: certificate has expired More details here:
> https://protect2.fireeye.com/v1/url?k=c5b9e4dd-9a22dc14-c5b86f92-
> 0cc47a336fae-d00fe4ef3dd99b9f&q=1&e=9a036cb4-fc47-47cb-a812-
> 14354795baa0&u=https%3A%2F%2Fcurl.haxx.se%2Fdocs%2Fsslcerts.html
> 
> curl performs SSL certificate verification by default, using a "bundle"
>  of Certificate Authority (CA) public keys (CA certs). If the default  bundle file
> isn't adequate, you can specify an alternate file  using the --cacert option.
> If this HTTPS server uses a certificate signed by a CA represented in  the
> bundle, the certificate verification probably failed due to a  problem with the
> certificate (it might be expired, or the name might  not match the domain
> name in the URL).
> If you'd like to turn off curl's verification of the certificate, use  the -k (or --
> insecure) option.
> 23:57:42 Failure (60) Retrying in 1 seconds...
> curl: (60) SSL certificate problem: certificate has expired More details here:
> https://protect2.fireeye.com/v1/url?k=bbfbff80-e460c749-bbfa74cf-
> 0cc47a336fae-4376695a932d0415&q=1&e=9a036cb4-fc47-47cb-a812-
> 14354795baa0&u=https%3A%2F%2Fcurl.haxx.se%2Fdocs%2Fsslcerts.html
> 
> curl performs SSL certificate verification by default, using a "bundle"
>  of Certificate Authority (CA) public keys (CA certs). If the default  bundle file
> isn't adequate, you can specify an alternate file  using the --cacert option.
> If this HTTPS server uses a certificate signed by a CA represented in  the
> bundle, the certificate verification probably failed due to a  problem with the
> certificate (it might be expired, or the name might  not match the domain
> name in the URL).
> If you'd like to turn off curl's verification of the certificate, use  the -k (or --
> insecure) option.
> 23:57:43 Failure (60) Retrying in 1 seconds...
> 23:57:44 Failed in the last attempt (curl -sSfL --connect-timeout 20 --retry 3 --
> retry-delay 0 --retry-max-time 60
> https://updates.jenkins.io/download/plugins/foo/latest/foo.hpi -o
> /var/lib/jenkins/plugins/foo.jpi)
> ./install-plugins.sh: line 63: $3: unbound variable
> 
> WAR bundled plugins:
> 
> 
> Installed plugins:
> ant:1.11
> antisamy-markup-formatter:2.1
> bootstrap4-api:4.5.3-1
> bouncycastle-api:2.18
> checks-api:1.1.1
> command-launcher:1.5
> description-setter:1.10
> display-url-api:2.3.4
> echarts-api:4.9.0-2
> external-monitor-job:1.7
> flot:1.0-SNAPSHOT (private-07/21/2017 10:45-dsl)
> font-awesome-api:5.15.1-1
> greenballs:1.15
> icon-shim:2.0.3
> jackson2-api:2.12.0
> javadoc:1.6
> jdk-tool:1.4
> jquery3-api:3.5.1-2
> junit:1.45
> ldap:1.11
> mailer:1.32.1
> matrix-auth:2.6.4
> matrix-project:1.18
> pam-auth:1.6
> pegdown-formatter:1.3
> plugin-util-api:1.5.0
> popper-api:1.16.0-7
> scm-api:2.6.4
> script-security:1.75
> snakeyaml-api:1.27.0
> structs:1.20
> trilead-api:1.0.4
> windows-slaves:1.7
> workflow-api:2.40
> workflow-step-api:2.23
> Cleaning up locks
> 
> -----------------
> 
> Please try either option 1 or 2 and let me know what happens.
> 
> By the way, you can shutdown Jenkins inside the docker container, and
> restart it, by doing this, inside the container, as root.
> $ service jenkins stop
> $ service jenkins start
> 
> Doing this might clear up the issue with plugins being present in the
> installation directory but not registered with the system.  You probably want
> to do this after installing plugins from the docker container command line.
> 
> You might also try to reset Jenkins from inside the web interface by going to
> "manage Jenkins", "Reload Configuration from Disk".
> 
> I hope this helps.  Let me know what you try, and what the eventual
> outcome is.
> 
> Best of luck!
>  -- Tim
> 
> P.S. If you decide to upgrade the version of Jenkins, make sure you check the
> Jenkins config.xml for items that are needed by Fuego for proper Jenkins
> operations.
> Sometimes, newer Jenkins versions will change the items in the config.xml
> file in a way that is not backwards compatible.  The config.xml that Fuego
> uses when it installs Jenkins into the docker container originally is in:
> fuego/frontend-install/config.xml
> 
> 
> > -----Original Message-----
> > From: Dhinakar Kalyanasundaram/FDS SW /SSIR/Staff Engineer/삼성전자
> > <[email protected]>
> >
> > Dear Fuego users,
> >
> >
> >
> > I am using Fuego VERSION=1.5.0 and Jenkins ver. 2.164.2
> >
> >
> >
> > I was trying to install a few Jira plugins to interface from Jenkins.
> >
> > After a while I noticed the below warning message (in many of my
> > plugins) and many of my already installed & setup plugins like ‘Ant’,
> > ‘Structs’, ‘Sidebar Link’, ‘Gerrit Trigger’ etc. were missing even though their
> folders and *.jpi files were still available under ‘/var/lib/jenkins/plugins’
> directory inside docker container.
> >
> >
> >
> > Warning: This plugin requires dependent plugins that require Jenkins
> > 2.249.1 or newer. Jenkins will refuse to load the dependent plugins
> requiring a newer version of Jenkins, and in turn loading this plugin will fail.
> >
> >
> >
> > Now, I see many of my above plugin’s next version listed under ‘Plugin
> Manager’ -> ‘Available’ with the above warning message.
> >
> > For e.g. Structs 1.20 version is already installed but not listed
> > under ‘Plugin Manager’ -> ‘Installed’ page but instead ‘Plugin Manager’ ->
> ‘Available’ is showing the next version of ‘Structs’ with the warning shown
> above (in red italics).
> >
> >
> >
> > I tried removing the newly installed plugins and restart Jenkins & docker
> container multiple times but still the issue was not solved.
> >
> > Any suggestions to fix this?
> >
> > Since I have lot of jobs configured with plugins like ‘Sidebar Link’, ‘Post
> Build Task’ etc. I cannot reinstall or upgrade fuego.
> >
> > Thanks in advance.
> >
> >
> >
> > Regards,
> >
> > Dhinakar
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> <http://ext.samsung.net/mail/ext/v1/external/status/update?userid=dhin
> > akar.k&do=bWFpbElEPTIwMjIwMjE3MTA1NjI3ZXBjYXM1cDFlMjR
> >
> hZDlmZGRjNTA1ZDAyNmRiZmU4NjQ4ZmU4YTVhYSZyZWNpcGllbnRBZGRyZX
> NzPVRpbS5CaXJkQHNvbnkuY29t>