RE[3]: Distribute continuous integration (build) to servers outside docker container
"dhinakar.k" <[email protected]> Fri, 15 Mar 2024 21:07:57 +0530
| Newsgroups | dev.linux.lists.fuego |
|---|---|
| Message-ID | <[email protected]> |
Hi Tim, Thanks a lot for your response. Please find below the details that you requested. =23 du -sh /var/lib/Jenkins 32G /var/lib/Jenkins =23 du -sh /var/lib/jenkins/* =7C sort -h 0 /var/lib/jenkins/secret.key.not-so-secret 4.0K /var/lib/jenkins/com.cloudbees.hudson.plugins.folder.config.Abstrac= tFolderConfiguration.xml 4.0K /var/lib/jenkins/com.mtvi.plateng.hudson.ldap.LdapMailAddressResolv= er.xml 4.0K /var/lib/jenkins/com.orctom.jenkins.plugin.buildtimestamp.BuildTime= stampWrapper.xml 4.0K /var/lib/jenkins/credentials.xml 4.0K /var/lib/jenkins/hudson.maven.MavenModuleSet.xml 4.0K /var/lib/jenkins/hudson.model.UpdateCenter.xml 4.0K /var/lib/jenkins/hudson.plugins.emailext.ExtendedEmailPublisher.xml 4.0K /var/lib/jenkins/hudson.plugins.git.GitSCM.xml 4.0K /var/lib/jenkins/hudson.plugins.git.GitTool.xml 4.0K /var/lib/jenkins/hudson.plugins.sidebar_link.SidebarLinkPlugin.xml 4.0K /var/lib/jenkins/hudson.plugins.sonar.SonarGlobalConfiguration.xml 4.0K /var/lib/jenkins/hudson.plugins.timestamper.TimestamperConfig.xml 4.0K /var/lib/jenkins/hudson.tasks.Mailer.xml 4.0K /var/lib/jenkins/hudson.tasks.Shell.xml 4.0K /var/lib/jenkins/hudson.triggers.SCMTrigger.xml 4.0K /var/lib/jenkins/identity.key.enc 4.0K /var/lib/jenkins/io.jenkins.plugins.junit.storage.JunitTestResultSt= orageConfiguration.xml 4.0K /var/lib/jenkins/jenkins.fingerprints.GlobalFingerprintConfiguratio= n.xml 4.0K /var/lib/jenkins/jenkins.install.InstallUtil.lastExecVersion 4.0K /var/lib/jenkins/jenkins.install.UpgradeWizard.state 4.0K /var/lib/jenkins/jenkins.model.ArtifactManagerConfiguration.xml 4.0K /var/lib/jenkins/jenkins.model.GlobalBuildDiscarderConfiguration.xm= l 4.0K /var/lib/jenkins/jenkins.model.JenkinsLocationConfiguration.xml 4.0K /var/lib/jenkins/jenkins.plugins.msginject.MsgInjectConfig.xml 4.0K /var/lib/jenkins/jenkins.security.ResourceDomainConfiguration.xml 4.0K /var/lib/jenkins/jenkins.security.apitoken.ApiTokenPropertyConfigur= ation.xml 4.0K /var/lib/jenkins/jenkins.tasks.filters.EnvVarsFilterGlobalConfigura= tion.xml 4.0K /var/lib/jenkins/jenkins.telemetry.Correlator.xml 4.0K /var/lib/jenkins/net.plavcak.jenkins.plugins.scmskip.SCMSkipBuildWr= apper.xml 4.0K /var/lib/jenkins/net.uaznia.lukanus.hudson.plugins.gitparameter.Git= ParameterDefinition.xml 4.0K /var/lib/jenkins/nodeMonitors.xml 4.0K /var/lib/jenkins/org.codefirst.SimpleThemeDecorator.xml 4.0K /var/lib/jenkins/org.jenkinsCi.plugins.projectDescriptionSetter.Des= criptionSetterWrapper.xml 4.0K /var/lib/jenkins/org.jenkinsci.plugins.ansible_tower.AnsibleTower.x= ml 4.0K /var/lib/jenkins/org.jenkinsci.plugins.corsfilter.AccessControlsFil= ter.xml 4.0K /var/lib/jenkins/org.jenkinsci.plugins.lucene.search.config.SearchB= ackendConfiguration.xml 4.0K /var/lib/jenkins/org.jenkinsci.plugins.workflow.flow.GlobalDefaultF= lowDurabilityLevel.xml 4.0K /var/lib/jenkins/org.quality.gates.jenkins.plugin.GlobalConfig.xml 4.0K /var/lib/jenkins/queue.xml 4.0K /var/lib/jenkins/queue.xml.bak 4.0K /var/lib/jenkins/scriptApproval.xml 4.0K /var/lib/jenkins/secret.key 4.0K /var/lib/jenkins/sidebar-link.xml 12K /var/lib/jenkins/gerrit-trigger.xml 16K /var/lib/jenkins/hudson.plugins.ansicolor.AnsiColorBuildWrapper.xml 20K /var/lib/jenkins/config.xml 44K /var/lib/jenkins/nodes 72K /var/lib/jenkins/secrets 356K /var/lib/jenkins/users 1.7M /var/lib/jenkins/logs 1.8M /var/lib/jenkins/luceneIndex 3.6M /var/lib/jenkins/updates 3.9M /var/lib/jenkins/plugins_backup 316M /var/lib/jenkins/plugins 316M /var/lib/jenkins/plugins_not_working 372M /var/lib/jenkins/plugins_old_2ndFeb2024 376M /var/lib/jenkins/plugins_working_2ndFeb2024 1.8G /var/lib/jenkins/userContent 11G /var/lib/jenkins/jobs 19G /var/lib/jenkins/workspace =23 du -sh /fuego-rw 13G /fuego-rw =23 du -sh /fuego-rw/buildzone/* =7C sort -h =23 du -sh /fuego-rw/logs/* = =7C sort -h =23 ftc list-nodes =7C wc -l =23 ftc list-jobs =7C wc -l =23 ft= c list-runs =7C wc -l 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Dhrystone-x86_= 64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.GLMark-aarch64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.GLMark-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.IOzone-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Interbench-x86= _64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Java-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.OpenSSL-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Stream-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Whetstone-x86_= 64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.bonnie-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.cyclictest-x86= _64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.dbench4-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.ebizzy-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.ffsb-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.fio-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.gtkperf-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.hackbench-x86_= 64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.himeno-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.iperf-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.linpack-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.lmbench2-x86_6= 4 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.nbench_byte-x8= 6_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.netperf-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.netpipe-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.signaltest-x86= _64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.tiobench-x86_6= 4 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.x11perf-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Dhrystone-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.GLMark-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.IOzone-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Interbench-x86_6= 4 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Java-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.OpenSSL-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Stream-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Whetstone-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.bonnie-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.cyclictest-x86_6= 4 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.dbench4-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.ebizzy-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.ffsb-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.fio-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.gtkperf-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.hackbench-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.himeno-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.iperf-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.linpack-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.lmbench2-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.nbench_byte-x86_= 64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.netperf-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.netpipe-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.signaltest-x86_6= 4 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.tiobench-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.x11perf-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.LTP-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.OpenSSL-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.aiostress-x86_6= 4 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.bc-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.bzip2-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.crashme-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.expat-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.fontconfig-x86_= 64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.ft2demos-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.glib-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.hello_world-x86= _64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.ipv6connect-x86= _64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.jpeg-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.linus_stress-x8= 6_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.netperf-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.pi_tests-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.rmaptest-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.scrashme-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.stress-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.synctest-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.zlib-x86_64 4.0K /fuego-rw/buildzone/README 12K /fuego-rw/buildzone/Functional.bc-x86_64 32K /fuego-rw/buildzone/Functional.hello_world-x86_64 32K /fuego-rw/buildzone/Functional.linus_stress-x86_64 32K /fuego-rw/buildzone/Functional.synctest-x86_64 36K /fuego-rw/buildzone/Functional.rmaptest-x86_64 48K /fuego-rw/buildzone/Benchmark.himeno-x86_64 48K /fuego-rw/buildzone/Benchmark.linpack-x86_64 48K /fuego-rw/buildzone/Functional.ipv6connect-x86_64 60K /fuego-rw/buildzone/Benchmark.Stream-x86_64 76K /fuego-rw/buildzone/Benchmark.Whetstone-x86_64 76K /fuego-rw/buildzone/Benchmark.ebizzy-x86_64 84K /fuego-rw/buildzone/Functional.aiostress-x86_64 168K /fuego-rw/buildzone/Benchmark.Dhrystone-x86_64 240K /fuego-rw/buildzone/Benchmark.tiobench-x86_64 244K /fuego-rw/buildzone/Functional.crashme-x86_64 488K /fuego-rw/buildzone/Benchmark.Interbench-x86_64 576K /fuego-rw/buildzone/Functional.scrashme-x86_64 760K /fuego-rw/buildzone/Benchmark.hackbench-x86_64 816K /fuego-rw/buildzone/Benchmark.signaltest-x86_64 868K /fuego-rw/buildzone/Functional.pi_tests-x86_64 900K /fuego-rw/buildzone/Benchmark.netpipe-x86_64 904K /fuego-rw/buildzone/Benchmark.cyclictest-x86_64 1004K /fuego-rw/buildzone/Functional.stress-x86_64 1.3M /fuego-rw/buildzone/Benchmark.ffsb-x86_64 1.6M /fuego-rw/buildzone/Benchmark.GLMark-aarch64 1.6M /fuego-rw/buildzone/Benchmark.GLMark-x86_64 1.7M /fuego-rw/buildzone/Benchmark.nbench_byte-x86_64 2.1M /fuego-rw/buildzone/Benchmark.iperf-x86_64 2.3M /fuego-rw/buildzone/Benchmark.gtkperf-x86_64 2.4M /fuego-rw/buildzone/Benchmark.bonnie-x86_64 2.5M /fuego-rw/buildzone/Benchmark.x11perf-x86_64 2.9M /fuego-rw/buildzone/Functional.bzip2-x86_64 3.0M /fuego-rw/buildzone/Functional.zlib-x86_64 3.1M /fuego-rw/buildzone/Benchmark.IOzone-x86_64 3.5M /fuego-rw/buildzone/Functional.jpeg-x86_64 5.2M /fuego-rw/buildzone/Benchmark.lmbench2-x86_64 6.1M /fuego-rw/buildzone/Benchmark.netperf-x86_64 6.1M /fuego-rw/buildzone/Functional.netperf-x86_64 7.0M /fuego-rw/buildzone/Functional.fontconfig-x86_64 9.1M /fuego-rw/buildzone/Benchmark.fio-x86_64 29M /fuego-rw/buildzone/Functional.expat-x86_64 33M /fuego-rw/buildzone/Benchmark.dbench4-x86_64 34M /fuego-rw/buildzone/Functional.ft2demos-x86_64 49M /fuego-rw/buildzone/Functional.glib-x86_64 74M /fuego-rw/buildzone/Benchmark.OpenSSL-x86_64 74M /fuego-rw/buildzone/Functional.OpenSSL-x86_64 160M /fuego-rw/buildzone/Benchmark.Java-x86_64 1.8G /fuego-rw/buildzone/X86-SSH-Test.default.Functional.kernel_build-x8= 6_64 2.4G /fuego-rw/buildzone/Functional.LTP-x86_64 Outside the container, can you run: =24 docker image ls nxo-container-backup-2ndsep2021 latest b6a3d16508e8 2 ye= ars ago 173GB fuego-base-image-20thaug2019-container latest b2cf22891ac6 4 ye= ars ago 2.41GB fuegocontainerbackup-4thmar2019 latest 04112c383476 5 ye= ars ago 3.34GB fuego-piper-image-22ndnov2021 latest 0121b1c342e6 6 yea= rs ago 10.7GB fuego-xyz-image-22ndnov2021 latest 0bf6b942db70 6 ye= ars ago 5.8GB fuego-dart-image-22ndnov2021 latest 1d2c6a99dc0a 6 year= s ago 1.66GB fuego-pavv-image-22ndnov2021 latest ba71ede1e872 6 ye= ars ago 4.63GB fuego-gpu-image-22ndnov2021 latest 42718d86dd09 6 years= ago 3.41GB Regarding below request, I just want to lock the Jenkins user interface/ da= shboard, so that none can see all the jobs, results, logs and other details= without logging in. So to start with even one user with login credentials = will help. But I don't know how Jenkins credentials relate to user accounts. I haven'= t ever set up Jenkins to handle multiple user accounts. Indeed, in my curr= ent 2.414.1 Jenkins setup I can see an option to list =22People=22 (which I= presume is registered user accounts), but I can find no way to create or m= anage new user accounts. Thanks a lot for all other details, they were very helpful. Regards, Dhinakar K. -----Original Message----- From: Bird, Tim =5Bmailto:Tim.Bird=40sony.com=5D=20 Sent: Wednesday, March 13, 2024 5:47 AM To: dhinakar.k <dhinakar.k=40samsung.com>; fuego+owner=40lists.linux.dev; f= uego=40lists.linuxfoundation.org Subject: RE: HTML message rejected: Re: Distribute continuous integration (= build) to servers outside docker container Hey Dhinakar, Sorry to not respond sooner. There was a lot to process in your email, and= some of it I don't have answers for, but I'll try to give some status and = feedback where I can. > -----Original Message----- > From: dhinakar.k <dhinakar.k=40samsung.com> Hi Tim, >=20 > I had sent a reply to your response on 3rd Feb.2024, not sure if it had r= eached you. > I am posting the same message below again. >=20 > Thanks a lot for your prompt reply. > Iam looking forward to try the updated docker container with an=20 > upgraded distribution base from Debian stretch to Debian bullseye, and fr= om a Jenkins version of 2.249.3 to 2.414.1. Please let me whenever it is av= ailable. OK - Will do. I haven't been able to allocate time for testing and fixing = it, so I didn't want to push it to the master branch. If I pushed somethin= g to a testing branch, would you be able to download from there and test it= ? One issue that came up is that some of the toolchains for some of my old bo= ards/distros are not supported (ie don't work) in Debian bullseye, and so I= was looking for solutions for that. =20 > We are migrating to another server, so Iam looking for best options to=20 > transfer all the content to the new server. Since the docker container=20 > size is huge I want to trim it down by removing unwanted old logs. It would be good to know where the container size is coming from. Is it in= the Fuego build directories, the Fuego run directories (where Fuego logs a= nd run artifacts are stored), or in the Jenkins build directories (where Je= nkins logs and artifacts are)? The materials under /fuego-rw should be on = the host system, and not inside the container itself (since these are volum= e mounts). But I don't know how this space is accounted to the container (= if any). It shouldn't be in any container image. The materials under /var/lib/jenkins are inside the container image. Can you run the following commands, inside your container, and send the res= ults: =23 du -sh /var/lib/jenkins =23 du -sh /var/lib/jenkins/* =7C sort -h =23 du -sh /fuego-rw =23 du -sh /fuego-rw/buildzone/* =7C sort -h =23 du -sh /fuego-rw/logs/* = =7C sort -h =23 ftc list-nodes =7C wc -l =23 ftc list-jobs =7C wc -l =23 ft= c list-runs =7C wc -l Outside the container, can you run: =24 docker image ls (and remove any lines not related to fuego images) But I don't know how Jenkins credentials relate to user accounts. I haven'= t ever set up Jenkins to handle multiple user accounts. Indeed, in my curr= ent 2.414.1 Jenkins setup I can see an option to list =22People=22 (which I= presume is registered user accounts), but I can find no way to create or m= anage new user accounts. > Also, I am worried about server crash and hard disk crash due to so many = read/write cycles (that happens during continuous integration). > Hence looking for the best way to back up the container and also to=20 > distribute the load & read/write cycles to another server (via an ssh=20 > agent). I tried with plain Jenkins via ssh credentials plugin, it seems f= easible. Hence if fuego upgrades to debian bullseye it would be really help= ful. OK - I'll keep plugging away at this. >=20 > Also, would like to know, when login credentials support will be=20 > available in fuego? Because as a security measure we are supposed to have= login credentials for each user. Will be available after debian upgrade? I believe you are referring to Jenkins credentials? It appears that Jenkin= s supports credentials in version 2.414, which is the new version of Jenkin= s in the new docker container (with the Debian upgrade). So I believe the = answer is yes. But I don't know how Jenkins credentials relate to user accounts. I haven'= t ever set up Jenkins to handle multiple user accounts. Indeed, in my curr= ent 2.414.1 Jenkins setup I can see an option to list =22People=22 (which I= presume is registered user accounts), but I can find no way to create or m= anage new user accounts. > With respect to removing the logs I will figure out the best way as I=20 > see so many options but not sure if that will reduce the container size d= irectly or need to run some container commands to trim it down. I will expl= ore that as well and email my observations in this forum. I have considered adding an =22rm-run=22 sub-command to ftc, which would al= low you to remove old runs. However, I'd like to see if it's the run data that is causing the size prob= lem that you are seeing. If you want to archive the data for an individual run, then you can use 'ft= c package-run' and move the resulting package to some long-term storage dev= ice (before, ostensibly, removing that run from the system.) I hope this is helpful. -- Tim >=20 > Regards, > Dhinakar K. >=20 > -----Original Message----- > From: Bird, Tim =5Bmailto:Tim.Bird=40sony.com=5D > Sent: Saturday, February 3, 2024 3:44 AM > To: dhinakar k <dhinakar.k=40gmail.com>; fuego+owner=40lists.linux.dev;= =20 > fuego=40lists.linuxfoundation.org > Subject: RE: HTML message rejected: Re: Distribute continuous=20 > integration (build) to servers outside docker container >=20 > Dhinakar, See my answers inline below.. >=20 > > -----Original Message----- > > From: dhinakar k <dhinakar.k=40gmail.com> Dear Fuego users, > > > > I have a requirement, so I thought of checking with the community if=20 > > it was feasible and if so what is the best or most efficient way to=20 > > implement it. > > We have fuego setup in a server and all our continuous integration=20 > > jobs (for multiple code bases) are setup in it. But the issue is=20 > > there are too many builds happening, which puts pressure on the hard=20 > > disk (so many read/writes) and also increases the docker container=20 > > size (because of build logs). > > > > Hence I would like to know if we can distribute the load to another=20 > > server, for e.g. job will be created on our main server but build=20 > > will happen on the agent on another server. The idea is to move out=20 > > all cpu, memory, storage intensive activity to agents (other=20 > > servers) on the network. The resultant logs can also be stored on=20 > > those servers (outside docker container) so that docker size won't=20 > > be growing rapidly. > > > > I tried creating a new node and setup an agent (connect via ssh) but=20 > > facing issues in installing the 'SSH Build Agents' plugin because of=20 > > compatibility issues (jenkins version needs to be upgraded). > > > > If anyone already tried and has a solution for this issue please let me= know. >=20 > Unfortunately, I have not tried this, so I'm not familiar with how Jenkin= s supports distributed jobs. >=20 > But I have a few ideas to share. >=20 > First, I'm working now on an update to the docker container, from a=20 > distribution base of Debian stretch to Debian bullseye, and from a=20 > Jenkins version of 2.249.3 to 2.414.1 (thanks to a patch provided by Fueg= o user Yoichi Tachibana). I have applied the patches, but found some issue= s with the build instructions for some packages working in Debian bullseye.= So I have not committed the changes to > the Fuego master branch yet. I hope to be able to do this soon. >=20 > This might help with the issue of not being able to use the ssh agent plu= gin for Jenkins. >=20 > Second, it may be possible to do some kind of proxy job submission=20 > thing, by creating proxy jobs on the main Jenkins server, but having the = jobs actually executed (=22built=22, in Jenkins terminology) on secondary s= ervers. >=20 > There are a number of ways of supporting remote execution, and how you=20 > implement it will determine where the CPU cycles and memory usage is=20 > during the job, and where the logs end up living (and taking up=20 > space.) >=20 > If you are using 'ttc' as your transport layer, recent versions of ttc=20 > have support for remote boards via ssh (configured in ttc.conf). But tha= t would leave logs on the master server. >=20 > There are multiple ways to configure Fuego on the secondary servers -=20 > for example using Docker containers or installing Fuego natively. In any= event, it may be possible to set up proxy jobs on the main server that cal= l 'ftc' on the secondary > servers, to run the jobs of interest. Depending on how this was structu= red > you may end up with build artifacts and logs on the secondary system,=20 > and only Jenkins artifacts and logs on the main system (which should save= space). >=20 > It might be good to get a more detailed description of your Jenkins setup= to be able to understand the options available. >=20 > > > > Also, what is the best way to backup the container after trimming it=20 > > down (remove unwanted files/logs etc. and reduce it's size)? >=20 > I'm not sure how to backup a container in Docker. There's the 'docker co= ntainer commit' > command, but I haven't used that. >=20 > In general, to trim down you data usage you will want to perform a few op= erations. > Inside Jenkins, you can delete individual builds. You can likely=20 > automate this (not have to do it manually through the UI), by using=20 > the remote API. That should work for the material that is inside the con= tainer. Lots of data, however, including the build artifacts and the logs = are natively on the host, and are only present in the container through bin= d mounts. >=20 > This means you can remove this data directly using file operations (e.g. = rm) on the host. > There are a few nuances to this, however, as some of the build directorie= s are referenced via symlinks. >=20 > 'ftc' currently supports removing jobs and nodes. I don't recall off the= top of my head if this includes removing all the builds for a job or not. > But in any case, it sounds like you only want to remove some of the build= s/runs, not the jobs themselves. > We could possibly support 'ftc rm-run', which could remove specific runs.= 'ftc query-runs' > could be used to generate a list of runs (e.g. that were created before a= certain date). >=20 > Please provide a few more details, and maybe we can talk through and/or d= evelop some solutions that will help meet your needs. >=20 > -- Tim >=20 > > > > On Thu, Feb 1, 2024 at 11:52=E2=80=AFPM=20<fuego+owner=40lists.linux.de= v>=20wrote:=0D=0A>=20>=20>=0D=0A>=20>=20>=20Greetings=21=0D=0A>=20>=20>=0D= =0A>=20>=20>=20This=20is=20the=20mlmmj=20program=20managing=20the=20<fuego= =40lists.linux.dev>=20=0D=0A>=20>=20>=20mailing=20list.=0D=0A>=20>=20>=0D= =0A>=20>=20>=20Your=20message=20to=20<fuego=40lists.linux.dev>=20was=20not= =20delivered=20to=20the=20=0D=0A>=20>=20>=20list=20because=20it=20contained= =20a=20HTML=20part.=20Only=20text/plain=20messages=20=0D=0A>=20>=20>=20are= =20allowed=20on=20this=20list.=0D=0A>=20>=20>=0D=0A>=20>=20>=20Please=20con= figure=20your=20mail=20client=20to=20only=20send=20plain=20text=20mail.=0D= =0A>=20>=20>=0D=0A>=20>=20>=20For=20your=20reference,=20the=20rejected=20me= ssage=20follows=20below.=0D=0A>=20>=20>=0D=0A>=20>=20>=0D=0A>=20>=20>=0D=0A= >=20>=20>=20----------=20Forwarded=20message=20----------=0D=0A>=20>=20>=20= From:=20dhinakar=20k=20<dhinakar.k=40gmail.com>=0D=0A>=20>=20>=20To:=20fueg= o+help=40lists.linux.dev,=20fuego=40lists.linuxfoundation.org,=20=0D=0A>=20= >=20>=20=22Bird,=20Timothy=22=20<Tim.Bird=40sony.com>=0D=0A>=20>=20>=20Cc:= =0D=0A>=20>=20>=20Bcc:=0D=0A>=20>=20>=20Date:=20Thu,=201=20Feb=202024=2023:= 51:30=20+0530=0D=0A>=20>=20>=20Subject:=20Re:=20Distribute=20continuous=20i= ntegration=20(build)=20to=20servers=20=0D=0A>=20>=20>=20outside=20docker=20= container=20Dear=20Fuego=20users,=0D=0A>=20>=20>=0D=0A>=20>=20>=20I=20have= =20a=20requirement,=20so=20I=20thought=20of=20checking=20with=20the=20commu= nity=20=0D=0A>=20>=20>=20if=20it=20was=20feasible=20and=20if=20so=20what=20= is=20the=20best=20or=20most=20efficient=20=0D=0A>=20>=20>=20way=20to=0D=0A>= =20>=20implement=20it.=0D=0A>=20>=20>=20We=20have=20fuego=20setup=20in=20a= =20server=20and=20all=20our=20continuous=20integration=20=0D=0A>=20>=20>=20= jobs=20(for=20multiple=20code=20bases)=20are=20setup=20in=20it.=20But=20the= =20issue=20is=20=0D=0A>=20>=20>=20there=20are=0D=0A>=20>=20too=20many=20bui= lds=20happening,=20which=20puts=20pressure=20on=20the=20hard=20disk=20(so= =20=0D=0A>=20>=20many=20read/writes)=20and=20also=20increases=20the=20docke= r=20container=20size=20(because=20of=20build=20logs).=0D=0A>=20>=20>=0D=0A>= =20>=20>=20Hence=20I=20would=20like=20to=20know=20if=20we=20can=20distribut= e=20the=20load=20to=20=0D=0A>=20>=20>=20another=20server,=20for=20e.g.=20jo= b=20will=20be=20created=20on=20our=20main=20server=20=0D=0A>=20>=20>=20but= =20build=20will=0D=0A>=20>=20happen=20on=20the=20agent=20on=20another=20ser= ver.=20The=20idea=20is=20to=20move=20out=20all=20=0D=0A>=20>=20cpu,=20memor= y,=20storage=20intensive=20activity=20to=20agents=20(other=20servers)=20on= =20=0D=0A>=20>=20the=20network.=20The=20resultant=20logs=20can=20also=20be= =20stored=20on=20those=20servers=0D=0A>=20(outside=20docker=20container)=20= so=20that=20docker=20size=20won't=20be=20growing=20rapidly.=0D=0A>=20>=20>= =0D=0A>=20>=20>=20I=20tried=20creating=20a=20new=20node=20and=20setup=20an= =20agent=20(connect=20via=20ssh)=20=0D=0A>=20>=20>=20but=20facing=20issues= =20in=20installing=20the=20'SSH=20Build=20Agents'=20plugin=20=0D=0A>=20>=20= >=20because=20of=0D=0A>=20>=20compatibility=20issues=20(jenkins=20version= =20needs=20to=20be=20upgraded).=0D=0A>=20>=20>=0D=0A>=20>=20>=20If=20anyone= =20already=20tried=20and=20has=20a=20solution=20for=20this=20issue=20please= =20let=20me=20know.=0D=0A>=20>=20>=0D=0A>=20>=20>=20Also,=20what=20is=20the= =20best=20way=20to=20backup=20the=20container=20after=20trimming=20it=20dow= n=20(remove=20unwanted=20files/logs=20etc.=20and=20reduce=20it's=20size)?= =0D=0A>=20>=20>=0D=0A>=20>=20>=20Regards,=0D=0A>=20>=20>=20Dhinakar=20K.=0D= =0A>=20>=20>=20Samsung=20India,=20Bengaluru.=0D=0A>=20>=20>=0D=0A>=20>=20>= =20On=20Thu,=20Feb=201,=202024=20at=2010:30=E2=80=AFPM=20dhinakar=20k=20<dh= inakar.k=40gmail.com>=20wrote:=0D=0A>=20>=20>>=0D=0A>=20>=20>>=20Dear=20Fue= go=20users,=0D=0A>=20>=20>>=0D=0A>=20>=20>>=20I=20have=20a=20requirement,= =20so=20I=20thought=20of=20checking=20with=20the=20community=20=0D=0A>=20>= =20>>=20if=20it=20was=20feasible=20and=20if=20so=20what=20is=20the=20best= =20or=20most=20efficient=20=0D=0A>=20>=20>>=20way=20to=0D=0A>=20>=20impleme= nt=20it.=0D=0A>=20>=20>>=20We=20have=20fuego=20setup=20in=20a=20server=20an= d=20all=20our=20continuous=20=0D=0A>=20>=20>>=20integration=20jobs=20(for= =20multiple=20code=20bases)=20are=20setup=20in=20it.=20But=20=0D=0A>=20>=20= >>=20the=20issue=20is=20there=20are=0D=0A>=20>=20too=20many=20builds=20happ= ening,=20which=20puts=20pressure=20on=20the=20hard=20disk=20(so=20=0D=0A>= =20>=20many=20read/writes)=20and=20also=20increases=20the=20docker=20contai= ner=20size=20(because=20of=20build=20logs).=0D=0A>=20>=20>>=0D=0A>=20>=20>>= =20Hence=20I=20would=20like=20to=20know=20if=20we=20can=20distribute=20the= =20load=20to=20=0D=0A>=20>=20>>=20another=20server,=20for=20e.g.=20job=20wi= ll=20be=20created=20on=20our=20main=20server=20=0D=0A>=20>=20>>=20but=20bui= ld=20will=0D=0A>=20>=20happen=20on=20the=20agent=20on=20another=20server.= =20The=20idea=20is=20to=20move=20out=20all=20=0D=0A>=20>=20cpu,=20memory,= =20storage=20intensive=20activity=20to=20agents=20(other=20servers)=20on=20= =0D=0A>=20>=20the=20network.=20The=20resultant=20logs=20can=20also=20be=20s= tored=20on=20those=20servers=0D=0A>=20(outside=20docker=20container)=20so= =20that=20docker=20size=20won't=20be=20growing=20rapidly.=0D=0A>=20>=20>>= =0D=0A>=20>=20>>=20I=20tried=20creating=20a=20new=20node=20and=20setup=20an= =20agent=20(connect=20via=20ssh)=20=0D=0A>=20>=20>>=20but=20facing=20issues= =20in=20installing=20the=20'SSH=20Build=20Agents'=20plugin=20=0D=0A>=20>=20= >>=20because=20of=0D=0A>=20>=20compatibility=20issues=20(jenkins=20version= =20needs=20to=20be=20upgraded).=0D=0A>=20>=20>>=0D=0A>=20>=20>>=20If=20anyo= ne=20already=20tried=20and=20has=20a=20solution=20for=20this=20issue=20plea= se=20let=20me=20know.=0D=0A>=20>=20>>=0D=0A>=20>=20>>=20Also,=20what=20is= =20the=20best=20way=20to=20backup=20the=20container=20after=20trimming=20it= =20down=20(remove=20unwanted=20files/logs=20etc.=20and=20reduce=20it's=20si= ze)?=0D=0A>=20>=20>>=0D=0A>=20>=20>>=20Regards,=0D=0A>=20>=20>>=20Dhinakar= =20K.=0D=0A>=20>=20>>=20Samsung=20India,=20Bengaluru.=0D=0A>=20=0D=0A>=20= =0D=0A>=20=0D=0A=0D=0A=0D=0A