RE: HTML message rejected: Re: Distribute continuous integration (build) to servers outside docker container

Hi Tim,

I had sent a reply to your response on 3rd Feb.2024, not sure if it had rea=
ched you.
I am posting the same message below again.

Thanks a lot for your prompt reply.
Iam looking forward to try the updated docker container with an upgraded di=
stribution base from Debian stretch to Debian bullseye, and from a Jenkins =
version of 2.249.3 to 2.414.1. Please let me whenever it is available. We a=
re migrating to another server, so Iam looking for best options to transfer=
 all the content to the new server. Since the docker container size is huge=
 I want to trim it down by removing unwanted old logs. Also, I am worried a=
bout server crash and hard disk crash due to so many read/write cycles (tha=
t happens during continuous integration). Hence looking for the best way to=
 back up the container and also to distribute the load & read/write cycles =
to another server (via an ssh agent). I tried with plain Jenkins via ssh cr=
edentials plugin, it seems feasible. Hence if fuego upgrades to debian bull=
seye it would be really helpful.

Also, would like to know, when login credentials support will be available =
in fuego? Because as a security measure we are supposed to have login crede=
ntials for each user. Will be available after debian upgrade?
With respect to removing the logs I will figure out the best way as I see s=
o many options but not sure if that will reduce the container size directly=
 or need to run some container commands to trim it down. I will explore tha=
t as well and email my observations in this forum.

Regards,
Dhinakar K.

-----Original Message-----
From: Bird, Tim =5Bmailto:Tim.Bird=40sony.com=5D=20
Sent: Saturday, February 3, 2024 3:44 AM
To: dhinakar k <dhinakar.k=40gmail.com>; fuego+owner=40lists.linux.dev; fue=
go=40lists.linuxfoundation.org
Subject: RE: HTML message rejected: Re: Distribute continuous integration (=
build) to servers outside docker container

Dhinakar,  See my answers inline below..

> -----Original Message-----
> From: dhinakar k <dhinakar.k=40gmail.com> Dear Fuego users,
>=20
> I have a requirement, so I thought of checking with the community if=20
> it was feasible and if so what is the best or most efficient way to=20
> implement it.
> We have fuego setup in a server and all our continuous integration=20
> jobs (for multiple code bases) are setup in it. But the issue is there=20
> are too many builds happening, which puts pressure on the hard disk=20
> (so many read/writes) and also increases the docker container size=20
> (because of build logs).
>=20
> Hence I would like to know if we can distribute the load to another=20
> server, for e.g. job will be created on our main server but build will=20
> happen on the agent on another server. The idea is to move out all=20
> cpu, memory, storage intensive activity to agents (other servers) on=20
> the network. The resultant logs can also be stored on those servers=20
> (outside docker container) so that docker size won't be growing=20
> rapidly.
>=20
> I tried creating a new node and setup an agent (connect via ssh) but=20
> facing issues in installing the 'SSH Build Agents' plugin because of=20
> compatibility issues (jenkins version needs to be upgraded).
>=20
> If anyone already tried and has a solution for this issue please let me k=
now.

Unfortunately, I have not tried this, so I'm not familiar with how Jenkins =
supports distributed jobs.

But I have a few ideas to share.

First, I'm working now on an update to the docker container, from a distrib=
ution base of Debian stretch to Debian bullseye, and from a Jenkins version=
 of 2.249.3 to 2.414.1 (thanks to a patch provided by Fuego user Yoichi Tac=
hibana).  I have applied the patches, but found some issues with the build =
instructions for some packages working in Debian bullseye.  So I have not c=
ommitted the changes to=20
the Fuego master branch yet.   I hope to be able to do this soon.

This might help with the issue of not being able to use the ssh agent plugi=
n for Jenkins.

Second, it may be possible to do some kind of proxy job submission thing, b=
y creating proxy jobs on the main Jenkins server, but having the jobs actua=
lly executed (=22built=22, in Jenkins terminology) on secondary servers.

There are a number of ways of supporting remote execution, and how you impl=
ement it will determine where the CPU cycles and memory usage is during the=
 job, and where the logs end up living (and taking up space.)

If you are using 'ttc' as your transport layer, recent versions of ttc have=
 support for remote boards via ssh (configured in ttc.conf).  But that woul=
d leave logs on the master server.

There are multiple ways to configure Fuego on the secondary servers - for e=
xample using Docker containers or installing Fuego natively.  In any event,=
 it may be possible to set up proxy jobs on the main server that call 'ftc'=
 on the secondary
servers, to run the jobs of interest.   Depending on how this was structure=
d
you may end up with build artifacts and logs on the secondary system, and o=
nly Jenkins artifacts and logs on the main system (which should save space)=
.

It might be good to get a more detailed description of your Jenkins setup t=
o be able to understand the options available.

>=20
> Also, what is the best way to backup the container after trimming it=20
> down (remove unwanted files/logs etc. and reduce it's size)?

I'm not sure how to backup a container in Docker.  There's the 'docker cont=
ainer commit'
command, but I haven't used that.

In general, to trim down you data usage you will want to perform a few oper=
ations.
Inside Jenkins, you can delete individual builds.  You can likely automate =
this (not have to do it manually  through the UI), by using the remote API.=
  That should work for the material that is inside the container.  Lots of =
data, however, including the build artifacts and the logs are natively on t=
he host, and are only present in the container through bind mounts.

This means you can remove this data directly using file operations (e.g. rm=
) on the host.
There are a few nuances to this, however, as some of the build directories =
are referenced via symlinks.

'ftc' currently supports removing jobs and nodes.  I don't recall off the t=
op of my head if this includes removing all the builds for a job or not.  B=
ut in any case, it sounds like you only want to remove some of the builds/r=
uns, not the jobs themselves.
We could possibly support 'ftc rm-run', which could remove specific runs.  =
'ftc query-runs'
could be used to generate a list of runs (e.g. that were created before a c=
ertain date).

Please provide a few more details, and maybe we can talk through and/or dev=
elop some solutions that will help meet your needs.

 -- Tim

>=20
> On Thu, Feb 1, 2024 at 11:52=E2=80=AFPM=20<fuego+owner=40lists.linux.dev>=
=20wrote:=0D=0A>=20>=0D=0A>=20>=20Greetings=21=0D=0A>=20>=0D=0A>=20>=20This=
=20is=20the=20mlmmj=20program=20managing=20the=20<fuego=40lists.linux.dev>=
=20=0D=0A>=20>=20mailing=20list.=0D=0A>=20>=0D=0A>=20>=20Your=20message=20t=
o=20<fuego=40lists.linux.dev>=20was=20not=20delivered=20to=20the=20=0D=0A>=
=20>=20list=20because=20it=20contained=20a=20HTML=20part.=20Only=20text/pla=
in=20messages=20are=20=0D=0A>=20>=20allowed=20on=20this=20list.=0D=0A>=20>=
=0D=0A>=20>=20Please=20configure=20your=20mail=20client=20to=20only=20send=
=20plain=20text=20mail.=0D=0A>=20>=0D=0A>=20>=20For=20your=20reference,=20t=
he=20rejected=20message=20follows=20below.=0D=0A>=20>=0D=0A>=20>=0D=0A>=20>=
=0D=0A>=20>=20----------=20Forwarded=20message=20----------=0D=0A>=20>=20Fr=
om:=20dhinakar=20k=20<dhinakar.k=40gmail.com>=0D=0A>=20>=20To:=20fuego+help=
=40lists.linux.dev,=20fuego=40lists.linuxfoundation.org,=20=0D=0A>=20>=20=
=22Bird,=20Timothy=22=20<Tim.Bird=40sony.com>=0D=0A>=20>=20Cc:=0D=0A>=20>=
=20Bcc:=0D=0A>=20>=20Date:=20Thu,=201=20Feb=202024=2023:51:30=20+0530=0D=0A=
>=20>=20Subject:=20Re:=20Distribute=20continuous=20integration=20(build)=20=
to=20servers=20=0D=0A>=20>=20outside=20docker=20container=20Dear=20Fuego=20=
users,=0D=0A>=20>=0D=0A>=20>=20I=20have=20a=20requirement,=20so=20I=20thoug=
ht=20of=20checking=20with=20the=20community=20if=20=0D=0A>=20>=20it=20was=
=20feasible=20and=20if=20so=20what=20is=20the=20best=20or=20most=20efficien=
t=20way=20to=0D=0A>=20implement=20it.=0D=0A>=20>=20We=20have=20fuego=20setu=
p=20in=20a=20server=20and=20all=20our=20continuous=20integration=20=0D=0A>=
=20>=20jobs=20(for=20multiple=20code=20bases)=20are=20setup=20in=20it.=20Bu=
t=20the=20issue=20is=20=0D=0A>=20>=20there=20are=0D=0A>=20too=20many=20buil=
ds=20happening,=20which=20puts=20pressure=20on=20the=20hard=20disk=20(so=20=
=0D=0A>=20many=20read/writes)=20and=20also=20increases=20the=20docker=20con=
tainer=20size=20(because=20of=20build=20logs).=0D=0A>=20>=0D=0A>=20>=20Henc=
e=20I=20would=20like=20to=20know=20if=20we=20can=20distribute=20the=20load=
=20to=20another=20=0D=0A>=20>=20server,=20for=20e.g.=20job=20will=20be=20cr=
eated=20on=20our=20main=20server=20but=20build=20=0D=0A>=20>=20will=0D=0A>=
=20happen=20on=20the=20agent=20on=20another=20server.=20The=20idea=20is=20t=
o=20move=20out=20all=20=0D=0A>=20cpu,=20memory,=20storage=20intensive=20act=
ivity=20to=20agents=20(other=20servers)=20on=20the=20network.=20The=20resul=
tant=20logs=20can=20also=20be=20stored=20on=20those=20servers=20(outside=20=
docker=20container)=20so=20that=20docker=20size=20won't=20be=20growing=20ra=
pidly.=0D=0A>=20>=0D=0A>=20>=20I=20tried=20creating=20a=20new=20node=20and=
=20setup=20an=20agent=20(connect=20via=20ssh)=20but=20=0D=0A>=20>=20facing=
=20issues=20in=20installing=20the=20'SSH=20Build=20Agents'=20plugin=20becau=
se=20of=0D=0A>=20compatibility=20issues=20(jenkins=20version=20needs=20to=
=20be=20upgraded).=0D=0A>=20>=0D=0A>=20>=20If=20anyone=20already=20tried=20=
and=20has=20a=20solution=20for=20this=20issue=20please=20let=20me=20know.=
=0D=0A>=20>=0D=0A>=20>=20Also,=20what=20is=20the=20best=20way=20to=20backup=
=20the=20container=20after=20trimming=20it=20down=20(remove=20unwanted=20fi=
les/logs=20etc.=20and=20reduce=20it's=20size)?=0D=0A>=20>=0D=0A>=20>=20Rega=
rds,=0D=0A>=20>=20Dhinakar=20K.=0D=0A>=20>=20Samsung=20India,=20Bengaluru.=
=0D=0A>=20>=0D=0A>=20>=20On=20Thu,=20Feb=201,=202024=20at=2010:30=E2=80=AFP=
M=20dhinakar=20k=20<dhinakar.k=40gmail.com>=20wrote:=0D=0A>=20>>=0D=0A>=20>=
>=20Dear=20Fuego=20users,=0D=0A>=20>>=0D=0A>=20>>=20I=20have=20a=20requirem=
ent,=20so=20I=20thought=20of=20checking=20with=20the=20community=20=0D=0A>=
=20>>=20if=20it=20was=20feasible=20and=20if=20so=20what=20is=20the=20best=
=20or=20most=20efficient=20way=20=0D=0A>=20>>=20to=0D=0A>=20implement=20it.=
=0D=0A>=20>>=20We=20have=20fuego=20setup=20in=20a=20server=20and=20all=20ou=
r=20continuous=20integration=20=0D=0A>=20>>=20jobs=20(for=20multiple=20code=
=20bases)=20are=20setup=20in=20it.=20But=20the=20issue=20is=20=0D=0A>=20>>=
=20there=20are=0D=0A>=20too=20many=20builds=20happening,=20which=20puts=20p=
ressure=20on=20the=20hard=20disk=20(so=20=0D=0A>=20many=20read/writes)=20an=
d=20also=20increases=20the=20docker=20container=20size=20(because=20of=20bu=
ild=20logs).=0D=0A>=20>>=0D=0A>=20>>=20Hence=20I=20would=20like=20to=20know=
=20if=20we=20can=20distribute=20the=20load=20to=20another=20=0D=0A>=20>>=20=
server,=20for=20e.g.=20job=20will=20be=20created=20on=20our=20main=20server=
=20but=20build=20=0D=0A>=20>>=20will=0D=0A>=20happen=20on=20the=20agent=20o=
n=20another=20server.=20The=20idea=20is=20to=20move=20out=20all=20=0D=0A>=
=20cpu,=20memory,=20storage=20intensive=20activity=20to=20agents=20(other=
=20servers)=20on=20the=20network.=20The=20resultant=20logs=20can=20also=20b=
e=20stored=20on=20those=20servers=20(outside=20docker=20container)=20so=20t=
hat=20docker=20size=20won't=20be=20growing=20rapidly.=0D=0A>=20>>=0D=0A>=20=
>>=20I=20tried=20creating=20a=20new=20node=20and=20setup=20an=20agent=20(co=
nnect=20via=20ssh)=20=0D=0A>=20>>=20but=20facing=20issues=20in=20installing=
=20the=20'SSH=20Build=20Agents'=20plugin=20=0D=0A>=20>>=20because=20of=0D=
=0A>=20compatibility=20issues=20(jenkins=20version=20needs=20to=20be=20upgr=
aded).=0D=0A>=20>>=0D=0A>=20>>=20If=20anyone=20already=20tried=20and=20has=
=20a=20solution=20for=20this=20issue=20please=20let=20me=20know.=0D=0A>=20>=
>=0D=0A>=20>>=20Also,=20what=20is=20the=20best=20way=20to=20backup=20the=20=
container=20after=20trimming=20it=20down=20(remove=20unwanted=20files/logs=
=20etc.=20and=20reduce=20it's=20size)?=0D=0A>=20>>=0D=0A>=20>>=20Regards,=
=0D=0A>=20>>=20Dhinakar=20K.=0D=0A>=20>>=20Samsung=20India,=20Bengaluru.=0D=
=0A=0D=0A=0D=0A