lucene3/lucene4 jenkins boxes - sha1 git errors

--0000000000008e7c8f06546f4760
Content-Type: text/plain; charset="UTF-8"

Hello everyone,

I cross-post to Solr and Lucene dev lists because we share ASF jenkins
boxes that
execute jenkins workflows.

There has been a series of odd exceptions from Jenkins, looking pretty much
like this:

...
> git reset --hard # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from
https://github.com/apache/lucene.git
...
Caused by: hudson.plugins.git.GitException: Command "git reset --hard"
returned status code 128:
stdout:
stderr: error: unable to read sha1 file of .asf.yaml
(48083930a50d886827c1f87e23875e5da98c58c6)
error: unable to read sha1 file of .dir-locals.el
(c51e1232603b85b8bc74fbed7d1de08186920379)
error: unable to read sha1 file of .git-blame-ignore-revs
(945e687d0f3cdfcc99cb69b8b35ab631a70371a7)
error: unable to read sha1 file of .gitattributes
(a3135003e80fa8f49fc0f2250f40b85cde12ebc5)
...

I've ssh'd to lucene3/lucene4 and it seems like something leaves the .git
folder of a workspace checkout in a broken state, leading to subsequent git
reset failures. I'm not sure if it's a jenkins task timeout kicking in or
something else.

There seems to be enough drive space on both boxes, although it's close to
full. Looking at asf INFRA jira, the same type of exception was always
caused by disk space running low so for now I've:

- removed all existing workspaces under
/home/jenkins/jenkins-agent/workspace/Lucene
- removed all existing workspaces under
/home/jenkins/jenkins-agent/workspace/Solr

I didn't know how to pause jenkins for this, apologies if you get an error
from a job that was running at the moment I removed those folders.

Oddly enough, I can't see the same error appearing on Solr jenkins runs -
could be that all the jobs there have a "wipe workspace before you start"
checkbox enabled... I'm really not sure, sorry.

Right now, df -h shows:

lucene3 /dev/sda2 503G 59G 419G 13% /
lucene4 /dev/sda2 503G 99G 379G 21% /

Let's see if this fixes the problem.

Dawid

--0000000000008e7c8f06546f4760
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><br><div>Hello everyone,</div><div><br></div><div>I cross-=
post to Solr and Lucene dev lists because we share ASF jenkins boxes that</=
div><div>execute jenkins workflows.</div><div><br></div><div>There has been=
 a series of odd exceptions from Jenkins, looking pretty much like this:</d=
iv><div><br></div><div>...</div><div>&gt; git reset --hard # timeout=3D10<b=
r>ERROR: Error fetching remote repo &#39;origin&#39;<br>hudson.plugins.git.=
GitException: Failed to fetch from <a href=3D"https://github.com/apache/luc=
ene.git">https://github.com/apache/lucene.git</a><br></div><div>...</div><d=
iv><span style=3D"background-color:transparent">Caused by: hudson.plugins.g=
it.GitException: Command &quot;git reset --hard&quot; returned status code =
128:</span><br></div><div>stdout: <br>stderr: error: unable to read sha1 fi=
le of .asf.yaml (48083930a50d886827c1f87e23875e5da98c58c6)<br>error: unable=
 to read sha1 file of .dir-locals.el (c51e1232603b85b8bc74fbed7d1de08186920=
379)<br>error: unable to read sha1 file of .git-blame-ignore-revs (945e687d=
0f3cdfcc99cb69b8b35ab631a70371a7)<br>error: unable to read sha1 file of .gi=
tattributes (a3135003e80fa8f49fc0f2250f40b85cde12ebc5)<br>...</div><div><br=
></div><div>I&#39;ve ssh&#39;d to lucene3/lucene4 and it seems like somethi=
ng leaves the .git folder of a workspace checkout in a broken state, leadin=
g to subsequent git reset failures. I&#39;m not sure if it&#39;s a jenkins =
task timeout kicking in or something else.<br><br>There seems to be enough =
drive space on both boxes, although it&#39;s close to full. Looking at asf =
INFRA jira, the same type of exception was always caused by disk space runn=
ing low so for now=C2=A0<span style=3D"background-color:transparent">I&#39;=
ve:</span></div><div><br>- removed all existing workspaces under /home/jenk=
ins/jenkins-agent/workspace/Lucene<br></div><div>- removed all existing wor=
kspaces under /home/jenkins/jenkins-agent/workspace/Solr</div><div><br></di=
v><div>I didn&#39;t know how to pause jenkins for this, apologies if you ge=
t an error from a job that was running at the moment I removed those folder=
s.</div><div><br></div><div>Oddly enough, I can&#39;t see the same error ap=
pearing on Solr jenkins runs - could be that all the jobs there have a &quo=
t;wipe workspace before you start&quot; checkbox enabled... I&#39;m really =
not sure, sorry.</div><div><br></div><div>Right now, df -h shows:</div><div=
><br></div><div><div style=3D"font-family:monospace;white-space:pre">lucene=
3
/dev/sda2       503G   59G  419G  13% /
<br></div><div style=3D"font-family:monospace;white-space:pre">lucene4
/dev/sda2       503G   99G  379G  21% /</div></div><div><br></div><div>Let&=
#39;s see if this fixes the problem.</div><div><br></div><div>Dawid</div></=
div>

--0000000000008e7c8f06546f4760--