Re: lucene3/lucene4 jenkins boxes - sha1 git errors

--000000000000f10f00065482ab70
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

https://ci-builds.apache.org/job/Lucene/job/Lucene-Check-main/

Seems like cleaning up disk space have helped!

D.



On Wed, Jun 17, 2026 at 11:58=E2=80=AFAM Michael Sokolov <[email protected]=
m> wrote:

> Thanks for digging into this, Dawid
>
> On Wed, Jun 17, 2026, 4:59=E2=80=AFAM Dawid Weiss <[email protected]>=
 wrote:
>
>>
>> Hello everyone,
>>
>> I cross-post to Solr and Lucene dev lists because we share ASF jenkins
>> boxes that
>> execute jenkins workflows.
>>
>> There has been a series of odd exceptions from Jenkins, looking pretty
>> much like this:
>>
>> ...
>> > git reset --hard # timeout=3D10
>> ERROR: Error fetching remote repo 'origin'
>> hudson.plugins.git.GitException: Failed to fetch from
>> https://github.com/apache/lucene.git
>> ...
>> Caused by: hudson.plugins.git.GitException: Command "git reset --hard"
>> returned status code 128:
>> stdout:
>> stderr: error: unable to read sha1 file of .asf.yaml
>> (48083930a50d886827c1f87e23875e5da98c58c6)
>> error: unable to read sha1 file of .dir-locals.el
>> (c51e1232603b85b8bc74fbed7d1de08186920379)
>> error: unable to read sha1 file of .git-blame-ignore-revs
>> (945e687d0f3cdfcc99cb69b8b35ab631a70371a7)
>> error: unable to read sha1 file of .gitattributes
>> (a3135003e80fa8f49fc0f2250f40b85cde12ebc5)
>> ...
>>
>> I've ssh'd to lucene3/lucene4 and it seems like something leaves the .gi=
t
>> folder of a workspace checkout in a broken state, leading to subsequent =
git
>> reset failures. I'm not sure if it's a jenkins task timeout kicking in o=
r
>> something else.
>>
>> There seems to be enough drive space on both boxes, although it's close
>> to full. Looking at asf INFRA jira, the same type of exception was alway=
s
>> caused by disk space running low so for now I've:
>>
>> - removed all existing workspaces under
>> /home/jenkins/jenkins-agent/workspace/Lucene
>> - removed all existing workspaces under
>> /home/jenkins/jenkins-agent/workspace/Solr
>>
>> I didn't know how to pause jenkins for this, apologies if you get an
>> error from a job that was running at the moment I removed those folders.
>>
>> Oddly enough, I can't see the same error appearing on Solr jenkins runs =
-
>> could be that all the jobs there have a "wipe workspace before you start=
"
>> checkbox enabled... I'm really not sure, sorry.
>>
>> Right now, df -h shows:
>>
>> lucene3 /dev/sda2 503G 59G 419G 13% /
>> lucene4 /dev/sda2 503G 99G 379G 21% /
>>
>> Let's see if this fixes the problem.
>>
>> Dawid
>>
>

--000000000000f10f00065482ab70
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><br></div><div><a href=3D"https://ci-builds.apache.or=
g/job/Lucene/job/Lucene-Check-main/">https://ci-builds.apache.org/job/Lucen=
e/job/Lucene-Check-main/</a></div><div><br></div><div>Seems like cleaning u=
p disk space have helped!</div><div><br></div><div>D.</div><div><br></div><=
div><br></div></div><br><div class=3D"gmail_quote gmail_quote_container"><d=
iv dir=3D"ltr" class=3D"gmail_attr">On Wed, Jun 17, 2026 at 11:58=E2=80=AFA=
M Michael Sokolov &lt;<a href=3D"mailto:[email protected]">msokolov@gmail.=
com</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"marg=
in:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1e=
x"><div dir=3D"auto">Thanks for digging into this, Dawid</div><br><div clas=
s=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Jun 17, 202=
6, 4:59=E2=80=AFAM Dawid Weiss &lt;<a href=3D"mailto:[email protected]"=
 target=3D"_blank">[email protected]</a>&gt; wrote:<br></div><blockquot=
e class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px s=
olid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><br><div>Hello eve=
ryone,</div><div><br></div><div>I cross-post to Solr and Lucene dev lists b=
ecause we share ASF jenkins boxes that</div><div>execute jenkins workflows.=
</div><div><br></div><div>There has been a series of odd exceptions from Je=
nkins, looking pretty much like this:</div><div><br></div><div>...</div><di=
v>&gt; git reset --hard # timeout=3D10<br>ERROR: Error fetching remote repo=
 &#39;origin&#39;<br>hudson.plugins.git.GitException: Failed to fetch from =
<a href=3D"https://github.com/apache/lucene.git" rel=3D"noreferrer" target=
=3D"_blank">https://github.com/apache/lucene.git</a><br></div><div>...</div=
><div><span style=3D"background-color:transparent">Caused by: hudson.plugin=
s.git.GitException: Command &quot;git reset --hard&quot; returned status co=
de 128:</span><br></div><div>stdout: <br>stderr: error: unable to read sha1=
 file of .asf.yaml (48083930a50d886827c1f87e23875e5da98c58c6)<br>error: una=
ble to read sha1 file of .dir-locals.el (c51e1232603b85b8bc74fbed7d1de08186=
920379)<br>error: unable to read sha1 file of .git-blame-ignore-revs (945e6=
87d0f3cdfcc99cb69b8b35ab631a70371a7)<br>error: unable to read sha1 file of =
.gitattributes (a3135003e80fa8f49fc0f2250f40b85cde12ebc5)<br>...</div><div>=
<br></div><div>I&#39;ve ssh&#39;d to lucene3/lucene4 and it seems like some=
thing leaves the .git folder of a workspace checkout in a broken state, lea=
ding to subsequent git reset failures. I&#39;m not sure if it&#39;s a jenki=
ns task timeout kicking in or something else.<br><br>There seems to be enou=
gh drive space on both boxes, although it&#39;s close to full. Looking at a=
sf INFRA jira, the same type of exception was always caused by disk space r=
unning low so for now=C2=A0<span style=3D"background-color:transparent">I&#=
39;ve:</span></div><div><br>- removed all existing workspaces under /home/j=
enkins/jenkins-agent/workspace/Lucene<br></div><div>- removed all existing =
workspaces under /home/jenkins/jenkins-agent/workspace/Solr</div><div><br><=
/div><div>I didn&#39;t know how to pause jenkins for this, apologies if you=
 get an error from a job that was running at the moment I removed those fol=
ders.</div><div><br></div><div>Oddly enough, I can&#39;t see the same error=
 appearing on Solr jenkins runs - could be that all the jobs there have a &=
quot;wipe workspace before you start&quot; checkbox enabled... I&#39;m real=
ly not sure, sorry.</div><div><br></div><div>Right now, df -h shows:</div><=
div><br></div><div><div style=3D"font-family:monospace;white-space:pre-wrap=
">lucene3
/dev/sda2       503G   59G  419G  13% /
<br></div><div style=3D"font-family:monospace;white-space:pre-wrap">lucene4
/dev/sda2       503G   99G  379G  21% /</div></div><div><br></div><div>Let&=
#39;s see if this fixes the problem.</div><div><br></div><div>Dawid</div></=
div>
</blockquote></div>
</blockquote></div>

--000000000000f10f00065482ab70--