Re: Very long delay for first write to big filesystem

--===============9055122267417960744==
Content-Type: multipart/alternative; boundary=001a11343b4e1fc48404f05d5c2f

--001a11343b4e1fc48404f05d5c2f
Content-Type: text/plain; charset=ISO-8859-1

Thx Andreas.

re: block bitmaps - yes that is what I really meant. My experience with
filesystems is mainly from CPM/BDOS, where "directory" and block mapping
are essentially synonymous.

And now I understand about the timing. Makes sense when you describe it
that way.

My system is ext4, although I doubt that I used "flex_bg" option, since
this was first created awhile back. I did try to run e4defrag. It simply
said that no defrag was needed.

So, now I'm only left in  need of a work-around. Perhaps a way to have the
system load the bitmaps at boot time in the background? It would need to be
done in such a way that it would not block any other access to that system.
Or, is there a better filesystem format that would not have this problem?
(Not a really great solution, since I would need to somehow/somewhere
backup my 7.5TB system first.)

It does seem strange that this hasn't become a more serious issue, as
typical filesystems are getting bigger now. And I can't imagine a really
large network server (10TB+) having to deal with this.

Again, thx for the response.

ken


On Sat, Jan 18, 2014 at 9:09 AM, Andreas Dilger <[email protected]> wrote:

> On Jan 17, 2014, at 9:32, Ken Bass <[email protected]> wrote:
> >
> > The problem/issue: there is a very long delay when my system does a
> write to the filesystem. The delay now is over 5 minutes (yes: minutes).
> This only happens on the first write after booting up the system, and only
> for large files - 1GB or more. This can be a serious problem since all
> access to any hard disk is blocked and will hang until the first write
> begins again.
> >
> > The prevailing thought at the time was this was associated with loading
> into memory the directory information looking for free space, which I would
> believe now.
>
> It isn't actually directory information that is being loaded, but rather
> the
> block bitmaps from each group, and each one needs a seek to read.
> This will take up to 7.5 TB / 128 MB/group / 100 seeks/sec = 600s
> if the filesystem is nearly full. After this point, the bitmaps are cached
> In memory and allocation is faster.
>
> > The filesystem in question is 7.5TB, with about 4TB used. There are over
> 250,000 files. I also have another system with 1TB total and 400GB used,
> with 65,000 files. This system, the smaller one, is beginning to show
> delays as well, although only a few seconds.
> >
> > This problem seems to involve several factors: the total size of the
> system; the current "fragmentation" of that system; and finally the amount
> of physical memory available.
> >
> > As to the last factor, the 7.5TB system has only 2GB of memory (I didn't
> think that it would need a lot since it is mostly being used as a file
> server). The "fragmentation" factor (I am only guessing here) occurs with
> having many files written and deleted over time.
> >
> > So my questions are: is there a solution or work around for this; and is
> this a bug, or perhaps an undesirable feature. If the latter, should this
> be reported (somewhere)?
>
> You might consider mounting the filesystem as ext4 instead of ext3.
> It will do a slightly better job of finding contiguous free space
> and avoid loading bitmaps that do not have enough space, but the
> physics of seeking to read bitmaps is still the same.
>
> If you format a new filesystem as ext4 (as opposed to just mounting the
> existing filesystem as ext4) you can use a new feature "flex_bg" that
> locates the block and inode bitmaps together so that they can be read
> without so much seeking. You'd need a spare disk to format and copy
> the data over to.
>
> Using ext4 is also more resistant to fragmentation over time.
>
> Cheers, Andreas
>
> > Any suggestions, tips, etc. greatly appreciated.
> >
> > TIA
> >
> > ken
> >
> > _______________________________________________
> > Ext3-users mailing list
> > [email protected]
> > https://www.redhat.com/mailman/listinfo/ext3-users
>

--001a11343b4e1fc48404f05d5c2f
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Thx Andreas.<br><br><div class=3D"gmail_extra">re: block b=
itmaps - yes that is what I really meant. My experience with filesystems is=
 mainly from CPM/BDOS, where &quot;directory&quot; and block mapping are es=
sentially synonymous.<br>
<br></div><div class=3D"gmail_extra">And now I understand about the timing.=
 Makes sense when you describe it that way.<br><br></div><div class=3D"gmai=
l_extra">My system is ext4, although I doubt that I used &quot;flex_bg&quot=
; option, since this was first created awhile back. I did try to run e4defr=
ag. It simply said that no defrag was needed.<br>
<br></div><div class=3D"gmail_extra">So, now I&#39;m only left in=A0 need o=
f a work-around. Perhaps a way to have the system load the bitmaps at boot =
time in the background? It would need to be done in such a way that it woul=
d not block any other access to that system. Or, is there a better filesyst=
em format that would not have this problem? (Not a really great solution, s=
ince I would need to somehow/somewhere backup my 7.5TB system first.)<br>
<br></div><div class=3D"gmail_extra">It does seem strange that this hasn&#3=
9;t become a more serious issue, as typical filesystems are getting bigger =
now. And I can&#39;t imagine a really large network server (10TB+) having t=
o deal with this.<br>
<br></div><div class=3D"gmail_extra">Again, thx for the response.<br><br>ke=
n<br><br></div><div class=3D"gmail_extra"><br></div><div class=3D"gmail_ext=
ra"><div class=3D"gmail_quote">On Sat, Jan 18, 2014 at 9:09 AM, Andreas Dil=
ger <span dir=3D"ltr">&lt;<a href=3D"mailto:[email protected]" target=3D"_b=
lank">[email protected]</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div class=3D"im">On Jan 17, 2014, at 9:32, =
Ken Bass &lt;<a href=3D"mailto:[email protected]">[email protected]</a>=
&gt; wrote:<br>

&gt;<br>
&gt; The problem/issue: there is a very long delay when my system does a wr=
ite to the filesystem. The delay now is over 5 minutes (yes: minutes). This=
 only happens on the first write after booting up the system, and only for =
large files - 1GB or more. This can be a serious problem since all access t=
o any hard disk is blocked and will hang until the first write begins again=
.<br>

&gt;<br>
&gt; The prevailing thought at the time was this was associated with loadin=
g into memory the directory information looking for free space, which I wou=
ld believe now.<br>
<br>
</div>It isn&#39;t actually directory information that is being loaded, but=
 rather the<br>
block bitmaps from each group, and each one needs a seek to read.<br>
This will take up to 7.5 TB / 128 MB/group / 100 seeks/sec =3D 600s<br>
if the filesystem is nearly full. After this point, the bitmaps are cached<=
br>
In memory and allocation is faster.<br>
<div class=3D"im"><br>
&gt; The filesystem in question is 7.5TB, with about 4TB used. There are ov=
er 250,000 files. I also have another system with 1TB total and 400GB used,=
 with 65,000 files. This system, the smaller one, is beginning to show dela=
ys as well, although only a few seconds.<br>

&gt;<br>
&gt; This problem seems to involve several factors: the total size of the s=
ystem; the current &quot;fragmentation&quot; of that system; and finally th=
e amount of physical memory available.<br>
&gt;<br>
&gt; As to the last factor, the 7.5TB system has only 2GB of memory (I didn=
&#39;t think that it would need a lot since it is mostly being used as a fi=
le server). The &quot;fragmentation&quot; factor (I am only guessing here) =
occurs with having many files written and deleted over time.<br>

&gt;<br>
&gt; So my questions are: is there a solution or work around for this; and =
is this a bug, or perhaps an undesirable feature. If the latter, should thi=
s be reported (somewhere)?<br>
<br>
</div>You might consider mounting the filesystem as ext4 instead of ext3.<b=
r>
It will do a slightly better job of finding contiguous free space<br>
and avoid loading bitmaps that do not have enough space, but the<br>
physics of seeking to read bitmaps is still the same.<br>
<br>
If you format a new filesystem as ext4 (as opposed to just mounting the<br>
existing filesystem as ext4) you can use a new feature &quot;flex_bg&quot; =
that<br>
locates the block and inode bitmaps together so that they can be read<br>
without so much seeking. You&#39;d need a spare disk to format and copy<br>
the data over to.<br>
<br>
Using ext4 is also more resistant to fragmentation over time.<br>
<br>
Cheers, Andreas<br>
<div class=3D"im"><br>
&gt; Any suggestions, tips, etc. greatly appreciated.<br>
&gt;<br>
&gt; TIA<br>
&gt;<br>
&gt; ken<br>
&gt;<br>
</div>&gt; _______________________________________________<br>
&gt; Ext3-users mailing list<br>
&gt; <a href=3D"mailto:[email protected]">[email protected]</a><br>
&gt; <a href=3D"https://www.redhat.com/mailman/listinfo/ext3-users" target=
=3D"_blank">https://www.redhat.com/mailman/listinfo/ext3-users</a><br>
</blockquote></div><br></div></div>

--001a11343b4e1fc48404f05d5c2f--


--===============9055122267417960744==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Ext3-users mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/ext3-users
--===============9055122267417960744==--