Fwd: Memory problems with dar

"Petr Skoda" <[email protected]> Sat, 20 Aug 2022 15:17:11 +0200 (CEST)

Newsgroups	gmane.comp.sysutils.backup.dar.support
Message-ID	<[email protected]>

Hi Denis and all,





I have searched details about a dar memory requirements and I am not sure I 
know the answer.

In the FAQ there is a link to script dar_rqck.bash for computation of amount
of memory based on equation one file=1300bytes.

In the Limitations file you write one file requires 650bytes (and 2x more 
for differential backup) - I guess that the 1300 in the script comes from 
this logic. But you write that is is already 850 for later dar versions.




However, it is not clear whether this is about infinit, 64 or 32 bit 
integers.   I could not find a table about requirements of modern dars in 
all this modes .   Furthermore in the FAQ you write that to overcome the 
memory problem you may compile with 64 or 32 bit integers. But this is 
already the default how the dar is compiled today !




So as I understand - my dar is 64 and 16GB of RAM in computer is not 
sufficient for extracting catalogue from the dar file I have created as 
described yesterday. if I count files I have 23901645 of them which would 
suggest 23901645*1700 (I take 850x2) =

38GB !!!  (I have only 16) .

So I am hopeless - and I am not allowed to activate swap (admins has good 
reasons for it ... to keep fast array of disk) 





But is the in memory catalog needed only for extracting it (by -C ) ?  It 
seems I am not able to test archive as well.

Only listing in serial mode     seems to be working   (cat betelgeuse-data.
1.dar|dar  -l - -0)




So I would ask you for a table of memory needed for all bit modes in modern 
dars and is there a way how to do catalog extraction or testing in some 
"serial" slow mode ... 


I see only way now to send my 3TB files to the original computer which has 
128GB of RAM or do the extraction of catalogue again as a part of dar 
creation on the original computer before sending it through nc to the lto 
machne (lto has a LTO6 tape but only 16GB of RAM).




Best regards,




Petr















---------- Původní e-mail ----------
Od: Petr Skoda <[email protected]>
Komu: [email protected]
Datum: 19. 8. 2022 21:10:52
Předmět: [Dar-support] Memory problems with dar
"
Hi Denis,




I have another problems with my backup (as described recently Ihave 
transfered over nc from one machine to another 


about 3TB dar  file.  But I am not able to extract catalog or test it on 16
GB machine. After some time dar it is killed with memory overflow. The 
machine may not have swap intentionally. I am using dar 2.7.7RC1  . The dar 
-V says 


  Integer size used            : 64 bits.

(default compilation) . In FAQ you write that the compilation should be --
enable-mode=64.   Isn't it that 64 bit shown on dar -V?




Is is possible to extract the catalogue or test file in some serial mode (-
0) ? That would not cosume a lot of memory ?

If I will have more slices - would it help when transforming it in serial 
mode ?




I have seen there is now 2.7.7 available ... If I compile it somehow to 
spare memory, can I use it on the file created with 2.7.7 RC1 as above or do
I need it resent  again from the original server ?





Thanks a lot













"

(unnamed) (text/plain, 9.2 KB)

From: "Petr Skoda" <[email protected]>
To: <[email protected]>
Date: Fri, 19 Aug 2022 21:07:37 +0200 (CEST)
Message-Id: <6vA.CLs3.UjaQzRXLSn.1Y}[email protected]>
Mime-Version: 1.0 (szn-mime-2.1.21)
 
 Content preview:  Hi Denis, I have another problems with my backup (as described
    recently Ihave transfered over nc from one machine to another about 3TB dar 
    file.  But I am not able to extract catalog or test it on 16 GB machine.
    After some time dar it is killed with memory overflow. The machine may not
    have swap intentionally. I am using [...] 
 
 Content analysis details:   (-0.2 points, 6.0 required)
 
  pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -0.0 SPF_PASS               SPF: sender matches SPF record
  0.0 SPF_HELO_NONE          SPF: HELO does not publish an SPF Record
  0.0 FREEMAIL_FROM          Sender email is commonly abused enduser mail
                             provider
                             [sezskoda[at]seznam.cz]
  0.0 HTML_MESSAGE           BODY: HTML included in message
 -0.1 DKIM_VALID_EF          Message has a valid DKIM or DK signature from
                             envelope-from domain
  0.1 DKIM_SIGNED            Message has a DKIM or DK signature, not necessarily
                             valid
 -0.1 DKIM_VALID             Message has at least one valid DKIM or DK signature
 -0.1 DKIM_VALID_AU          Message has a valid DKIM or DK signature from
                             author's domain
 -0.0 T_SCC_BODY_TEXT_LINE   No description available.
X-Headers-End: 1oP7LT-0008UK-75
Subject: [Dar-support] Memory problems with dar
X-BeenThere: [email protected]
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: "need help compiling or using dar ? This mailing list is for you."
 <dar-support.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/options/dar-support>,
 <mailto:[email protected]?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=dar-support>
List-Post: <mailto:[email protected]>
List-Help: <mailto:[email protected]?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/dar-support>,
 <mailto:[email protected]?subject=subscribe>
Reply-To: "need help compiling or using dar ? This mailing list is for you."
 <[email protected]>
Content-Type: multipart/mixed; boundary="===============7960543681240749719=="
Errors-To: [email protected]

--===============7960543681240749719==
Content-Type: multipart/alternative;
	boundary="=_4e6055727cf3e2f102331cd4=9c3cabf4-4d5e-5edf-94e5-812b55935fae_="

--=_4e6055727cf3e2f102331cd4=9c3cabf4-4d5e-5edf-94e5-812b55935fae_=
Content-Type: text/plain;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable


Hi Denis,




I have another problems with my backup (as described recently Ihave 
transfered over nc from one machine to another 


about 3TB dar=C2=A0 file.=C2=A0 But I am not able to extract catalog or te=
st it on 16
GB machine. After some time dar it is killed with memory overflow. The 
machine may not have swap intentionally. I am using dar 2.7.7RC1=C2=A0 . T=
he dar 
-V says 


=C2=A0 Integer size used=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 : 64 bits.

(default compilation) . In FAQ you write that the compilation should be --=

enable-mode=3D64.=C2=A0=C2=A0 Isn't it that 64 bit shown on dar -V?




Is is possible to extract the catalogue or test file in some serial mode (=
-
0) ? That would not cosume a lot of memory ?

If I will have more slices - would it help when transforming it in serial =

mode ?




I have seen there is now 2.7.7 available ... If I compile it somehow to =

spare memory, can I use it on the file created with 2.7.7 RC1 as above or =
do
I need it resent=C2=A0 again from the original server ?





Thanks a lot













--=_4e6055727cf3e2f102331cd4=9c3cabf4-4d5e-5edf-94e5-812b55935fae_=
Content-Type: text/html;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><body><div>Hi Denis,</div><div><br></div><div>I have another problem=
s with my backup (as described recently Ihave transfered over nc from one =
machine to another <br></div><div>about 3TB dar&nbsp; file.&nbsp; But I am=
 not able to extract catalog or test it on 16GB machine. After some time d=
ar it is killed with memory overflow. The machine may not have swap intent=
ionally. I am using dar 2.7.7RC1&nbsp; . The dar -V says <br></div><div> &=
nbsp; Integer size used&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp; : 64 bits.</div><div>(default compilation) . In FAQ you wr=
ite that the compilation should be --enable-mode=3D64.&nbsp;&nbsp; Isn't i=
t that 64 bit shown on dar -V?</div><div><br></div><div>Is is possible to =
extract the catalogue or test file in some serial mode (-0) ? That would n=
ot cosume a lot of memory ?</div><div>If I will have more slices - would i=
t help when transforming it in serial mode ?</div><div><br></div><div>I ha=
ve seen there is now 2.7.7 available ... If I compile it somehow to spare =
memory, can I use it on the file created with 2.7.7 RC1 as above or do I n=
eed it resent&nbsp; again from the original server ?<br></div><div><br></d=
iv><div>Thanks a lot<br></div><div><br></div><div><br></div><div><br></div=
><div><br></div></body></html>
--=_4e6055727cf3e2f102331cd4=9c3cabf4-4d5e-5edf-94e5-812b55935fae_=--



--===============7960543681240749719==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline


--===============7960543681240749719==--