Re: How to connect to backblaze?
Scott Hannahs via Duplicity-talk <[email protected]> Mon, 29 Sep 2025 16:55:01 -0400
| Newsgroups | gmane.comp.sysutils.backup.duplicity.general |
|---|---|
| Message-ID | <[email protected]> |
--===============7575997454329470476== Content-Type: multipart/alternative; boundary="Apple-Mail=_B593A03F-EBB4-4D42-B480-BED4E6999230" --Apple-Mail=_B593A03F-EBB4-4D42-B480-BED4E6999230 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Ede, I have boto3 version 1.34.144 installed I am using duplicity version 3.0.4 I think that is the difference where I am accessing the Dreamhost with = the old style S3 with the bucket name in the target and trying to change = to the bucket name in the URL. But the obvious solution is not working? That should be the first command version: nice -n19 /opt/sw/bin/duplicity backup --full-if-older-than 49D = --num-retries 5 --tempdir /var/tmp/duplicity --volsize 500 --encrypt-key = XXXXXXXX --sign-key YYYYYYYY --gpg-binary /opt/sw/bin/gpg --force = --verbosity Notice --s3-multipart-chunk-size 500 --s3-endpoint-url = https://<BUCKETNAME>.s3.us-east-005.backblazeb2.com = <http://s3.us-east-005.backblazeb2.com/> <FOLDER TO BACKUP> = s3://<FOLDERNAME> So I put the host in the s3-endpoint-url parameter and the the target of = s3://<FOLDERNAME> I have the .boto configuration file with the =E2=80=9Caws_access_key_id=E2= =80=9D and =E2=80=9Caws_secret_access_key=E2=80=9D set. One question I have is if I use Backblaze for an S3 backup and Dreamhost = for an S3 backup I am exporting a different BOTO_CONFIG variable for = each with the hopefully correct Key and Secret Key. I am getting a warning about another backend, but I think that isn=E2=80=99= t the problem. Import of duplicity.backends._testbackend Failed Is there anyway to test the boto connection and see how it is failing. = If I move the config file I get a warning about no such config file so = it seems to be finding it. But the =E2=80=9CHeadBucket=E2=80=9D = operation is failing? Am I not using the correct target? I am using s3://<directory> but = s3:<directory> complains it is not a URL. Even though the bucket name = is in the URL, should it be in the target specification as well? That help on the Boto3 issue needs some examples? -Scott > On Sep 29, 2025, at 11:03=E2=80=AFAM, edgar.soldin--- via = Duplicity-talk <[email protected]> wrote: >=20 > assuming you use a recent duplicity 3.x . we retired the old s3/boto = backend as it was deprecated long ago and switched to boto3 as default. = that changed the parameter syntax. read > https://duplicity.us/stable/duplicity.1.html#a-note-on-amazon-s3 >=20 > main difference boto3:// does not want a host in the url but as extra = parameter instead. >=20 > does that help? ..ede/duply.net <http://duply.net/> >=20 > On 9/29/2025 4:49 PM, Scott Hannahs via Duplicity-talk wrote: >> I am trying to switch cloud providers from Dreamhost/Wasabi to = BackBlaze due to the pricing structure being much better. >> The Backblaze B2 storage is S3 compliant and I think Dreamhost is = just repackaaging it for their DreamObjects and adding an extra = surcharge. >> it uses a global bucket naming scheme in the domain name but I = can=E2=80=99t get the S3 backend to work. I gather that the B2 backend = is no longer supported? The S3 should be successful here AFAICT. >> The domain <BUCKETNAME>.s3.us-east-005.backblazeb2.com = <http://s3.us-east-005.backblazeb2.com/> = <http://s3.us-east-005.backblazeb2.com = <http://s3.us-east-005.backblazeb2.com/>> does exist and I can ping it. >> My command line (minus a bunch of exclude statements: >> nice -n19 /opt/sw/bin/duplicity backup --full-if-older-than 49D = --num-retries 5 --tempdir /var/tmp/duplicity --volsize 500 --encrypt-key = XXXXXXXX --sign-key YYYYYYYY --gpg-binary /opt/sw/bin/gpg --force = --verbosity Notice --s3-multipart-chunk-size 500 --s3-endpoint-url = https://<BUCKETNAME>.s3.us-east-005.backblazeb2.com <FOLDER TO BACKUP> = s3://<FOLDERNAME> >> I have also tried the following >> nice -n19 /opt/sw/bin/duplicity backup --full-if-older-than 49D = --num-retries 5 --tempdir /var/tmp/duplicity --volsize 500 --encrypt-key = XXXXXXXX --sign-key YYYYYYYY --gpg-binary /opt/sw/bin/gpg --force = --verbosity Notice --s3-multipart-chunk-size 500 --s3-endpoint-url = https://s3.us-east-005.backblazeb2.com <FOLDER TO BACKUP> = s3://<BUCKETNAME>/<FOLDERNAME> >> I get the error: >> ClientError: An error occurred (403) when calling the HeadBucket = operation: Forbidden for both. >> If I use Verbosity=3DDebug >> Backtrace of previous error: Traceback (innermost last): >> File = "/opt/sw/lib/python3.10/site-packages/duplicity/backend.py", line 375, = in inner_retry >> return fn(self, *args) >> File = "/opt/sw/lib/python3.10/site-packages/duplicity/backend.py", line 631, = in list >> return [tobytes(x) for x in self.backend._list()] >> File = "/opt/sw/lib/python3.10/site-packages/duplicity/backends/s3_boto3_backend.= py", line 209, in _list >> self.reset_connection() >> File = "/opt/sw/lib/python3.10/site-packages/duplicity/backends/s3_boto3_backend.= py", line 105, in reset_connection >> self.s3.meta.client.head_bucket(Bucket=3Dself.bucket_name) >> File "/opt/sw/lib/python3.10/site-packages/botocore/client.py", = line 565, in _api_call >> return self._make_api_call(operation_name, kwargs) >> File "/opt/sw/lib/python3.10/site-packages/botocore/client.py", = line 1021, in _make_api_call >> raise error_class(parsed_response, operation_name) >> botocore.exceptions.ClientError: An error occurred (403) when = calling the HeadBucket operation: Forbidden >> This seems to work with the older Dreamhost S3 storage so there is = something in my setup that is wrong? Any advice from someone = successfullly using Backblaze would be greatly appreciated. >> -Scott >> _______________________________________________ >> Duplicity-talk mailing list >> [email protected] >> https://lists.nongnu.org/mailman/listinfo/duplicity-talk >=20 >=20 > _______________________________________________ > Duplicity-talk mailing list > [email protected] <mailto:[email protected]> > https://lists.nongnu.org/mailman/listinfo/duplicity-talk --Apple-Mail=_B593A03F-EBB4-4D42-B480-BED4E6999230 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"content-type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"overflow-wrap: break-word; = -webkit-nbsp-mode: space; line-break: = after-white-space;">Ede,<div><br></div><div>I have boto3 = version 1.34.144 installed<div>I am using duplicity version = 3.0.4</div><div><br></div><div>I think that is the difference where I am = accessing the Dreamhost with the old style S3 with the bucket name in = the target and trying to change to the bucket name in the URL. But = the obvious solution is not working?</div><div><br></div><div>That = should be the first command version:</div><blockquote style=3D"margin: 0 = 0 0 40px; border: none; padding: 0px;"><div><span style=3D"caret-color: = rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Menlo;">nice -n19 = /opt/sw/bin/duplicity backup --full-if-older-than 49D --num-retries 5 = --tempdir /var/tmp/duplicity --volsize 500 --encrypt-key XXXXXXXX = --sign-key YYYYYYYY --gpg-binary /opt/sw/bin/gpg --force --verbosity = Notice --s3-multipart-chunk-size 500 --s3-endpoint-url = https://<b><u><BUCKETNAME>.</u></b></span><a = href=3D"http://s3.us-east-005.backblazeb2.com/" style=3D"font-family: = Menlo;"><b>s3.us-east-005.backblazeb2.com</b></a><span = style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: = Menlo;"> <FOLDER TO BACKUP> = <u><b>s3://<FOLDERNAME></b></u></span></div></blockquote><div><div><= br></div><div>So I put the host in the <span style=3D"caret-color: = rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Menlo;">s3-endpoint-url = parameter and the the target of </span><u style=3D"caret-color: = rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: = Menlo;"><b>s3://<FOLDERNAME></b></u></div><div><br></div><div>I = have the .boto configuration file with the =E2=80=9Caws_access_key_id=E2=80= =9D and =E2=80=9Caws_secret_access_key=E2=80=9D = set.</div><div><br></div><div>One question I have is if I use Backblaze = for an S3 backup and Dreamhost for an S3 backup I am exporting a = different BOTO_CONFIG variable for each with the hopefully correct Key = and Secret Key.</div><div><br></div><div>I am getting a warning about = another backend, but I think that isn=E2=80=99t the = problem.</div><div><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>Import of = duplicity.backends._testbackend Failed</div><div><br></div><div>Is there = anyway to test the boto connection and see how it is failing. If I = move the config file I get a warning about no such config file so it = seems to be finding it. But the =E2=80=9CHeadBucket=E2=80=9D = operation is failing?</div><div><br></div><div>Am I not using the = correct target? I am using s3://<directory> but = s3:<directory> complains it is not a URL. Even though the = bucket name is in the URL, should it be in the target specification as = well?</div><div><br></div><div>That help on the Boto3 issue needs some = examples?</div><div><br></div><div>-Scott</div><div><br></div><div><br><bl= ockquote type=3D"cite"><div>On Sep 29, 2025, at 11:03=E2=80=AFAM, = edgar.soldin--- via Duplicity-talk <[email protected]> = wrote:</div><br class=3D"Apple-interchange-newline"><div><meta = charset=3D"UTF-8"><span style=3D"caret-color: rgb(0, 0, 0); font-family: = LucidaGrande; font-size: 14px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">assuming you use a = recent duplicity 3.x . we retired the old s3/boto backend as it was = deprecated long ago and switched to boto3 as default. that changed the = parameter syntax. read</span><br style=3D"caret-color: rgb(0, 0, 0); = font-family: LucidaGrande; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: LucidaGrande; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline = !important;"> </span><a = href=3D"https://duplicity.us/stable/duplicity.1.html#a-note-on-amazon-s3" = style=3D"font-family: LucidaGrande; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: = 0px;">https://duplicity.us/stable/duplicity.1.html#a-note-on-amazon-s3</a>= <br style=3D"caret-color: rgb(0, 0, 0); font-family: LucidaGrande; = font-size: 14px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><br style=3D"caret-color: rgb(0, 0, 0); font-family: = LucidaGrande; font-size: 14px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span style=3D"caret-color: rgb(0, 0, 0); font-family: = LucidaGrande; font-size: 14px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">main difference boto3:// = does not want a host in the url but as extra parameter = instead.</span><br style=3D"caret-color: rgb(0, 0, 0); font-family: = LucidaGrande; font-size: 14px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><br style=3D"caret-color: rgb(0, 0, 0); font-family: = LucidaGrande; font-size: 14px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span style=3D"caret-color: rgb(0, 0, 0); font-family: = LucidaGrande; font-size: 14px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">does that help? = ..ede/</span><a href=3D"http://duply.net/" style=3D"font-family: = LucidaGrande; font-size: 14px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: = 0px;">duply.net</a><br style=3D"caret-color: rgb(0, 0, 0); font-family: = LucidaGrande; font-size: 14px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><br style=3D"caret-color: rgb(0, 0, 0); font-family: = LucidaGrande; font-size: 14px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span style=3D"caret-color: rgb(0, 0, 0); font-family: = LucidaGrande; font-size: 14px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">On 9/29/2025 4:49 PM, = Scott Hannahs via Duplicity-talk wrote:</span><br style=3D"caret-color: = rgb(0, 0, 0); font-family: LucidaGrande; font-size: 14px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><blockquote type=3D"cite" style=3D"font-family: = LucidaGrande; font-size: 14px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;">I am trying to switch cloud providers from = Dreamhost/Wasabi to BackBlaze due to the pricing structure being much = better.<br>The Backblaze B2 storage is S3 compliant and I think = Dreamhost is just repackaaging it for their DreamObjects and adding an = extra surcharge.<br>it uses a global bucket naming scheme in the domain = name but I can=E2=80=99t get the S3 backend to work. I gather that = the B2 backend is no longer supported? The S3 should be successful = here AFAICT.<br>The domain <BUCKETNAME>.<a = href=3D"http://s3.us-east-005.backblazeb2.com/">s3.us-east-005.backblazeb2= .com</a><span class=3D"Apple-converted-space"> </span><<a = href=3D"http://s3.us-east-005.backblazeb2.com/">http://s3.us-east-005.back= blazeb2.com</a>> does exist and I can ping it.<br>My command = line (minus a bunch of exclude statements:<br> nice = -n19 /opt/sw/bin/duplicity backup --full-if-older-than 49D --num-retries = 5 --tempdir /var/tmp/duplicity --volsize 500 --encrypt-key XXXXXXXX = --sign-key YYYYYYYY --gpg-binary /opt/sw/bin/gpg --force --verbosity = Notice --s3-multipart-chunk-size 500 --s3-endpoint-url = https://<BUCKETNAME>.s3.us-east-005.backblazeb2.com <FOLDER TO = BACKUP> s3://<FOLDERNAME><br>I have also tried the = following<br> nice -n19 /opt/sw/bin/duplicity backup = --full-if-older-than 49D --num-retries 5 --tempdir /var/tmp/duplicity = --volsize 500 --encrypt-key XXXXXXXX --sign-key YYYYYYYY --gpg-binary = /opt/sw/bin/gpg --force --verbosity Notice --s3-multipart-chunk-size 500 = --s3-endpoint-url https://s3.us-east-005.backblazeb2.com <FOLDER TO = BACKUP> s3://<BUCKETNAME>/<FOLDERNAME><br>I get the = error:<br> ClientError: An error occurred (403) when = calling the HeadBucket operation: Forbidden for both.<br>If I use = Verbosity=3DDebug<br> Backtrace of previous error: = Traceback (innermost last):<br> File = "/opt/sw/lib/python3.10/site-packages/duplicity/backend.py", line 375, = in inner_retry<br> return = fn(self, *args)<br> File = "/opt/sw/lib/python3.10/site-packages/duplicity/backend.py", line 631, = in list<br> return [tobytes(x) = for x in = self.backend._list()]<br> File = "/opt/sw/lib/python3.10/site-packages/duplicity/backends/s3_boto3_backend.= py", line 209, in _list<br> = self.reset_connection()<br>  = ;File = "/opt/sw/lib/python3.10/site-packages/duplicity/backends/s3_boto3_backend.= py", line 105, in reset_connection<br> = self.s3.meta.client.head_bucket(Bucket=3Dself.bucket_name)<br>= File = "/opt/sw/lib/python3.10/site-packages/botocore/client.py", line 565, in = _api_call<br> return = self._make_api_call(operation_name, = kwargs)<br> File = "/opt/sw/lib/python3.10/site-packages/botocore/client.py", line 1021, in = _make_api_call<br> raise = error_class(parsed_response, = operation_name)<br> botocore.exceptions.Clien= tError: An error occurred (403) when calling the HeadBucket operation: = Forbidden<br>This seems to work with the older Dreamhost S3 storage so = there is something in my setup that is wrong? Any advice from = someone successfullly using Backblaze would be greatly = appreciated.<br>-Scott<br>_______________________________________________<= br>Duplicity-talk mailing = list<br>[email protected]<br>https://lists.nongnu.org/mailman/list= info/duplicity-talk<br></blockquote><br style=3D"caret-color: rgb(0, 0, = 0); font-family: LucidaGrande; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><br style=3D"caret-color: rgb(0, 0, 0); = font-family: LucidaGrande; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: LucidaGrande; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline = !important;">_______________________________________________</span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: LucidaGrande; = font-size: 14px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span style=3D"caret-color: rgb(0, 0, 0); font-family: = LucidaGrande; font-size: 14px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">Duplicity-talk mailing = list</span><br style=3D"caret-color: rgb(0, 0, 0); font-family: = LucidaGrande; font-size: 14px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><a href=3D"mailto:[email protected]" style=3D"font-family: = LucidaGrande; font-size: 14px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: = 0px;">[email protected]</a><br style=3D"caret-color: rgb(0, 0, = 0); font-family: LucidaGrande; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><a = href=3D"https://lists.nongnu.org/mailman/listinfo/duplicity-talk" = style=3D"font-family: LucidaGrande; font-size: 14px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: = 0px;">https://lists.nongnu.org/mailman/listinfo/duplicity-talk</a></div></= blockquote></div><br></div></div></body></html>= --Apple-Mail=_B593A03F-EBB4-4D42-B480-BED4E6999230-- --===============7575997454329470476== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KRHVwbGljaXR5 LXRhbGsgbWFpbGluZyBsaXN0CkR1cGxpY2l0eS10YWxrQG5vbmdudS5vcmcKaHR0cHM6Ly9saXN0 cy5ub25nbnUub3JnL21haWxtYW4vbGlzdGluZm8vZHVwbGljaXR5LXRhbGsK --===============7575997454329470476==--