Re: Citus Data + ZODB = Petabytes of data???

Le 05/10/19 =C3=A0 16:34, Joni Orponen a =C3=A9crit=C2=A0:
> On Fri, May 10, 2019 at 4:28 PM Christopher Lozinski <lozinski@specialtyj=
obmarkets.com <mailto:[email protected]>> wrote:
>=20
>     One of the limitations of the ZODB is how big the data can be.=C2=A0
>     PostrgeSQL helps, but you are still limited to the size of a single t=
able.=C2=A0
>=20
>     Usually PostgreSQL table size is limited to 16 to 64 Terabytes
>=20
>=20
> Are there even any known production ZODB installations exceeding one tera=
byte in size?

Yes, we have 2: currently 1.7TB & 2TB, growing quite fast.
And this is after compression.

A 3rd production instance was migrated to NEO recently. It powers a website=
 for ~1 million customers. The FileStorage would exceed 1TB this month; we =
packed it quite often. The NEO DB currently takes 600GB on-disk and it is f=
oreseen to exceed 10TB in a few years.

> As of about a year ago Neo, a close cousin of the ZODB, had 80 TB databas=
es in production and 160TB in test.  By now that number must be larger.  Bu=
t it is GPL.=20

Hmm, we don't have so big production DB yet, but that may be the case in ne=
ar future.

We made 2 tests:
- 100TB: https://www.nexedi.com/blog/NXD-Document.Blog.NEO.100T
- 1PB, with 10% compressible data (200TB on-disk with 1 replica, i.e. somet=
hing like RAID1 but at NEO level)

Julien

--=20
You received this message because you are subscribed to the Google Groups "=
zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to [email protected].
To view this discussion on the web visit https://groups.google.com/d/msgid/=
zodb/209668e5-dea4-3bf3-bc0c-274334f7f9a6%40nexedi.com.
For more options, visit https://groups.google.com/d/optout.