Citus Data + ZODB = Petabytes of data???
Christopher Lozinski <[email protected]> Fri, 10 May 2019 16:28:24 +0200
| Newsgroups | gmane.comp.web.zope.zodb |
|---|---|
| Message-ID | <[email protected]> |
--Apple-Mail=_B41EE495-8F12-4074-B13A-B8298CEA2A6B Content-Type: text/plain; charset="UTF-8" One of the limitations of the ZODB is how big the data can be. PostrgeSQL helps, but you are still limited to the size of a single table. Usually PostgreSQL table size is limited to 16 to 64 Terabytes https://link.springer.com/content/pdf/bbm%3A978-1-4302-0018-5%2F1.pdf <https://link.springer.com/content/pdf/bbm:978-1-4302-0018-5/1.pdf> Well now there is PostgreSQL sharding from Citus Data, now owned by Microsoft. You get a PostgreSQL api, but they split the data across multiple computers. One hashes on a single column. In this case it would be the object id, and they do the two phase commit. Do you think this will work with the ZODB? Anyone tried it? I introduced them to the Plone foundation people. Maybe they will be sponsoring that conference. Comments? Chris -- You received this message because you are subscribed to the Google Groups "zodb" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/zodb/2F249A63-9EA8-4228-8217-A01EBB0B338C%40specialtyjobmarkets.com. For more options, visit https://groups.google.com/d/optout. --Apple-Mail=_B41EE495-8F12-4074-B13A-B8298CEA2A6B Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="UTF-8" <html><body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webk= it-line-break: after-white-space;" class=3D"">One of the limitations of the= ZODB is how big the data can be. <div class=3D"">PostrgeSQL helps, bu= t you are still limited to the size of a single table. </div><div clas= s=3D""><br class=3D""></div><div class=3D"">Usually PostgreSQL table size i= s limited to 16 to 64 Terabytes</div><div class=3D""><br class=3D""></div><= div class=3D""><a href=3D"https://link.springer.com/content/pdf/bbm:978-1-4= 302-0018-5/1.pdf" class=3D"">https://link.springer.com/content/pdf/bbm%3A97= 8-1-4302-0018-5%2F1.pdf</a></div><div class=3D""><br class=3D""></div><div = class=3D"">Well now there is PostgreSQL sharding from Citus Data, now owned= by Microsoft. You get a PostgreSQL api, </div><div class=3D"">= but they split the data across multiple computers. One hashes on a si= ngle column. In this case it would be the object id, and they do the = two phase commit. </div><div class=3D""><br class=3D""></div><div clas= s=3D"">Do you think this will work with the ZODB? Anyone tried it?</d= iv><div class=3D""><br class=3D""></div><div class=3D"">I introduced them t= o the Plone foundation people. Maybe they will be sponsoring that con= ference. </div><div class=3D""><br class=3D""></div><div class=3D"">Co= mments?</div><div class=3D""><br class=3D""></div><div class=3D"">Chris</di= v><div class=3D""><br class=3D""></div><div class=3D""><br class=3D""></div= ></body></html> <p></p> -- <br /> You received this message because you are subscribed to the Google Groups &= quot;zodb" group.<br /> To unsubscribe from this group and stop receiving emails from it, send an e= mail to <a href=3D"mailto:[email protected]">zodb+unsubscri= [email protected]</a>.<br /> To view this discussion on the web visit <a href=3D"https://groups.google.c= om/d/msgid/zodb/2F249A63-9EA8-4228-8217-A01EBB0B338C%40specialtyjobmarkets.= com?utm_medium=3Demail&utm_source=3Dfooter">https://groups.google.com/d/msg= id/zodb/2F249A63-9EA8-4228-8217-A01EBB0B338C%40specialtyjobmarkets.com</a>.= <br /> For more options, visit <a href=3D"https://groups.google.com/d/optout">http= s://groups.google.com/d/optout</a>.<br /> --Apple-Mail=_B41EE495-8F12-4074-B13A-B8298CEA2A6B--