Re: how to split backend storages of ZEO

On Thu, Aug 18, 2016 at 11:55 AM, Sanjay Rao <[email protected]>
wrote:

> I am trying
>
> How to serve multiple file storages from a single ZEO server (and
> reference them from a single cllient application with multiple nodes)
>

OK. ZEO storage servers can serve multiple storages. I consider this a bit
of a misfeature.

In the storage server configuration, you can define multiple storages. In
the storage tags, you can include a name, like this:

   <filestorage SOMENAME>
      path /var/tmp/Data.fs
   </filestorage>

(The default name is "1".)

When defining a client storage, you use the storage option, as in:

      <zeoclient>
          server SERVERADDR
          storage SOMENAME
      </zeoclient>

If an application is using multiple databases, then you should be able use
multiple zodb configuration tags in the same configuration file/string.

   <zodb SOMENAME>
      <zeoclient>
          server SERVERADDR
          storage SOMENAME
      </zeoclient>
   </zodb>

   <zodb OTHERNAME>
      <zeoclient>
          server SERVERADDR
          storage OTHERNAME
      </zeoclient>
   </zodb>

Whether this works for your application may depend on how it manages it's
ZConfig.

Personally, I typically have ZODB configs separate from other application
configs and use ZODB.config.databasesFromString to load the configuration.
This creates collection of linked databases, and returns the first database
defined.  The other databases are accessible via the first database's
databases attribute.  To use the databases, open a connection on one of the
databases and then use that connection's get_connection method to get other
linked connections.  Done of this is documented.  (Documentation is going
to be one of my next big projects.)

But I consider multi-database configurations an advanced feature that
should only be used when the need is great. For example, when using a
multi-database configuration, you need to be much more careful about
garbage collection.



>
> I want to store some blobs in database and application will keep deleting
> these blobs and then creating new blobs. Application can create/delete 100s
> of blobs per minute. I do not want to store these blobs in main database to
> avoid frequent packing.
>

I'm not sure how this follows. I assume by blobs, you mean ZODB.blob.Blob
objects, not big strings.



> plus we want to store settings and records in separate storage. In case of
> database corruptions, this will let our customers start their system again
> with settings storage only.usually customers are OK if their records are
> gone but their workflow is live again.
>

I haven't seen database corruption in many years.  At ZC, when we bought
decent hardware, we didn't have a problem with corruption, and we didn't
experience any corruption issues using AWS Elastic Block Storage. Have you
had a problem with database corruption?



>
> on another side,
> Will this design give a better write performance of DB ?
>

No, especially if all of the storages are served by a single server.

There's a potential to have a performance benefit if storages are served by
separate servers and if most transactions only access one of the storages.

From an operational point of view, I prefer to manage separate storages in
separate storages in separate storage servers, as it makes deployment and
management easier to automate.

Jim

-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.