Re: Exploring using Rust for implementing a faster ZEO server.

On Sun, Sep 25, 2016 at 12:58 PM, Jason Madden
<[email protected]> wrote:
>
>> On Sep 25, 2016, at 11:30, Jim Fulton <[email protected]> wrote:
>>
>> I'm exploring using Rust to implement a faster ZEO server.  To do a
>> quick performance comparison, I implemented a file-storage iterator in
>> Rust. It's about 20x faster (40x excluding system time) than the
>> Python version.
>>
>> Here's a write-up:
>>
>> http://jimfulton.info/site/2016/Sep/25/experiment-compare-zodb-file-storage-iteration-with-python-and-rust/
>>
>
> Interesting results!
>
> I'm very curious where PyPy 5.4 falls in the spectrum. Is the file-storage file you used available somewhere to test with?

No. It has some sensitive information in it. But it was handy. :)

> Or if not, would you consider adding a PyPy column to the results table? (The iteration may need to be run a few times for the JIT to fully kick in.)

Is the JIT compilation cached somewhere? I assume not. OTOH, I suppose
it would kick in early enough in the run, which iterates over 4452
transactions and 190689 records.  The real, user, and system times for
PyPy2 v5.4.1 were: 1.3, 1.1, and .236 seconds, which is almost twice
as fast as ordinary Python 2, but still much slower than Rust.

I had to run the Python version with Python 2, because we decided to
use different FileStorage magic strings for Python 2 and Python 3 for
some reason, even though the file formats are the same.

>
> The obvious consequence of writing a ZEO server in another language is that it's not running Python code. This implies (1) client-side conflict resolution

which ZEO 5 has.

> and (2) no "stackable" storages, only built-in storages.

Right. My plan was to only have FileStorage 2, at least initially.

https://github.com/jimfulton/filestorage2

> For example, you'd lose zc.zlibstorage and cipher.encrypingstorage---but those are only needed on the server to do conflict resolution, I think.

Yup.  And also garbage collection, which I would hope to support via
external GC.

I also plan to update ZEO to optionally support using msgpack rather
than pickle in the network protocol. This will allow non-Python
servers, and would allow Python 3 servers for Python 2 clients,
assuming that client-side conflict resolution and external GC are
used.

> The only other commonly used ZEO stackable storage I'm aware of is ZRS[1]. That could be built in,

Yup.

> but looks to be a sizable code base.

<shrug> The core isn't that big. It's certainly smaller and much
simpler that the ZEO server itself.  I expect I'll be able to reuse
existing Python ZEO integration-test code.

Jim

-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.