Re: Re: asyncio-based ZEO implementation implemented

On Tue, Jun 21, 2016 at 4:10 AM, Christopher Lozinski
<[email protected]> wrote:
>
>
> On Sunday, June 19, 2016 at 9:12:45 PM UTC+2, Jim Fulton wrote:
>>
>>
>> This is a large pull request, so I get why nobody might want to review
>> it.
>
>
> So how does one review ZEO?  Well the first step is to understand
> FileStorage.

a) The text you quoted above was about reviewing a pull request, not ZEO.

b) The pull request didn't touch FileStorage because FileStorage isn't
part of ZEO.

Of course, knowledge of ZODB broadly is necessary.

...

> Let me ask the obvious question.  If the Python GIl prevents multiple
> threads from executing at the same time,
> why does filestorage use multiple threads?  Would it not be easier to just
> have a single thread.

The GIL prevents multiple threads from executing Python code at the
same time.  It doesn't prevent multiple threads from executing C code
at the same time.

When this was written, a common configuration was a RAID
configurations, with multiple read heads that could be moved
independently.  At the time, seeking/moving a read head could be very
expensive.  Reading in parallel could be a big win.

As an aside, this is an effect which could only be measured with
benchmarks against large databases (like the ones we were working
with).

Today, with the prevalence of SSDs and networked file systems (e.g.
EBS) it's not clear that this still provides and advantage. The only
way to be sure would be to do a realistic test with a large database
and realistic access patterns.

> Answer: The problem is with the OS.  It blocks on writing to a file.

The issue wasn't writing, it was reading. The OS blocks in both cases.
At the time, time to seek in preparation for a read could take
milliseconds (and eternity IOW).

>  So it
> makes sense to have a separate thread
> for writing transactions on Data.fs

It wouldn't matter as we block waiting for the write to complete
anyway.  We currently get an exclusive lock when writing. This is
probably unnecessary and will likely be relaxed, especially with the
change to always use loadBefore.


> and on the lock file.

We never write to the lock file after acquiring the lock when the
storage is opened.

> Question: So why does ZODB filestorage support multiple threads?

To allow parallel I/O operations, which are mostly not restricted by the GIL.

>  Is it so
> that every transaction gets its own thread with its own cache, and version
> of objects?

It's because clients of storages are typically multi-threaded.

Note that at ZC we had database servers that often used ~200% CPU,
meaning that they were using multiple cores. This was possible because
a lot of work was done in C code unhindered by the GIL.

None of this has anything to do with this PR BTW.

Jim

-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.