Re: Trellis-fork

Committed the thread-safety fix.

On 2009-06-23, Sergey Schetinin <[email protected]> wrote:
> Oh yeah, one more thing. We could try to use deadlock-breaking and
>  given that automatic retries are OK with Trellis, it would work. But
>  lets just not do that. :)
>
>
>  On 2009-06-23, Sergey Schetinin <[email protected]> wrote:
>  > Next thing I want to do it introduce thread safety. There's some code
>  >  to help do that in Trellis already, but I plan to remove that
>  >  implement a very simple model. If I am not mistaken, Phillip planned
>  >  for cells to have locks in their manager attribute, so that
>  >  ctrl.lock(cell) uses that just like other managers, calling __enter__
>  >  / __exit__. For locks that acquires and releases them so, given
>  >  ctrl.lock is called in all appropriate places the access to cells is
>  >  synchronous.
>  >
>  >  The problem with this is that if different cells have different locks
>  >  as their managers the deadlocks are very likely. There's no way of
>  >  telling what cells will be used in a transaction and in what order so
>  >  two concurrent transactions are very likely to get into a deadlock.
>  >
>  >  If all cells have the same lock as their manager there are no
>  >  deadlocks, but the whole ctrl.lock() business is pointless as there's
>  >  only one lock to acquire and release.
>  >
>  >  Another possible case is when there are two or more sets of cells that
>  >  form disjoint networks. All cells in each of these sets would have the
>  >  same lock. This way there's a guarantee of no deadlocks, but the only
>  >  reason to use locks in this case is if there will be concurrent
>  >  transactions on the same cell network.
>  >
>  >  I think the best tradeoff for now is a GIL-like solution -- just
>  >  disable all this cell.manager / ctrl.lock stuff and just hold the
>  >  global lock in ctrl._process. This doesn't support the last case
>  >  described above, but let's first have an actual use case for that and
>  >  see how it could be supported then. One simple solution that could
>  >  work for that is to have a controller per cell group / network with
>  >  its own lock.
>  >
>  >  Another problem is how cells store their values -- they change them
>  >  in-place and log that for undo. The problem is that if that cell is
>  >  accessed from another thread, and from *outside* a transaction the
>  >  changed value will be seen -- that's not good at all. For example see
>  >  _ReadValue.get_value implementation:
>  >
>  >     def get_value(self):
>  >         if ctrl.active:
>  >             # if atomic, make sure we're locked and consistent
>  >             used(self)
>  >         return self._value
>  >
>  >  If not atomic there's no guarantee of getting valid value which isn't
>  >  easily resolved.
>  >
>  >  One way to fix that is to store changes in a different attribute and
>  >  read that only if inside transaction. A development of this approach
>  >  is to let the controller store all of cell values. This also makes way
>  >  for properly concurrent transactions without locking because the
>  >  controller can store the consistent data snapshot in one location and
>  >  let the transactions work on a copy. Each transaction would get its
>  >  own copy so there's no need to lock at this point. At commit time
>  >  however the changes need to be merged, and because all of value
>  >  storage is handled by controller it can verify at this point if other
>  >  transactions changed any inputs used by this transaction since it
>  >  started. (For more on this see
>  >  http://code.google.com/p/trellis-fork/wiki/Transactions ) Reads from
>  >  outside transactions would just get a value from the latest committed
>  >  snapshot.
>  >
>  >  The unknown in this approach is how fast is it for single threaded
>  >  applications. I don't see any reasons for it too especially slow, but
>  >  it could be a bit slower than current implementation. It might be
>  >  faster too though. If anyone could work on performance measuring for
>  >  Trellis I'd really appreciate that.
>  >
>  >  Also, one more thing to mention in regards to thread safety -- even
>  >  with the GIL-like fix in place the controller has to be thread-local
>  >  because even while there's only one concurrent transaction possible
>  >  not all operations on controller are transactional: ctrl.active checks
>  >  in other threads will happen outside transactions and need to return
>  >  False.
>  >
>  >
>  >  On 2009-06-22, Sergey Schetinin <[email protected]> wrote:
>  >  > I've just brought the ForkChanges page up to date.
>  >  >  http://code.google.com/p/trellis-fork/wiki/ForkChanges
>  >  >
>  >  >  Any feedback on changes (also the "future changes" section) is welcome.
>  >  >
>  >  >
>  >  >  Also the "Transaction" page have grown into a quite long overview of
>  >  >  what's similar between Trellis rules and concurrent transactions and
>  >  >  some ideas on how Trellis could evolve. Still incomplete, but about
>  >  >  2/3 done, I think. If you see any mistakes in terminology or factual
>  >  >  errors, please report.
>  >  >
>  >  > http://code.google.com/p/trellis-fork/wiki/Transactions
>  >  >
>  >  >
>  >  >
>  >  >
>  >  >
>  >  > On 2009-06-17, Sergey Schetinin <[email protected]> wrote:
>  >  >  > On 2009-06-12, Sergey Schetinin <[email protected]> wrote:
>  >  >  >  > This is a thread for trellis-fork updates and discussion. At least it
>  >  >  >  >  should suffice at first.
>  >  >  >  >
>  >  >  >  >  http://code.google.com/p/trellis-fork/
>  >  >  >  >  http://pypi.python.org/pypi/Trellis-fork
>  >  >  >
>  >  >  >
>  >  >  > I've already finished quite a lot of work and test finally pass (had
>  >  >  >  to make a few edits, see commit log) and the changes to the code
>  >  >  >  structure are bigger than I imagined at first, but I'm quite happy
>  >  >  >  with the state of things overall.
>  >  >  >
>  >  >  >  Anyway, I'll work on it some more before reporting on the progress, so
>  >  >  >  if you want to watch the changes as they happen see
>  >  >  >  http://code.google.com/p/trellis-fork/updates/list
>  >  >  >  or the rss feed: http://code.google.com/feeds/p/trellis-fork/updates/basic
>  >  >  >
>  >  >  >  The actual reason for this email is that I want to get some feedback
>  >  >  >  on an article I started to write, explaining the use of transactions
>  >  >  >  in Trellis, it's work in progress, but it's something I think that can
>  >  >  >  help a lot with understanding Trellis.
>  >  >  >
>  >  >  >  http://code.google.com/p/trellis-fork/wiki/Transactions
>  >  >  >  Does it help?
>  >  >  >  Should it be more verbose or the maybe the opposite?
>  >  >  >  Any factual mistakes, typos, hints on structure, things to mention etc?
>  >  >  >
>  >  >  >
>  >  >  >
>  >  >  >
>  >  >  >  --
>  >  >  >  Best Regards,
>  >  >  >  Sergey Schetinin
>  >  >  >
>  >  >  >  http://s3bk.com/ -- S3 Backup
>  >  >  >  http://word-to-html.com/ -- Word to HTML Converter
>  >  >  >
>  >  >
>  >  >
>  >  >  --
>  >  >  Best Regards,
>  >  >  Sergey Schetinin
>  >  >
>  >  >  http://s3bk.com/ -- S3 Backup
>  >  >  http://word-to-html.com/ -- Word to HTML Converter
>  >  >
>  >
>  >
>  >
>  > --
>  >
>  > Best Regards,
>  >  Sergey Schetinin
>  >
>  >  http://s3bk.com/ -- S3 Backup
>  >  http://word-to-html.com/ -- Word to HTML Converter
>  >
>
>
>
> --
>
> Best Regards,
>  Sergey Schetinin
>
>  http://s3bk.com/ -- S3 Backup
>  http://word-to-html.com/ -- Word to HTML Converter
>


-- 
Best Regards,
Sergey Schetinin

http://s3bk.com/ -- S3 Backup
http://word-to-html.com/ -- Word to HTML Converter