(original) (raw)


On 10/10/2016 09:36 PM, Chris Angelico wrote:
Hmm. Here's a naughty, and maybe dangerous, theory. Obtain a "memory  
deallocation lock". While it is held (by any thread - it's a guard,  
more than a lock), Py\_DECREF will not actually deallocate memory -  
objects can fall to zero references without being wiped. Once the  
lock/guard is freed/cleared, anything that had fallen to zero is now  
deallocated. This probably would mean stuffing them onto a list of  
"doomed objects", and upon release of the guard, any doomed objects  
that still have no refs would get deallocated.

If this worked, this would actually be really easy with my current "buffered reference counting" approach.� I literally already have a list of doomed objects, which a separate thread queues up for each thread to process later.

(This was a necessary part of "buffered reference counting", in which reference count changes are stored in a transaction log and later committed by a single thread.� Since there's only one thread making reference count changes, there's no contention, so it doesn't have to use atomic incr/decr, which is a big performance win.)

But I don't think this fixes the problem.� Consider:
  1. Thread A calls Q = PyList\_GetItem(L, 0), which returns a borrowed reference.� Thread A then gets suspended, before it has a chance to Py\_INCREF(Q).
  2. Thread B does L.clear(), the reference count of Q goes to 0, Q is added to the doomed objects list.
  3. Thread A gets to run again.� It does Py\_INCREF(Q); Q's refcount is now back up to 1, as if Q had been resurrected.
  4. At some point in the future the "memory deallocation lock" is released and Q is deleted.
If you say "well, just look at the reference count, and if it's 1 throw it off the doomed objects list", keep in mind that I no longer have accurate real-time reference counts.� These hacks where we play games with the reference count are mostly removed in my branch.

Also, I don't know when it would ever be safe to release the "memory deallocation lock".� Just because it's safe for your thread doesn't mean it's safe for another thread.� And if you do it on a thread-by-thread basis, in the above example it might be safe from thread B's perspective to release its "memory deallocation lock", but as illustrated that can have an effect on thread A.

I appreciate the brainstorming but I'm not currently sanguine about this idea,


/arry