[Python-Dev] Slides from today's parallel/async Python talk (original) (raw)

Trent Nelson trent at snakebite.org
Thu Mar 14 23:23:37 CET 2013

On Thu, Mar 14, 2013 at 12:59:57PM -0700, Stefan Ring wrote:

_> Yup, in fact, if I hadn't come up with the read[gf]sword() trick, > my only other option would have been TLS (or the GetCurrentThreadId > /pthreadself() approach in the presentation). TLS is fantastic, > and it's definitely an intrinsic part of the solution (the "Y" part > of "if we're a parallel thread, do Y"), but it definitely more > costly than a simple FS/GS register read.

I think you should be able to just take the address of a static _thread variable to achieve the same thing in a more portable way.

Sure, but, uh, that's kinda' trivial in comparison to all the wildly
unportable Windows-only functionality I'm using to achieve all of
this at the moment :-)

For the record, here are all the Windows calls I'm using that have
no *direct* POSIX equivalent:

    Interlocked singly-linked lists:
        - InitializeSListHead()
        - InterlockedFlushSList()
        - QueryDepthSList()
        - InterlockedPushEntrySList()
        - InterlockedPushListSList()
        - InterlockedPopEntrySlist()

    Synchronisation and concurrency primitives:
        - Critical sections
            - InitializeCriticalSectionAndSpinCount()
            - EnterCriticalSection()
            - LeaveCriticalSection()
            - TryEnterCriticalSection()
        - Slim read/writer locks (some pthread implements have
          rwlocks)*:
            - InitializeSRWLock()
            - AcquireSRWLockShared()
            - AcquireSRWLockExclusive()
            - ReleaseSRWLockShared()
            - ReleaseSRWLockExclusive()
            - TryAcquireSRWLockExclusive()
            - TryAcquireSRWLockShared()
        - One-time initialization:
            - InitOnceBeginInitialize()
            - InitOnceComplete()
        - Generic event, signalling and wait facilities:
            - CreateEvent()
            - SetEvent()
            - WaitForSingleObject()
            - WaitForMultipleObjects()
            - SignalObjectAndWait()

    Native thread pool facilities:
        - TrySubmitThreadpoolCallback()
        - StartThreadpoolIo()
        - CloseThreadpoolIo()
        - CancelThreadpoolIo()
        - DisassociateCurrentThreadFromCallback()
        - CallbackMayRunLong()
        - CreateThreadpoolWait()
        - SetThreadpoolWait()

    Memory management:
        - HeapCreate()
        - HeapAlloc()
        - HeapDestroy()

    Structured Exception Handling (#ifdef Py_DEBUG):
        - __try/__except

    Sockets:
        - ConnectEx()
        - AcceptEx()
        - WSAEventSelect(FD_ACCEPT)
        - DisconnectEx(TF_REUSE_SOCKET)
        - Overlapped WSASend()
        - Overlapped WSARecv()


Don't get me wrong, I grew up with UNIX and love it as much as the
next guy, but you can't deny the usefulness of Windows' facilities
for writing high-performance, multi-threaded IO code.  It's decades
ahead of POSIX.  (Which is also why it bugs me when I see select()
being used on Windows, or IOCP being used as if it were a poll-type
"generic IO multiplexor" -- that's like having a Ferrari and speed
limiting it to 5mph!)

So, before any of this has a chance of working on Linux/BSD, a lot
more scaffolding will need to be written to provide the things we
get for free on Windows (threadpools being the biggest freebie).


    Trent.

More information about the Python-Dev mailing list