Hmm. But what I did was look up the RLock section of the manual to answer the question "does this support the context manager protocol", and the answer appeared to be no. In a reference manual I would expect all the important features of an object to be described in that object's documentation, or linked to therefrom.
Here is a patch that adds an note about using Locks, RLocks, Conditions, and Semaphores as context managers to each of their descriptions as well as a link to the "Using locks, conditions, and semaphores in the with statement" section.