[medusa] Strange exception in ZServer/medusa/asynchat.py (original) (raw)

Skip Montanaro skip@m...
Tue, 25 May 1999 21:56:52 -0400

I'm using Medusa by way of Zope's ZServer. I have ZServer sitting behind an Apache proxy and am bombing it from a threaded client script that reads a web server log file as input and throws requests at the Apache server. I can vary the number of threads to adjust the load. I normally run it with five simultaneous threads.

by default, the system runs briefly then (as they say), "Whammo! Blammo!", I get the following error:

Unhandled exception in thread: Traceback (innermost last): File "/home/dolphin/skip/src/Zope/ZServer/PubCore/ZServerPublisher.py", line 97, in init response._finish() File "/home/dolphin/skip/src/Zope/ZServer/HTTPResponse.py", line 209, in _finish self.stdout.close() File "/home/dolphin/skip/src/Zope/ZServer/HTTPResponse.py", line 235, in close self._channel.push(CallbackProducer(self._channel.done)) File "/home/dolphin/skip/src/Zope/ZServer/HTTPServer.py", line 307, in push if send: self.initiate_send() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 200, in initiate_send self.refill_buffer() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 184, in refill_buffer self.producer_fifo.pop() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 255, in pop del self.list[0] IndexError: list assignment index out of range

If you look at the code in asynchat.fifo.pop, the error can only be explained by the fact that two threads are modifying the fifo simultaneously, since access to self.list[0] succeeded in the statement before the statement that throws the IndexError:

def pop (self): if self.list: result = (1, self.list[0]) # non-null list here del self.list[0] # empty list here else: result = (0, None) return result

I modified the fifo class to lock access to its list:

class fifo: def init (self, list=None): self.lock = thread.allocate_lock() self.lock.acquire() if not list: self.list = [] else: self.list = list self.lock.release()

def len (self): self.lock.acquire() l = len(self.list) self.lock.release() return l

def first (self): self.lock.acquire() item = self.list[0] self.lock.release() return item

def push (self, data): self.lock.acquire() self.list.append (data) self.lock.release()

def pop (self): self.lock.acquire() if self.list: result = (1, self.list[0]) del self.list[0] else: result = (0, None) self.lock.release() return result

but this didn't help:

Unhandled exception in thread: Traceback (innermost last): File "/home/dolphin/skip/src/Zope/ZServer/PubCore/ZServerPublisher.py", line 97, in init response._finish() File "/home/dolphin/skip/src/Zope/ZServer/HTTPResponse.py", line 209, in _finish self.stdout.close() File "/home/dolphin/skip/src/Zope/ZServer/HTTPResponse.py", line 235, in close self._channel.push(CallbackProducer(self._channel.done)) File "/home/dolphin/skip/src/Zope/ZServer/HTTPServer.py", line 307, in push if send: self.initiate_send() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 200, in initiate_send self.refill_buffer() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 175, in refill_buffer p = self.producer_fifo.first() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 254, in first item = self.list[0] IndexError: list index out of range

This error also looks like a threading problem. When refill_buffer tests self.producer.fifo, it sees something, but by the time it calls fifo.first, the list has evaporated:

while 1: if len(self.producer_fifo): # non-null list here p = self.producer_fifo.first() # null list here

Any ideas? Is there a way to easily turn off threading? It seems that the data structures are not properly protected.

Skip Montanaro | Mojam: "Uniting the World of Music" http://www.mojam.com/ skip@m... | Musi-Cal: http://www.musi-cal.com/ 518-372-5583

eGroups.com home: http://www.egroups.com/group/medusa http://www.egroups.com - Simplifying group communications