Issue 14436: SocketHandler sends obejcts while they cannot be unpickled on receiver's side (original) (raw)

I decided to use SocketHandler in multi-processes application. Log server, sending data, logging simple strings - works fine.

The problem is with own classes (or external libraries). Looks like SocketHandler creates pickles that cannot be unpickled then on receiver's side (it does not contain my own classes in PYTHONPATH).

The issue happens only when I use recommened way of passing parameters:

class LibInfo( object ): """Library info taken from xml""" def init(self, libName=None, xmlName=None, packName=None): self.libName = libName self.xmlName = xmlName self.packName = packName

def repr(self): return "L=%s X=%s P=%s" % (self.libName,self.xmlName,self.packName)

myObj = LibInfo("1", "2", "3") logging.info("Simple data: %s", myObj)

Traceback (most recent call last): File "/opt/python2.7/logging/handlers.py", line 563, in emit s = self.makePickle(record) File "/opt/python2.7/logging/handlers.py", line 533, in makePickle s = cPickle.dumps(record.dict, 1) PicklingError: Can't pickle <class 'LibInfo'>: attribute lookup LibInfo failed

these two lines work properly:

logging.info("Simple data: %s", str(myObj) ) logging.info("Simple data: %s" % myObj)

This would be not that critical: I could convert all passed parameters to strings. The issue is with external libraries. That I cannot control.

I think SocketHandler should make record with all parameters resolved to final string.

I'm not sure this was fixed in an optimal way.

We have a set of processes using the SocketHandler to send log records to a single log process. Some of these log records have the msg attribute as a dictionary that contained a variety of extra information about the event that was being logged. After this change, our process receiving the events and logging to a file stopped working correctly because it was expecting a dictionary for the msg but was always receiving a string. (I have since made it smarter).

The fix for this ticket makes sense when you don't have a guarantee of being able to unpickle the msg on the receiving end, but it also limits some of the adaptability to pass objects using the SocketHandler.

Some possible improvements:

  1. Add a flag to SocketHandler on whether or not it should force the msg to a string
  2. If it's it a built-in picklable type, don't force to string, else force msg to string

Suggestion 2 has some drawbacks in that you'd have to loop over lists, dictionaries, etc to verify everything inside there is also picklable. Also it would prevent you from sending custom objects across.