[Python-bugs-list] [ python-Bugs-467384 ] provide a documented serialization func (original) (raw)

noreply@sourceforge.net noreply@sourceforge.net
Fri, 12 Oct 2001 12:29:07 -0700

Bugs item #467384, was opened at 2001-10-02 19:25 You can respond by visiting: http://sourceforge.net/tracker/?func=detail&atid=105470&aid=467384&group_id=5470

Category: None Group: Feature Request Status: Open Resolution: None Priority: 5 Submitted By: paul rubin (phr) Assigned to: Nobody/Anonymous (nobody) Summary: provide a documented serialization func

Initial Comment: It would be nice if there was a documented library function for serializing Python basic objects (numbers, strings, dictionaries, and lists). By documented I mean the protocol is specified in the documentation, precisely enough to write interoperating implementations in other languages.

Code-wise, the marshal.dumps and loads functions do what I want, but their data format is (according to the documentation) intentionally not specified, because the format might change in future Python versions. Maybe that doc was written long enough ago that it's ok to freeze the marshal format now, and document it? I just mean for the basic types listed above. Stuff like code objects don't have to be specified. In fact it would be nice if there was a flag to the loads and dumps functions to refuse to marshal/ unmarshal those objects.

Pickle/cpickle aren't really appropriate for what I'm asking, since they're complicated (they try to handle class instances, circular structure, etc.) and anyway they're not documented either.

The XDR library is sort of ok, but it's written in Python (i.e. slow) and it doesn't automatically handle compound objects.

Thanks

Comment By: paul rubin (phr) Date: 2001-10-12 12:29

Message: Logged In: YES user_id=72053

I just want to be able to do convenient transfers of python data to other programs including over the network. XMLRPC is excessive bloat in my opinion. Sending a number like 12345678 should take at most 5 bytes (a type byte and a 4-byte int) instead of 12345678. For long ints (300 digits) it's even worse.

The marshal format is fine, and writing a PEP would solve the doc problem, but the current marshal doc says the non-specification is intentional. Writing it in a PEP means not just documenting--it means asking the language maintainers to freeze the marshal format of certain types, instead of reserving the right to change the format in future versions. Writing the PEP only makes sense if you're willing to freeze the format for those types (the other types can stay undocumented). Is that ok with you?

Thanks Paul

Comment By: Guido van Rossum (gvanrossum) Date: 2001-10-12 07:33

Message: Logged In: YES user_id=6380

Paul, I don't understand the application that you are envisioning. If you think that the marshal format is what you want, why don't you write a PEP that specifies the format? That would solve the documentation problem.

Comment By: Martin v. L�wis (loewis) Date: 2001-10-12 02:39

Message: Logged In: YES user_id=21627

Well, then I guess you need to specify your requirements more clearly. XML-RPC was precisely developed to be something simple for primitive types and structures that is sufficiently well-specified to allow interoperation between various languages.

I don't see why extending the data 'by an order of magnitude' would be a problem per se, nor do I see why 'requiring a complicated parser' is a problem if the implementation already does all the unpacking for you under the hoods.

Furthermore, I believe it is simply not true that XML-RPC expands the representation by an order of magnitude. For example, the Python Integer object 1 takes 12 bytes in its internal representation (plus the overhead that malloc requires); the XML-RPC representation '1' also uses 12 bytes. In short, you need to say as precise as possible what it is that you want, or you won't get it. Also, it may be that you have conflicting requirements (e.g. 'compact, binary', and 'simple, easily processible in different languages'); then you won't get it either. For a marshalling format that is accessible from different languages, you better specify it first, and implement it then.

Comment By: paul rubin (phr) Date: 2001-10-11 22:12

Message: Logged In: YES user_id=72053

I haven't looked at xmlrpclib, but I'm looking for a simple, compact, binary representation, not something that needs a complicated parser and expands the data by an order of magnitude.

Comment By: Martin v. L�wis (loewis) Date: 2001-10-05 17:10

Message: Logged In: YES user_id=21627

So what's wrong with xmlrpclib?

You can respond by visiting: http://sourceforge.net/tracker/?func=detail&atid=105470&aid=467384&group_id=5470