Issue 4051: UNICODE macro in cPickle conflicts with Windows define (original) (raw)
Created on 2008-10-06 10:28 by eckhardt, last changed 2022-04-11 14:56 by admin. This issue is now closed.
Messages (8)
Author: Ulrich Eckhardt (eckhardt)
Date: 2008-10-06 10:28
In several places, Python is using TCHAR-based win32 APIs which has several consequences:
- The actually used function depends on the _UNICODE macro, while the using code often assumes that this macro is not defined. That means that not only the code changes when the macro is defined but that it simply breaks.
- Using the ANSI-variants of functions causes overhead when a Unicode string has to be transcoded and possibly data loss, because the ANSI encoding can't represent the full Unicode range. Note that I'm assuming an NT based system (i.e. NT, win2k etc), because the systems where the ANSI encoding is native (win9x, ME(?) etc) have been dropped from the supported platforms.
- The main reason for me wanting a build independent of TCHAR is a port to MS Windows CE, which doesn't support the ANSI encoding at all. This means that a TCHAR-clean build for desktop win32 is a necessary part of the port to CE, but even without the port it would be useful to skip the conversion overhead and at the same time gain functionality. Concerning the CE port, see this thread as a reference: http://mail.python.org/pipermail/python-dev/2008-September/082529.html
Note: I'm adding this tracker item mainly in order to attach patches for review, so they don't get lost like it's possible on the mailinglist.
Author: Ulrich Eckhardt (eckhardt)
Date: 2008-10-06 10:35
Add a patch that fixes a warning.
Author: Martin v. Löwis (loewis) * 
Date: 2008-10-06 19:24
Please create a separate tracker item per individual change (or per group by enourmously similar changes). Each change will be reviewed separately, and will get accepted or rejected separately.
This specific patch (Python-unicode-redefine-warning) is fine, but I'd rather wait until after the release of Python 3.0, in order to be able to merge the changes better into the 3k branch.
Author: Roumen Petrov (rpetrov) *
Date: 2008-10-07 20:23
What about to substitute UNICODE macro as example with RAWUNICODE in the source ?
Author: Martin v. Löwis (loewis) *
Date: 2008-10-07 20:34
What about to substitute UNICODE macro as example with RAWUNICODE in the source ?
That would work as well. As it stands, the macro names are equivalent to the ones in pickle.py, which is a useful property to have. So I'm in favor of the proposed change.
Author: Roumen Petrov (rpetrov) *
Date: 2008-10-10 22:39
For the protocol I attach part for PythonCE-2.5-20061219.patch. Both patches address UNICODE name clash in Modules/cPickle.c.
For windows programs UNICODE is more commonly used define and may be python C-code only has to avoid issues like this one.
I couldn't found where python C-code use so called "TCHAR meaning" and the first sentence from comment in Ulrich patch looks irrelevant to the issue.
Also Ulrich patch is far from complete "TCHAR" implementation in python. May be issue title has to be changed to address more precise problem.
Author: Martin v. Löwis (loewis) *
Date: 2008-10-12 17:36
I couldn't found where python C-code use so called "TCHAR meaning" and the first sentence from comment in Ulrich patch looks irrelevant to the issue.
I was confused by Ulrich's terminology at first, also (claiming that Python doesn't use TCHAR, so how could it depend on it); I now think Ulrich is nearly correct. E.g. in 2.6, we call, in posixmodule.c, CreateProcess. Now, CreateProcess is defined in the SDK as
#ifdef UNICODE #define CreateProcess CreateProcessW #else #define CreateProcess CreateProcessA #endif // !UNICODE
However, it is documented as
BOOL WINAPI CreateProcess( __in_opt LPCTSTR lpApplicationName, __inout_opt LPTSTR lpCommandLine, __in_opt LPSECURITY_ATTRIBUTES lpProcessAttributes, __in_opt LPSECURITY_ATTRIBUTES lpThreadAttributes, __in BOOL bInheritHandles, __in DWORD dwCreationFlags, __in_opt LPVOID lpEnvironment, __in_opt LPCTSTR lpCurrentDirectory, __in LPSTARTUPINFO lpStartupInfo, __out LPPROCESS_INFORMATION lpProcessInformation );
So according to the documentation, this is a function that takes TCHAR parameters, hence it is a "TCHAR-based (windows) API". The fact that the Windows SDK actually defines it differently is a technical hack; for all practical purposes, it behaves as if it really was a TCHAR API.
Also Ulrich patch is far from complete "TCHAR" implementation in python.
See .
May be issue title has to be changed to address more precise problem.
Certainly.
Author: Martin v. Löwis (loewis) *
Date: 2009-01-02 20:40
Thanks for the patch. Committed as r68173.
History
Date
User
Action
Args
2022-04-11 14:56:40
admin
set
github: 48301
2009-01-02 20:40:31
loewis
set
status: open -> closed
messages: +
2008-12-30 13:31:24
loewis
set
priority: high
assignee: loewis
resolution: accepted
stage: commit review
2008-10-12 17:37:39
loewis
set
title: use of TCHAR under win32 -> UNICODE macro in cPickle conflicts with Windows define
2008-10-12 17:36:31
loewis
set
messages: +
2008-10-10 22:39:08
rpetrov
set
files: + PythonCE-2.5-20061219-cPickle.c-patch
messages: +
2008-10-07 20:34:09
loewis
set
messages: +
2008-10-07 20:23:48
rpetrov
set
nosy: + rpetrov
messages: +
2008-10-06 19:24:42
loewis
set
nosy: + loewis
messages: +
2008-10-06 10:35:19
eckhardt
set
files: + Python-unicode-redefine-warning.0.patch
keywords: + patch
messages: +
2008-10-06 10:28:45
eckhardt
create