Issue 22846: distutils needlessly fails to build apsw (original) (raw)
Created on 2014-11-11 08:29 by mark, last changed 2022-04-11 14:58 by admin. This issue is now closed.
Messages (10)
Author: Mark Summerfield (mark) *
Date: 2014-11-11 08:29
When I try to build APSW (http://rogerbinns.github.io/apsw/index.html) with Python 3.3 or 3.4 on Debian stable 64-bit I get the error output shown below.
I dug into the source and it seems that the problem is that distutils/ccompiler.py's gen_preprocess_options() functions expects to get a sequence of macros and each macro must be a 1 or 2 item tuple.
But for some reason during the APSW build it gets a 2 item list which it then chokes on.
Now, the code really does need a tuple because in some cases it uses Python's % print formatting option as in "-D%s=%s" % macro -- and this won't work if macro is a list.
I solved this problem for me by adding two lines, shown here in context:
pp_opts = []
for macro in macros:
if isinstance(macro, list): # NEW
macro = tuple(macro) # NEWI don't know how safe or wise a fix this is, but it did work for me for a locally built (from www.python.org tarball) 3.3 and 3.4.
$ /home/mark/opt/python34/bin/python3 setup.py fetch --all build --enable-all-extensions install running fetch Getting download page to work out current SQLite version Fetching https://sqlite.org/download.html Version is 3.8.7.1 Getting the SQLite amalgamation Fetching https://sqlite.org/2014/sqlite-autoconf-3080701.tar.gz Length: 1998389 SHA1: 5601be1263842209d7c5dbf6128f1cc0b6bbe2e5 MD5: 8ee4541ebb3e5739e7ef5e9046e30063 Checksums verified Running configure to work out SQLite compilation flags setup.py:53: DeprecationWarning: 'U' mode is deprecated f=open(name, mode) running build running build_ext SQLite: Using amalgamation /home/mark/zip/apsw-3.8.7.1-r1/sqlite3/sqlite3.c setup.py:624: ResourceWarning: unclosed file <_io.TextIOWrapper name=4 encoding='UTF-8'> for part in shlex.split(os.popen("icu-config --cppflags", "r").read()): setup.py:637: ResourceWarning: unclosed file <_io.TextIOWrapper name=4 encoding='UTF-8'> for part in shlex.split(os.popen("icu-config --ldflags", "r").read()): ICU: Added includes, flags and libraries from icu-config building 'apsw' extension Traceback (most recent call last): File "setup.py", line 862, in 'win64hackvars': win64hackvars} File "/home/mark/opt/python34/lib/python3.4/distutils/core.py", line 148, in setup dist.run_commands() File "/home/mark/opt/python34/lib/python3.4/distutils/dist.py", line 955, in run_commands self.run_command(cmd) File "/home/mark/opt/python34/lib/python3.4/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/home/mark/opt/python34/lib/python3.4/distutils/command/build.py", line 126, in run self.run_command(cmd_name) File "/home/mark/opt/python34/lib/python3.4/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/home/mark/opt/python34/lib/python3.4/distutils/dist.py", line 974, in run_command cmd_obj.run() File "setup.py", line 661, in run v=beparent.run(self) File "/home/mark/opt/python34/lib/python3.4/distutils/command/build_ext.py", line 339, in run self.build_extensions() File "/home/mark/opt/python34/lib/python3.4/distutils/command/build_ext.py", line 448, in build_extensions self.build_extension(ext) File "/home/mark/opt/python34/lib/python3.4/distutils/command/build_ext.py", line 503, in build_extension depends=ext.depends) File "/home/mark/opt/python34/lib/python3.4/distutils/ccompiler.py", line 566, in compile depends, extra_postargs) File "/home/mark/opt/python34/lib/python3.4/distutils/ccompiler.py", line 341, in _setup_compile pp_opts = gen_preprocess_options(macros, incdirs) File "/home/mark/opt/python34/lib/python3.4/distutils/ccompiler.py", line 1061, in gen_preprocess_options % macro) TypeError: bad macro definition '['_FORTIFY_SOURCE', '2']': each element of 'macros' list must be a 1- or 2-tuple
Author: Ned Deily (ned.deily) * 
Date: 2014-11-11 09:15
Have you reported this problem to the author of apsw? It seems like figuring out why apsw is apparently creating an incorrect setup.py configuration should be a first step before suggesting a change to Distutils. You might want to supply the values from your system for:
icu-config --cppflags icu-config --ldflags
FWIW, with a current Debian testing, I get the following failure with either Python 3.4 or 2.7:
running build_ext SQLite: Using amalgamation /tmp/e/apsw-3.8.7.1-r1/sqlite3/sqlite3.c
icu-config: Can't find /usr/lib/i386-linux-gnu/libicuuc.so - ICU prefix is wrong.
Try the --prefix= option
or --detect-prefix
(If you want to disable this check, use the --noverify option)
icu-config: Exitting.
icu-config: Can't find /usr/lib/i386-linux-gnu/libicuuc.so - ICU prefix is wrong.
Try the --prefix= option
or --detect-prefix
(If you want to disable this check, use the --noverify option)
icu-config: Exitting.
ICU: Unable to determine includes/libraries for ICU using icu-config ICU: You will need to manually edit setup.py or setup.cfg to set them
So there may be issues with icu-config on Debian. (Also, Python 3.3 is now in security-fix only mode.)
Author: Ned Deily (ned.deily) *
Date: 2014-11-11 09:25
Another data point: apsw appears to build OK on OS X with a MacPorts-supplied icu, including icu-config.
Author: Mark Summerfield (mark) *
Date: 2014-11-11 09:35
The first person I asked was the author of APSW (Roger Binns). He told me:
"The ultimate cause of that is some interaction with the compilation environment. Some sort of CFLAGS is ultimately ending up in some Python code like above when it should be [ ('_FORTIFY_SOURCE', '2') ]. Note this is not part of the APSW source - it is something external."
"I have seen it before when using the Ubuntu PPA build service. When building locally everything was fine, but the build service injected _FORTIFY_SOURCE like above and got it wrong. I presume your version of Debian is doing something similar. Sadly I have no idea how to fix it."
So clearly he believes it is not a problem with his setup.py file.
Also, it strikes me as a bit unpythonic that a function should demand a specific type (i.e., tuple) especially when this is just for the convenience of being able to use % formatting.
I'm not asking or expecting you to add my change to distutils; but at least now if someone encounters the same problem, they will have a potential fix.
Author: Mark Summerfield (mark) *
Date: 2014-11-11 09:36
Here are the flags you asked for:
$ icu-config --cppflags -D_FORTIFY_SOURCE=2 -D_REENTRANT -I/usr/include $ icu-config --ldflags -Wl,-z,relro -ldl -lm -L/usr/lib/x86_64-linux-gnu -licui18n -licuuc -licudata -ldl -lm
Author: Marc-Andre Lemburg (lemburg) *
Date: 2014-11-11 09:57
Shouldn't this be fixed in the APSW setup.py ?
The patch is you are proposing looks harmless, but it can also mask programming errors in setup.py.
Author: Antoine Pitrou (pitrou) *
Date: 2014-11-11 10:07
The ultimate cause of that is some interaction with the compilation environment. Some sort of CFLAGS is ultimately ending up in some Python code like above when it should be [ ('_FORTIFY_SOURCE', '2') ]. Note this is not part of the APSW source - it is something external.
That's as unhelpful as a bug report can get. If ICU doesn't provide the right types, the setup.py should fix them up. (assuming ICU provides the preprocessor flags here - I haven't check)
Author: Antoine Pitrou (pitrou) *
Date: 2014-11-11 10:07
I suggest closing.
Author: Ned Deily (ned.deily) *
Date: 2014-11-11 10:10
I was able to reproduce the behavior you saw with an older Debian system. The following patch to the apsw setup.py file seems to fix the problem:
--- apsw-3.8.7.1-r1/setup.py 2014-11-04 19:23:36.000000000 -0800 +++ apsw-3.8.7.1-r1_PATCHED/setup.py 2014-11-11 02:01:16.000000000 -0800 @@ -628,7 +628,7 @@ elif part.startswith("-D"): part=part[2:] if '=' in part:
part=part.split('=', 1)
part=tuple(part.split('=', 1)) else: part=(part, '1') ext.define_macros.append(part)
Also, requiring a tuple is the documented behavior of Distutils:
https://docs.python.org/2/distutils/apiref.html#distutils.core.Extension
I agree that the issue should be closed and am closing it.
Author: Mark Summerfield (mark) *
Date: 2014-11-11 10:18
I've notified APSW's author and I'm sure he'll fix it. Thanks!
History
Date
User
Action
Args
2022-04-11 14:58:10
admin
set
github: 67035
2014-11-11 10🔞31
mark
set
messages: +
2014-11-11 10:10:12
ned.deily
set
status: pending -> closed
messages: +
stage: resolved
2014-11-11 10:07:38
pitrou
set
status: open -> pending
nosy: + ncoghlan
messages: +
resolution: not a bug
2014-11-11 10:07:04
pitrou
set
nosy: + pitrou
messages: +
2014-11-11 09:57:28
lemburg
set
nosy: + lemburg
messages: +
2014-11-11 09:36:47
mark
set
messages: +
2014-11-11 09:35:09
mark
set
messages: +
2014-11-11 09:25:12
ned.deily
set
messages: +
2014-11-11 09:15:53
ned.deily
set
nosy: + ned.deily
messages: +
versions: - Python 3.3
2014-11-11 08:29:28
mark
create