Issue 32350: pip can't handle MSVC warnings containing special characters (original) (raw)

When trying to install pylint using pip on Windows 10, the installation of wrapt (a dependency of pylint) fails because a special character in an MSVC warning can't be decoded to utf-8.

Below is the relevant part of the console output:

Exception: Traceback (most recent call last): File "C:\Users(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\compat_init_.py", line 73, in console_to_str return s.decode(sys.stdout.encoding) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 51: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\basecommand.py", line 215, in main status = self.run(options, args) File "C:\Users(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\commands\install.py", line 342, in run prefix=options.prefix_path, File "C:\Users(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\req\req_set.py", line 784, in install **kwargs File "C:\Users(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\req\req_install.py", line 878, in install spinner=spinner, File "C:\Users(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\utils_init.py", line 676, in call_subprocess line = console_to_str(proc.stdout.readline()) File "C:\Users(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\compat_init.py", line 75, in console_to_str return s.decode('utf_8') UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 51: invalid start byte

I changed line 73 in \pip\compat_init_.py to: print("!! TEST !! : ", s). The full console output after that change is in the attachment. The relevant part is:

!! TEST !! : b'C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.11.25503\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Users\(...)\AppData\Local\Programs\Python\Python36\include -IC:\Users\(...)\AppData\Local\Programs\Python\Python36\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.11.25503\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.11.25503\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\winrt" /Tcsrc/wrapt/_wrappers.c /Fobuild\temp.win-amd64-3.6\Release\src/wrapt/_wrappers.obj\r\n' !! TEST !! : b'_wrappers.c\r\n' !! TEST !! : b"src/wrapt/wrappers.c(195): warning C4244: 'return'\xff: conversion de 'Py_hash_t' en 'long', perte possible de donn\x82es\r\n" error Exception: Traceback (most recent call last): File "C:\Users(...)\AppData\Local\Programs\Python\Python36\Lib\site-packages\pip\compat_init.py", line 74, in console_to_str return s.decode(sys.stdout.encoding) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 51: invalid start byte

As a workaround I've changed the original line 73 in \pip\compat_init_.py :

return s.decode(sys.stdout.encoding)

to:

return s.decode(sys.stdout.encoding, "replace")

(thanks to dieter on comp.lang.python for his help). I don't have the knowledge to propose a patch.

This is a pip issue, not a Python issue. It's a known issue in the current version of pip, and should be fixed in the development version (via pull request https://github.com/pypa/pip/pull/4486).

If you like, you can try the development version of pip. At the moment there is no planned date for the next release of pip, but we're hoping to put together a release in the not too distant future.