Issue 32409: venv activate.bat is UTF-8 encoded but uses current console codepage (original) (raw)

process

Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: Jac0, Martin Bijl-Schwab, eryksun, ezio.melotti, miss-islington, pablogsal, paul.moore, steve.dower, tim.golden, vinay.sajip, vstinner, zach.ware
Priority: normal Keywords: patch

Created on 2017-12-22 10:36 by Jac0, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 5757 merged steve.dower,2018-02-19 18:11
PR 5765 merged miss-islington,2018-02-20 01:25
PR 5765 merged miss-islington,2018-02-20 01:25
PR 5766 merged miss-islington,2018-02-20 01:26
PR 10295 merged python-dev,2018-11-02 10:46
PR 10377 merged miss-islington,2018-11-07 16:50
PR 10403 merged pablogsal,2018-11-07 22:59
PR 10405 merged miss-islington,2018-11-07 23:55
PR 8321 LorenzMende,2019-04-09 13:15
Messages (18)
msg308931 - (view) Author: Jaakko Roponen (Jac0) Date: 2017-12-22 10:36
Let's say I have a folder c:\test-ä in Windows Now if I run: py -m venv env and activate: env\scripts\activate and check: where python the result is incorrectly just: C:\Users\Username\AppData\Local\Programs\Python\Python36\python.exe If I run: path the result is: PATH=C:\test-├ñ\env\Scripts;... So clearly the encoding is broken for the folder name. I can fix this by changing activate.bat character encoding to OEM-US and then replacing "test-├ż" by "test-ä". If I now activate and run: where python the result is (as should be): C:\test-ä\env\Scripts\python.exe C:\Users\Username\AppData\Local\Programs\Python\Python36\python.exe By running: path I get: PATH=C:\test-ä\env\Scripts;... So looks good here as well. I suggest that what ever is creating activate.bat file, is using incorrect character encoding for the creation of the file. If this is somehow platform specific, there could be a guide in the venv documentation about how to fix this.
msg308934 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-12-22 13:24
The CMD shell decodes batch scripts using the attached console's output codepage, which defaults to OEM. OTOH, venv writes the replacement values for the template activate.bat as UTF-8 (codepage 65001), which is correct and should not be downgraded to OEM. Instead, the batch script could temporarily switch the console to codepage 65001. Then restore the previous codepage at the end. For example: @echo off for /f "tokens=2 delims=:" %%a in ('"%SystemRoot%\System32\chcp.com"') do ( set "CODEPAGE=%%a" ) "%SystemRoot%\System32\chcp.com" 65001 > nul [rest of script] "%SystemRoot%\System32\chcp.com" %CODEPAGE% > nul set "CODEPAGE=" :END
msg312356 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-02-19 16:57
Eryk's solution seems to be best, so I'll add that.
msg312389 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-02-20 01:25
New changeset 6240917b773b52f8883387b9e3a5f327a4372068 by Steve Dower in branch 'master': bpo-32409: Ensures activate.bat can handle Unicode contents (GH-5757) https://github.com/python/cpython/commit/6240917b773b52f8883387b9e3a5f327a4372068
msg312390 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-02-20 01:45
New changeset a3d6c1b23b8a49b5003fdbd115d3598fe3d4c4bf by Steve Dower (Miss Islington (bot)) in branch '3.7': bpo-32409: Ensures activate.bat can handle Unicode contents (GH-5765) https://github.com/python/cpython/commit/a3d6c1b23b8a49b5003fdbd115d3598fe3d4c4bf
msg312393 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-02-20 02:18
New changeset 8e149ff481acbb3889c825b8bf7b10aa191f09a7 by Steve Dower (Miss Islington (bot)) in branch '3.6': bpo-32409: Ensures activate.bat can handle Unicode contents (GH-5766) https://github.com/python/cpython/commit/8e149ff481acbb3889c825b8bf7b10aa191f09a7
msg329141 - (view) Author: Martin Bijl-Schwab (Martin Bijl-Schwab) * Date: 2018-11-02 16:41
it does not work as expected on swiss german (and likely other internationalised) windows systems. See https://bugs.python.org/issue32409
msg329142 - (view) Author: Martin Bijl-Schwab (Martin Bijl-Schwab) * Date: 2018-11-02 16:42
I meant https://bugs.python.org/issue35148
msg329425 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2018-11-07 16:49
New changeset c64583b6d3e8516a8cd2b5f84fc1e300bfac2206 by Vinay Sajip (samstagern) in branch 'master': bpo-32409: Fix regression in activate.bat on international Windows (GH-10295) https://github.com/python/cpython/commit/c64583b6d3e8516a8cd2b5f84fc1e300bfac2206
msg329431 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2018-11-07 19:26
New changeset 881e273c795f2f5154b8afebfa299f0e830f3712 by Vinay Sajip (Miss Islington (bot)) in branch '3.7': bpo-32409: Fix regression in activate.bat on international Windows (GH-10295) (GH-10377) https://github.com/python/cpython/commit/881e273c795f2f5154b8afebfa299f0e830f3712
msg329440 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-11-07 21:39
This commit broke a lot of the Windows buildbots: https://buildbot.python.org/all/#/builders/32/builds/1707 https://buildbot.python.org/all/#/builders/113/builds/733 https://buildbot.python.org/all/#/builders/40/builds/1137 https://buildbot.python.org/all/#/builders/130/builds/420 https://buildbot.python.org/all/#/builders/3/builds/1634 Can someone work on a patch soon? If not we need to revert the commits in all the branches that are failed to build :(
msg329447 - (view) Author: Martin Bijl-Schwab (Martin Bijl-Schwab) * Date: 2018-11-07 22:18
I will have a look.
msg329450 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-11-07 23:55
New changeset 6843ffe4533e9f2cde036296fd932fef6f059687 by Pablo Galindo in branch 'master': Revert "bpo-32409: Fix regression in activate.bat on international Windows (GH-10295)" (GH-10403) https://github.com/python/cpython/commit/6843ffe4533e9f2cde036296fd932fef6f059687
msg329451 - (view) Author: miss-islington (miss-islington) Date: 2018-11-08 00:14
New changeset 3ba5e253de1a52e9692aa645c95072705f08b7bb by Miss Islington (bot) in branch '3.7': Revert "bpo-32409: Fix regression in activate.bat on international Windows (GH-10295)" (GH-10403) https://github.com/python/cpython/commit/3ba5e253de1a52e9692aa645c95072705f08b7bb
msg329464 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-08 08:48
Pablo Galindo reverted the change because it broke Windows buildbots, and we have a policy to revert a change breaking buildbots if the regression cannot be fixed "quickly" event: https://pythondev.readthedocs.io/ci.html#revert-on-fail Is someone working on investigating the bug? Do you need help to reproduce the bug? Copy of the test_venv error: ====================================================================== ERROR: test_unicode_in_batch_file (test.test_venv.BasicTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "D:\buildarea\3.x.bolen-windows8\build\lib\test\test_venv.py", line 302, in test_unicode_in_batch_file out, err = check_output( File "D:\buildarea\3.x.bolen-windows8\build\lib\test\test_venv.py", line 37, in check_output raise subprocess.CalledProcessError( TypeError: __init__() takes from 3 to 5 positional arguments but 6 were given https://buildbot.python.org/all/#/builders/32/builds/1707
msg329481 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-11-08 16:38
That error is a bug in the test, but it only shows up on an error path anyway. Without removing the extra None we don't get to see the actual error output. I can't look into this over the next week or two, but a quick glance at the original PR looks like a lot of quotes are missing around executable paths, so maybe it was that?
msg332324 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2018-12-22 06:09
See also bpo-35558.
msg348505 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-07-26 16:39
This appears to be completely resolved now, but if it's not, please ping with the details and we'll reopen.
History
Date User Action Args
2022-04-11 14:58:56 admin set github: 76590
2019-07-26 16:39:28 steve.dower set status: open -> closedmessages: + stage: patch review -> resolved
2019-04-09 13:15:58 LorenzMende set pull_requests: + <pull%5Frequest12669>
2018-12-22 06:09:56 vinay.sajip set messages: +
2018-11-08 16:38:08 steve.dower set messages: +
2018-11-08 08:48:59 vstinner set messages: +
2018-11-08 00:14:05 miss-islington set nosy: + miss-islingtonmessages: +
2018-11-07 23:55:54 miss-islington set pull_requests: + <pull%5Frequest9689>
2018-11-07 23:55:43 pablogsal set messages: +
2018-11-07 22:59:27 pablogsal set stage: resolved -> patch reviewpull_requests: + <pull%5Frequest9688>
2018-11-07 22🔞18 Martin Bijl-Schwab set messages: +
2018-11-07 21:39:10 pablogsal set status: closed -> opennosy: + pablogsalmessages: +
2018-11-07 19:26:17 vinay.sajip set messages: +
2018-11-07 16:50:26 miss-islington set pull_requests: + <pull%5Frequest9679>
2018-11-07 16:49:25 vinay.sajip set nosy: + vinay.sajipmessages: +
2018-11-02 16:42:55 Martin Bijl-Schwab set messages: +
2018-11-02 16:41:28 Martin Bijl-Schwab set nosy: + Martin Bijl-Schwabmessages: +
2018-11-02 10:46:53 python-dev set pull_requests: + <pull%5Frequest9604>
2018-02-20 02🔞45 steve.dower set messages: +
2018-02-20 01:45:58 steve.dower set status: open -> closedresolution: fixedstage: patch review -> resolved
2018-02-20 01:45:06 steve.dower set messages: +
2018-02-20 01:26:32 miss-islington set pull_requests: + <pull%5Frequest5545>
2018-02-20 01:25:42 miss-islington set pull_requests: + <pull%5Frequest5544>
2018-02-20 01:25:40 miss-islington set pull_requests: + <pull%5Frequest5543>
2018-02-20 01:25:29 steve.dower set messages: +
2018-02-19 18:11:12 steve.dower set keywords: + patchstage: needs patch -> patch reviewpull_requests: + <pull%5Frequest5535>
2018-02-19 16:57:10 steve.dower set assignee: steve.dowermessages: + versions: + Python 3.8
2017-12-22 13:24:49 eryksun set title: venv activation doesn't work, if project is in a Windows folder that has latin-1 supplement characters (such as ä,ö,å) in its path -> venv activate.bat is UTF-8 encoded but uses current console codepagecomponents: + Library (Lib), Unicode, Windows, - Extension Modulesnosy: + ezio.melotti, eryksun, paul.moore, tim.golden, vstinner, zach.ware, steve.dowerversions: + Python 3.7messages: + stage: needs patch
2017-12-22 10:36:15 Jac0 create