| msg269604 - (view) |
Author: Anilyka Barry (abarry) *  |
Date: 2016-06-30 17:16 |
| Specifically, test_random and test_sax both fail because of git's newline preferences. Mine is set to convert \r\n to \n on push and the other way around on pull. Relevant output: test_random: ....EException ignored in: <_io.FileIO name='E:\\GitHub\\cpython\\lib\\test\\randv2_32.pck' mode='rb' closefd=True> ResourceWarning: unclosed file <_io.BufferedReader name='E:\\GitHub\\cpython\\lib\\test\\randv2_32.pck'> ...........................EException ignored in: <_io.FileIO name='E:\\GitHub\\cpython\\lib\\test\\randv2_32.pck' mode='rb' closefd=True> ResourceWarning: unclosed file <_io.BufferedReader name='E:\\GitHub\\cpython\\lib\\test\\randv2_32.pck'> ............................ ====================================================================== ERROR: test_bug_1727780 (__main__.MersenneTwister_TestBasicOps) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\GitHub\cpython\lib\test\test_random.py", line 178, in test_bug_1727780 r = pickle.load(f) ImportError: No module named 'random\r' ====================================================================== ERROR: test_bug_1727780 (__main__.SystemRandom_TestBasicOps) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\GitHub\cpython\lib\test\test_random.py", line 178, in test_bug_1727780 r = pickle.load(f) ImportError: No module named 'random\r' ---------------------------------------------------------------------- Ran 61 tests in 1.480s FAILED (errors=2) test_sax: (Passing tests omitted) ====================================================================== FAIL: test_expat_binary_file (__main__.ExpatReaderTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\GitHub\cpython\lib\test\test_sax.py", line 836, in test_expat_binary_file self.assertEqual(result.getvalue(), xml_test_out) AssertionError: b'\n<HTML xmlns:pp="http://www.isogen.com/pa[1508 chars]TML>' != b'\r\n<HTML xmlns:pp="http://www.isogen.com/[1736 chars]TML>' ====================================================================== FAIL: test_expat_binary_file_bytes_name (__main__.ExpatReaderTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\GitHub\cpython\lib\test\test_sax.py", line 874, in test_expat_binary_file_bytes_name self.assertEqual(result.getvalue(), xml_test_out) AssertionError: b'\n<HTML xmlns:pp="http://www.isogen.com/pa[1508 chars]TML>' != b'\r\n<HTML xmlns:pp="http://www.isogen.com/[1736 chars]TML>' ====================================================================== FAIL: test_expat_binary_file_int_name (__main__.ExpatReaderTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\GitHub\cpython\lib\test\test_sax.py", line 886, in test_expat_binary_file_int_name self.assertEqual(result.getvalue(), xml_test_out) AssertionError: b'\n<HTML xmlns:pp="http://www.isogen.com/pa[1508 chars]TML>' != b'\r\n<HTML xmlns:pp="http://www.isogen.com/[1736 chars]TML>' ====================================================================== FAIL: test_expat_binary_file_nonascii (__main__.ExpatReaderTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\GitHub\cpython\lib\test\test_sax.py", line 862, in test_expat_binary_file_nonascii self.assertEqual(result.getvalue(), xml_test_out) AssertionError: b'\n<HTML xmlns:pp="http://www.isogen.com/pa[1508 chars]TML>' != b'\r\n<HTML xmlns:pp="http://www.isogen.com/[1736 chars]TML>' ====================================================================== FAIL: test_expat_inpsource_byte_stream (__main__.ExpatReaderTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\GitHub\cpython\lib\test\test_sax.py", line 1053, in test_expat_inpsource_byte_stream self.assertEqual(result.getvalue(), xml_test_out) AssertionError: b'\n<HTML xmlns:pp="http://www.isogen.com/pa[1508 chars]TML>' != b'\r\n<HTML xmlns:pp="http://www.isogen.com/[1736 chars]TML>' ====================================================================== FAIL: test_expat_inpsource_character_stream (__main__.ExpatReaderTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\GitHub\cpython\lib\test\test_sax.py", line 1066, in test_expat_inpsource_character_stream self.assertEqual(result.getvalue(), xml_test_out) AssertionError: b'\n<HTML xmlns:pp="http://www.isogen.com/pa[1508 chars]TML>' != b'\r\n<HTML xmlns:pp="http://www.isogen.com/[1736 chars]TML>' ====================================================================== FAIL: test_expat_inpsource_filename (__main__.ExpatReaderTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\GitHub\cpython\lib\test\test_sax.py", line 1015, in test_expat_inpsource_filename self.assertEqual(result.getvalue(), xml_test_out) AssertionError: b'\n<HTML xmlns:pp="http://www.isogen.com/pa[1508 chars]TML>' != b'\r\n<HTML xmlns:pp="http://www.isogen.com/[1736 chars]TML>' ====================================================================== FAIL: test_expat_inpsource_sysid (__main__.ExpatReaderTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\GitHub\cpython\lib\test\test_sax.py", line 1025, in test_expat_inpsource_sysid self.assertEqual(result.getvalue(), xml_test_out) AssertionError: b'\n<HTML xmlns:pp="http://www.isogen.com/pa[1508 chars]TML>' != b'\r\n<HTML xmlns:pp="http://www.isogen.com/[1736 chars]TML>' ====================================================================== FAIL: test_expat_inpsource_sysid_nonascii (__main__.ExpatReaderTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\GitHub\cpython\lib\test\test_sax.py", line 1040, in test_expat_inpsou rce_sysid_nonascii self.assertEqual(result.getvalue(), xml_test_out) AssertionError: b'\n<HTML xmlns:pp="http://www.isogen.com/pa[1508 chars]TML>' != b'\r\n<HTML xmlns:pp="http://www.isogen.com/[1736 chars]TML>' ====================================================================== FAIL: test_expat_text_file (__main__.ExpatReaderTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\GitHub\cpython\lib\test\test_sax.py", line 847, in test_expat_text_fi le self.assertEqual(result.getvalue(), xml_test_out) AssertionError: b'\n<HTML xmlns:pp="http://www.isogen.com/pa[1508 chars]TML>' != b'\r\n<HTML xmlns:pp="http://www.isogen.com/[1736 chars]TML>' ---------------------------------------------------------------------- Ran 171 tests in 0.200s FAILED (failures=10) Traceback (most recent call last): File "E:\GitHub\cpython\lib\runpy.py", line 184, in _run_module_as_main "__main__", mod_spec) File "E:\GitHub\cpython\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "E:\GitHub\cpython\lib\test\test_sax.py", line 1261, in test_main() File "E:\GitHub\cpython\lib\test\test_sax.py", line 1258, in test_main XmlReaderTest) File "E:\GitHub\cpython\lib\test\support\__init__.py", line 1837, in run_unittest _run_suite(suite) File "E:\GitHub\cpython\lib\test\support\__init__.py", line 1812, in _run_suite raise TestFailed(err) test.support.TestFailed: multiple errors occurred |
|
|
| msg269636 - (view) |
Author: Martin Panter (martin.panter) *  |
Date: 2016-07-01 01:05 |
| If it were me, I might try to use a version of Git that doesn’t mess with the files. But if this becomes a common problem (esp with the move from Mercurial to Git), maybe we can add a “.gitattributes” file <https://git-scm.com/docs/gitattributes>. We aleady seem to have “.hgeol”. I presume this is going to affect earlier versions too. |
|
|
| msg269638 - (view) |
Author: Anilyka Barry (abarry) * |
Date: 2016-07-01 01:36 |
| I agree, but I wouldn't expect my line ending preferences to make tests fail. I think that with the move to GitHub already underway, it's not unreasonable to (at least consider to) add a .gitattributes file. Added Brett to nosy since he's responsible for PEP 512. (I think this should be changed before the final announcement that the cpython repo is on GitHub, but there's no rush until then; although I'm certainly not going to complain if it's done before ;) |
|
|
| msg269664 - (view) |
Author: R. David Murray (r.david.murray) * |
Date: 2016-07-01 12:38 |
| Yes .gitattributes looks like the correct solution. The problem is that most of the text files we *want* git to transform per platform, it's just a few that we don't. Mercurial actually added support for this for us when we switched to it. |
|
|
| msg273652 - (view) |
Author: Anilyka Barry (abarry) * |
Date: 2016-08-25 14:38 |
| Here are three patches for this: - add_gitattributes_1.patch adds `.gitattributes`, with all the newline preferences in `.hgeol` (with the exception of the "native" rule at the end, which in git is the default newline preference). - fix_newlines_1.patch fixes test_sax to properly handle CRLF (since *.xml files may have either LF or CRLF), as well as a single batch file which wasn't using CRLF line endings for some reason. - test_random_warning_1.patch is somewhat unrelated, but it makes sure that failures in the `pickle.load` call no longer show ResourceWarnings on stderr. The two tests which used to fail no longer do so. Maybe add_gitattributes_1.patch should be backported to 2.7, but probably not the other ones. |
|
|
| msg273657 - (view) |
Author: Anilyka Barry (abarry) * |
Date: 2016-08-25 15:24 |
| Whoops, turns out I was using the wrong approach for binary files. Here come add_gitattributes_2.patch and fix_newlines_2.patch (thanks Zachary for pointing this out). |
|
|
| msg273668 - (view) |
Author: Jim Jewett (Jim.Jewett) * |
Date: 2016-08-25 16:39 |
| I think if the test is ensuring one line-ending type, then there should be another test ensuring the other... but I think it would be best if there were a single test file that had both types of line-endings, just to ensure that the code doesn't cheat by reading only the first line and assuming that the rest of the data is consistent. |
|
|
| msg273752 - (view) |
Author: Martin Panter (martin.panter) * |
Date: 2016-08-27 02:47 |
| Emanuel: fix_newlines_2.patch seems to change the contents of batch from CRLF to LF newlines, undoing revision 640ccb924b5f. This seems like a step in the wrong direction. Notice $ means LF, and ^M$ means CRLF: $ curl https://bugs.python.org/file44225/fix_newlines_2.patch | cat -A . . . diff --git a/Tools/unicode/genwincodecs.bat b/Tools/unicode/genwincodecs.bat$ . . . -c:\python30\python genwincodec.py 720 > build/cp720.py^M$ . . . +c:\python30\python genwincodec.py 720 > build/cp720.py$ However, the test_sax change may be sensible. test_random_warning_1.patch looks okay. |
|
|
| msg273781 - (view) |
Author: Anilyka Barry (abarry) * |
Date: 2016-08-27 14:25 |
| Martin: Indeed, seems like it's backwards for some reason. I'm not sure what happened when I regenerated the index; I removed the patches now anyway. I think the .gitattributes patch would be fine to go on its own. It's my understanding that line endings don't matter for XML files, so it would probably be a better move to fix xml.sax instead of the test. I'll likely open a new issue if we decide to do that. |
|
|
| msg290085 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-03-24 11:42 |
| In the master branch, .gitattributes now contains: --- *.pck binary Lib/test/cjkencodings/* binary Lib/test/decimaltestdata/*.decTest binary Lib/test/sndhdrdata/sndhdr.* binary Lib/test/test_email/data/msg_26.txt binary Lib/test/xmltestdata/* binary Lib/venv/scripts/nt/* binary Lib/test/coding20731.py binary --- History of the file: --- $ git log --follow .gitattributes commit 060d2d776a29341c079cce37220324f9775140ba Author: INADA Naoki <methane@users.noreply.github.com> Date: Sun Mar 5 08:49:45 2017 +0900 remove merge=union attribute for Misc/NEWS (GH-460) Github doesn't support it (ref. isaacs/github#487). So it can't ease conflict on Github. Additionally, it can make trouble when cherry-pick. (ref. GH-212) commit 2771304357607aa62801a67acc1e3c7c8ec489ce Author: Benjamin Peterson <benjamin@python.org> Date: Wed Feb 22 22:38:48 2017 -0800 mark various test data binary (#233) commit 2c700af5a761f83f303cc7295b660ac31f7d4ed1 Author: INADA Naoki <methane@users.noreply.github.com> Date: Tue Feb 21 18:39:41 2017 +0900 .gitattribute -> .gitattributes (GH-213) commit 3f3d0364a91aa8b89e175a49f93ad04dd4676186 Author: INADA Naoki <methane@users.noreply.github.com> Date: Tue Feb 21 18:17:06 2017 +0900 Reduce conflict on Misc/NEWS (GH-212) use "union" merge strategy for Misc/NEWS. --- test_random and test_sax now pass on my Windows VM... but still fail on the "AMD64 Windows8 3.x" buildbot :-( http://buildbot.python.org/all/builders/AMD64%20Windows8%203.x/builds/461/steps/test/logs/stdio Is it possible that Git on this buildbot ignores .gitattributes? Maybe the Git version is too old? |
|
|
| msg290094 - (view) |
Author: Steve Dower (steve.dower) * |
Date: 2017-03-24 16:17 |
| More likely it needs a fresh clone to fix up those files, unless they've changed since the attributes file was added. |
|
|
| msg290617 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-03-27 14:11 |
| New changeset cf57fe13b44e3994096ae79f32c11475f333a94f by Victor Stinner in branch '3.5': bpo-27425: Add .gitattributes, fix Windows tests (#844) https://github.com/python/cpython/commit/cf57fe13b44e3994096ae79f32c11475f333a94f |
|
|
| msg295571 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-06-09 20:43 |
| Wait, why is this issue still open? Tests now pass on Windows on all branches. The issue can be closed, except if I missed something? |
|
|
| msg295663 - (view) |
Author: Zachary Ware (zach.ware) * |
Date: 2017-06-10 19:58 |
| New changeset 6b6e68776663c0dda04b6a36609297728da2ae9e by Zachary Ware in branch 'master': bpo-27425: Be more explicit in .gitattributes (GH-840) https://github.com/python/cpython/commit/6b6e68776663c0dda04b6a36609297728da2ae9e |
|
|
| msg295666 - (view) |
Author: Zachary Ware (zach.ware) * |
Date: 2017-06-10 20:39 |
| New changeset 964c261dc9a6a901f50d5596d88248bfc4251a55 by Zachary Ware in branch '3.6': [3.6] bpo-27425: Be more explicit in .gitattributes (GH-840) (GH-2083) https://github.com/python/cpython/commit/964c261dc9a6a901f50d5596d88248bfc4251a55 |
|
|
| msg295667 - (view) |
Author: Zachary Ware (zach.ware) * |
Date: 2017-06-10 20:40 |
| New changeset afa6a389606402779c5048a7211edbf2678cecff by Zachary Ware in branch '3.5': [3.5] bpo-27425: Be more explicit in .gitattributes (GH-840) (GH-2084) https://github.com/python/cpython/commit/afa6a389606402779c5048a7211edbf2678cecff |
|
|
| msg295675 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-06-10 22:00 |
| Zach, do you want to update Python 2.7 for your latest change as well? |
|
|
| msg295684 - (view) |
Author: Zachary Ware (zach.ware) * |
Date: 2017-06-11 03:19 |
| > Zach, do you want to update Python 2.7 for your latest change as well? PR 2086 is in for that, but it's a huge diff due to line endings changing. As noted on the PR, I want Benjamin to give his OK before I merge it. |
|
|
| msg295728 - (view) |
Author: Zachary Ware (zach.ware) * |
Date: 2017-06-11 19:19 |
| New changeset 5fe8ac69f99d58232a5234cfca78f05f7223a782 by Zachary Ware in branch '2.7': [2.7] bpo-27425: Be more explicit in .gitattributes (GH-840) (GH-2086) https://github.com/python/cpython/commit/5fe8ac69f99d58232a5234cfca78f05f7223a782 |
|
|
| msg295729 - (view) |
Author: Zachary Ware (zach.ware) * |
Date: 2017-06-11 19:23 |
| It still might be nice to adjust some of the tests to not care about how line endings are checked in, but I think we're in a good enough place now with .gitattributes that we can close this issue. Any other changes can be done in a new issue. |
|
|
| msg295762 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-06-12 09:36 |
| I'm happy to see this "old" issue now fixed :-) |
|
|
| msg296519 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2017-06-21 05:21 |
| Installed 3.6.2rc1 is showing the same errors for test_random and test_sax. #30716. |
|
|
| msg296520 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2017-06-21 05:23 |
| Why de we would 'want' /r/n on Windows? What beside Notepad can't handle /n? |
|
|
| msg296534 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-06-21 09:27 |
| I reproduce the bug after installing Python 3.6.2rc1 on Windows. I reopen the issue and set the priority to release blocker. |
|
|
| msg297465 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2017-07-01 01:48 |
| So do we have a resolution or resolutions for this yet? And is bpo-30716 truly a duplicate? If so, let's use one or the other. |
|
|
| msg297469 - (view) |
Author: Steve Dower (steve.dower) * |
Date: 2017-07-01 04:40 |
| There's nothing that needs to block the release as far as I'm concerned. The main fix was that I needed a fresh clone on my build machine to pick up the gitattribute changes. The other bug is still open to enhance some tests and skip one in the release tree, but nothing release blocking. I don't see why this one is now either. |
|
|
| msg297478 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2017-07-01 07:15 |
| There is still the problem with test_winreg freezing CommandPrompt, #30715. For me, this is the worst test bug ever. |
|
|
| msg297490 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2017-07-01 18:02 |
| OK, based on Steve's response, I am closing this again as it sounds like it should not be an issue going forward with releases and also not for dev builds as long as people do a fresh clone if necessary. Further discussion on hardening the tests can take place over on bpo-30716. |
|
|