Issue 36876: [subinterpreters] Global C variables are a problem (original) (raw)

Issue36876

Created on 2019-05-10 19:01 by eric.snow, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 13372 closed eric.snow,2019-05-17 00:46
PR 13531 closed eric.snow,2019-05-23 22:55
PR 15760 closed eric.snow,2019-09-09 11:30
PR 15877 merged eric.snow,2019-09-10 15:31
PR 16017 merged eric.snow,2019-09-12 09:23
PR 16058 merged eric.snow,2019-09-12 14:41
PR 16328 merged vinay.sajip,2019-09-22 04:41
PR 16841 merged eric.snow,2019-10-18 21:07
PR 22841 merged eric.snow,2020-10-21 00:18
PR 23045 merged eric.snow,2020-10-30 20:53
PR 23431 merged eric.snow,2020-11-20 21:52
PR 23918 merged eric.snow,2020-12-24 05:30
PR 23929 merged eric.snow,2020-12-25 02:10
PR 31225 merged eric.snow,2022-02-09 01:19
PR 31239 merged eric.snow,2022-02-09 23:26
PR 31264 merged eric.snow,2022-02-10 22:01
Messages (22)
msg342119 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-05-10 19:01
We still have a bunch of "global" C variables (static globals, static locals, maybe thread-local storage) in our code-base that break the isolation between interpreters. I added Tools/c-globals/check-c-globals.py a while back to help identify such variables, however more have crept in. I also did not take static locals or thread-locals into account. To address the above, we should do the following: 1. update check-c-globals.py to identify static locals (and thread-locals) 2. deal with any identified globals * move them to _PyRuntimeState (or thread-locals to PyThreadState, etc.) * ignore them by adding them to Tools/c-globals/ignored-globals.txt 3. add check-c-globals.py to "make check" 4. (if "make check" isn't already there), ensure check-c-globals.py is run at some point in CI Separately, we should move fields out of _PyRuntimeState into PyInterpreterState wherever possible. That can also be done at step 2 if it's not too much work.
msg342125 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-05-10 20:13
Also, Tools/c-globals/ignored-globals.txt is a bit out of date (some vars have been removed, renamed, or moved to another file). That should get cleaned up. It might also make sense to update check-c-globals.py to verify that all variables in ignored-globals.txt actually exist.
msg352013 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-09-11 18:49
New changeset ee536b2020b1f0baad1286dbd4345e13870324af by Eric Snow in branch 'master': bpo-36876: Add a tool that identifies unsupported global C variables. (#15877) https://github.com/python/cpython/commit/ee536b2020b1f0baad1286dbd4345e13870324af
msg352032 - (view) Author: David Bolen (db3l) * Date: 2019-09-11 23:37
The new test_check_c_globals.ActualChecks test is failing with an "unexpected success" on the bolen-ubuntu buildbot (under Ubuntu 18.04.3). I can reproduce the failure in a manually built tree.
msg352063 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-09-12 09:03
@db3l, I'll take a look right away.
msg352085 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-09-12 09:51
New changeset 64535fc6c0712caef0bc46be30e661f7ccf8280e by Eric Snow in branch 'master': bpo-36876: Skip test_check_c_globals for now. (gh-16017) https://github.com/python/cpython/commit/64535fc6c0712caef0bc46be30e661f7ccf8280e
msg352208 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-09-12 15:59
New changeset 088b63ea7a8331a3e34bc93c3b873c60354b4fad by Eric Snow in branch 'master': bpo-36876: Fix the globals checker tool. (gh-16058) https://github.com/python/cpython/commit/088b63ea7a8331a3e34bc93c3b873c60354b4fad
msg354928 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-10-19 02:00
New changeset e4c431ecf50def40eb93c3969c1e4eeaf7bf32f1 by Eric Snow in branch 'master': bpo-36876: Re-organize the c-analyzer tool code. (gh-16841) https://github.com/python/cpython/commit/e4c431ecf50def40eb93c3969c1e4eeaf7bf32f1
msg356181 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-11-07 10:09
New changeset 9def81aa52adc3cc89554156e40742cf17312825 by Vinay Sajip in branch 'master': bpo-36876: Moved Parser/listnode.c statics to interpreter state. (GH-16328) https://github.com/python/cpython/commit/9def81aa52adc3cc89554156e40742cf17312825
msg357351 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-11-23 00:41
Thanks, Vinay!
msg357352 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-11-23 00:42
FYI, others have been tackling this in separate issues (e.g. Victor, anyone relative to PEP 384).
msg357353 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-11-23 00:43
And I *am* still working on the c-analyzer + a test to that runs it, so we can keep from adding more globals. :)
msg368910 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-15 01:37
More and more C extensions are converted to multiphase initialization API (PEP 489) and their global variables are moved into a new module state. bpo-39465 will make _Py_IDENTIFIER compatible with subinterpreters. See also bpo-40521 for caches like free lists and Unicode interned strings.
msg379390 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2020-10-23 00:42
New changeset 345cd37abe324ad4f60f80e2c3133b8849e54e9b by Eric Snow in branch 'master': bpo-36876: Fix the C analyzer tool. (GH-22841) https://github.com/python/cpython/commit/345cd37abe324ad4f60f80e2c3133b8849e54e9b
msg379999 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2020-10-30 21:46
New changeset 4fe72090deb7fb7bc09bfa56c92f6b3b0967d395 by Eric Snow in branch 'master': bpo-36876: Small adjustments to the C-analyzer tool. (GH-23045) https://github.com/python/cpython/commit/4fe72090deb7fb7bc09bfa56c92f6b3b0967d395
msg381511 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2020-11-20 22:39
New changeset 9f02b479e6b6b48d0c2aad621978cff82e530b15 by Eric Snow in branch 'master': bpo-36876: [c-analyzer tool] Tighten up the results and output. (GH-23431) https://github.com/python/cpython/commit/9f02b479e6b6b48d0c2aad621978cff82e530b15
msg383698 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2020-12-24 18:04
New changeset 7ec59d8861ef1104c3028678b2cacde4c5693e19 by Eric Snow in branch 'master': bpo-36876: [c-analyzer tool] Add a "capi" subcommand to the c-analyzer tool. (gh-23918) https://github.com/python/cpython/commit/7ec59d8861ef1104c3028678b2cacde4c5693e19
msg383778 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2020-12-25 22:57
New changeset 5ae9be68d9f1a628fdc920b647257f94afb77887 by Eric Snow in branch 'master': bpo-36876: [c-analyzer tool] Additional CLI updates for "capi" command. (gh-23929) https://github.com/python/cpython/commit/5ae9be68d9f1a628fdc920b647257f94afb77887
msg393785 - (view) Author: Dennis Sweeney (Dennis Sweeney) * (Python committer) Date: 2021-05-17 05:50
I'm getting the following FutureWarning on a certain regular expression. I think it just needs "[]" to become "\[\]". 0:05:36 load avg: 0.00 [ 53/427] test_check_c_globals ...\Tools\c-analyzer\c_common\tables.py:236: FutureWarning: Possible nested set at position 12 _COLSPEC_RE = re.compile(textwrap.dedent(r'''
msg412886 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2022-02-09 01:38
New changeset 77bab59c8a1f04922bb975cc4f11e5323d1d379d by Eric Snow in branch 'main': bpo-36876: Update the c-analyzer whitelist. (gh-31225) https://github.com/python/cpython/commit/77bab59c8a1f04922bb975cc4f11e5323d1d379d
msg412957 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2022-02-10 00:11
New changeset cb68788dcadf43b47292bab7816a5ed9efa69730 by Eric Snow in branch 'main': bpo-36876: Minor cleanup to c-analyzer "ignored" data.' (gh-31239) https://github.com/python/cpython/commit/cb68788dcadf43b47292bab7816a5ed9efa69730
msg413029 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2022-02-10 23:14
New changeset 80e4f262aa27a39abf3fadc19a6323fea4607a8f by Eric Snow in branch 'main': bpo-36876: Make sure the c-analyzer is checking all the source files.' (gh-31264) https://github.com/python/cpython/commit/80e4f262aa27a39abf3fadc19a6323fea4607a8f
History
Date User Action Args
2022-04-11 14:59:15 admin set github: 81057
2022-02-10 23:14:23 eric.snow set messages: +
2022-02-10 22:01:51 eric.snow set pull_requests: + <pull%5Frequest29430>
2022-02-10 00:11:02 eric.snow set messages: +
2022-02-09 23:26:00 eric.snow set pull_requests: + <pull%5Frequest29409>
2022-02-09 01:38:05 eric.snow set messages: +
2022-02-09 01:19:33 eric.snow set pull_requests: + <pull%5Frequest29396>
2021-05-17 05:50:14 Dennis Sweeney set nosy: + Dennis Sweeneymessages: +
2020-12-25 22:57:36 eric.snow set messages: +
2020-12-25 02:10:34 eric.snow set pull_requests: + <pull%5Frequest22779>
2020-12-24 18:04:44 eric.snow set messages: +
2020-12-24 05:30:08 eric.snow set pull_requests: + <pull%5Frequest22769>
2020-11-20 22:39:37 eric.snow set messages: +
2020-11-20 21:52:28 eric.snow set pull_requests: + <pull%5Frequest22322>
2020-10-30 21:46:55 eric.snow set messages: +
2020-10-30 20:53:12 eric.snow set pull_requests: + <pull%5Frequest21964>
2020-10-23 00:42:58 eric.snow set messages: +
2020-10-21 00🔞03 eric.snow set pull_requests: + <pull%5Frequest21795>
2020-05-15 01:41:00 db3l set nosy: - db3l
2020-05-15 01:37:33 vstinner set messages: +
2020-05-15 00:37:07 vstinner set components: + Subinterpreters, - Interpreter Coretitle: Global C variables are a problem. -> [subinterpreters] Global C variables are a problem
2020-02-07 15:21:02 maciej.szulik set nosy: + maciej.szulik
2019-11-23 00:43:11 eric.snow set messages: +
2019-11-23 00:42:18 eric.snow set messages: +
2019-11-23 00:41:07 eric.snow set messages: +
2019-11-07 16:01:05 yan12125 set nosy: - yan12125
2019-11-07 10:09:19 vinay.sajip set messages: +
2019-10-19 02:00:07 eric.snow set messages: +
2019-10-18 21:07:27 eric.snow set pull_requests: + <pull%5Frequest16393>
2019-09-22 04:41:55 vinay.sajip set pull_requests: + <pull%5Frequest15904>
2019-09-17 16:37:28 vinay.sajip set nosy: + vinay.sajip
2019-09-12 15:59:41 eric.snow set messages: +
2019-09-12 14:41:00 eric.snow set pull_requests: + <pull%5Frequest15681>
2019-09-12 09:51:03 eric.snow set messages: +
2019-09-12 09:23:14 eric.snow set pull_requests: + <pull%5Frequest15640>
2019-09-12 09:03:11 eric.snow set messages: +
2019-09-12 03:43:33 yan12125 set nosy: + yan12125
2019-09-11 23:37:17 db3l set nosy: + db3lmessages: +
2019-09-11 18:49:48 eric.snow set messages: +
2019-09-10 15:31:41 eric.snow set pull_requests: + <pull%5Frequest15518>
2019-09-09 11:30:34 eric.snow set pull_requests: + <pull%5Frequest15413>
2019-08-22 02🔞01 phsilva set nosy: + phsilva
2019-05-23 22:55:58 eric.snow set pull_requests: + <pull%5Frequest13445>
2019-05-17 00:46:59 eric.snow set keywords: + patchstage: needs patch -> patch reviewpull_requests: + <pull%5Frequest13283>
2019-05-11 16:09:44 christian.heimes set nosy: + christian.heimes
2019-05-10 22:19:01 pablogsal set nosy: + pablogsal
2019-05-10 20:13:09 eric.snow set messages: +
2019-05-10 19:01:42 eric.snow create