Issue 13475: Add '--mainpath'/'--nomainpath' command line options to override sys.path[0] initialisation (original) (raw)
Messages (37)
Author: Alyssa Coghlan (ncoghlan) * 
Date: 2011-11-25 00:46
PEP 395 spends a lot of time discussing ways that the current automatic initialisation of sys.path[0] can go wrong, and even the proposed improvements in that PEP don't claim to fix the default behaviour for every possible scenario (just many of the most common ones).
The unittest module gets around similar problems with test autodiscovery by providing an explicit "-t" (for 'toplevel') command line option.
While '-t' is not available for the main interpreter executable (and nor is '-d' for directory), '-p' for "path0" is a possibility.
Directory execution (for example) would then be equivalent to:
python -p dirname dirname/main.py
A separate '--nopath0' option could also be provided to explicitly disable path initialisation based on the script being executed (see http://lists.debian.org/debian-python/2011/11/msg00058.html)
Author: Eric Snow (eric.snow) *
Date: 2011-11-25 06:09
+1 Both the -p and --nopath0 would be great additions and a good match for PEP 395.
So "-p ." would be equivalent to the current implicit sys.path[0] initialization, right? Would any other effects happen with "-p" than sys.path[0] initialization? I'll follow up with my one concern (the implicit "-p .") in a follow-up message.
Author: Eric Snow (eric.snow) *
Date: 2011-11-25 08:52
The current behavior is an implicit "-p .", which causes all sorts of hard-to-figure-out problems, most of which PEP 395 is rightly trying to fix. I'm suggesting that the next step would be to make "--nopath0" the default (rendering the flag unnecessary). Since this changes the status quo, I'll try to make the case here that it's worth it.
First of all, the current implicit sys.path[0] initialization for scripts has been around since forever. Why would we change it now? The inspiration is three-fold: we said we got rid of implicit relative imports already, PEP 395 already has made a case for changing the status quo (though not to this degree), and a "-p" flag provides a simple mechanism to get the current behavior explicitly.
Like Nick says in PEP 395, "PEP 328 eliminated implicit relative imports from imported modules. This PEP proposes that the de facto implicit relative imports from main modules that are provided by the current initialisation behaviour for sys.path[0] also be eliminated."
I'm just saying we should go all the way, and that the "-p" flag would allow that. As far as I can tell, there are two meaningful use cases for the currently implicit sys.path[0]: When you are first learning Python (not actually using it)... When you are working on a new project... Let's look at both, first relative to the currently implicit "-p .", then to the explicit site.
- (current behavior) implicit sys.path[0] initialization
When a beginner is first learning Python, they learn at the interactive prompt or by running a single script. Pretty quickly they learn about writing their own modules and using import to load them.
Where do the modules live? In the CWD from which they are calling the Python executable. The current implicit "-p ." means they can import their modules right there, and not have to learn yet about sys.path. That's nice since they can concentrate their focus.
But later they will run into hard-to-figure-out problems when they start relying on what are effectively implicit relative imports. Or they will run into one of the problems that PEP 395 explains, which are likewise head-scratchers. All of this can be pretty frustrating when you still don't know about things like the import machinery.
Consider how that beginner uses the implicit sys.path[0] behavior on a project, once they have some experience (hopefully I don't misrepresent something Nick explained to me). The src/ directory of the project won't be under sys.path yet. So to quickly test out their script, they change to src/ and run their script from there (something PEP 395 rightfully proposes fine-tuning). They conveniently don't have to manually fiddle with sys.path. That's a pretty neat trick.
The problem I have with this is that there's no difference between the sys.path workaround for the script and it's expected normal usage. If you aren't aware of what's going on behind-the-scenes, you may end up with some confusing import exceptions later on.
- explicit sys.path[0] initialization
With a -p flag, it's really easy to say "-p ." and get the previous implicit behavior. Consider the impact on our beginner. The try to run their script that imports modules in their CWD, but end up with an ImportError because we no longer have an implicit "-p .".
How is this an improvement? No more mysterious default behavior. There's only one way it will fail and it will fail the same way every time. They'll learn pretty quickly (if not taught already) you have to use the -p flag if you want to import modules that aren't in the "standard" places. The ImportError message, special-cased for imports in main, could help in that regard.
For the "src/" use-case, the developer won't be able to rely on the implicit behavior anymore. A simple, explicit "-p ." fills the gap and makes it clear that they are doing something unusual. On top of that, they can use -p with the path to the "src/" directory from anywhere and not have to change their directory. All the better if -p takes advantage of the proposals in PEP 395.
In the end, this boils down to opt-in vs. opt-out. Either you opt-in to the sys.path[0] initialization by using -p, or you opt-out of the default implicit "-p ." by passing "--nopath0". It's a case of "explicit is better than implicit" without a clear reason, to me, that the status quo is worth the exception anymore in the face of the alternative.
My only concern would be backwards compatibility. How much do scripts out there rely on the implicit sys.path[0] initialization that aren't already broken by the removal of implicit relative imports? I simply don't know.
However, I do know that having the -p flag that Nick has recommended would be awesome.
Author: Éric Araujo (eric.araujo) *
Date: 2011-11-25 16:53
I don’t understand all the issues and it’s too late for me to read all the thread, but I hope these comments are helpful:
If our goal is to help naïve users, then one or two new options wouldn’t help (people want to run “python path/to/thing.py” and want it to Just Work™)
Getting a very helpful error message is second-best to Just Working™
This probably needs a wider audience, python-ideas or python-dev, and maybe a PEP to explain all the issues and all the considered solutions.
Author: Eric Snow (eric.snow) *
Date: 2011-11-25 19:22
First of all, I don't want Nick's proposal in this issue (the -p and --nopath0 flags) to be misunderstood because of me. His is a great idea that will make a really useful shortcut available and will not change any current behavior.
My (overly) long message is an attempt to justify going with a more drastic variation, to which your response applies exclusively. I stand by what I said, but I would be quite happy with the original proposal, to which your response does not apply. :)
Just wanted to make all that clear so I don't get in the way of a good thing. :)
Your comments are spot on. Your concerns are exactly the ones I would need to assuage before my alternative would be acceptable. When I get a chance I'll take this to python-ideas.
Author: Alyssa Coghlan (ncoghlan) *
Date: 2011-11-26 05:19
Yeah, sorry Eric (Snow), you're trying to bite off way more than is reasonable by proposing to removing sys.path[0] auto-initialisation. While it has its problems, it's also far too entrenched in people's expections of Python's behaviour to consider removing at this late date. The earliest such a radical change could likely be considered is Python 4.
Éric (Araujo): the "just do the right thing" aspect is covered by PEP 395. This is an orthogonal proposal that allows power users the ability to override the automatic initialisation to handle those cases where it goes wrong.
That said, I may still make this suggestion part of PEP 395 rather than a distinct proposal, even though there's no direct dependency (in either direction) between this suggestion and the other ideas in that PEP.
Author: Eric Snow (eric.snow) *
Date: 2011-11-26 05:21
Yeah, the more I think about it, the more I agree it's Python 4 fodder.
Author: Éric Araujo (eric.araujo) *
Date: 2011-11-26 12:50
Eric: Actually I posted my comments after reading only Nick’s message, not yours, so no worry :)
Nick:
the "just do the right thing" aspect is covered by PEP 395. This is great. Thanks for clarifying.
This is an orthogonal proposal that allows power users the ability to override the automatic initialisation to handle those cases where it goes wrong. If I’m understanding correctly, #12934 is one of those issues. I agree that an option would be better than nothing, but 1) it would not help with older Pythons 2) it would not help naïve users 3) it would complicate the command-line interface of Python. (Not saying I’m against the idea, just listing cons.)
Author: Alyssa Coghlan (ncoghlan) *
Date: 2011-11-27 00:30
Zbigniew posted a nice summary of some of the issues sys.path[0] autoinitialisation can cause to python-dev: http://mail.python.org/pipermail/python-dev/2011-November/114668.html
Author: Matthew Woodcraft (mattheww)
Date: 2011-11-27 20:42
The proposed --nopath0 option is something I've wished I had in the past.
If this is added, it would be good if it could be given a single-letter form too, because it's an option that would be useful in #! lines (they don't reliably support using more than one command-line argument, and single-letter switches can be combined while long-form ones can't).
Author: Alyssa Coghlan (ncoghlan) *
Date: 2011-11-29 10:48
I realised "-P" is available for use as the short form of the "suppress sys.path[0] initialisation" option.
So "-p" would override the calculation of sys.path[0], while "-P" would switch it off completely (similar to the way "-S" and "-E" switch off other aspects of the normal initialisation process).
Whether or not to provide "--path0" and "--nopath0" as long form alternatives would then be a separate question.
Author: Eric Snow (eric.snow) *
Date: 2012-05-26 18:45
Here's a stab at implementing -p/-P. There are a couple warnings that I'm not sure about and I've undoubtedly missed some detail, but it should be pretty close.
Author: Antoine Pitrou (pitrou) *
Date: 2012-05-26 19:07
Before this is assigned a short option form, I would like to ask whether anybody but experts will be able to make a proper use of this option. (I also don't understand what it adds over PYTHONPATH)
As for the patch, it lacks error checking when calling C API functions.
Author: Eric Snow (eric.snow) *
Date: 2012-05-26 22:20
Before this is assigned a short option form, I would like to ask whether anybody but experts will be able to make a proper use of this option.
Do you mean relative to a long form? And what would constitute improper use for the option?
As Nick noted earlier in this issue, the implicit setting of sys.path[0] is a convenience for some cases, and I can appreciate that. My complaint is that it is not an obvious behavior. Furthermore, in an import system that has eliminated implicit relative imports it can lead to confusing behavior. I consider Nick's proposal the best solution given the constraints we have.
If people do not use it, the status quo is unaffected. If they do use it, they will be explicitly affecting sys.path[0]. If they use it without knowing what it's for, it will at worst cause imports to happen from unexpected sources (or not happen at all). To me this doesn't seem that different from the way things are now when someone doesn't understand about sys.path[0] initialization.
(I also don't understand what it adds over PYTHONPATH)
It provides an explicit alternative to the default implicit insertion to sys.path[0]. If the default behavior were no implicit initialization, then I'd agree that PYTHONPATH is sufficient.
As for the patch, it lacks error checking when calling C API functions.
Ah, yes. Spaced it. Patch updated (and warnings fixed).
This brings me to ask what the behavior should be when we have errors come back from those C API functions? In the patch I just have it fall back to the default sys.path[0] behavior. However, wouldn't an error indicate a deeper problem? If so, shouldn't Py_FatalError() be called?
Author: Antoine Pitrou (pitrou) *
Date: 2012-05-26 22:27
Before this is assigned a short option form, I would like to ask whether anybody but experts will be able to make a proper use of this option.
Do you mean relative to a long form?
Yes.
And what would constitute improper use for the option?
As a replacement of PYTHONPATH or "setup.py install" or "setup.py develop", I guess.
As Nick noted earlier in this issue, the implicit setting of sys.path[0] is a convenience for some cases, and I can appreciate that. My complaint is that it is not an obvious behavior. Furthermore, in an import system that has eliminated implicit relative imports it can lead to confusing behavior.
Agreed. I'm not arguing against the principle.
(I also don't understand what it adds over PYTHONPATH)
It provides an explicit alternative to the default implicit insertion to sys.path[0]. If the default behavior were no implicit initialization, then I'd agree that PYTHONPATH is sufficient.
Ah, indeed, I'd missed that.
This brings me to ask what the behavior should be when we have errors come back from those C API functions? In the patch I just have it fall back to the default sys.path[0] behavior.
Well, no, errors should not pass silently. The error should be propagated. Here, it probably means print the error and abort (or whatever strategy the rest of the function adopts).
However, wouldn't an error indicate a deeper problem? If so, shouldn't Py_FatalError() be called?
Py_FatalError() is a low-level exit; it dumps core. PyErr_Print() followed by a proper exit may be better.
Author: Eric Snow (eric.snow) *
Date: 2012-05-26 23:55
did you mean to change the title? this isn't about overriding sys.path, but rather just about explicitly dictating the initialization of sys.path[0].
Author: Antoine Pitrou (pitrou) *
Date: 2012-05-27 00:01
did you mean to change the title?
No, I think it's a bug of the roundup e-mail gateway.
Author: Alyssa Coghlan (ncoghlan) *
Date: 2012-05-27 06:28
FWIW, I now think this should only be a long option. Short options are precious, and this is an unusual enough use case that I'm not yet sure it deserves one.
In particular, we may decide to use "-p" later for adding directories to sys.path, rather than specifically overriding sys.path[0]
Author: Alyssa Coghlan (ncoghlan) *
Date: 2012-05-27 06:29
FWIW, I now think this should only be a long option. Short options are precious, and this is an unusual enough use case that I'm not yet sure it deserves one.
In particular, we may decide to use "-p" later for adding directories to sys.path, rather than specifically overriding sys.path[0]
Author: Eric Snow (eric.snow) *
Date: 2012-05-28 02:03
Long options only would be fine with me. So "--path0" and "--nopath0"?
Author: Eric Snow (eric.snow) *
Date: 2012-05-30 04:17
here's an updated patch with long options only. I'm only 95% confident about my changes to getopt.c...there's probably a better way to do it. However, the patch stands on its own two feet.
Author: Eric Snow (eric.snow) *
Date: 2012-06-12 14:25
any chance on this for 3.3?
Author: Alyssa Coghlan (ncoghlan) *
Date: 2012-06-13 13:49
I was tempted to just add this (perhaps as a -X option) but, on reflection, I'm going to go with "No, not for 3.3".
I want to take a long hard look at the whole sys.path[0] initialisation process when I update PEP 395 to account for namespace packages, and it doesn't make sense to skip ahead with part of the conclusion. (I may even end up splitting that PEP into two pieces)
I'll also look into the embedding API, to see if/how embedding applications can fully control sys.path initialisation.
Python has survived without this feature for the last couple of decades, there's no need to be hasty in adding it now.
Author: Eric Snow (eric.snow) *
Date: 2012-07-10 05:25
Your plan sounds good, Nick.
Author: Gregory P. Smith (gregory.p.smith) *
Date: 2012-08-17 00:39
I'd also like a command line flag to override PYTHONPATH (which could also be used in combination with -E so that you could still set the PYTHONPATH while ignoring everything else). I'll file a separate feature request for that.
Author: Kristján Valur Jónsson (kristjan.jonsson) *
Date: 2012-11-23 09:52
Butting in here: Early on in the startup, a search is made for "site.py" using automatic sys.path settings. Do the suggestions here propose to override this? I know there is a -s flag, but all these flags start to be confusing.
I have been looking for ways to completely ignore automatic settings for sys.path, either from internal heuristics (either build environment or installed environment) or PYTHONPATH. The reason for this is that as a developer in a large project, one that has many branches, each with their potentially different version of python compiled and with their own libraries, and where a "standard" python distribution may even be installed on the machine as well, we still want to be able to run scripts with each branch' own python with relative simplicity with some sort of guarantee that we are not inadvertently pulling in modules from other, unrelated branches, or god forbid, the "installed" python.
PEP 405 seemed to go a long way, although it didn't provide a means in the pyvenv.cfg to completely override sys.path from the startup.
I wonder if there isn't a synergy here with PEP 405, e.g. would it make sense to have a pyvenv.cfg file next to the main.py file?
Author: Eric Snow (eric.snow) *
Date: 2012-11-28 07:54
This proposal only affects the initialization of sys.path[0], and not any of the other sys.path entries made by site.py or otherwise.
Author: Kristján Valur Jónsson (kristjan.jonsson) *
Date: 2012-11-28 09:55
before or after running site.py?
Author: Alyssa Coghlan (ncoghlan) *
Date: 2012-11-28 09:59
I'd suggest this would be applied at the same time as when sys.path[0] gets set now, but that kind of messy complication is why I decided we really needed to step back and start trying to clean up what we already have before we started layering yet more complications onto the initialisation process.
Author: Kristján Valur Jónsson (kristjan.jonsson) *
Date: 2012-11-28 10:33
Absolutely, Nick. Perhaps we ought to start gathering a list of real world requirements, i.e. how do people run, or want to run, python?
Author: Matthew Woodcraft (mattheww)
Date: 2014-04-18 18:26
For the record: the '-I' option (#16499) in Python 3.4 disables sys.path[0] initialisation (among other things).
Author: Alyssa Coghlan (ncoghlan) *
Date: 2018-03-18 13:28
This question recently came up again over in https://bugs.python.org/issue33053#msg314040.
With the assorted startup refactorings that were landed for 3.7, it likely makes sense to take another run at this for 3.8.
Author: Alyssa Coghlan (ncoghlan) *
Date: 2018-03-21 12:12
Over in https://bugs.python.org/issue33053#msg314192, I came up with --basepath <dir> as another possible spelling (--no-basepath would then be an option for turning it off.
The main argument against that name is that we use "base" to mean something different in the sys module: "base_prefix" and "base_exec_prefix" refer to the info for the parent environment when in a virtual environment.
Another alternative would be "--mainpath", since what we're really overriding is the default path entry that's normally derived from the location of __main__. As an added bonus, we could define "main path" as a glossary entry, rather than having to talk about "sys dot path zero" all the time.
Author: Alyssa Coghlan (ncoghlan) *
Date: 2019-04-08 11:49
Retitled the issue using the --mainpath spelling, as I wanted to link to it from the PEP 582 discussion, and that's my current favourite from the various ideas I've had since first filing the issue.
(I decided I didn't like --basepath because we already have sys.base_prefix that refers to something entirely different)
Author: STINNER Victor (vstinner) *
Date: 2022-02-24 00:32
See also "Python flag/envvar not to put current directory to sys.path (but don’t ignore PYTHONPATH)" discussion: https://discuss.python.org/t/python-flag-envvar-not-to-put-current-directory-to-sys-path-but-dont-ignore-pythonpath/4235/2
Author: STINNER Victor (vstinner) *
Date: 2022-02-24 01:55
The problem of long option names is that they cannot be used in a Unix shebang: "#!/usr/bin/python3 --long-option".
I propose to add the -P option to not add sys.path[0]: I wrote GH-31542.
Author: STINNER Victor (vstinner) *
Date: 2022-02-24 02:55
See also bpo-16202 "sys.path[0] security issues".
History
Date
User
Action
Args
2022-04-11 14:57:24
admin
set
github: 57684
2022-02-24 02:55:12
vstinner
set
messages: +
2022-02-24 01:55:08
vstinner
set
messages: +
2022-02-24 01:54:09
vstinner
set
stage: patch review
pull_requests: + <pull%5Frequest29665>
2022-02-24 00:32:02
vstinner
set
nosy: + vstinner
messages: +
2021-11-26 14:29:46
nisaac
set
nosy: + nisaac
2019-04-08 11:49:07
ncoghlan
set
type: enhancement
title: Add '-p'/'--path0' command line option to override sys.path[0] initialisation -> Add '--mainpath'/'--nomainpath' command line options to override sys.path[0] initialisation
messages: +
versions: + Python 3.9, - Python 3.4
2018-03-21 12:26:54
jwilk
set
nosy: + jwilk
2018-03-21 12:12:51
ncoghlan
set
messages: +
2018-03-18 13:28:01
ncoghlan
set
messages: +
2014-05-23 00:29:13
berker.peksag
link
2014-04-18 18:26:06
mattheww
set
messages: +
2012-11-28 10:33:59
kristjan.jonsson
set
messages: +
2012-11-28 09:59:36
ncoghlan
set
messages: +
2012-11-28 09:55:55
kristjan.jonsson
set
messages: +
2012-11-28 07:54:39
eric.snow
set
messages: +
2012-11-23 09:52:39
kristjan.jonsson
set
messages: +
2012-11-23 09:38:04
kristjan.jonsson
set
nosy: + kristjan.jonsson
2012-08-17 00:39:39
gregory.p.smith
set
nosy: + gregory.p.smith
messages: +
2012-07-10 05:25:53
eric.snow
set
messages: +
2012-06-17 05:49:30
ncoghlan
set
assignee: ncoghlan ->
2012-06-13 13:49:16
ncoghlan
set
assignee: ncoghlan
messages: +
versions: + Python 3.4
2012-06-12 14:25:42
eric.snow
set
messages: +
2012-05-30 04:17:27
eric.snow
set
files: + issue13475_long_only.diff
messages: +
2012-05-28 02:03:19
eric.snow
set
messages: +
2012-05-27 06:29:28
ncoghlan
set
messages: +
2012-05-27 06:28:20
ncoghlan
set
messages: +
2012-05-27 00:01:59
pitrou
set
title: Add '-p'/'--path0' command line option to override sys.path -> Add '-p'/'--path0' command line option to override sys.path[0] initialisation
2012-05-27 00:01:32
pitrou
set
messages: +
2012-05-26 23:55:11
eric.snow
set
messages: +
2012-05-26 22:27:42
pitrou
set
messages: +
title: Add '-p'/'--path0' command line option to override sys.path[0] initialisation -> Add '-p'/'--path0' command line option to override sys.path
2012-05-26 22:20:32
eric.snow
set
files: + issue13475_2.diff
messages: +
2012-05-26 19:07:15
pitrou
set
nosy: + pitrou
messages: +
2012-05-26 19:01:14
eric.snow
set
files: + issue13475_1.diff
2012-05-26 19:00:56
eric.snow
set
files: - issue13475_1.diff
2012-05-26 18:57:10
eric.snow
set
files: + issue13475_1.diff
2012-05-26 18:56:44
eric.snow
set
files: - issue13475_1.diff
2012-05-26 18:45:42
eric.snow
set
files: + issue13475_1.diff
keywords: + patch
messages: +
2011-11-29 10:48:09
ncoghlan
set
messages: +
2011-11-27 20:42:44
mattheww
set
nosy: + mattheww
messages: +
2011-11-27 00:30:41
ncoghlan
set
messages: +
2011-11-26 23:33:12
Arfrever
set
nosy: + Arfrever
2011-11-26 12:50:02
eric.araujo
set
messages: +
2011-11-26 05:21:26
eric.snow
set
messages: +
2011-11-26 05:19:47
ncoghlan
set
messages: +
2011-11-25 19:22:05
eric.snow
set
messages: +
2011-11-25 16:53:02
eric.araujo
set
nosy: + eric.araujo
messages: +
2011-11-25 08:53:01
eric.snow
set
messages: +
2011-11-25 06:09:29
eric.snow
set
messages: +
2011-11-25 02:01:48
eric.snow
set
nosy: + eric.snow
2011-11-25 00:46:33
ncoghlan
create