Issue 32300: print(os.environ.keys()) should only print the keys (original) (raw)

Issue32300

Created on 2017-12-13 08:37 by Aaron.Meurer, last changed 2022-04-11 14:58 by admin.

Messages (20)
msg308190 - (view)	Author: Aaron Meurer (Aaron.Meurer)	Date: 2017-12-13 08:37
Take the following scenario which happened to me recently. I am trying to debug an issue on Travis CI involving environment variables. Basically, I am not sure if an environment variable is being set correctly. So in my code, I put print(os.environ.keys()) The reason I put keys() was 1, I didn't care about the values, and 2, I have secure environment variables set on Travis. To my surprise, in the Travis logs, I found something like this KeysView(environ({'TRAVIS_STACK_FEATURES': 'basic cassandra chromium couchdb disabled-ipv6 docker docker-compose elasticsearch firefox go-toolchain google-chrome jdk memcached mongodb mysql neo4j nodejs_interpreter perl_interpreter perlbrew phantomjs postgresql python_interpreter rabbitmq redis riak ruby_interpreter sqlite xserver', 'CI': 'true', ..., 'MANPATH': '/home/travis/.nvm/versions/node/v7.4.0/share/man:/home/travis/.kiex/elixirs/elixir-1.4.5/man:/home/travis/.rvm/rubies/ruby-2.4.1/share/man:/usr/local/man:/usr/local/clang-3.9.0/share/man:/usr/local/share/man:/usr/share/man:/home/travis/.rvm/man'})) So instead of just printing the keys like I asked for, it printed the whole environment, plus "KeysView(environ(". Included here was my secure environment variable. Now, fortunately, Travis hides the contents of secure environment variables in the logs, but it didn't used to (https://blog.travis-ci.com/2017-05-08-security-advisory). Aside from being a potential security issue, it's just annoying that it prints the whole environment. The values are much larger than the keys. With a normal dictionary, print(d.keys()) just prints the keys: >>> print(dict(a=1, b=2).keys()) dict_keys(['a', 'b'])
msg308199 - (view)	Author: Cheryl Sabella (cheryl.sabella) *	Date: 2017-12-13 13:30
For your current situation, list(os.environ) or iter(os.environ) both return keys only. It looks like the __repr__ on the class for os.environ is printed for os.environ (which is expected). For os.environ.keys(), the same __repr__ is wrapped as a KeysView, for os.environ.values(), it's wrapped as a ValuesView, and os.environ.items(), as an ItemsView. In 2.7, os.environ.keys() print just keys.
msg308200 - (view)	Author: R. David Murray (r.david.murray) *	Date: 2017-12-13 13:50
This is a consequence of the repr used by KeysView, which it inherits from MappingView. I agree that the result is surprising, but there may not be a generic fix. It's not entirely clear what KeysView should do here, but presumably we could at least add a repr to environ's use of it that would produce something useful. Note that a workaround when doing something like a travis debug is to print os.environ._data.keys(), which will print the bytes versions of the keys. I wouldn't recommend depending on that continuing to, though ;)
msg308219 - (view)	Author: STINNER Victor (vstinner) *	Date: 2017-12-13 16:59
Usually, I use print(sorted(os.environ)) since I prefer a sorted list and it prevents such issue :-) David: > I agree that the result is surprising, but there may not be a generic fix. What about something like: vstinner@apu$ ./python -c 'import os; print(os.environ.keys())' <KeysView ['LS_COLORS', 'XDG_MENU_PREFIX', 'LANG', 'GDM_LANG', 'HISTIGNORE', 'HISTCONTROL', 'DISPLAY', 'HOSTNAME', 'EDITOR', 'COLORTERM', 'DESKTOP_AUTOSTART_ID', 'USERNAME', 'XDG_VTNR', 'SSH_AUTH_SOCK', 'XDG_SESSION_ID', 'USER', 'DESKTOP_SESSION', 'GTK2_RC_FILES', 'PWD', 'SSH_ASKPASS', 'HOME', 'JOURNAL_STREAM', 'XDG_SESSION_TYPE', 'XDG_DATA_DIRS', 'XDG_SESSION_DESKTOP', 'LOADEDMODULES', 'MAIL', 'WINDOWPATH', 'VTE_VERSION', 'TERM', 'SHELL', 'QT_IM_MODULE', 'XMODIFIERS', 'XDG_CURRENT_DESKTOP', 'XDG_SEAT', 'SHLVL', 'PYTHONPATH', 'WINDOWID', 'MODULEPATH', 'GDMSESSION', 'GNOME_DESKTOP_SESSION_ID', 'LOGNAME', 'DBUS_SESSION_BUS_ADDRESS', 'XDG_RUNTIME_DIR', 'XAUTHORITY', 'PATH', 'PS1', 'MODULESHOME', 'MAKEFLAGS', 'HISTSIZE', 'SESSION_MANAGER', 'LESSOPEN', 'BASH_FUNC_module%%', 'BASH_FUNC_scl%%', '_', 'OLDPWD']> vstinner@apu$ git diff diff --git a/Lib/_collections_abc.py b/Lib/_collections_abc.py index a5c7bfcda1..5e524dffbf 100644 --- a/Lib/_collections_abc.py +++ b/Lib/_collections_abc.py @@ -719,6 +719,9 @@ class KeysView(MappingView, Set): def __iter__(self): yield from self._mapping + def __repr__(self): + return '<KeysView %r>' % list(self) + KeysView.register(dict_keys) list(key_view) is valid. I mimicked dict views implementation: static PyObject * dictview_repr(_PyDictViewObject *dv) { PyObject *seq; PyObject *result; seq = PySequence_List((PyObject *)dv); if (seq == NULL) return NULL; result = PyUnicode_FromFormat("%s(%R)", Py_TYPE(dv)->tp_name, seq); Py_DECREF(seq); return result; }
msg308220 - (view)	Author: STINNER Victor (vstinner) *	Date: 2017-12-13 17:00
If we decide to change abc.KeysView.__repr__, IMHO we should also modify abc.ValuesView.__repr__, and maybe also abc.ItemsView.__repr__.
msg308223 - (view)	Author: R. David Murray (r.david.murray) *	Date: 2017-12-13 17:21
Agreed about the other classes if we change this. Your solution looks reasonable to me.
msg308241 - (view)	Author: STINNER Victor (vstinner) *	Date: 2017-12-13 20:08
Cheryl: would you like to work on a PR? If yes, tests are needed.
msg308244 - (view)	Author: Serhiy Storchaka (serhiy.storchaka) *	Date: 2017-12-13 20:29
Don't make KeysView.__repr__ and ValuesView.__repr__ containing the lists of all keys and values. This will make the repr of the view of a mapping which is a proxy of an external DB containing the full content of that DB, which can be gigabytes. See for example rejected .
msg308249 - (view)	Author: Aaron Meurer (Aaron.Meurer)	Date: 2017-12-13 21:04
So the best fix is to just override keys() in the _Environ class, so that it returns an EnvironKeysView class that overrides __repr__?
msg308251 - (view)	Author: STINNER Victor (vstinner) *	Date: 2017-12-13 21:29
> Don't make KeysView.__repr__ and ValuesView.__repr__ containing the lists of all keys and values. This will make the repr of the view of a mapping which is a proxy of an external DB containing the full content of that DB, which can be gigabytes. See for example rejected . The current implementation displays repr(self._mapping): class MappingView(Sized): ... def __repr__(self): return '{0.__class__.__name__}({0._mapping!r})'.format(self) For your proxy example: what is the proxy? The KeysView subtype, or the mapping? repr(list), repr(dict) and repr(set) all render their full content in the result, no? repr() on a list of 1 million of items *will* render all items, even if nobody wants to read such very long string :-) If you want to get a smarter __repr__() than the default implementation: just override __repr__(), no? I don't see why KeysView must be special.
msg308266 - (view)	Author: Serhiy Storchaka (serhiy.storchaka) *	Date: 2017-12-13 23:30
shelve.Shelf is the example of such kind.
msg308309 - (view)	Author: Cheryl Sabella (cheryl.sabella) *	Date: 2017-12-14 16:05
Victor, Thanks! Yes, I can work this. It may take a few days for me to get to it.
msg308314 - (view)	Author: STINNER Victor (vstinner) *	Date: 2017-12-14 16:28
Currently, repr(Shelf.keys()) doesn't dump the content of the shelf: >>> import pickle, shelve >>> s=shelve.Shelf({b'key': pickle.dumps('value')}) >>> k=s.keys() >>> k KeysView(<shelve.Shelf object at 0x7fba1f6189b0>) >>> list(k) ['key'] >>> list(s.values()) ['value'] I understand that changing KeysView.__repr__() changes repr(Shelf.keys()) behaviour.
msg308315 - (view)	Author: STINNER Victor (vstinner) *	Date: 2017-12-14 16:29
Cheryl: "Thanks! Yes, I can work this. It may take a few days for me to get to it." It seems like we have a disagreement on how to fix this issue. We should first agree on the best solution.
msg308321 - (view)	Author: Serhiy Storchaka (serhiy.storchaka) *	Date: 2017-12-14 17:14
Options: 1. Use the subclass of KeyView with overridden __repr__ for os.environ.keys(). This will add a code for pretty rare use case. 2. Remove os.environ.__repr__. You can use repr(os.environ.copy()) or repr(dict(os.environ)) if you want to see a content. 3. Do not change anything. You can use repr(list(os.environ.keys())) if you want to see only keys.
msg308334 - (view)	Author: Aaron Meurer (Aaron.Meurer)	Date: 2017-12-14 20:37
Serhiy, isn't option 4? 4. Make KeysView.__repr__ show list(self). Add a custom wrapper for Shelf's KeysView so that it doesn't do this. This seems to be what Victor is suggesting. It makes the most sense to me for the common (i.e., default) case to be to show the keys (and just the keys), and for use cases that want otherwise to subclass and modify.
msg308335 - (view)	Author: Serhiy Storchaka (serhiy.storchaka) *	Date: 2017-12-14 20:43
With option 4 we need to add a custom wrapper not only for Shelf's KeysView, but for KeysView of all similar proxy classes, including classes in third-party code not available for us. This is impossible.
msg308338 - (view)	Author: Aaron Meurer (Aaron.Meurer)	Date: 2017-12-14 20:53
Can't third party code write their own proxies? Why do we have to do that?
msg308357 - (view)	Author: Cheryl Sabella (cheryl.sabella) *	Date: 2017-12-15 00:48
I'm missing something here, so please forgive me for asking, but why is it bad to add: def keys(self): return self._data.keys() def values(self): return self._data.values() def items(self): return self._data.items() to the _Environ class?
msg308358 - (view)	Author: Cheryl Sabella (cheryl.sabella) *	Date: 2017-12-15 00:51
Never mind. I see it's in bytes. I missed that the first time. But, if it wasn't in bytes, would that be an OK solution? Or is it bad to override those methods in that class?