Remove outdated Tools/scripts/ scripts (original) (raw)

September 29, 2022, 2:17pm 1

Hi,

The Python Security Response Team received a report of a shell code injection vulnerability in the Tools/scripts/get-remote-certificate.py script. I decided to fix it: issue #97612.

Honestly, I don’t think that it’s still relevant to maintain such script in Python, and I would suggest to remove it in Python 3.12. For example, this one is a thin wrapper to the openssl s_client -connect HOST:PORT -showcert command. IMO using directly the openssl command is simple enough.

There are 74 Python scripts in Tools/scripts/. How many are outdated? The dutree.py comes with a documentation (an email) written in… 1992. Many scripts are short examples of what can be done in a few lines of Python to enhance some existing Unix commands. The which.py script can now be replaced with the shutil.which() function which has a better implementation.

I’m not sure how to proceed. Should we go through each script and decide if it should be removed or not? Or remove all of them and then wait for complains? :smiley:

Some scripts are used to build Python, like check_extension_modules.py and deepfreeze.py. Maybe they should live in a separated directory? Is win_add2path.py used by the Windows installer?

If some scripts are useful, maybe the code should be moved into the stdlib? For example, should diff.py becomes python3 -m difflib CLI? Currently, python3 -m difflib runs a self-test, IMO such test should belong to Lib/test/test_difflib.py instead.

Or should we move these scripts to a separated project/Git repository?

Well, what do you think?

Victor

barry (Barry Warsaw) September 29, 2022, 3:40pm 2

gpshead (Gregory P. Smith) September 29, 2022, 3:47pm 3

+1

LOL diff.py, let’s just get rid of that while also moving -m difflib test code into a unit test. No need for -m difflib to do anything. We’d have to support it’s use as a diff tool forever I’d we did that. Every command line system has a diff tool already.

mwichmann (Mats Wichmann) September 29, 2022, 4:08pm 4

Just as a kind of data point, a “mature” (20+ yrs) project (written in Python) I work on has been moving old snippets of this nature off into a separate repository of unsupported tools and tricks, and out of the main repo where it might end up getting distributed with the project and considered something that needs supporting.

steve.dower (Steve Dower) September 29, 2022, 4:19pm 5

Okay, consider yourself corrected. (Though only for the executable and Nuget packages, not the Store package.)

But I’d be more than happy to remove them. There’s so little benefit in including them, and non-zero risk, as it turns out.

ericvsmith (Eric V. Smith) September 29, 2022, 5:06pm 6

I think they should all be removed. Too much risk, for things that are never looked at.

If something is used to build Python, that’s another story. But I think it still doesn’t need to be installed on end user systems.

nad (Ned Deily) September 29, 2022, 6:01pm 7

I would have no objection to also removing them from our macOS installers. I added them years ago as a convenience to users but deliberately did not make them very visible. I don’t recall ever seeing any feedback from users; I doubt anyone would miss them there. And if anyone does, I suppose they could be bundled up as a PyPI-installable package.

barry (Barry Warsaw) September 29, 2022, 6:07pm 8

JFDI! I removed Tools/pynche and nobody complained :wink:

steve.dower (Steve Dower) September 29, 2022, 6:38pm 9

What else should go from the Tools directory (when installed for users)?

guido (Guido van Rossum) September 29, 2022, 6:57pm 10

I think we should remove everything from those directories that isn’t needed to build. I don’t know how to tell what that is. For sure the Tools/demo directory can go – there’s even a demo of remote execution there! :slight_smile:

There are also some benchmarks (ccbench, iobench, stringbench) that probably ought to be moved to pyperformance so they can be used and maintained.

Etc., etc.

malemburg (Marc-André Lemburg) September 29, 2022, 7:18pm 11

I think we need to be a bit more careful here, since some scripts are
useful to (re)generate source files, update or fix them.

None of these need to be user installed. At least on Unix platforms,
“make install” does not install them anywhere, AFAIK.

brandtbucher (Brandt Bucher) September 29, 2022, 8:12pm 12

Just my data point: besides the tools that are required to build, regenerate, etc., the only thing I use here is summarize_stats.py.

It converts the counters dumped by --with-pystats builds into a nice markdown format. I use it at least several times monthly.

steve.dower (Steve Dower) September 29, 2022, 8:24pm 13

I filed https://github.com/python/cpython/issues/97649 to remove the files from being installed on Windows. Not planning to touch anything in the repository Tools directory.

gpshead (Gregory P. Smith) September 29, 2022, 8:28pm 14

I suggest any of us who actually use these tools (aside from the ones invoked by build scripts which are thus clearly referred to by our build systems) edit them to add a top level comment with our name and date describing what we use it for and why. Consider it a freshness indicator. We don’t need to be shipping/installing most of these, but that at least helps indicate which subset are owned and used.

brettcannon (Brett Cannon) September 29, 2022, 8:58pm 15

At worst we can write up the list of files on a whiteboard at the core dev sprints and people can erase the files and directories they want to keep. That might the fastest way to get consensus on the list of things to drop.

gpshead (Gregory P. Smith) September 29, 2022, 9:22pm 16

File a GitHub issue with the current list as check box bullet items and let all of us committers modify the checkboxes for a week. That way it is global.

vstinner (Victor Stinner) September 30, 2022, 10:59am 17

vstinner (Victor Stinner) September 30, 2022, 12:37pm 18

Here is a first PR to remove 42 scripts, IMO the least controversial scripts: gh-97669: Remove outdated example scripts by vstinner · Pull Request #97674 · python/cpython · GitHub

I prefer to write separated PRs for the remaining scripts that I plan to remove.

vstinner (Victor Stinner) September 30, 2022, 1:15pm 19

It’s more complicated than what I expected: diff.py and ndiff.py are copied in the difflib documentation. I will propose a separated PR for these scripts.

eryksun (Eryk Sun) September 30, 2022, 1:43pm 20

“win_add2path.py” is not used by the installer. I don’t know whether any project uses this script. PR 1645 improves it in several ways, but this PR has been open for over 5 years without getting merged. The one thing this script does that the installer doesn’t do is add the user scripts directory to PATH. That’s for --user installations, which pip enables automatically if it isn’t permitted to install to the system site packages.