[Python-Dev] Yearly PyPI breakage (original) (raw)

Donald Stufft donald at stufft.io
Thu May 5 19:07:52 EDT 2016

On May 5, 2016, at 6:22 PM, Stefan Krah <stefan at bytereef.org> wrote:

Nick Coghlan <ncoghlan gmail.com> writes: I know you're not happy with myself and the other distutils-sig folks regarding the decision to deprecate and remove automatic link spidering, More accurately: Not happy with the removal of the checksummed "explicit" mode. What I would have preferred is a FreeBSD-like ports system. FreeBSD has been used in huge and secure installations, so the I don't buy the reliability and security arguments that are used in favor of centralization. But centralization seems to be a goal in and of itself these days (and that isn't limited to Python).

Let me say outright that there is nothing wrong with a well designed, ports like system. However, PyPI was not that and the vast bulk of packages were not using it in that fashion. At the time of PEP 470 being written a mere 59 out of 65,232 (at the time) projects were utilizing the feature that allowed people to safely do a ports like system. That's roughly 0.09% of all of the projects on PyPI which meant those projects were in a very serious minority. Thus it is only natural that the expectations end users had when installing software was that it was coming from PyPI, not from some external host and they made decisions based around those expectations and when those expectations were violated they came to the people operating PyPI and developing pip and were at best very confused, and at worst irate at us.

It's also true that there are benefits and cons to both a repository centric approach and a ports style approach, however since we were attempting to do both in one repository it typically meant that we got all of the cons of both systems, but we couldn't ever take advantage of any of the unique benefits of either system since it would only apply to some fraction of the projects. This meant that improvements that would help people installing 99.11% of projects would end up getting blocked by 0.09% of projects.

At the end of the day, attempting to both things in one repository just wasn't working out very well in the general case, so we needed (IMO) to pick one way and stop supporting the other way. I personally think it's a pretty obvious choice to go the way that we did, given that the overwhelming majority of projects were already being used that way.

That being said, I don't think the characterization of "centralizing" is really accurate here. People are still free to host their projects wherever they want and pip (and easy_install) can be used to install them. All it requires is telling pip where it needs to look in addition to the default location (PyPI), which can be configured via the CLI, environment variables, or a number of configuration files at the system, user, and virtual environment level. You're free to continue providing a page on PyPI to enable discovery of your project and you can include in that page instructions on how to install your project.

nor with the PSF regarding the current Terms of Service around uploads to PyPI, but that doesn't make it OK to start off-topic threads on python-dev just because you're a CPython core developer in addition to being a PyPI user. Alexander thought otherwise: https://mail.python.org/pipermail/python-dev/2015-October/141840.html "Given that ensurepip is part of stdlib, I am not sure this is an accurate statement. Even if it was, did you make any effort to discuss the proposal outside of a small group subscribed to distutils ML?" I completely agree with that.

We've sort of hijacked the PEP process to do something that it wasn't really meant to do. Changes like PEP 470 have nothing to do with what Python itself does and thus it's really outside of the realm of where python-dev really needs a say or notification. In Alexander's example of PEP 495 it was talking about adding a new API to Python itself, thus it makes sense that python-dev should be told about that PEP, and in cases where we needed to change the standard library we've done that discussion on python-dev (such as with PEP 453).

The python-dev list is not the list to notify people of changes to things in all corners of the Python ecosystem. If people want to get involved or keep abreast of the changes in the external projects of PyPI, pip, setuptools, etc then they should subscribe to the places that those changes get discussed-- which is distutils-sig. Packaging already has enough problems with getting bogged down in "why wasn't I consulted" and bikeshedding, I don't think that inviting folks (and nothing against python-dev here) a chance to try and re-legislate an agreed upon change after the fact is going to provide much in the way of benefits and will instead just contribute further to stymying efforts to move packaging forward. If it was decided that we really need all PEPs announced on python-dev and python-dev given a chance to weigh in, I would advocate for packaging to stop using the PEP process all together and create our own process to mirror the PEP process because again, we're not using the PEP process because these things naturally fall under what the PEP process was designed to cover, we're using it because it was there already and close enough that we could hijack it and it was moderately easier to just do that. If the downsides of using the PEP process become greater than the effort to come up with our own process then I think it's a no brainer to just do our own thing.

Fredrik Lundh is also affected (and might not have received any mail, same as me): https://pypi.python.org/pypi/PIL

I've personally emailed the PIL offer several times through various email addresses over several years and have never gotten a response. At this point I can only assume either they are willfully ignoring those messages or they've completely changed their contact information in a way that I have not been able to find updated contacted information for at all.

Technically, he didn't gloat, but suddenly legal advice was apparently readily available.

https://pypi.python.org/pypi/m3-cdecimal

I'm not going to respond to the rest of this, because I don't think it's possible for you and me to have a constructive discussion on these other issues. However, I do want to point out that the PSF is the legal entity behind PyPI itself and Van is both a board member and a lawyer. I don't recall what exactly was discussed with who about m3-cdecimal, but it's completely reasonable that when a legal question comes up with PyPI that we consult the PSF board members, particularly the ones who are lawyers. I do recall that the last time m3-cdecimal came up [1] you (again) brought up issues you had with PyPI in an inappropriate venue and as far as I know, you never actually used any of the contact channels that you were offered to try and resolve the issue that you had with a project on PyPI.

[1] https://bugs.python.org/issue22483

Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20160505/881c096a/attachment.sig>

More information about the Python-Dev mailing list