[Python-Dev] typeshed for 3rd party packages (original) (raw)

M.-A. Lemburg mal at egenix.com
Wed Apr 22 12:46:16 CEST 2015

On 21.04.2015 18:08, Guido van Rossum wrote:

On Tue, Apr 21, 2015 at 12:33 AM, M.-A. Lemburg <mal at egenix.com> wrote:

On 21.04.2015 05:37, Guido van Rossum wrote: On Mon, Apr 20, 2015 at 4:41 PM, Jack Diederich <jackdied at gmail.com> wrote:

* Uploading stubs for other people's code is a terrible idea. Who do I contact when I update the interface to my library? The random Joe who "helped" by uploading annotations three months ago and then quit the internet? I don't even want to think about people maliciously adding stubs to PyPI.

We're certainly not planning to let arbitrary people upload stubs for arbitrary code to PyPI that will automatically be used by the type checkers. (We can't stop people from publishing their stubs, just as you can't stop people from writing blog posts or stackoverflow answers with examples for your library.) The actual plan is to have a common repository of stubs (a prototype exists at https://github.com/JukkaL/typeshed) but we also plan some kind of submission review. I've proposed that when submitting stubs for a package you don't own, the typeshed owners ask the package owner what their position is, and we expect the answers to fall on the following spectrum: - I don't want stubs uploaded for my package - I'll write the stubs myself - I want to review all stubs that are uploaded for my package before they are accepted - Please go ahead and add stubs for my package and let me know when they're ready - Go ahead, I trust you This seems a reasonable due diligence policy that avoids the scenarios you're worried about. (Of course if you refuse stubs a black market for stubs might spring into existence. That sounds kind of exciting... :-) Hmm, that's the first time I've heard about this. I agree with Jack that it's a terrible idea to allow this for 3rd party packages. If people want to contribute stubs, they should contribute them to the package in the usual ways, not in a side channel. The important part missing in the above setup is maintenance and to some extent an external change of the API definitions. Both require active participation in the package project, not the separated setup proposed above, to be effective and actually work out in the long run. For the stdlib, typeshed looks like a nice idea to spread the workload. I hesitate to speak for others, but IIUC the reason why typeshed was started is that companies like PyCharm and Google (and maybe others) are already creating their own stubs for 3rd party packages, because they have a need to type-check code that uses 3rd party packages. Their use cases are otherwise quite different (the user code type-checked by PyCharm is that of PyCharm users, and the code type-checked by Google is their own proprietary code) but they both find themselves needing stubs for commonly used 3rd party packages. mypy finds itself in a similar position. Think of it this way. Suppose you wrote an app that downloaded some files from the web using the popular Requests package. Now suppose you wanted to run mypy over your app. You're willing to do the work of adding signatures to your own app, and presumably there are stubs for those parts of the stdlib that you're using, but without stubs for Requests, mypy won't do a very good job type-checking your calls into Requests. It's not rocket science to come up with stubs for Requests (there aren't that many classes and methods) but the Requests package is in maintenance mode, and while they respond quickly to security issues they might take their time to release a new version that includes your stub files, and until there are a lot of people clamoring for stubs for Requests, they might not care at all.

For projects in maintenance mode, it does make sense indeed.

For active ones, I think you'd quickly run into a situation similar to translation projects: there are always parts which haven't been translated yet or where the translation no longer matches the original intent.

Unlike with translations, where missing or poor ones don't have much effect on the usefulness of the software, a type checker would complain loudly and probably show lots of false positives (if you read a type bug as "positive"), causing the usual complaints by users to the software authors.

I don't really think that users would instead complain to the type checker authors or find the actual source of the problem which are the broken stub files.

OTOH, if the type checkers are written in a way where they can detect authoritative stubs compared to non-authoritative ones and point users to the possible type stub file problem, this could be resolved, I guess.

The stub files would then need an "authoritative" flag and probably also be versioned to get this working.

So what does Requests have to lose if, instead of including the stubs in Requests, they let the typeshed people distribute stubs for Requests? Presumably having the stubs in typeshed means that PyCharm and mypy (and the 50 other type-checkers that are being written right now :-) can give

Hmm, with 50 type-checkers around it sounds like the above idea wouldn't work by simple convention. I guess a PEP would be needed to standardize it instead.

better diagnostics for code using Requests, and once in a while this may save a user of Requests from doing something dumb and blaming Requests. The only downside would be if something was missing in the stubs and users would get incorrect error messages from their favorite type checker. But it's a long stretch to see this rain down on Requests' reputation -- more likely the type checker will be blamed, so type checker authors/distributors will be vigilant before distributing stubs.

As I've explained above, in my experience, people (*) often first go to the authors of the software and not do research to find out that the tool they were using has a problem (via the non-authoritative stub files it's using).

(*) More experienced users of pylint like tools will probably think twice due to the many false positives these tools tend to generate. I'm not sure whether people using type checkers would have the same approach, though, esp. not if they are coming from the land of statically typed languages.

OTOH if you prefer to make and distribute your own stubs, type checkers will use those, and there won't be a need to include stubs in typeshed when a package already provides stubs.

And if you really don't want anything to do with stubs for your package, just tell the typeshed owners and your wish will be respected.

While this sounds like a fair deal, I think you're underestimating the social pressure this can impose on the software authors and this is really the main reason why I think we need to approach this carefully.

I'm not really worried about the technical side of things with the approach, but more with the social side of forcing everyone to provide stubs or use type annotations in their code. As always, I'm probably too worried ;-), but if there's something we could do to avoid it, I believe we should.

Another question: Will these stubs also work for closed-source software, i.e. commercial Python extensions such as the ones eGenix is selling ?

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Apr 22 2015)

Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

More information about the Python-Dev mailing list