[Proposal] Include abstract dependencies in install_requires format in Pipfile.lock · Issue #98 · pypa/pipfile (original) (raw)

Summary: It would be nice if Pipfile.lock contained the abstract dependencies in a way that could be used in install_requires in setup.py, as this would allow better tooling in the near term for library developers

This is intended as an actionable follow-up to #27, https://github.com/kennethreitz/pipenv/issues/209.

To summarize many earlier discussions, across applications and libraries, we can taxonomize three different kinds of abstract dependencies.

  1. For applications, the abstract dependencies required to run the application
  2. For libraries, the abstract dependencies that consumers of the library must also add
  3. For both applications and libraries, the abstract dependencies required for a development environment – these also almost always include respectively (1) or (2), since you usually can't run tests without the actual dependencies
  4. The concrete dependencies required to run an application, corresponding to (1)
  5. The concrete dependencies to develop an application or a library, corresponding to (3)

Currently, (1) and (3) are handled via Pipfile (or requirements.in). (2) is handled by setup.py. (4) and (5) are handled by Pipfile.lock (or requirements.txt).

In practice, however, (1) and (2) above are often managed in very similar ways. Also, the requirements around (5) are nearly identical for both libraries and applications.

This mostly works, except there is proximate room for improvement in tooling support. Specifically, my user workflow for adding a dependency per (2) to a library closely resembles that for adding a dependency per (1) for an application. I want to add the dependency to the abstract specification, then lock down a version for my development environment per (5) to have a reproducible build environment (so I can e.g. separate out test failures from dep version bumps from those relating to my own code). While this is easy for an application with Pipfile, it's not really possible for a library using setup.py, since setup.py is arbitrary Python rather than structured data.

In an ideal world, this would be a non-issue if we had a setuptools entry in pyproject.toml, but we don't have that right now. This would also be less of an issue if we could just include packages from Pipfile in setup.py, but barring near-universal adoption of pyproject.toml, there's no real way to access a TOML parser in setup.py.

That leaves a last option, which is realizable – as Pipfile.lock is just JSON, if the abstract dependencies from Pipfile are made available in Pipfile.lock, then it would be straightforward to just forward through those dependencies from setup.py with an appropriately configured package manifest.

This would mitigate the flaws in tooling available for Python library development right now. Once better solutions become available in the future, as this is something that would be handled within packages, it would be straightforward to remove this section from Pipfile.lock and move people to newer tooling instead.