How to Manage Python Projects With pyproject.toml (original) (raw)

Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: Everyday Project Packaging With pyproject.toml

The pyproject.toml file simplifies Python project configuration by unifying package setup, managing dependencies, and streamlining builds. In this tutorial, you’ll learn how it can improve your day-to-day Python setup by exploring its key use cases, like configuring your build system, installing packages locally, handling dependencies, and publishing to PyPI.

By the end of this tutorial, you’ll understand that:

To get the most out of this tutorial, you should be familiar with the basics of Python. You should know how to import modules and install packages with pip. You should also be able to navigate the terminal and understand how to create virtual environments.

The pyproject.toml package configuration file is the relatively new (circa 2016) standard in the Python ecosystem, intended to unify package configuration. It’s also supported by many major tools for managing your Python projects. Some of the project management tools that support the pyproject.toml file are pip, Setuptools, Poetry, Flit, pip-tools, Hatch, PDM, and uv.

The pyproject.toml file is a configuration file written in the TOML syntax. For many Python project management needs, a minimal pyproject.toml file doesn’t have to contain a lot of information:

Different tools have different requirements, but the name and version of your project are officially the only required fields in the [project] table. Typically, you’ll want to include more fields, but if you only want to include a minimal pyproject.toml file, then that’s all you’ll need to get started. Just include this file at the root of your project.

To understand more about why using a pyproject.toml file may be useful, you’ll explore a sample CLI application to show you how the pyproject.toml file fits into a standard project workflow.

Take the Quiz: Test your knowledge with our interactive “How to Manage Python Projects With pyproject.toml” quiz. You’ll receive a score upon completion to help you track your learning progress:

How to Manage Python Projects With pyproject.toml

Interactive Quiz

How to Manage Python Projects With pyproject.toml

In this quiz, you'll test your understanding of Python's pyproject.toml file, which simplifies Python project configuration by unifying package setup, managing dependencies, and streamlining builds.

Setting Up a Python Project With pyproject.toml

The example project you’ll work with in this tutorial is inspired by the classic cowsay program. The example project is called snakesay and—once installed—you can run it with the ssay command:

As you can see, the program takes a string argument and echoes it back with a bit of ASCII art.

The structure of the example project follows a popular pattern for Python projects:

snakesay-project/ ← The project root │ ├── snakesay/ ← The main module of this project │ ├── __init__.py │ ├── __main__.py ← The entry point to snakesay │ └── snake.py ← The core of the program │ ├── .gitignore ├── LICENSE ├── pyproject.toml ← What this tutorial is about └── README.md

The directory snakesay-project is the root location of your project. The main package, where most of the code goes, is the snakesay directory.

At the root level of the project, you’ve got the star of this tutorial, the pyproject.toml file. In this project, the pyproject.toml file currently contains the following content:

As the tutorial progresses, you’ll examine what all this means in more detail. You’ll also expand this pyproject.toml to include more tables and fields. As it stands, this pyproject.toml file includes:

With this pyproject.toml file, you’ve already defined all the configuration you need to build and run your project.

The [tools.setuptools.packages.find] table isn’t required since the value of ["."] is the default. Even though it’s the default, sometimes Setuptools can’t find other modules in the project root, and explicitly setting the where key can help with this.

Setuptools has various defaults for package discovery, which include the current project layout and the src layout.

If you want to customize the package discovery defaults, then bear in mind that the [tools.setuptools.packages.find] table is Setuptools-specific, and other build tools will have different tables and fields to configure the build process.

Why Setuptools in the first place? Setuptools is the fallback build system when using pip if you don’t include the [build-system] table. So it’s a good choice for a build backend if you’re not sure what to use. It’s been a default build system for Python for a long time, and it’s well supported by the Python packaging ecosystem.

The build backend of Setuptools is setuptools.build_meta. This is a Python module that implements the build backend interface, as defined in PEP 517.

Most of the code in the snakesay example project isn’t relevant to this tutorial, as you’re just interested in the pyproject.toml file. You can still download the source code of this project, if you’re interested. One relevant file is the entry point, which determines how your project behaves once installed:

All this file does is join the command-line arguments into a string and use them to call the say() function in the snake module.

If you don’t install this project, then to call your snakesay program you’ll need to navigate to the snakesay-project directory and execute the following command:

The -m flag tells the Python executable to treat the following argument as the dotted path to the module.

Since you’re calling the directory snakesay as a module, it runs the __main__.py file in that directory. If the directory didn’t have a __main__.py, then it couldn’t be directly executed as you’ve done above.

To set up the project so that you can call it from the command line with just a ssay command—as it’s configured to do in the pyproject.toml file—you’ll need to install the project.

Understanding Why You Should Install Your Python Project

One of the main reasons you’d want to install your package right off the bat is to take full advantage of the import system. Installing your project is generally the recommended way to work with Python projects and comes with other benefits that you’ll see later in the tutorial.

If you don’t install your project, then your project is coupled to its location on your file system. This may seem fine at first, but it’s often the root of much confusion when working with Python imports.

For example, you may think that you can just run the __main__.py file directly as a script:

As you can see, there are issues when importing. The key to understanding the problem is in understanding the module search path. When you try to import something in Python, such as import math, the Python interpreter looks in the module search path.

There are various locations in the module search path. If you run a Python file as a script, so with no -m flag, then the script location is added to the module search path. If you run a Python module with the -m flag, then the module itself is added to the search path.

To be able to import snakesay, there needs to be a snakesay module in the search path. In the example above, when you run the __main__.py file as a script, there’s no snakesay module in that directory so you get a ModuleNotFoundError.

Now that you understand some of the reasons why you’d want to install your project, you’ll see how to do that next.

Installing Your pyproject.toml Project With pip

To install your project with pip, first navigate to the snakesay-project directory and optionally create a virtual environment there. Once ready, you can install your project:

That’s it—you’ve successfully installed your package locally. You’re now ready to take full advantage of Python’s powerful import system. Now you can rely on the snakesay module always being available wherever in the file system you happen to be. So, in the same way that you can always import math from wherever you are, now you’ll be able to import snakesay from wherever, too.

Try running the __main__.py file directly now, as you did before it was installed, and it should work! Also, if snakesay is installed correctly in the active Python interpreter, then you’ll be able to start a REPL session from anywhere in the file system and always be able to import snakesay.

This isn’t only handy for executing your program, but it’s also very useful for making your imports consistent in your project. Instead of resorting to relative imports—which can get hard to maintain—absolute imports will work predictably now. So, even if you’re deeply nested within the snakesay module, or you’re in a completely different module, you can always from snakesay import snake without issues.

As you may have noticed, the command used to install the project included the -e flag to instruct pip to install the package as an editable install.

An editable install means that if you edit the source code of the package, then you won’t need to reinstall the package to see those changes when you run the program. If you want to install an established project and don’t foresee any changes, then you can omit this flag.

There’s one thing to keep in mind with an editable install. If any changes are made to the pyproject.toml file itself, then you’ll often need to reinstall the package. This is because the pyproject.toml file is used to configure the build process, and if you change the fundamental configuration of the project, then you’ll need to completely rebuild the package.

pip manages the overall installation process, ensuring that dependencies like Setuptools are available. Setuptools handles the build process, which involves finding packages, configuring metadata, creating links for editable mode, and setting up entry point scripts. Both tools leverage the pyproject.toml file to configure the process.

Another great thing about installing your project is that you can now run the ssay command from the command line. In the next section, you’ll see how the pyproject.toml file is used to configure scripts.

Using pyproject.toml to Configure Scripts

With the snakesay package installed, you’ll see that you can now call the command from the command line directly with the ssay command:

You have this command available from the command line because the pyproject.toml file defines a section, which your build backend reads to create executable commands:

The [project.scripts] table has one entry with a key of ssay and a value of "snakesay.__main__:main". With this information, your build backend will create an executable command to run the main() function in the __main__ submodule of snakesay, which you just installed.

These values can be customized to whatever you want, and you can add as many as you want, as long as they point to a Python callable, such as a function. You just need to make sure the target callable is a procedure—it shouldn’t take arguments.

You’ll typically want to automate certain common tasks in your project. Maybe that will involve easy commands for formatting, linting and testing. While there are many tools that automate all or parts of this process for you, you can leverage the pyproject.toml file for quite a lot. Anything you can do in a Python function, you can attach a script to. So, with some imagination, there’s little that you couldn’t do.

At this stage, you might be pining after a single solution for project management, but the truth is that you’ll have to decide on a set of tools that works for you, and there’s always going to be a bit of a learning curve. You’ll explore why this is so in a bit, but for now, you’ll explore how to manage dependencies with the pyproject.toml file.

Managing Dependencies With a pyproject.toml File

If all you need is to migrate from a basic requirements.txt file, then you’ll be able to replace that with pyproject.toml. As you’ve seen, pip and Setuptools are almost always available as part of a default Python installation. These tools can take care of most dependency management tasks together with the pyproject.toml file, leaving you with one single file to manage your project.

Take the current state of the example project snakesay, which doesn’t have any dependencies. Imagine that you wanted to enhance the project with some fancy terminal magic with a library like Rich. Since you foresee working on this longer than a few minutes, you may want to add in a some development tools, such as Black and isort:

File Changes (diff) pyproject.toml

As you can see, you’ve added a list with one item in the dependencies key of the [project] table. In the [project.optional-dependencies] table, you’ve added a dev key with a list of two more dependencies. In this case, these dependencies are only needed if you’re developing the project. Note that the dev key is arbitrary—you can call it anything you like.

Now, if you install your project with pip, the Rich library will automatically be installed as part of the regular installation. Nice! If you want to install the optional dependencies too, then you can call:

Appending [dev] to the directory with the pyproject.toml file will include the optional dependencies you’ve specified under dev, along with the core dependencies specified in the [project] table. Very nice!

If you’re not ready to give up your requirements.txt file just yet, don’t worry! You can still use a traditional requirements.txt file and reference that file from pyproject.toml:

File Changes (diff) pyproject.toml

With this setup, all you’d need is a couple of files at the root of your project to define the requirements:

Then, when you install your project, the dependencies will be read from the requirements.txt and requirements-dev.txt files.

You’ve already set your dependency versions to be compatible with a specific version, but if you wanted to take another step towards ensuring reproducibility, you could use a lock file.

There’s a proposal for a standard lock file in PEP 751, but it’s not yet been accepted. If you need a lock file right now, then perhaps the most straightforward way is to use pip-tools, which has some instructions on how it recommends structuring requirements with pyproject.toml, as you’ve done with dynamic fields above.

To see some of the lively debate around standards in Python when it comes to how to specify dependencies in pyproject.toml file, check out the forum thread Development Dependencies In pyproject.toml.

If you’ve been slightly overwhelmed by the number of tools and standards in the Python packaging ecosystem, you’re not alone. In the next section, you’ll explore the context of the pyproject.toml file in Python packaging.

Understanding the Context of pyproject.toml in Python Packaging

Having spent some time in the Python ecosystem, you’ve probably noticed that there are a few tools and workflows out there to pick and choose from. This can put a lot of people off. It’s natural to want one universal standard.

To understand why Python is the way it is, it can be helpful to understand the context in which Python has evolved. So, in this section, you’ll get an overview of how Python packaging has developed, and why the pyproject.toml file is a step in the right direction.

Python was conceived by Guido van Rossum at the tail end of the 1980s, during a time when the World Wide Web hadn’t yet opened to the public. In these times, packaging wasn’t as much as a concern as it is today. Back then, developers often shared code by emailing files or physically passing around disks within the office. Not many would have imagined how the web would revolutionize society.

When designing Python, Guido van Rossum adopted the Unix philosophy as a practical and time-saving approach. This philosophy embraces developing tools that tackle specific, often simple, tasks. These tools work well in isolation but can also be combined to solve more complex problems.

So, perhaps it’s no surprise that many Python packaging tools have evolved to reflect the Unix philosophy. There are tools for each packaging sub-task, and you can mix and match them to suit your needs.

The first build system for Python was distutils, which was included with Python from version 1.6 in the year 2000. It was the defacto build system for Python for four years.

In 2003, the Python Package Index (PyPI) was introduced to provide a central repository for Python packages. This was a big step forward for Python packaging, as it would eventually become the central location for developers to share their packages.

Perhaps with the rise of PyPI and the internet in general, the limitations of distutils were becoming obvious, such as rudimentary dependency management and tedious configuration. The Setuptools project was introduced in 2004 to address these limitations, eventually becoming the default build system.

pip was introduced in 2008 as a package installer for Python. It was designed to be a replacement for easy_install, which was the default package installer for Setuptools. pip was designed to be more user-friendly and to have a more consistent interface.

In 2011, the Python Packaging Authority (PyPA) was formed to take over the maintenance of key packaging tools like pip and Setuptools. PyPA has since brought various other tools under its umbrella.

While Setuptools was a big improvement over distutils, it still had limitations. As Python adoption grew, so did the special requirements of building and distributing Python packages. Over time, various tools emerged to address these limitations, some of which you’ll explore in the next section. That said, it was often difficult to switch between tools, as each tool had its own configuration format.

The pyproject.toml file was introduced to provide a single configuration file that could be used to specify build system configuration in an explicit way. It allows you to switch out the build backend entirely without having to completely rewrite your configuration. Its status as a project configuration file also makes it a convenient place to specify other project metadata and tool configuration.

Software development in general tends to progress this way. Often, you only get to fully know the problem after trying to solve it for a while. As society and development practices evolve, so must the tools being used.

Python’s ecosystem is robust in part because of the many tools that have been developed to solve specific problems. When the problem space evolves rapidly, it’s often easier to create new tools to solve these emerging challenges rather than trying to retrofit old, monolithic ones.

In the next section, you’ll explore a selection of the many tools that have survived the test of time and which also leverage the pyproject.toml file.

There are now several tools to enhance specific areas of your project management, many of which can leverage the pyproject.toml file. In this section, you’ll explore some of these tools.

For advanced dependency management, you can use pip-tools, which is a set of utilities to manage and lock dependencies. It was originally based on the requirements.txt file, but can also be configured to leverage the pyproject.toml. This is a PyPA project, so it’s a good choice if you want to stay within the official PyPA ecosystem.

For automating testing and script running, you can use tox. This tool can leverage the pyproject.toml file to configure its behavior. Typically, this involves using a special table within the pyproject.toml file. For example, tox uses the [tool.tox] table to define its settings:

This table specifies that tox should use version 4.19 or higher, and that it should run the 3.13, 3.12, and type environments.

This [tool.<TOOL>] format is also leveraged to configure other tools not directly related to building. For example, development tools like Black and isort:

To see more tools that leverage the pyproject.toml file, you can check out the Awesome pyproject.toml repository which lists tools that support the pyproject.toml file.

Even though having choice between small tools is great, there are some tools that attempt to be one-stop solutions for managing Python projects. These tools aim to simplify the process of managing Python projects and provide a more unified experience. They even provide their own build backend. Some of these tools include:

These tools take care of all steps of the development process such as creating a new project, managing dependencies, managing environments, building, and publishing.

The tools listed here use the pyproject.toml file for configuration. So even if you don’t want to fully commit, by leveraging the pyproject.toml file you can try out these tools now or later without having to fully rewrite your configuration.

Another tool worthy of mention is another PyPA project: Flit. Flit is a minimal and easy-to-use project manager that has straightforward commands to build, install, and publish your package, but isn’t as all-encompassing as something like Poetry.

As you can see, there are many tools available to manage your Python project. The pyproject.toml file is a key part of all the tools mentioned in this section, providing a standard way to configure your project and making it easier to switch between tools.

Building and Distributing Your pyproject.toml Python Project

When it comes time to distribute your Python project, pyproject.toml plays a very important role. After all, the pyproject.toml file was initially adopted to configure the build process, and it’s the build process that creates the package that you distribute.

For distribution to PyPI, or any other package index, you’ll want to add some more fields to your pyproject.toml file to configure your package for distribution. These fields will help users understand what your package is about, who maintains it, and how to install it. In some cases, certain fields will also determine how your package is distributed:

As you can see, there are a few new fields in the [project] table. All of these are useful to help users understand your package and to help package indexes categorize your package. The classifiers field is particularly important, as it helps package indices categorize your package.

PyPI even supports a "Private :: Do Not Upload" classifier, which will prevent your package from being uploaded to PyPI if you or someone on your team accidentally tries to upload it.

Also, note the inclusion of two new dependencies in the [project.optional-dependencies] table. These are build and Twine. Both of these are PyPA projects that are the recommended tools for building and uploading packages to PyPI.

The build project provides a build front end that makes it straightforward to build your package, creating both source distributions and wheel distributions. Being a front end, it doesn’t actually build your package, but it provides a consistent interface to build backends like Setuptools.

You’ve configured your project for distribution. To actually build your distributable package with Setuptools and build, you can run the following command:

You’ll see that a dist directory has been created with both a source distribution and a wheel distribution.

To upload your package to PyPI you’ll be using Twine. To upload with Twine, you can run the following command:

This command will upload all the distributions in the dist directory to PyPI. You’ll be prompted to enter your PyPI username and password, though this behavior can be configured.

Twine is agnostic to pyproject.toml or any other build configuration file as Twine’s only focus is uploading distributions to the PyPI. That is, Twine doesn’t care how you build your package, it just uploads it. Since pyproject.toml is used to configure the build process, Twine doesn’t need to know about it.

Congratulations, you’ve managed an entire project lifecycle using only the pyproject.toml file to configure it. In the final section, you’ll take a look at the full pyproject.toml file example.

Understanding a Full pyproject.toml Example

Here’s a full example of a pyproject.toml file that includes all the fields you’ve seen in this tutorial—plus a few more:

Feel free to use this example as a starting point for your own projects. You can also check out other project templates such as Cookiecutter PyPackage for more inspiration on how to structure your project with pyproject.toml.

Conclusion

In this tutorial, you’ve learned how to manage Python projects with the pyproject.toml file. Along the way, you’ve:

With this knowledge, you’re well on your way to managing Python projects with pyproject.toml and simplifying your Python project management workflow.

Frequently Asked Questions

Now that you have some experience with using pyproject.toml in Python, you can use the questions and answers below to check your understanding and recap what you’ve learned.

These FAQs are related to the most important concepts you’ve covered in this tutorial. Click the Show/Hide toggle beside each question to reveal the answer.

You use a pyproject.toml file to configure Python projects, specifying build systems, dependencies, and project metadata in a standardized format.

You create a pyproject.toml file by placing it at the root of your project and including necessary configuration tables, such as [build-system] and [project].

You structure a Python project with a root directory containing essential files like pyproject.toml, source code directories, and optional files like README and LICENSE.

You should include build system requirements, project metadata, dependencies, and optional configurations for scripts and additional tools in your pyproject.toml file.

You use pyproject.toml for comprehensive project configuration, including dependencies, whereas requirements.txt lists only the dependencies.

Take the Quiz: Test your knowledge with our interactive “How to Manage Python Projects With pyproject.toml” quiz. You’ll receive a score upon completion to help you track your learning progress:

How to Manage Python Projects With pyproject.toml

Interactive Quiz

How to Manage Python Projects With pyproject.toml

In this quiz, you'll test your understanding of Python's pyproject.toml file, which simplifies Python project configuration by unifying package setup, managing dependencies, and streamlining builds.

Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: Everyday Project Packaging With pyproject.toml