Pyproject Toml
Mar 26th, 2021 - written by Kimserey with .
Few weeks ago we talked about setuptools and setup.py. We demonstrated the configuration of a simple setup.py file in order to package a project. Although setup.py used to be the default way of packaging applications, Python has progressed to offer a more generic approach described in PEP 517 and PEP 518 with the pyproject.toml file. In today’s post we will look at the usage of pyproject.toml in the context of package management.
pyproject.toml
In our previous post, we looked at how we could use setup.py to build source distribution and wheels. We used python3 setup.py sdist bdist_wheel command to do so. This implied that we had setuptools installed and that prior hand we knew that our build would be using setup tools.
To understand the problem we can think about what pip does when installing a package from source tree, it will unzip the sdist then create a wheel with python3 setup.py bdist_wheel. This command is problematic as from users machine, on installation, it’s not possible to know what tool is required to execute this command, we can’t be sure whether this uses setuptool, or other build system like flit or poetry. This has always be a problem hence the reason why setuptools was installed by default on all virtual environment. On top of that version of the packaging tool had no way to be specified.
There are two problems solved for the build:
- the first problem is exposing the tools required to build the package (without having to look into
setup.py), - providing a way for build system to be built independently and hooked up easily into the build chain.
The first issue is addressed in PEP 518 which introduces a setting in pyproject.toml file use for “Specifying Minimum Build System Requirements for Python Projects”. The pyproject.toml is a file that would be at the root of the project. This fille will contain requirements for the build under build-system:
1
2
[build-system]
requires = ["setuptools", "wheel"]
By adding this file, although it is an extra file, it allows pip to know prior hand the requirements to build the package which is important for pip install step.
Now that the first issue is addressed, the second issue is addressed by PEP 517 which specifies a way of creating “A build-system independent format for source trees”. This specification defines a minimal interface for a tool like pip to interact with the package source tree. From a user perspective, all we need to do is to specify the build-backend in pyproject.toml to point to the module containing the definition of PEP-517, for example for setuptools it would be:
1
2
3
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"
If we look into setuptools/setuptools/build_meta.py, we will find the definition of the following methods:
build_wheelcreating the wheel,build_sdistcreating the source distribution,get_requires_for_build_wheelreturning the list of requirements to build the wheel,prepare_metadata_for_build_wheel,get_requires_for_build_sdistreturning the list of requirements to build the source distribution.
Here we have been using setuptools but if we were using flit we could have specified the build system:
1
2
3
4
5
6
7
8
[build-system]
requires = ["flit_core >=2,<4"]
build-backend = "flit_core.buildapi"
[tool.flit.metadata]
module = "my_package"
author = "Kimserey Lam"
author-email = "[email protected]"
If we look into flit_core/flit_core/buildapi.py we would see the same PEP 517 implementation. Or even with poetry:
1
2
3
4
5
6
7
8
9
[build-system]
requires = ["poetry_core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
[tool.poetry]
name = "my_package"
version = "0.0.1"
description = "hello"
authors = [ "kimserey <[email protected]>" ]
If we look into poetry-core/poetry/core/masonry/api.py we would see PEP 517 implementation.
Another detail that come up by comparing the different pyproject.toml, we can see that different build system uses pyproject.toml to define extra configuration metadata in their own ways. With poetry and flit for example, the tools allow us to define every metadata that would be defined in setup.py allowing us to completely get rid of setup.py. PEP 517 allows the build system to express themselves in complete freedom as shown by the different tomls where different metadata attributes are used.
Lastly the whole point of providing an interface and hooks is to create a unified way of building packages without being tied to any build system implementation and this is brought by pep517 package:
1
pip install pep517
Using pep517.build, we are able to build all three examples, setuptools, flit or poetry with the same following command:
1
python -m pep517.build .
And that concludes today’s post! We now know the purpose of pyproject.toml. I hope you liked this post and I see you on the next one!