Packaging a Python project with setuptools

Date Thu 08 August 2024 By Emmanuel Fleury Category programming Tags python / setuptools / packaging

I do not like when things are not properly organized. I like to have a clear structure in my projects, and I like to have a clear way to package them. Lately, Python decided to reunify its packaging tooling around pyproject.toml. Unfortunately, only few documentations are available on how to use it properly and how to mix it with other features such as testing or documentation. In this article, I will present a simple scaffolding for a Python project that uses setuptools as a builder, pytest for testing and sphinx for documentation.

Project Hierarchy

Our project will only contain one module and one function for the sake of simplicity. The hierarchy of the project will be as follows:

myproject.git/
├── myproject/
│   ├── __init__.py
│   └── hello.py
├── tests/
│   ├── __init__.py
│   └── test_hello.py
├── LICENSE.md
├── README.md
└── pyproject.toml

The content of myproject/hello.py is the following:

"""A Hello module in the myproject project"""

def hello() -> None:
    """Hello function"""
    print("Hello World!")

The content of tests/test_hello.py is the following:

"""A test case to check the myproject.hello module"""

import pytest
from myproject.hello import hello

def test_hello(capfd):
    """Test the hello function"""
    hello()
    out, err = capfd.readouterr()
    assert out == "Hello World!\n"
    assert err == ""

Then, myproject/__init__.py will only contain the version of the project:

"""Main module for myproject"""

import myproject.hello

__version__ = "0.1.0"

The other tests/__init__.py is empty. The LICENSE.md contains a BSD-2-Clause license, the README.md contains the project description and pyproject.toml contains the following:

[build-system]
build-backend = "setuptools.build_meta"
requires = ["setuptools"]

[project]
name = "myproject"
version = "0.1.0"
description = "My scaffolding project example"
authors = [
  {name = "John Doe", email = "john.doe@springfield.org"},
]
license = "BSD-2-Clause"
license-files = ["LICENSE.md"]
readme = "README.md"
dependencies = [
  'importlib-metadata; python_version<"3.10"', # Metadata access
  "pytest", # Testing framework
]

Note: The importlib-metadata dependency is required for Python versions prior to 3.10. It is used by setuptools to access metadata in pyproject.toml.

It is also important to note that the requirements.txt file is not used anymore. The dependencies are declared in pyproject.toml and only there. The syntax used to describe the dependencies here is also the one from PEP 508, so nothing new to learn.

Build the Project

As usual in Python development, we will use a virtual environment to isolate the project dependencies. First, we create and activate the virtual environment:

#> python3 -m venv .venv
#> source .venv/bin/activate

Note: The virtual environment is created in the .venv directory at the root of the project. You can choose another name if you want, but this one is quite standard. And, finally, you can use deactivate to quit.

Then, we get all the project dependencies inside our virtual environment and install the project in editable mode:

#> pip install -e ./
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done

...8<...

Successfully built myproject
Installing collected packages: myproject
  Attempting uninstall: myproject
    Found existing installation: myproject 0.1.0
    Uninstalling myproject-0.1.0:
      Successfully uninstalled myproject-0.1.0
Successfully installed myproject-0.1.0

Note: The -e option is for editable mode. It means that the project is installed in development mode, and any change in the source code is immediately available without reinstalling the project each time you change the code. You may omit this option if you are installing the project in a production environment.

Then, we can run the tests:

#> pytest
============================= test session starts ==============================
platform linux -- Python 3.12.4, pytest-8.3.2, pluggy-1.5.0
rootdir: .../myproject.git
configfile: pyproject.toml
collected 1 item                                                               

tests/test_hello.py .                                                    [100%]

============================== 1 passed in 0.07s ===============================

We are half-way through our goals now. We have a project with a module, a test suite, and a way to build and test it. The next step is to add code coverage and documentation generation. But before that, lets be perfectionist and try to get rid of some annoying technicalities.

One version to rule them all

You may have noticed that the version of the project is defined in pyproject.toml but also in myproject/__init__.py. It is a common practice to have only one place where the version of the project is defined because it prevents inconsistencies. For that, we will use the pyproject.toml dynamic variables as follows:

...
[project]
dynamic = ["version"] # Variables handled in dynamic metadata
...
[tool.setuptools.dynamic]
version = {attr = "myproject.__version__"} # Get version from package dynamically
...

And we remove the old version = "0.1.0" line from pyproject.toml. Now, the version of the project is only defined in myproject/__init__.py.

Much cleaner, isn’t it? Lets go for code coverage now.

Add Code Coverage

This one will be quite easy. We use the pytest-cov plugin and we only modify pyproject.toml to add the plugin and give it a few options:

...
dependencies = [
  "pytest", # Testing framework
  "pytest-cov", # Code coverage plugin
]

[tool.pytest.ini_options]
addopts = [ # Add options for pytest
  "--cov=myproject",
  "--cov-report=html",
  "--cov-report=term",
  "--cov-branch",
]

Then, do not forget to install the new dependencies:

#> pip install -e ./

And, we can run the tests with code coverage:

#> pytest
============================= test session starts ==============================
platform linux -- Python 3.12.4, pytest-8.3.2, pluggy-1.5.0
rootdir: .../myproject.git
configfile: pyproject.toml
plugins: cov-5.0.0
collected 1 item                                                               

tests/test_hello.py .                                                    [100%]

---------- coverage: platform linux, python 3.12.4-final-0 -----------
Name                    Stmts   Miss Branch BrPart  Cover
---------------------------------------------------------
myproject/__init__.py       2      0      0      0   100%
myproject/hello.py          2      0      0      0   100%
---------------------------------------------------------
TOTAL                       4      0      0      0   100%
Coverage HTML written to dir htmlcov

============================== 1 passed in 0.07s ===============================

Note: The code coverage HTML report is available in the htmlcov directory. You can open the index.html in your browser to see the coverage report.

The option --cov-branch is used to check the coverage of the branches in the code. It give a much more accurate view of the code coverage than the simple line coverage and can capture some corner cases that could escape to too simple tests.

So, lets go for the last step, automate this damn documentation generation which is always a pain to write and maintain if it is outside of the code.

Automated Documentation Generation

Documentation generation can be achieved with several tools, but the most common one seems to be Sphinx. So, we will use it here with its ‘ReadTheDocs’ theme. First, lets declare the dependency in pyproject.toml:

dependencies = [
  "sphinx", # Documentation generator
  "sphinx_rtd_theme", # ReadTheDocs theme for Sphinx
]

Then, do not forget to install the new dependencies, ‘again’:

#> pip install -e ./

Unfortunately, Sphinx does not provide yet a way to fit all its configuration in pyproject.toml. There are some ongoing projects to make this possible, but they seems to be in early stage and I wanted something that works now. So, we will use the standard method to setup Sphinx (note that it may change in the future). First, we will use the scaffolding provided by Sphinx:

#> sphinx-quickstart docs
Welcome to the Sphinx 7.4.7 quickstart utility.

Please enter values for the following settings (just press Enter to
accept a default value, if one is given in brackets).

Selected root path: docs

You have two options for placing the build directory for Sphinx output.
Either, you use a directory "_build" within the root path, or you separate
"source" and "build" directories within the root path.
> Separate source and build directories (y/n) [n]: n

The project name will occur in several places in the built documentation.
> Project name: myproject
> Author name(s): John Doe
> Project release []: 0.1.0

If the documents are to be written in a language other than English,
you can select a language here by its language code. Sphinx will then
translate text that it generates into that language.

For a list of supported codes, see
https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-language.
> Project language [en]: en

Creating file docs/conf.py.
Creating file docs/index.rst.
Creating file docs/Makefile.
Creating file docs/make.bat.

Finished: An initial directory structure has been created.

You should now populate your master file docs/index.rst and create other documentation
source files. Use the Makefile to build the docs, like so:
   make builder
where "builder" is one of the supported builders, e.g. html, latex or linkcheck.

Right now, the generated documentation will not be really interesting. It will requires some more work to be useful. First, we will modify docs/conf.py to customize a few things. Add the following lines:

import myproject
...
release = myproject.__version__
...
extensions = ["sphinx.ext.autodoc"]
...
html_theme = "sphinx_rtd_theme"
...

So, the first and the second line fix the version of the project and prevent inconsistencies (yes, again…). The third line adds the autodoc extension to the Sphinx configuration to grab documentation directly from the docstrings in the code. Finally, the last line sets the theme of the documentation to the ‘ReadTheDocs’ theme.

Before keep going, we need to generate a couple of rst files from the docstrings in our code. We will use the sphinx-apidoc tool for that and we execute it from the root of the project:

#> sphinx-apidoc -o docs/ myproject

The -o docs option tells Sphinx to put the generated files in the docs directory. The myproject argument is the name of the package to document. It will generate modules.rst and myproject.rst in docs/ that we will have to include in docs/index.rst as follow:

myproject documentation
=======================

This is our wonderful project documentation.

.. toctree::
   :maxdepth: 2
   :caption: Contents:

   modules

Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

The .. toctree:: directive tells Sphinx to include the modules.rst file in the documentation. The :maxdepth: 2 option tells Sphinx to include only the first two levels of the documentation. The :caption: Contents: option sets the title of the included section.

Now, we can finally build the documentation:

#> cd docs
#> make html
Running Sphinx v7.4.7
loading translations [en]... done
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
writing output... 
building [html]: targets for 0 source files that are out of date
updating environment: 0 added, 0 changed, 0 removed
reading sources... 
looking for now-outdated files... none found
no targets are out of date.
build succeeded.

The HTML pages are in _build/html.
#> firefox _build/html/index.html

Of course, the autogenerated Makefile can be modified to include the call to sphinx-apidoc automatically. But, this is left as an exercise to the reader.

And, voilà! We have a project with a module, a test suite, code coverage, and documentation generation. All of this is automated and can be run with a few commands and is up-to-date with the last standard of Python packaging (see: this blog post). I hope you found this useful.

Create a Starter Script

Finally, we can add a main script as an entry point to the myprogram project. We need to add a __main__.py file in the myproject package:

import argparse

from myproject import __version__
from myproject.hello import hello


def main():
    """
    Starting point of the program.
    """
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "-V",
        "--version",
        help="display the version of the program",
        action="store_true",
    )
    args = parser.parse_args()
    if args.version:
        print(f"MyProgram: version {__version__}")

    # Call our awesome hello function
    hello()

    return 0

Finally, we add a [project.scripts] section in the pyproject.toml to add the name of our main script and its entry point in our code:

[project.scripts]
myproject = "myproject.__main__:main"

That’s it! Now, you can run your program with:

#> myproject
Hello World!
#> myproject --version
MyProgram: version 0.1.0

Code analysis with pylint

To finish, we can add a code analysis tool to our project. We will use pylint here. It is a very powerful tool that can help you to improve the quality of your code. It checks for coding standards, errors, and many other issues.

First, we need to add a .pylintrc file at the root of the project to:

[MAIN]
recursive = yes

[MESSAGES CONTROL]
disable =

Then, we can run the analysis on our project:

#> pylint --enable-all-extensions myproject
************* Module myproject.__main__
myproject/__main__.py:1:0: C0199: First line empty in module docstring (docstring-first-line-empty)
myproject/__main__.py:11:0: C0199: First line empty in function docstring (docstring-first-line-empty)

-------------------------------------------------------------------
Your code has been rated at 8.67/10

We can then decide to fix the problem or ignore all the C0199 (docstring-first-line-empty) warnings by adding it to the disable option in the .pylintrc file.

[MAIN]
recursive = yes

[MESSAGES CONTROL]
disable = docstring-first-line-empty
#> pylint --enable-all-extensions myproject

-------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 8.67/10, +1.33)

Another way to ignore a warning on a specific occurrence is to add a comment in the code:

...
def main():
    # pylint: disable=docstring-first-line-empty
    """
    Starting point of the program.
    """
...

pylint package comes also with a pyreverse tool that can generate UML diagrams of your project. It can be used to visualize the class hierarchy of your project and the packages dependencies. As our example code has no classe, we will focus on packages only (but classes are working the same way). pyreverse generates a packages.dot file that can be converted to a PNG image with the dot tool from the graphviz package:

#> pyreverse myproject
#> dot -Tpng packages.dot > packages.png

PyReverse package graph