Package HowTo

This is a brief introduction to python packages. How to make them, how to install them, how to upload them, how to maintain them, how to work with them. For details consult https://packaging.python.org, or https://py-pkgs.org, or other sites.

Before you create a package please consider building a virtual environment for your project.

Why packages?

When working on your project, you typically end up with some scripts, functions and classes that could be of more general interest. For example, if you would use this code in some other projects as well. As a first step, you make some modules, i.e. separate python files, where you collect this code. These modules can be easily imported from other scripts in the very same directory.

For example, consider a module addition.py and a script analyze.py both in the same directory:

├── addition.py
└── analyze.py

In addition.py we define a function add_two():

def add_two(x):
    return x + 2

We can use this function in analyze.py by importing it from addition.py:

from addition import add_two

x = 5
y = add_two(x)

This works, however, only if scripts and modules are in the same directory (or if modules are in sub-directories). To make modules available in other places of your file system or even for other people, you need to turn the modules into packages.

Minimal package

First we make a project directory for our new package. Usually the name of the project directory is the same as the one of the package. Here, however, we call the project directory packagehowto and the name of the package numerix.

A package is a directory, and the name of the package is the name of this directory, here numerix. Inside the package directory we put all the modules, for now just addition.py.

What makes this directory a package is the presence of a file named __init__.py. This file is executed when the package is imported. For now we leave it empty.

The package directory resides in the src/ directory of the project. The layout of your package project then looks like this:

packagehowto/
├── pyproject.toml
└── src/
    └── numerix
        ├── __init__.py
        └── addition.py

A package also needs a pyproject.toml file. This file contains some metadata about the package and informations about how to build a package. As the bare minimum the content of the pyproject.toml file specifies setuptools to be used for building the project:

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

With this pyproject.toml file, and addition.py and __init__.py in the src/numerix directory, you can install the numerix package on your machine and import it from wherever you want.

To install the project, go to the project root, here packagehowto/ and run from your shell

pip3 install .

This installs the package of the current project folder . somewhere in your home directory. From anywhere in your home file system you now can import this package. The import line of our analyze.py file needs to look like this:

from numerix.addition import add_two

From the addition module of the numerix package the function add_two is imported.

You would need to reinstall the package whenever you change your package, for exmple when you add a new function or when you just fix some package code. This is tedious, and that is why there is a -e option ("editable install") for pip install. So install your package with

pip3 install -e .

Then all future changes on your package are immediately available without the need to reinstall the package again.

The __init__.py file

This file is your package. In fact, you could write a package that only has an __init__.py file in the package directory. All code you want to make available in you package could be placed in the __init__.py file. For example, if you define an add_four() function in __init__.py like this:

def add_four(x):
    return x + 4

then it can be imported directly from the package:

from numerix import add_four

Alternatively, you can define all your functions in module files, like we did with the function add_two() in the addition.py module. In the __init__.py file you can import this function and this way make it available directly from the package. With this line in __init__.py

from .addition import add_two

(note the . in front of addition - this makes it a relative import) you can import add_two() without specifying the module:

from numerix import add_two

Multiple modules within a package

You can have as many modules as you need in your package. Let's add a module numbers.py to the numerix package

    └── numerix
        ├── __init__.py
        ├── addition.py
        └── numbers.py

with the following content:

from .addition import add_two

def three():
    return add_two(1)

This makes a function three() available that can be imported via

from numerix.numbers import three

Note, that the numbers.py module imports a function from the addition.py module via a relative import with the . in front of addition.

And of course you can also import this function in __init__.py so that it can be imported from the package directly.

Versioning

Your package needs a version number (for details see below). But how to choose the right version number? A nice and brief overview is given at the Python Packaging User Guide.

In short, for semantic versioning the version number is composed of three integer numbers called major, minor, and patch, separated by a dot. These numbers are incremented in the following way:

• major, when incompatible API changes have been introduced,
• minor, when functionality is added in a backwards-compatible manner, and
• patch, for backwards-compatible bug fixes.

Package version

For uploading your package to PyPi you need to specify a version number in the pyproject.toml file. However, your package should also provide a __version__ variable. And some scripts of your package might want to know about the version number as well when you call them with a `--version' argument.

For the latter two cases, you can define a __version__ variable in one of the modules, or even in a dedicated module, that you then import whereever you need it. Even better is to define it in the __init__.py file:

__version__ = '1.4.2'
""" Current version of the numerix package as string 'x.y.z'. """

__year__ = '2024'
""" Year of the current numerix version as string. """

Then a script can easily check the package's version like this:

from numerix import __version__

print(__version__)

Other modules of the package need to import it from the __init__.py file by a relative import to be able to use it:

from .__init__ import __version__


def about():
    print(f'{__name__} {__version__}')

But how do you get the very same version into the pyproject.toml file? It would be a very bad idea to write it there directly and update it whenever you change it in the __init__.py file. The version number should be set only in one place.

This is possible, of course. You need to specify the version field as dynamic in the [project] section. This tells the package tool that the version will be set dynamically by some code. There are several options to do so. The simplest one is to add a [tool.setuptools.dynamic] section where the version is read as an attribute from the numerix package, that is its __init__.pyfile:

...

[project]
name = "numerix"
dynamic = ["version"]
...

[tool.setuptools.dynamic]
version = {attr = "numerix.__version__"}

Distribute your package

Full pyproject.toml file...

README.md

LICENSE

PyPi

python3 -m build

python3 -m twine upload dist/*

token

Using poetry

You have now seen how you can do everything "from scratch". This is of course good to know. But there are tools to significantly simplify package creation and management, such as poetry. Its worth checking it out. Heres a nice demo: https://python-poetry.org/docs/basic-usage/

Unit tests

pytest

coverage

pytest -v --cov-report html --cov numerix tests/

Documentation