8. Continuous integration and deployment


If you’ve gotten this far, you now have a working knowledge of how to create a fully-featured Python package! We went through quite a lot to get here: we learned about package structure, developed source code, created tests, wrote documentation, and learned how to release new versions of a package.

As you continue to develop your package into the future it would be helpful to automate many of these workflows so you and your collaborators can focus more on writing code and less on the nuances of packaging and testing. This is where continuous integration (CI) and continuous deployment (CD) come in! CI/CD generally refers to the automated testing, building, and deployment of software. In this chapter we’ll first introduce CI/CD and walk through how to set it up with the GitHub Actions service. After that, we’ll show how to set up CI/CD for a Python package, demonstrating concepts using the pycounts package we’ve been developing throughout this book.

Note

This chapter requires basic familiarity with Git and GitHub or similar version control tools. To learn more about Git and GitHub, we recommend the following resources: Happy Git and GitHub for the useR2 and Research Software Engineering with Python: Using Git at the Command Line3.

8.1. An introduction to CI/CD

Continuous integration (CI) refers to the process of automatically evaluating your code as it is updated by yourself and contributors, to try and catch any potential issues your updates have caused. A CI workflow typically includes automatic execution of many of the steps we’ve seen throughout this book, such as running tests, calculating code coverage, and building documentation, among others.

Continuous deployment (CD) is the process of automating the deployment of new versions of your software to e.g., PyPI, from changes that have made it through CI.

CI/CD can automate the packaging workflows that we’ve done manually throughout this book and can ultimately save you time and help you release new versions of your package quickly. CI/CD also helps others contribute to your package, because the process of updating your package is automated and doesn’t depend on one person’s expert knowledge (i.e., yours) of how to make releases manually. Even if your package won’t be updated very often, setting up CI/CD is still beneficial because it means you don’t have to remember all the manual steps required to make a release of your package (which can be daunting and deter you from wanting to update and maintain your package).

8.2. CI/CD tools

You could manually write and execute a CI/CD workflow by, for example, writing scripts that execute all of the steps we’ve walked through in previous chapters (i.e., running tests, building documented, build and release distributions, etc.). However, this process is not efficient or scalable, and it does not work well if more than one person (i.e., you) is contributing to your code.

It is therefore more common to use a CI/CD service to implement CI/CD. These services essentially do what we described above but in an automated manner; we define a workflow which these services will automatically run at certain “trigger events” which we can also define (for example, merging new code into the “main” branch of a GitHub repository might trigger the automatic deployment of a new version of the software).

There are many CI/CD services out there — such as GitHub Actions, Travis CI, and CircleCi. We’ll be using GitHub Actions in this chapter which is a service for executing CI/CD workflows for software stored in a GitHub repository. We’ll introduce how to use GitHub Actions in the next section.

Note

GitHub Actions is free for public repositories and includes a generous amount of free minutes for private repositories. Read more in the GitHub Actions documentation.

8.3. Introduction to GitHub Actions

8.3.1. Key concepts

GitHub Actions is a service for executing CI/CD workflows. The general idea is to create a set of commands that GitHub Actions will run on our behalf. We call this set of commands a “workflow”. A GitHub Actions workflow is defined in a .yml file and contains the set of “actions” we want GitHub Actions to run for us (such as running our tests with pytest or building our documentation with sphinx). Actions are organized as “steps” in a workflow (e.g., step 1: run tests, step 2: build documentation), which in turn are organized into “jobs” (e.g., job 1: continuous integration). A workflow is executed on a machine provided by GitHub Actions called a “runner”, when triggered by a particular “event” (like merging code into the main branch of a repository).

That’s a lot to take in, but don’t worry! All this terminology is summarized in Table 8.1 and we’ll walk through an example of using GitHub Actions which refers to this terminology in the next section.

Table 8.1 Terminology used in GitHub Actions.

Keyword

Description

Actions

Individual tasks you want to perform.

Workflow

A collection of actions (specified together in one file).

Event

Something that triggers the running of a workflow.

Runner

A machine that can run the Github Action(s).

Job

A set of steps executed on the same runner.

Step

A set of commands or actions which a job executes.

8.3.2. A toy example

In this section we’ll walk through a simple example of running a workflow with GitHub Actions. The workflow will contains actions that simply print some things to the GitHub Actions runner’s terminal using the echo command, when a change is made to a repository’s content.

  • Step 1

    Create a new repository on GitHub named anything you like (we called our repository “actions-example”). Click on the “Actions” tab, and then click the “Set up this workflow” button as shown in Fig. 8.1.

Setting up our first GitHub Actions workflow.

Fig. 8.1 Setting up our first GitHub Actions workflow.

  • Step 2

    Commit the .yml file that has been created for you to your repository by clicking “Start commit” and then “Commit new file”. Then, open the file on GitHub. You should see the contents below. All the terminology we defined in Section 8.3.1 exists in this workflow file and each line is commented to describe exactly what it does. For example, we can see that:

    • This workflow is triggered on any push or pull request made to the main branch.

    • It consists of one job called build.

    • The job will run on ubuntu-latest (the latest version of the Ubuntu runner GitHub Actions provides).

    • It contains three steps. The first step “checks out” the repository — this is necessary for the runner to access your repository. The second step will print “Hello, world!” to the runner’s terminal. The final step will print two lines. We’ll discuss steps and how to write them in Section 8.3.3.

    # This is a basic workflow to help you get started with Actions
    
    # Name of the workflow
    name: CI
    
    # Controls when the workflow will run
    on:
      # Triggers the workflow on push or pull request events but only for 
      # the main branch
      push:
        branches: [ main ]
      pull_request:
        branches: [ main ]
    
      # Allows you to run this workflow manually from the Actions tab
      workflow_dispatch:
    
    # A workflow run is made up of one or more jobs that can run 
    # sequentially or in parallel
    jobs:
      # This workflow contains a single job called "build"
      build:
        # The type of runner that the job will run on
        runs-on: ubuntu-latest
    
        # Steps represent a sequence of tasks that will be executed as 
        # part of the job
        steps:
          # Checks-out your repository so your job can access it
          - uses: actions/checkout@v2
    
          # Runs a single command using the runners shell
          - name: Run a one-line script
            run: echo Hello, world!
    
          # Runs a set of commands using the runners shell
          - name: Run a multi-line script
            run: |
              echo Add other actions to build,
              echo test, and deploy your project.
    
  • Step 3

    Now, go to the “Actions” tab of your repository. You should see one workflow run, as in Fig. 8.2. This workflow ran because in Step 2 we committed our workflow .yml file to the main branch of our repository, and the workflow is triggered to execute on any push or pull request with the main branch.

Our first GitHub Actions workflow.

Fig. 8.2 Our first GitHub Actions workflow.

  • Step 4

    Look at the logs of the executed workflow by clicking on the “Create blank.yml” workflow, then clicking the “build” job in the left-hand panel. Click on arrows inside the build logs to examine their output. You should be able to see output printed to the screen for the “Run a one-line script” and “Run a multi-line script” steps in our workflow, as shown in Fig. 8.3.

The logs of our first GitHub Actions workflow.

Fig. 8.3 The logs of our first GitHub Actions workflow.

We’ll practice writing workflows for implementing CI and CD for a Python package in the following sections of this chapter, but at this point, the high-level concepts to be aware of are:

  • A workflow is a set of commands that are triggered to execute by certain events (like a push to the main branch of a repository).

  • A workflow is run on a machine called a runner, which uses a particular operating system and is hosted by the CI/CD service.

  • A workflow contains one or more jobs.

  • Each job contains one or more steps to execute. In GitHub Actions, steps comprise either “actions” or “commands” as we’ll discuss in the next section.

8.3.3. Actions and commands

As we saw in the workflow file in Section 8.3.2, a step in a GitHub Actions can be an “action” (specified with the keyword uses) or a “command” (specified with keyword run). We’ll briefly explain the difference between these two concepts here.

Steps that use command line commands consist of a name and a run key, as shown in the example below:

# Runs a single command using the runners shell
- name: Run a one-line script
  run: echo Hello, world!

Anything after the run key will be executed at the runner’s command line. You can run multiple commands in a single step using the | character:

# Runs a set of commands using the runners shell
- name: Run a multi-line script
  run: |
    echo Add other actions to build,
    echo test, and deploy your project.

In contrast to commands, actions are reusable units of code that perform a particular task without having to write out any commands. You’ll typically use actions that have been created by others and shared on the GitHub Marketplace. Actions are specified with the uses keyword, followed by the name of the action you want to use. The @ symbol is used to specify which version of the action you want to use, like in the example below:

# Checks-out your repository so your job can access it
- uses: actions/checkout@v2

Some actions can also be configured with inputs using the with key, as in the example below:

# Set up a Python environment for use in actions
- uses: actions/setup-python@v2
  with:
    python-version: 3.9

8.4. Setting up continuous integration

Now that we have a basic familiarity with GitHub Actions, in this section we’ll build up a continuous integration workflow for a Python package. We’ll create this workflow for the pycounts package we’ve been developing throughout this book. However, it will be applicable to any Python package and it should be straightforward to see how you can modify it to your needs.

Our goal here is to create a CI workflow that will install our package with poetry, run our package’s tests with pytest, and build its documentation with sphinx, every time someone makes a push or pull request of changes to the main branch of the pycounts GitHub repository. These are steps we, or a collaborator, would usually perform locally every time our package is changed, so it makes sense to automate them.

8.4.1. Set up

To set up a workflow with GitHub Actions, we need to create a workflow file. Workflow files are .yml files located in a .github/workflows/ directory in the root package directory. We’ll call our file ci-cd.yml. You can create that file in an editor of your choice, or by running the following commands at the command line, from your root package directory:

mkdir -p .github/workflows
touch .github/workflows/ci-cd.yml

Your package directory structure should now look something like this:

pycounts
├── .github             <--------
│   └── workflows       <--------
│       └── ci-cd.yml   <--------
├── .readthedocs.yml
├── CHANGELOG.md
├── CONDUCT.md
├── CONTRIBUTING.md
├── docs
│   └── ...
├── LICENSE
├── README.md
├── poetry.lock
├── pyproject.toml
├── src
│   └── ...
└── tests
    └── ...

Open this new ci-cd.yml file in an editor. We are going to set up a CI workflow that triggers when someone makes a push or pull-request to the main branch of our repository. To set this up, copy and paste the following text into ci-cd.yml:

Tip

When building our CI workflow, we’ll be using the same syntax and terminology we described previously in Section 8.3. Don’t be afraid to revise that section as needed.

name: ci-cd

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

Now we need to set up the steps that will be executed if one of the above trigger events occurs. GitHub Actions essentially provides you with a blank operating system of your choice (a “runner”) which we need to set up based on what steps we are going to want it to execute. In our case, we need to install Python and install poetry on the runner, so that we can then install our packages and run its tests and build its documentation. Thus, our setup will involve the steps below, for which we’ve indicated whether step will use an action or command(s) (Section 8.3.3):

  1. Specify an operating system. We’ll be using Ubuntu with the syntax runs-on: ubuntu-latest. MacOS and Windows are also available if you wish to test your package on those systems, but Ubuntu is a good default to use — see the GitHub Actions documentation);

  2. Install Python (action: actions/setup-python@v2);

  3. Checkout our repository so we can access its contents (action: actions/checkout@v2);

  4. Install poetry (action: snok/install-poetry@v1); and,

  5. Use poetry to install pycounts (command: poetry install).

We’ll add all these steps to our workflow in a job called “ci”:

name: ci-cd

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  ci:
    # Step 1. Set up operating system
    runs-on: ubuntu-latest
    steps:
    # Step 2. Set up Python 3.9
    - uses: actions/setup-python@v2
      with:
        python-version: 3.9
    # Step 3. Check-out repository so we can access its contents
    - uses: actions/checkout@v2
    # Step 4. Install poetry
    - uses: snok/install-poetry@v1
    # Step 5. Install our pycounts package
    - name: Install package
      run: poetry install

The above steps will set up our system in preparation for:

  1. Running pycounts’s unit tests with pytest;

  2. Checking the code coverage of our tests; and,

  3. Checking that pycounts’s documentation builds correctly.

We’ll create each of these steps in the following sections.

8.4.2. Running tests

Remember all the hard work we put into writing tests for our package back in the Chapter 5: Testing? Well, we likely want to make sure that these tests (and any others that we add) continue to pass for any new changes proposed to our package.

Recall that we used pytest as the testing framework for our pycounts package. This is listed as a development dependency of our package, so it will already be installed on our runner from “Step 5” when we execute poetry install. Therefore, we just need to add a new step to our workflow with a command to run pytest. Because our runner is not using a conda virtual environment, poetry sets one up automatically when poetry install is executed. We need to explicitly tell poetry to use this virtual environment it set up for us by prefixing commands with poetry run, as we do below:

    # Step 6. Run tests for pycounts
    - name: Test with pytest
      run: poetry run pytest tests/ --cov=pycounts --cov-report=xml

Note

We could install conda on our runner and set up a virtual environment if we wanted to. But this is a lot of overhead for a workflow that is just going to run tests and build documentation, so we’ve decided not to do that here.

Note that in the command above we are also obtaining our test coverage through the --cov argument and outputting a report to .xml format with the --cov-report argument (these require the pytest-cov package which we used and added as a dependency of our package in Section 5.5.3). In the next section, we will integrate another service called Codecov into our workflow that will automatically record test coverage for us using the .xml report.

8.4.3. Recording code coverage

In the previous step we ran the tests for our pycounts package. However, if someone adds new code to your package but forgets to write tests for that new code, your existing tests will still pass, but the coverage will be reduced. So, we probably want to track code coverage in our CI workflow.

To do this, we could print the coverage to our runner’s build log, but having the coverage buried in those logs is not overly helpful. Instead, it’s common to use a service like Codecov to track our code coverage for us. To set up Codecov, first create a Codecov account by linking it with your GitHub account, as described in the Codecov documentation (Codecov also supports GitLab and Bitbucket). Once you’ve done this, Codecov automatically syncs with all the repositories that you have access to. Now, to use Codecov to automatically track code coverage as part of our CI workflow we can use the action they’ve created called codecov/codecov-action@v2, as below:

Attention

If your GitHub repository is private, you’ll need to provide an “upload token” to allow Codecov to access it as described in the Codecov documentation.

    # Step 7. Use Codecov to track coverage
    - uses: codecov/codecov-action@v2
      with:
        file: ./coverage.xml    # coverage report
        fail_ci_if_error: true  # terminate workflow if an error occurs

With this step in our workflow, coverage will automatically be recorded for each new proposed change to our code. Codecov can show you whether coverage has increased or decreased, by how much, and will link to relevant areas of your package’s source code. This information will automatically appear on any pull request someone makes to the main branch on GitHub, or it can be viewed anytime on the Codecov website; for example, Fig. 8.4 shows the coverage dashboard for a package called pypkgs, where coverage decreased significantly after the most recent commit.

Example of the Codecov dashboard linked to a repository called `pypkgs`. Coverage decreased significantly after the most recent commit.

Fig. 8.4 Example of the Codecov dashboard linked to a repository called pypkgs. Coverage decreased significantly after the most recent commit.

8.4.4. Build documentation

The final step we’ll add to our CI workflow will be to check that our documentation builds without issue. We’ll use the same make html command we’ve used throughout this book to build documentation with sphinx to do this:

    # Step 8. Build documentation
    - name: Build documentation
      run: poetry run make html --directory docs/

8.4.5. Testing continuous integration

We’ve now set up our CI pipeline! Our final .github/workflows/ci-cd.yml file looks like this:

name: ci-cd

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  ci:
    # Step 1. Set up operating system
    runs-on: ubuntu-latest
    steps:
    # Step 2. Set up Python 3.9
    - uses: actions/setup-python@v2
      with:
        python-version: 3.9
    # Step 3. Check-out repository so we can access its contents
    - uses: actions/checkout@v2
    # Step 4. Install poetry
    - uses: snok/install-poetry@v1
    # Step 5. Install our pycounts package
    - name: Install package
      run: poetry install
    # Step 6. Run tests for pycounts
    - name: Test with pytest
      run: poetry run pytest tests/ --cov=pycounts --cov-report=xml
    # Step 7. Use Codecov to track coverage
    - uses: codecov/codecov-action@v2
      with:
        file: ./coverage.xml    # coverage report
        fail_ci_if_error: true  # terminate workflow if an error occurs
    # Step 8. Build documentation
    - name: Build documentation
      run: poetry run make html --directory docs/

We’re now ready to test out our workflow! Let’s go ahead and commit our workflow file to version control and push it to GitHub. This will trigger our workflow because we configured it to run when someone pushes to the “main” branch of our repository.

git add .github/workflows/ci-cd.yml
git commit -m "build: add CI workflow"
git push

Now if we go to our pycounts GitHub repository and click on the “Actions” tab, we should see our workflow, as shown in Fig. 8.5:

Successfully run continuous integration workflow on GitHub.

Fig. 8.5 Successfully run continuous integration workflow on GitHub.

We can investigate the build logs by clicking the “ci” job as in Fig. 8.6:

Continuous integration workflow logs.

Fig. 8.6 Continuous integration workflow logs.

This workflow will trigger anytime someone makes a push or pull request with the main branch. Now you, or your collaborators, don’t have to worry about remembering all these steps or running them manually! In the next section, we’ll take this automation to the next level, and set up a workflow to automatically deploy a new version of our package if proposed changes pass the CI workflow.

8.5. Setting up continuous deployment

In the previous step, we set up CI for our package to check that tests run, code coverage is stable, and documentation still builds, whenever we make a push or pull request with new changes to the main branch of our repository.

In this section we’ll set up continuous deployment (CD). If the changes we push to our repository pass our CI, then we want a CD workflow that will automatically:

  1. Create a new version of our pycounts package;

  2. Build new distributions (i.e., sdist and wheel);

  3. Upload the distributions to TestPyPI and test that the package can be installed successfully; and,

  4. Upload the distributions to PyPI.

We’ll build up that CD workflow in this section.

8.5.1. Set up

To set up CD, we’ll add to the .github/workflows/ci-cd.yml workflow file we created in Section 8.4. Our aim here is to add a new job called “cd” to this workflow that will trigger a deployment of our package each time updated code is pushed to the main branch of our repository. We only want this job to execute if:

  1. The “ci” job passes — we don’t want to deploy a new version of our package if it isn’t passing CI. We can specify this constraint using the needs keyword in GitHub Actions (documentation); and,

  2. Code is pushed to the main branch — we don’t want to deploy a new version of our package when a pull request is opened, we only want to deploy a new version when a pull request is merged into the main branch or changes are pushed to the main branch directly. We can specify this constraint using the if syntax in GitHub Actions (documentation).

We can set up a new job and add the two constraints above with the following syntax:

name: ci-cd

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  ci:
    # ...
    # CI steps hidden
    # ...
  cd:
    # Only run this job if the "ci" job passes
    needs: ci
    # Only run this job if new work is pushed to "main"
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'

Now we can set up our CD workflow. In GitHub Actions, each job runs on a fresh runner which we need to set up from scratch. Once we’ve done that, our CD workflow will effectively comprise all the steps we walked through manually in Chapter 7: Releasing and versioning. Below we list all the steps required to set up the CD workflow:

  1. Specify an operating system. We’ll be using Ubuntu again with the syntax runs-on: ubuntu-latest;

  2. Install Python (action: actions/setup-python@v2);

  3. Check out our repository so we can access its contents (action: actions/checkout@v2);

  4. Make a new release of pycounts (command: semantic-release publish, this uses the Python Semantic Release (PSR) tool which we described in Section 7.2.2 and will describe again below);

  5. Upload new release to TestPyPI (action: pypa/gh-action-pypi-publish@release/v1);

  6. Test that the new package version installs successfully from TestPyPI (command: pip install; and,

  7. Upload new release to PyPI (action: pypa/gh-action-pypi-publish@release/v1).

Steps 1-3 are the same as we set up previously for our CI workflow, so we can just copy and paste them into our “cd” job as below. The only new code here is that we specify the input parameter fetch-depth: 0 for the actions/checkout@v2 action. This input parameter will allow the PSR tool to access the full history of commits in our repository, so that it can determine how to bump the package’s version. Without this parameter, PSR can only access the single most recent commit message.

name: ci-cd

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  ci:
    # ...
    # CI steps hidden
    # ...
  cd:
    # Only run this job if the "ci" job passes
    needs: ci
    # Only run this job if the "main" branch changes
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    # Step 1. Set up operating system
    runs-on: ubuntu-latest
    steps:
    # Step 2. Set up Python 3.9
    - uses: actions/setup-python@v2
      with:
        python-version: 3.9
    # Step 3. Check-out repository so we can access its contents
    - uses: actions/checkout@v2
      with:
        fetch-depth: 0

We’ll discuss steps 4-7 in the sections below.

8.5.2. Automatically creating a new package version

As we saw in Chapter 7: Releasing and versioning, there’s a few key steps to go through when creating a new version of your package:

  1. Document what’s changed in the CHANGELOG.md;

  2. Bump the package version number;

  3. Tag a new release on GitHub; and,

  4. Build a new sdist and wheel.

In Section 7.2.2 we introduced the Python Semantic Release (PSR) tool, which can automatically bump your package’s version number based on keywords it finds in commit messages.

However, PSR can do more than this! In fact, it can do all the steps we list above. In Section 7.2.2 we saw how PSR is configured using a table called [tool.semantic_release] in the pyproject.toml. To configure it to perform all the steps above, we need to add a few keys to that table as follows:

[tool.semantic_release]
version_variable = "pyproject.toml:version" # version location
branch = "main"                             # branch to make releases of
changelog_file = "CHANGELOG.md"             # changelog file
build_command = "pip install poetry && poetry build"  # build dists
dist_path = "dist/"                         # where to put dists
upload_to_pypi = false                      # don't auto-upload to PyPI
remove_dist = false                         # don't remove dists

We’ve added comments above to clarify what each key is doing in the table and we describe them in Table 8.2. You can also read more about these configuration options in the PSR documentation.

Table 8.2 Description of Python Semantic Release configuration options.

Key

Description

version_variable

Location of version number for PSR to bump.

branch

Branch where releases should be made from.

changelog_file

Location of changelog file for PSR to update using commit messages.

build_command

How to build new distributions for the release.

dist_path

Location of distributions after running build_command.

upload_to_pypi

Whether to upload to PyPI. Default is true, but we want to upload to TestPyPI first to test things out, so we’ve turned this off.

remove_dist

Whether to remove distributions at dist_path after upload. We turned this off because we want to upload these distributions to TestPyPI and PyPI ourselves.

With PSR configured in our pyproject.toml, we can now add it as a step to our CD workflow. In Section 7.2.2 we used the command semantic-release version to get PSR to automatically update our version based on keywords it finds in all the commit messages that have been made since the last tag of your package. In our CD workflow, we’ll be using the slightly different command semantic-release publish which will bump our version, update our changelog, tag a new release on GitHub, and build a new sdist and wheel. Because PSR will directly modifying our CHANGELOG.md file and tagging a new release for us in our GitHub repository, we need to configure the Git credentials of the runner machine. This involves setting the username as github-actions and email address as github-actions@github.com. If you don’t configure these credentials, the workflow won’t be able to make changes to our repository, as you can read more about in the GitHub documentation.

    # Step 4. Use PSR to make release
    - name: Python Semantic Release
      run: |
          pip install python-semantic-release
          git config user.name github-actions
          git config user.email github-actions@github.com
          semantic-release publish

We’ll see PSR in action shortly, but let’s first configure the rest of our CD workflow file.

8.5.3. Uploading to TestPyPI and PyPI

The PSR step will create an sdist and wheel for our new package version. What we need to do next in our CD workflow is try uploading those distributions to TestPyPI. This step is not strictly necessary but it’s a good idea because it can help catch any unexpected errors before we upload our package to PyPI.

Rather than write the code needed to do all this from scratch, we’ll use the pypa/gh-action-pypi-publish@release/v1 action. This action relies on token authentication with TestPyPI (rather than the classic username and password authentication). To use the action, you’ll need to log-in to TestPyPI, create an API token, and add the token as a secret called TEST_PYPI_API_TOKEN to your GitHub repository, as shown in Fig. 8.7.

Adding the TestPyPI API token secret to our GitHub repository.

Fig. 8.7 Adding the TestPyPI API token secret to our GitHub repository.

To use the pypa/gh-action-pypi-publish@release/v1 action, we can add the following step to our CD workflow. Note how we configure the action to use the token user method, specify password as the TEST_PYPI_API_TOKEN we just added to our repository, and we point the action to the TestPyPI repository (repository_url: https://test.pypi.org/legacy/).

    # Step 5. Publish to TestPyPI
    - uses: pypa/gh-action-pypi-publish@release/v1
      with:
        user: __token__
        password: ${{ secrets.TEST_PYPI_API_TOKEN }}
        repository_url: https://test.pypi.org/legacy/

The above action will publish the new version of your package to TestPyPI. We now want to test that we can install the package correctly from TestPyPI using the following command:

    # Step 6. Test install from TestPyPI
    - name: Test install from TestPyPI
      run: |
          pip install \
          --index-url https://test.pypi.org/simple/ \
          --extra-index-url https://pypi.org/simple \
          pycounts

Finally, the last step in our CD workflow will be publishing our package to PyPI. This uses the same pypa/gh-action-pypi-publish@release/v1 action as earlier, and will require you to obtain a token from PyPI and add the token as PYPI_API_TOKEN to your GitHub repository, as shown in Fig. 8.8.

Adding the PyPI API token secret to our GitHub repository.

Fig. 8.8 Adding the PyPI API token secret to our GitHub repository.

    # Step 7. Publish to PyPI
    - uses: pypa/gh-action-pypi-publish@release/v1
      with:
        user: __token__
        password: ${{ secrets.PYPI_API_TOKEN }}

8.5.4. Testing continuous deployment

We’ve now set up our CD workflow! Our final .github/workflows/ci-cd.yml file looks like this:

name: ci-cd

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  ci:
    # ...
    # CI steps same as before
    # ...
  cd:
    # Only run this job if the "ci" job passes
    needs: ci
    # Only run this job if new work is pushed to "main"
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    # Step 1. Set up operating system
    runs-on: ubuntu-latest
    steps:
    # Step 2. Set up Python 3.9
    - uses: actions/setup-python@v2
      with:
        python-version: 3.9
    # Step 3. Check-out repository so we can access its contents
    - uses: actions/checkout@v2
      with:
        fetch-depth: 0
    # Step 4. Use PSR to make release
    - name: Python Semantic Release
      run: |
          pip install python-semantic-release
          git config user.name github-actions
          git config user.email github-actions@github.com
          semantic-release publish
    # Step 5. Publish to TestPyPI
    - uses: pypa/gh-action-pypi-publish@release/v1
      with:
        user: __token__
        password: ${{ secrets.TEST_PYPI_API_TOKEN }}
        repository_url: https://test.pypi.org/legacy/
    # Step 6. Test install from TestPyPI
    - name: Test install from TestPyPI
      run: |
          pip install \
          --index-url https://test.pypi.org/simple/ \
          --extra-index-url https://pypi.org/simple \
          pycounts
    # Step 7. Publish to PyPI
    - uses: pypa/gh-action-pypi-publish@release/v1
      with:
        user: __token__
        password: ${{ secrets.PYPI_API_TOKEN }}

We’re now ready to test out our full CI/CD workflow! Let’s go ahead and commit our new workflow file and our pyproject.toml file (which we changed when we added the configuration options for PSR) to version control and push it to GitHub. This will trigger our CI/CD workflow because we configured it to run when someone pushes to the “main” branch of our repository. For the sake of example, we’ll include the “feat” keyword in our commit message to trigger PSR to make a minor release of our package from these changes (we described what keywords trigger particular version bumps in Section 7.2.2).

git add .github/workflows/ci-cd.yml pyproject.toml
git commit -m "feat: add CI/CD workflow"
git push

Now if we go to our pycounts GitHub repository and click on the “Actions” tab, we should see a new run of our workflow as shown in Fig. 8.9:

Continuous deployment workflow on GitHub.

Fig. 8.9 Continuous deployment workflow on GitHub.

If we click on the workflow we will see it was composed of two jobs, “ci” and “cd”, each of which ran successfully, as shown in Fig. 8.10:

Successfully run continuous deployment workflow on GitHub.

Fig. 8.10 Successfully run continuous deployment workflow on GitHub.

If we click on the “cd” job to view the build log, we can see that PSR parsed our commit message — “feat: add CI/CD workflow” — to determine that our package should be bumped with a minor release from 0.2.0 to 0.3.0 as shown in Fig. 8.11:

The Python semantic release tool automatically bumped the package version from 0.2.0 to 0.3.0.

Fig. 8.11 The Python semantic release tool automatically bumped the package version from 0.2.0 to 0.3.0.

PSR also automatically updated our changelog and tagged a new release of our package, as shown in Fig. 8.12 and Fig. 8.13 respectively.

The Python semantic release tool automatically updated the changelog and added an entry for v0.3.0 based on commit messages.

Fig. 8.12 The Python semantic release tool automatically updated the changelog and added an entry for v0.3.0 based on commit messages.

The Python semantic release tool automatically created tagged release v0.3.0.

Fig. 8.13 The Python semantic release tool automatically created tagged release v0.3.0.

Finally, we can see from the build logs that the new version of our package was released to TestPyPI and PyPI, as shown in Fig. 8.14:

Deployment of new package version 0.3.0 to PyPI.

Fig. 8.14 Deployment of new package version 0.3.0 to PyPI.

It didn’t take too long for us to implement a CD workflow that completely automates all the steps we would usually have to perform manually when publishing a new release of our package. You may choose, or need, to use different tools and commands to what we’ve used here to implement CD for your packages in the future. But hopefully you can see the kinds of things that are possible with CD and how it can be useful to quickly deploy new releases of your package, how it saves you from having to remember all the commands you’d need to run to do this yourself, and how it lowers the barrier for potential collaborators to contribute to your package.

8.6. Summary

In this chapter we created CI/CD workflows for our pycounts package. What we’ve shown here is just one example and one set of tools for implementing CI/CD — but after reading this chapter, the hope is that you can appreciate the utility of CI/CD and the kinds of workflows that you can set up for your packages in the future.

It’s important to note that when using version control in this book, we’ve been directly modifying the main branch of our repository. However, changes to your package are more typically made on branches. Branches isolate your changes so you can develop your package without affecting the existing, stable version. Only when you’re happy with your changes and they’ve passed CI, do you merge them into the existing source, and trigger a CD workflow (if it exists). In collaborative environments, this is typically done via a “pull request”, which you can read more about in the GitHub documentation. Open-source projects are built off pull requests; go visit your favorite Python package repository on GitHub and click the “Pull requests” tab to see what and how collaborators are merging changes into the package, and the kind of CI/CD workflows that are set up to handle them.

Finally, while we developed our workflow file from scratch here, the py-pkgs-cookiecutter template we used to set up our package in Section 3.2.2 can make the workflow .yml file for you. Recall from Section 3.2.2 that one of the py-pkgs-cookiecutter prompts was as follows:

Select include_github_actions:
1 - no
2 - ci
3 - ci+cd
Choose from 1, 2, 3 [1]:

In the future, you can include a workflow file for CI or CI+CD by selecting an appropriate response.

Congratulations on making it to the end of the book and happy packaging!