Testing a Python CLI and Automating CI - 6/9

Packaging gives us an installable command. Tests give us evidence that its contracts survive change: options still select the expected output, failures return non-zero status codes, and the package works on the Python versions and operating systems we claim to support.

The main design question is where to place the test boundary. A unit test should not depend on a public HTTP service being online or returning exactly the same body. That is a separate integration concern.

Series

Make the external boundary replaceable

The CLI currently calls requests.get() directly. A small parameter gives tests control over that boundary without introducing a mocking framework:

def main(argv=None, *, request_get=requests.get) -> int:
    args = build_parser().parse_args(argv)

    try:
        response = request_get(args.url, timeout=10)
        response.raise_for_status()
    except requests.RequestException as error:
        print(f"tihttp: request failed: {error}", file=sys.stderr)
        return 1

    print_response(response, args.section)
    return 0

The default behavior is unchanged in normal use. Tests can now supply a function with the same small contract: accept a URL and timeout, then return a response-like object.

Test behavior, not the network

Pytest’s capsys fixture captures standard output and standard error. Combined with the injected request function, it lets us test the CLI deterministically:

from tihttp.cli import main


class StubResponse:
    headers = {"Content-Type": "application/json"}
    text = '{"ok": true}'

    def raise_for_status(self) -> None:
        pass


def stub_get(url: str, *, timeout: int) -> StubResponse:
    assert url == "https://example.test"
    assert timeout == 10
    return StubResponse()


def test_body_only(capsys):
    status = main(
        ["--body-only", "https://example.test"],
        request_get=stub_get,
    )

    captured = capsys.readouterr()
    assert status == 0
    assert captured.out == '{"ok": true}\n'
    assert captured.err == ""

This test checks the behavior we own: argument handling, the call made at the HTTP boundary, output selection, and the return code. It does not test Requests or the availability of example.test.

Failure behavior deserves its own test:

import requests


def time_out(url: str, *, timeout: int):
    raise requests.Timeout("server did not respond")


def test_timeout_is_reported_on_stderr(capsys):
    status = main(["example.test"], request_get=time_out)

    captured = capsys.readouterr()
    assert status == 1
    assert captured.out == ""
    assert "request failed" in captured.err

A smaller number of focused tests is more useful than one test that contacts a live API and sometimes fails for reasons outside the project. A separate, explicitly marked integration test can still exercise real HTTP when that confidence is worth the extra latency and instability.

Recreate the test environment with tox

Tox turns the test command and supported interpreter set into project configuration:

[tox]
envlist = py36, py37, py38, py39

[testenv]
deps = pytest
commands =
    python -m pytest {posargs}

Tox creates each environment, installs the package, and then runs the configured command. This catches packaging omissions which a repository-root import can hide. Part 8 will separately smoke-test the wheel itself.

Developers can run the full matrix or one environment:

$ python -m tox
$ python -m tox -e py39 -- tests/test_cli.py

Choose a CI matrix, do not merely maximize it

Every matrix entry costs time and creates another place for external failures. For this small, pure-Python CLI, a reasonable policy is to test every supported Python version on Linux and smoke-test the oldest and newest versions across the other operating systems:

name: CI

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        include:
          - {os: ubuntu-latest, python: "3.7"}
          - {os: ubuntu-latest, python: "3.8"}
          - {os: ubuntu-latest, python: "3.9"}
          - {os: macos-latest, python: "3.9"}
          - {os: windows-latest, python: "3.6"}

    runs-on: ${{ matrix.os }}

    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python }}
      - run: python -m pip install tox
      - run: python -m tox -e py

The exact matrix should follow the project’s support policy. CI is the executable form of that policy, not a substitute for deciding it.

Tests now protect behavior and packaging. Part 7 turns the same care toward the human-facing interface.

Series

Make the external boundary replaceable

Test behavior, not the network

Recreate the test environment with tox

Choose a CI matrix, do not merely maximize it

Further reading