The Coding Lab
<- All posts
Python

When sys.argv Stops Scaling - 2/9

Reading command-line arguments with sys.argv, defining useful exit codes, and seeing why manual CLI parsing becomes difficult to maintain.

The first version of tinyHTTPie can contact one URL, but that URL is embedded in the source code. The next step is to let the caller provide input without editing the program.

Python exposes the process arguments through sys.argv. It is a useful low-level mechanism, and implementing a tiny parser by hand reveals exactly which responsibilities a command-line parser must handle.

Series

  1. From a Script to a Command-Line Tool
  2. When sys.argv Stops Scaling
  3. Designing a CLI with argparse
  4. Why a Bash Installer Becomes a Liability
  5. Packaging a Python CLI with Console Scripts
  6. Testing a Python CLI and Automating CI
  7. Documentation as Part of the Product
  8. Building a Reliable PyPI Release Pipeline
  9. Publishing a Pure-Python CLI with Conda

What sys.argv contains

sys.argv is a list of strings. The first item is the script name; the remaining items are supplied by the caller:

import sys

print(sys.argv)
$ python tihttp.py -H example.com
['tihttp.py', '-H', 'example.com']

That makes a positional URL easy to read: sys.argv[1]. It does not make the input safe. The element may be missing, there may be too many values, or an option may appear before it. Indexing the list directly turns normal usage mistakes into tracebacks.

A small manual parser

The following version accepts one URL and two flags: -H for headers only and -B for the body only. With neither or both flags, it prints both sections.

import sys
from typing import List, Optional
from urllib.parse import urlsplit

import requests


USAGE = "usage: tihttp [-H] [-B] URL"
VALID_FLAGS = {"-H", "-B"}


def normalize_url(value: str) -> str:
    candidate = value if "://" in value else f"https://{value}"
    parsed = urlsplit(candidate)

    if parsed.scheme not in {"http", "https"} or not parsed.netloc:
        raise ValueError(f"unsupported URL: {value!r}")

    return candidate


def main(argv: Optional[List[str]] = None) -> int:
    arguments = list(sys.argv[1:] if argv is None else argv)
    flags = {item for item in arguments if item.startswith("-")}
    urls = [item for item in arguments if not item.startswith("-")]

    if flags - VALID_FLAGS or len(urls) != 1:
        print(USAGE, file=sys.stderr)
        return 2

    try:
        response = requests.get(normalize_url(urls[0]), timeout=10)
        response.raise_for_status()
    except (ValueError, requests.RequestException) as error:
        print(f"tihttp: {error}", file=sys.stderr)
        return 1

    show_headers = "-H" in flags or not flags or flags == VALID_FLAGS
    show_body = "-B" in flags or not flags or flags == VALID_FLAGS

    if show_headers:
        for name, value in sorted(response.headers.items()):
            print(f"{name}: {value}")

    if show_headers and show_body:
        print()

    if show_body:
        print(response.text)

    return 0


if __name__ == "__main__":
    raise SystemExit(main())

This is still a small program, but several useful contracts have appeared:

  • Usage errors go to standard error and return status 2.
  • Request or URL failures return status 1.
  • Successful execution returns 0.
  • HTTP requests have a timeout and unsuccessful status codes are not treated as success.
  • The URL scheme is normalized in one function rather than scattered across branches.

Exit codes matter because shell scripts and CI systems consume them even when a human never reads the message.

A subtle Boolean trap

It is tempting to test several prefixes like this:

if ("http://" or "https://") not in value:
    ...

That does not check both strings. The or expression returns the first truthy value, so Python effectively evaluates only "http://" not in value. A tuple-aware method such as value.startswith(("http://", "https://")) would work, but parsing the URL gives us clearer validation and access to its scheme and host.

Why manual parsing stops scaling

The HTTP request is now shorter than the code needed to interpret its arguments. We still have no generated help page, long option names, -- separator, structured validation, or consistent error messages. Supporting combined short options such as -HB would add another custom rule.

This is the useful limit of sys.argv: it exposes the raw input correctly, but it does not define a command-line language for us. In part 3, argparse will replace most of this bookkeeping with a declarative interface.

Further reading

<< section 1 | section 3 >>