Turning a Website into an API

Revision note: I originally published this article in 2021. The project has since moved from Python 3.9, requirements files, and Heroku to Python 3.13, pyproject.toml, uv, and Docker Hub releases. I also revised the article to cover the typed response models, fixture-based tests, caching, failure handling, health and metadata endpoints, and more resilient parser that were added later.

GitHub Trending presents useful information about popular repositories and developers, but it presents that information as HTML for people rather than as a supported JSON API for programs. This project turns that page into a small FastAPI service.

At first glance, the task sounds like “scrape a page and return a dictionary.” The more interesting problem is the boundary around it. The service must translate client parameters into an upstream request, interpret HTML that it does not control, normalize the result into a stable response, and distinguish its own failures from failures at GitHub.

That makes the application an adapter, not merely a proxy:

1. FastAPI validates the client’s path and query parameters.
2. aiohttp requests the corresponding GitHub Trending page.
3. Beautiful Soup and lxml parse its repository or developer cards.
4. Pydantic validates the normalized response.
5. FastAPI serializes that response as JSON and documents it through OpenAPI.

The API exposes separate resources because repository and developer trends have different response shapes and filters:

Endpoint	Purpose	Optional filters
/repositories	Trending repositories across languages	since, spoken_language_code
/repositories/{prog_lang}	Repository trends for one programming language	since, spoken_language_code
/developers	Trending developers across languages	since
/developers/{prog_lang}	Developer trends for one programming language	since
/health	Lightweight process health	none
/metadata	Version, links, cache TTL, and supported parameter counts	none

There is no hosted public instance at the moment, so the examples below assume that the service is running locally.

Develop the parser against saved HTML

A scraper should not need a live network request for every development cycle. I started by saving a GitHub Trending response locally with HTTPie:

$ http --body https://github.com/trending > repositories.html

The downloaded document was several hundred kilobytes, but the desired data lived in a small number of repeated repository cards. Each card was represented by an article element with the Box-row class.

The parser can express that structure directly:

import bs4


def make_soup(raw_html: str) -> bs4.element.ResultSet:
    soup = bs4.BeautifulSoup(raw_html, "lxml")
    return soup.select("article.Box-row")

An earlier implementation first split the page into lines and copied the text between occurrences of article. That appeared to reduce the input before parsing, but it coupled the scraper to whitespace and string layout as well as to the DOM. Parsing once and selecting the desired nodes is both clearer and less fragile.

Selectors do not make the upstream contract stable, however. GitHub later changed repository headings from an h1 to an h2. The current parser deliberately accepts both forms and verifies that the candidate link looks like a repository path:

from typing import Optional

import bs4


def find_repository_link(
    article: bs4.element.Tag,
) -> Optional[bs4.element.Tag]:
    headings = article.find_all(["h1", "h2"], class_="h3")
    for heading in headings:
        link = heading.find("a", href=True)
        if link and link["href"].count("/") >= 2:
            return link
    return None

The same defensive rule applies to optional values. A repository can have no description, no detected programming language, or no “built by” section. Numeric labels may contain commas, while some cards may omit a stars or forks link. The parser converts those cases into None or an empty list instead of letting one incomplete card fail the entire response.

Here is a condensed part of the repository extraction:

def scraping_repositories(
    articles: bs4.element.ResultSet,
    since: str,
) -> list[dict]:
    repositories = []

    for rank, article in enumerate(articles, start=1):
        repo_link = find_repository_link(article)
        if repo_link is None:
            continue

        relative_url = repo_link["href"]
        username, repository_name = relative_url.strip("/").split("/", 1)

        language_tag = article.find(
            "span",
            itemprop="programmingLanguage",
        )
        language = (
            language_tag.get_text(strip=True)
            if language_tag
            else None
        )

        repositories.append(
            {
                "rank": rank,
                "username": username,
                "repositoryName": repository_name,
                "url": "https://github.com" + relative_url,
                "language": language,
                "since": since,
            }
        )

    return repositories

The full implementation also extracts descriptions, language colors, star and fork counts, period-specific stars, and contributor data. I keep those details in small lookup and conversion helpers so the main loop describes the response rather than the mechanics of every selector.

Treat HTML samples as contract tests

The saved pages became more valuable when they moved from development aids into the test suite. The project now keeps repository and developer HTML fixtures together with their expected JSON output. Parameterized tests run the same parser against normal cards and edge cases such as missing descriptions, stars, forks, avatars, and popular repositories:

import json

import pytest


@pytest.mark.parametrize(
    "input_html, expected_json",
    [
        ("data/repodata1.html", "data/repodata1.json"),
        ("data/repodata3.html", "data/repodata3.json"),
        ("data/repodata4.html", "data/repodata4.json"),
    ],
)
def test_repository_scraping(input_html, expected_json):
    with open(input_html) as html_file:
        articles = make_soup(html_file.read())

    with open(expected_json) as json_file:
        expected = json.load(json_file)

    assert scraping_repositories(articles, since="daily") == expected

These tests are deterministic, quick, and polite to the upstream service because they never contact GitHub. They also preserve examples of markup that once broke the parser. A live smoke test can answer whether GitHub changed today, but it should not replace fixtures that explain exactly which HTML shapes the parser supports.

The route tests isolate the other half of the adapter. They replace the outbound request with a fake response, then verify URL construction, query parameters, cache reuse, response validation, and error status codes. This separation makes failures easier to locate: either the HTML-to-data contract changed, or the HTTP-to-API contract changed.

Make the API contract explicit with FastAPI

FastAPI derives validation and OpenAPI documentation from ordinary Python type annotations. String-backed Enums restrict the accepted values while still serializing them as strings:

from enum import Enum


class AllowedDateRanges(str, Enum):
    daily = "daily"
    weekly = "weekly"
    monthly = "monthly"

An invalid value such as since=yearly is rejected with a 422 response before the scraper runs. The same Enum values appear in the generated interactive documentation:

Request validation is only half of an API contract. Pydantic response models describe what successful calls return and catch mismatches between the parser and the public schema:

from pydantic import BaseModel


class Contributor(BaseModel):
    username: str
    url: str
    avatar: str


class Repository(BaseModel):
    rank: int
    username: str
    repositoryName: str
    url: str
    description: str | None
    language: str | None
    totalStars: int | None
    forks: int | None
    starsSince: int | None
    since: str
    builtBy: list[Contributor]

The route combines the input and output contracts:

@app.get(
    "/repositories/{prog_lang}",
    response_model=list[Repository],
)
async def trending_repositories_by_language(
    prog_lang: AllowedProgrammingLanguages,
    since: AllowedDateRanges | None = None,
    spoken_language_code: AllowedSpokenLanguages | None = None,
) -> list[Repository]:
    payload = build_payload(since, spoken_language_code)
    url = f"https://github.com/trending/{prog_lang.value}"
    return await get_repository_trends(url, payload)

Using .value here matters. The Enum member is part of the Python interface; its value is the string understood by the upstream URL. Keeping that translation at the boundary prevents internal type choices from leaking into an HTTP request.

Async I/O, failures, and caching

Each trend request waits on another HTTP server. That makes aiohttp a natural fit for the outbound boundary: while one request waits for GitHub, the event loop can make progress on other work. The parsing step remains synchronous. Adding async does not make Beautiful Soup parse HTML faster.

The client sets a user agent and a ten-second total timeout, checks non-success status codes, and translates aiohttp and timeout exceptions into an application-specific UpstreamRequestError. The FastAPI layer then maps that failure to 502 Bad Gateway:

try:
    raw_html = await get_request(
        url,
        compress=True,
        params=payload,
    )
except UpstreamRequestError as request_error:
    raise fastapi.HTTPException(
        status_code=502,
        detail="Unable to fetch GitHub Trending data",
    ) from request_error

This status code tells clients that the API itself was reachable but could not obtain a valid response from its upstream dependency. Returning an error message with status 200 would make that distinction invisible to clients and monitoring.

The original project included a small local comparison between synchronous Requests calls and asynchronous aiohttp calls. It showed higher throughput for the asynchronous version under that particular workload:

The chart is useful as an experiment, not as a general benchmark. It combines application work, local concurrency, GitHub latency, connection setup, and a small sample size. The reusable conclusion is narrower: asynchronous I/O can overlap waiting, while parser performance must be measured and optimized separately.

The current implementation also keeps fetched HTML in a five-minute in-memory cache. The cache key includes both the upstream URL and a sorted tuple of query parameters:

CACHE_TTL_SECONDS = 300
CacheKey = tuple[str, tuple[tuple[str, str], ...]]

_trending_html_cache: dict[CacheKey, tuple[float, str]] = {}


async def fetch_trending_html(
    url: str,
    payload: dict[str, str],
) -> str:
    cache_key = (url, tuple(sorted(payload.items())))
    cached_at, cached_html = _trending_html_cache.get(
        cache_key,
        (0.0, ""),
    )

    age = time.monotonic() - cached_at
    if cached_html and age < CACHE_TTL_SECONDS:
        return cached_html

    raw_html = await get_request(
        url,
        compress=True,
        params=payload,
    )
    _trending_html_cache[cache_key] = (time.monotonic(), raw_html)
    return raw_html

time.monotonic() is the right clock for elapsed-time comparisons because wall-clock adjustments cannot move it backwards. Caching the raw page avoids repeated upstream traffic, while still letting the parser apply the current extraction rules on every API request.

This cache is intentionally small in scope: every process owns its own dictionary, a restart clears it, and it does not coordinate simultaneous misses for the same key. Those properties are acceptable for a compact service. A multi-worker or multi-instance deployment would need a shared cache only if traffic and measurements justify the added operational dependency.

One similar tradeoff remains in the HTTP client: the implementation creates and closes a ClientSession for each upstream request. That makes ownership simple, but it gives up connection-pool reuse between calls. An application-scoped session would be the next improvement if outbound connection setup becomes measurable.

Package, verify, and release the service

The application now declares its runtime and development dependencies in pyproject.toml. uv resolves them into uv.lock, synchronizes the local environment, and runs commands inside it:

$ uv sync --locked --dev
$ uv run pytest
$ uv run pre-commit run --all-files
$ uv run python -m uvicorn app.main:app \
    --host 127.0.0.1 --port 1313 --reload

The distinction between declaration and resolution is useful: pyproject.toml states the project’s dependency ranges and tool configuration, while the committed lockfile records the resolved environment used by local development and CI.

For distribution, the project builds a container from python:3.13-slim. It installs the application from its project metadata, creates a dedicated system user, exposes the service port, and includes a health check against /health:

FROM python:3.13-slim

WORKDIR /github-trending-api

COPY ./pyproject.toml ./README.md ./
COPY ./app ./app

RUN pip install --no-cache-dir . && \
    groupadd --system app && \
    useradd --system --gid app \
        --home-dir /github-trending-api app

USER app

EXPOSE 5000

CMD ["sh", "-c", "python -m uvicorn app.main:app --host 0.0.0.0 --port=${PORT:-5000}"]

Running as a non-root user reduces the privileges available to the application inside the container. The /health endpoint is deliberately lightweight: it proves that the process can answer HTTP requests without making every container probe depend on GitHub. /metadata serves a different purpose by exposing the API version, documentation links, cache TTL, and parameter counts to clients and operators.

The GitHub Actions workflow closes the lifecycle:

• Pull requests and pushes to main run pre-commit, the pytest suite with coverage, and a Docker build.
• A semantic version tag runs the verification again.
• After the tests pass, separate jobs publish latest and versioned images to Docker Hub and create a GitHub release.
• Docker Hub credentials live in repository secrets rather than in the workflow file.

Heroku was useful for the first version, but the current project deliberately stops at a portable container image. Deployment is then a decision for the image consumer rather than a provider-specific assumption embedded in the repository.

What this project taught me

The interesting part of turning a website into an API is not the happy-path selector. It is deciding what the service promises when the input page is incomplete, slow, unavailable, or structurally different.

The current design makes those decisions visible: Enums validate client input, fixtures preserve known HTML contracts, optional fields tolerate incomplete cards, Pydantic models stabilize output, upstream failures become 502 responses, a short cache reduces repeated work, health endpoints support container operation, and the release workflow verifies the artifact before publication.

There are still useful hardening steps for a larger deployment. I would reuse an application-scoped aiohttp session, add telemetry for unexpectedly empty result sets, and revisit the process-local cache if the service ran with multiple workers. Those are scaling decisions, not prerequisites for understanding the architecture.

The source code and release history show both versions of the project: the small 2021 experiment and the more deliberate service it became.

Further reading

• FastAPI: path parameters with Enums
• FastAPI: concurrency and async/await
• Beautiful Soup: searching the tree
• aiohttp client quickstart
• uv: working on projects
• pytest: parametrizing tests
• Dockerfile best practices
• GitHub Actions: publishing Docker images