Skip to content

Debugging Testcontainers

Debugging failing Testcontainer tests can be tricky. The code is running in separate ephemeral Docker containers that are immediately deleted after the test run finishes.

Below are some debugging and exploratory testing tips to help you debug failing Testcontainer tests.

1. Inspect container logs

Logs are the main source of information when debugging Testcontainers. Generally, you should be able to pinpoint any problem by looking at the container logs in the same way as you'd investigate a problem in a production environment. If you find it difficult to understand how the system behaves from the logs, it's a sign that the logging is insufficient and needs improvement.

By default, tomodachi_testcontainers will forward all container logs to Python's standard logger as INFO logs when containers stop. See Forward Container Logs to pytest section for more information and examples of configuring pytest to show the logs.

Running Testcontainer tests is a great way to do exploratory testing of the system, check out if log messages are meaningful, and it's easy to understand what the system is doing.

2. Pause a test with a breakpoint and inspect running containers

Testcontainers are ephemeral - they're removed immediately after the test run finishes. Sometimes, it's helpful to inspect the state of running containers, e.g., manually check the contents of a database, S3 buckets, message queues, or application logs.

To do that, pause the execution of a test with a breakpoint and manually inspect running containers:

import httpx
import pytest


@pytest.mark.asyncio(loop_scope="session")
async def test_healthcheck_passes(http_client: httpx.AsyncClient) -> None:
    response = await http_client.get("/health")

    # The breakpoint will pause the execution of the test
    # and allow you to inspect running Docker containers.
    breakpoint()

    assert response.status_code == 200
    assert response.json() == {"status": "ok"}

3. Use helper containers and tools for exploratory testing

When logs are insufficient to understand what's going on, it's helpful to use other helper containers and tools for inspecting container state, e.g., what's in the database, S3 buckets, message queues, etc.

Pause a test with a breakpoint and inspect running containers with other tools, for example:

  • Use AWS CLI with aws --endpoint-url=http://localhost:<port> to inspect the state of LocalStack or Moto containers. Find out LocalStack or Moto port in the debug console output or inspect the containers with docker ps.
  • Moto provides a convenient web UI dashboard. Find the link to the Moto dashboard in the pytest console output.
  • Use the DynamoDBAdminContainer to inspect the state of DynamoDB tables.

4. Attach a remote debugger to a running container

As a last resort, you can attach a remote debugger to an application running in a remote container, e.g., to a TomodachiContainer running your application code.

If using VScode, see the VSCode documentation of attaching a remote debugger to a running process over HTTP.

"""An example of attaching a debugger to a running Tomodachi container.

Generally you won't need a debugger in the testcontainer often,
because you should be able to detect most issues by checking the logs,
in the same way as you would to when investigating an issue in a production environment.
"""

from typing import AsyncGenerator, Generator

import httpx
import pytest
import pytest_asyncio

from tomodachi_testcontainers import DockerContainer, TomodachiContainer


@pytest.fixture(scope="module")
def tomodachi_container(testcontainer_image: str) -> Generator[DockerContainer, None, None]:
    with (
        (
            TomodachiContainer(testcontainer_image)
            # Bind debugger port.
            .with_bind_ports(5678, 5678)
            # Explicitly install debugpy. Adding the debugpy to dev dependencies in pyproject will not work
            # because the image is using the 'release' target which doesn't include dev dependencies.
            # Adding the debugpy to production dependencies is not recommended.
            .with_command(
                'bash -c "pip install debugpy; python -m debugpy --listen 0.0.0.0:5678 -m tomodachi run src/healthcheck.py --production"'  # pylint: disable=line-too-long
            )
        ) as container
    ):
        yield container


@pytest_asyncio.fixture(scope="module", loop_scope="session")
async def http_client(tomodachi_container: TomodachiContainer) -> AsyncGenerator[httpx.AsyncClient, None]:
    async with httpx.AsyncClient(base_url=tomodachi_container.get_external_url()) as client:
        yield client


@pytest.mark.asyncio(loop_scope="session")
async def test_healthcheck_passes(http_client: httpx.AsyncClient) -> None:
    # To start the debugging, place a breakpoint in the test and in the production code.
    # If using VSCode, run the test in the debugger and then attach the remote debugger on container port 5678.

    # https://code.visualstudio.com/docs/python/debugging#_debugging-by-attaching-over-a-network-connection
    # See .vscode/launch.example.json.

    # Timeout set to None to avoid getting a TimeoutError while working in the debugger.
    response = await http_client.get("/health", timeout=None)

    # Set a breakpoint after sending the HTTP request to the container, e.g. on the next line.
    # It will trigger the breakpoint set in the production code and won't stop the running containers while debugging.
    assert response.status_code == 200
    assert response.json() == {"status": "ok"}

5. Inspect Testcontainers with Testcontainers Desktop App

Testcontainers Desktop app allows you to prevent container shutdown so you can inspect and debug them, and it has some other extra features.