Automating Regression Tests in Python with Pytest

Table of Contents

1. What is Regression Testing?
2. Why Automate Regression Tests?
3. Why Pytest for Regression Testing?
4. Setting Up Your Environment
5. Writing Your First Regression Test with Pytest
6. Advanced Pytest Features for Regression Testing
7. Organizing Test Suites for Scalability
8. Integrating with CI/CD Pipelines
9. Best Practices for Effective Regression Testing
10. Conclusion
11. References

What is Regression Testing?

Regression testing is a software testing technique that ensures recent code changes (e.g., new features, bug fixes, refactoring) do not adversely affect existing functionality. The goal is to catch “regressions”—unintended side effects that break previously working features.

Types of Regression Testing:

• Corrective Regression Testing: Runs existing tests after a bug fix to ensure the fix didn’t introduce new issues.
• Progressive Regression Testing: Adds new tests for new features and runs them alongside existing tests.
• Complete Regression Testing: Runs all existing tests (time-consuming but thorough, often used for critical releases).
• Selective Regression Testing: Runs a subset of tests targeting areas affected by recent changes (faster than complete regression).

Example: Imagine a banking app with a “transfer funds” feature. After adding a “scheduled transfers” feature, regression testing would verify that one-time transfers, account balances, and transaction history still work as expected.

Why Automate Regression Tests?

Manual regression testing is common in small projects but becomes impractical as applications scale. Here’s why automation is essential:

Automated regression tests act as a safety net, giving developers confidence to refactor or add features without fear of breaking existing functionality.

Why Pytest for Regression Testing?

Pytest is a popular Python testing framework known for its simplicity, flexibility, and rich ecosystem. Here’s why it’s ideal for regression testing:

1. Simple Syntax

Pytest uses plain Python functions with names prefixed by test_ (e.g., def test_addition()), making it easy to write and read. No boilerplate (unlike unittest, Python’s built-in framework).

2. Powerful Features

• Fixtures: Reusable setup/teardown logic (e.g., initializing a database connection).
• Parametrization: Test multiple inputs with a single test function.
• Rich Plugins: Extend functionality with plugins like pytest-html (reports), pytest-xdist (parallel runs), and pytest-mock (mocking).
• Detailed Reports: Clear, color-coded output highlighting failures and errors.

3. Compatibility

Works with existing unittest tests and other Python testing tools (e.g., doctest).

4. Scalability

Easily organizes tests into directories, modules, and classes, supporting large codebases.

Setting Up Your Environment

Let’s set up pytest to start writing regression tests.

Prerequisites

• Python 3.6+ (check with python --version).
• pip (Python package installer, included with Python 3.4+).

Install Pytest

Run this command in your terminal:

pip install pytest  

Verify installation:

pytest --version  
# Output: pytest 7.4.0 (or similar)  

Writing Your First Regression Test with Pytest

Let’s walk through a practical example. We’ll build a simple calculator app, write regression tests, and simulate a regression to see how pytest catches it.

Step 1: Create the Application Code

First, create a calculator.py file with basic arithmetic functions:

# calculator.py  

def add(a: int, b: int) -> int:  
    return a + b  

def subtract(a: int, b: int) -> int:  
    return a - b  

Step 2: Write Your First Test

Create a tests directory (pytest convention) and add a test file test_calculator.py:

# tests/test_calculator.py  

from calculator import add  

def test_add_positive_numbers():  
    result = add(2, 3)  
    assert result == 5, "2 + 3 should equal 5"  

def test_add_negative_numbers():  
    result = add(-1, -1)  
    assert result == -2, "-1 + (-1) should equal -2"  

• Key Points:
  • Test functions start with test_.
  • assert statements validate expected outcomes.

Step 3: Run the Tests

Execute pytest from the project root:

pytest  

Output:

============================= test session starts ==============================  
collected 2 items  

tests/test_calculator.py ..                                              [100%]  

============================== 2 passed in 0.01s ===============================  

All tests pass—great!

Step 4: Simulate a Regression

Now, let’s intentionally break the add function to simulate a regression. Modify calculator.py:

# calculator.py (with bug)  

def add(a: int, b: int) -> int:  
    return a * b  # Oops! Accidentally used * instead of +  

def subtract(a: int, b: int) -> int:  
    return a - b  

Step 5: Run Tests Again

Rerun pytest:

pytest  

Output:

============================= test session starts ==============================  
collected 2 items  

tests/test_calculator.py FF                                              [100%]  

=================================== FAILURES ===================================  
________________________________ test_add_positive_numbers ________________________________  

    def test_add_positive_numbers():  
        result = add(2, 3)  
>       assert result == 5, "2 + 3 should equal 5"  
E       AssertionError: 2 + 3 should equal 5  
E       assert 6 == 5  

tests/test_calculator.py:4: AssertionError  
________________________________ test_add_negative_numbers ________________________________  

    def test_add_negative_numbers():  
        result = add(-1, -1)  
>       assert result == -2, "-1 + (-1) should equal -2"  
E       AssertionError: -1 + (-1) should equal -2  
E       assert 1 == -2  

tests/test_calculator.py:8: AssertionError  
=========================== short test summary info ============================  
FAILED tests/test_calculator.py::test_add_positive_numbers - AssertionError: 2 + 3 should equal 5  
FAILED tests/test_calculator.py::test_add_negative_numbers - AssertionError: -1 + (-1) should equal -2  
============================== 2 failed in 0.02s ===============================  

Pytest catches the regression! Fix the add function (replace * with +), and rerun tests—they’ll pass again.

Advanced Pytest Features for Regression Testing

Pytest’s advanced features make regression testing more efficient and maintainable. Let’s explore key ones.

1. Fixtures: Reusable Setup/Teardown

Fixtures are functions that provide test data or setup/teardown logic. Use @pytest.fixture to define them.

Example: Create a fixture to initialize a calculator instance.

First, update calculator.py to use a class (more realistic for larger apps):

# calculator.py  

class Calculator:  
    def add(self, a: int, b: int) -> int:  
        return a + b  

    def subtract(self, a: int, b: int) -> int:  
        return a - b  

Now, create a conftest.py file in the tests directory (shared fixtures for all tests):

# tests/conftest.py  

import pytest  
from calculator import Calculator  

@pytest.fixture  
def calculator():  
    """Fixture to provide a Calculator instance."""  
    return Calculator()  

Use the fixture in tests (tests/test_calculator.py):

# tests/test_calculator.py  

def test_add_positive_numbers(calculator):  # Fixture injected here  
    result = calculator.add(2, 3)  
    assert result == 5  

def test_subtract(calculator):  
    result = calculator.subtract(5, 3)  
    assert result == 2  

Fixtures eliminate redundant code and ensure consistent setup across tests.

2. Parametrization: Test Multiple Inputs

Use @pytest.mark.parametrize to test multiple input-output pairs with a single test function.

Example: Test add with positive, negative, and zero values:

# tests/test_calculator.py  

import pytest  

@pytest.mark.parametrize("a, b, expected", [  
    (2, 3, 5),    # Positive numbers  
    (-1, -1, -2), # Negative numbers  
    (0, 0, 0),    # Zero  
    (5, -3, 2),   # Mixed signs  
])  
def test_add_parametrized(calculator, a, b, expected):  
    assert calculator.add(a, b) == expected  

Run tests:

pytest -v  # -v for verbose output  

Output shows 4 tests (one per parametrized input), all passing.

3. Markers: Group and Select Tests

Markers categorize tests (e.g., slow, integration) to run subsets of tests.

Example: Mark slow tests and run only fast ones.

First, define a marker in pytest.ini (project root):

# pytest.ini  
[pytest]  
markers =  
    slow: marks tests as slow (deselect with -m "not slow")  

Mark a test:

# tests/test_calculator.py  

import time  

@pytest.mark.slow  
def test_large_numbers(calculator):  
    time.sleep(2)  # Simulate slow test  
    assert calculator.add(10**6, 10**6) == 2 * 10**6  

Run only fast tests:

pytest -m "not slow"  

4. Plugins for Enhanced Workflow

• pytest-html: Generate HTML reports (useful for sharing results).
  Install: pip install pytest-html
  Run: pytest --html=report.html

• pytest-xdist: Run tests in parallel (speed up large test suites).
  Install: pip install pytest-xdist
  Run: pytest -n auto (uses all CPU cores).

• pytest-mock: Simplify mocking (e.g., mock external APIs).
  Install: pip install pytest-mock

Organizing Test Suites for Scalability

As your codebase grows, organize tests to keep them maintainable. Here’s a recommended structure:

my_project/  
├── calculator.py          # Application code  
├── pytest.ini             # Pytest configuration  
├── requirements.txt       # Dependencies (e.g., pytest)  
└── tests/                 # All tests live here  
    ├── conftest.py        # Shared fixtures  
    ├── test_calculator.py # Unit tests for calculator  
    ├── integration/       # Integration tests  
    │   └── test_api.py  
    └── regression/        # Dedicated regression tests  
        └── test_critical_paths.py  

Integrating with CI/CD Pipelines

Regression tests should run automatically on every code change. Let’s integrate pytest with GitHub Actions (a CI/CD tool).

Step 1: Create a Workflow File

Add .github/workflows/pytest.yml to your repo:

name: Run Pytest  

on: [push, pull_request]  # Run on pushes and PRs  

jobs:  
  test:  
    runs-on: ubuntu-latest  
    steps:  
      - name: Checkout code  
        uses: actions/checkout@v4  

      - name: Set up Python  
        uses: actions/setup-python@v4  
        with:  
          python-version: "3.10"  

      - name: Install dependencies  
        run: |  
          python -m pip install --upgrade pip  
          pip install pytest pytest-html  

      - name: Run pytest  
        run: pytest --html=report.html --self-contained-html  

      - name: Upload report  
        uses: actions/upload-artifact@v3  
        with:  
          name: pytest-report  
          path: report.html  

Step 2: Push to GitHub

Commit and push your code. GitHub Actions will:

1. Spin up a Ubuntu VM.
2. Install Python and dependencies.
3. Run pytest and generate an HTML report.
4. Upload the report as an artifact (downloadable from the Actions tab).

Now, regression tests run on every code change—no manual effort needed!

Best Practices for Effective Regression Testing

To maximize the value of your regression tests:

1. Keep Tests Independent

Tests should not rely on each other (e.g., one test’s output shouldn’t affect another). Use fixtures to reset state between tests.

2. Test Both Happy Paths and Edge Cases

Cover typical user scenarios (happy paths) and edge cases (e.g., invalid inputs, large numbers).

3. Keep Tests Fast

Slow tests discourage frequent runs. Optimize by:

• Using mocks for external services (e.g., APIs, databases).
• Avoiding unnecessary setup (e.g., reuse database connections).

4. Maintain Readable Tests

Use descriptive test names (e.g., test_add_negative_numbers instead of test_add2). Add comments for complex logic.

5. Avoid Flaky Tests

Flaky tests (pass/fail randomly) erode trust. Fix them by:

• Ensuring deterministic behavior (e.g., avoid race conditions).
• Using retries for flaky external dependencies (via pytest-rerunfailures plugin).

6. Regularly Review and Update Tests

As the codebase evolves, update tests to reflect new features or deprecated functionality. Remove obsolete tests.

Conclusion

Automating regression tests with pytest is a game-changer for Python developers. It provides a scalable, efficient way to ensure code changes don’t break existing functionality, enabling faster development and higher-quality software.

By leveraging pytest’s features—fixtures, parametrization, markers, and plugins—you can build a robust regression testing suite. Integrating with CI/CD pipelines ensures tests run on every code change, catching regressions early.

Start small: write tests for critical features, then expand. Your future self (and team) will thank you!

References

• Pytest Official Documentation
• GitHub Actions Documentation
• Python Testing with pytest (Book by Brian Okken)
• pytest Fixtures: Explicit, Modular, Scalable
• pytest Plugins