py4u guide

Advanced Python Techniques: Decorators and Generators

Python’s reputation as a beginner-friendly language often overshadows its depth as a tool for professional developers. Beyond basic syntax and loops, Python offers advanced features that enable elegant, efficient, and maintainable code. Two such cornerstones are **decorators** and **generators**. Decorators empower you to modify function behavior dynamically, while generators revolutionize iteration by enabling lazy evaluation. In this blog, we’ll dive deep into both concepts, exploring their mechanics, use cases, and how they can elevate your Python programming skills.

Table of Contents

  1. Introduction
  2. Decorators: Modifying Function Behavior
  3. Generators: Efficient Iteration with Yield
  4. When to Use Decorators vs. Generators
  5. Conclusion
  6. References

Decorators: Modifying Function Behavior

2.1 What Are Decorators?

In Python, functions are first-class citizens, meaning they can be passed as arguments, returned from other functions, and assigned to variables. Decorators leverage this to modify or enhance the behavior of other functions without permanently altering their source code. Think of decorators as “wrappers” that add functionality (e.g., logging, timing, or authentication) to existing functions.

2.2 How Decorators Work: A Simple Example

Let’s start with a basic decorator that logs when a function is called.

Step 1: Define the Decorator

A decorator is a function that takes a function as input and returns a new function (the “wrapper”) that includes the enhanced behavior.

def log_function_call(func):
    def wrapper():
        print(f"Calling function: {func.__name__}")
        func()  # Call the original function
        print(f"Function {func.__name__} finished execution")
    return wrapper

Here, log_function_call is the decorator. It takes func (the function to decorate) and returns wrapper, which adds logging before and after func runs.

Step 2: Apply the Decorator

To use the decorator, pass the target function to the decorator and reassign the result to the original function name:

def greet():
    print("Hello, World!")

# Apply the decorator manually
greet = log_function_call(greet)

Now, when we call greet(), the wrapper executes:

greet()
# Output:
# Calling function: greet
# Hello, World!
# Function greet finished execution

2.3 Decorator Syntax: The @ Symbol

Python simplifies decorator application with the @ syntax. Placing @decorator_name above a function definition is equivalent to manually wrapping the function (as in Section 2.2).

Rewriting the previous example with @:

@log_function_call
def greet():
    print("Hello, World!")

greet()  # Same output as before

This is cleaner and more readable, especially when applying multiple decorators.

2.4 Advanced Decorators: Passing Arguments

What if you want a decorator to accept arguments (e.g., a custom log message)? This requires a decorator factory—a function that returns a decorator.

Example: A Repeat Decorator

Let’s create a decorator that runs a function n times:

def repeat(n):
    def decorator(func):
        def wrapper(*args, **kwargs):
            for _ in range(n):
                func(*args, **kwargs)  # Pass args/kwargs to the original function
        return wrapper
    return decorator

Here’s how to use it:

@repeat(n=3)  # Decorator factory: returns a decorator configured to run 3 times
def greet(name):
    print(f"Hello, {name}!")

greet("Alice")
# Output:
# Hello, Alice!
# Hello, Alice!
# Hello, Alice!
  • *args and **kwargs ensure the wrapper works with functions that take positional or keyword arguments.

2.5 Chaining Multiple Decorators

You can apply multiple decorators to a single function. The order of decorators matters: they execute from bottom to top.

Example: Logging + Timing

Let’s chain a logging decorator and a timing decorator (to measure execution time):

import time

def log_decorator(func):
    def wrapper(*args, **kwargs):
        print(f"Calling {func.__name__}...")
        result = func(*args, **kwargs)
        print(f"{func.__name__} returned: {result}")
        return result
    return wrapper

def time_decorator(func):
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = func(*args, **kwargs)
        end = time.perf_counter()
        print(f"{func.__name__} took {end - start:.4f} seconds")
        return result
    return wrapper

# Apply decorators: time_decorator runs first, then log_decorator
@log_decorator
@time_decorator
def slow_function(seconds):
    time.sleep(seconds)
    return "Done"

slow_function(0.5)
# Output:
# Calling slow_function...
# slow_function took 0.5002 seconds
# slow_function returned: Done

Key Note: Decorators are applied in reverse order of their @ symbols. Here, @time_decorator is applied first (innermost), then @log_decorator (outermost).

2.6 Class Decorators

Decorators can also be defined as classes. A class decorator implements the __call__ method, which acts as the wrapper.

Example: A Counter Decorator

Track how many times a function is called:

class CountCalls:
    def __init__(self, func):
        self.func = func
        self.call_count = 0  # Track calls

    def __call__(self, *args, **kwargs):
        self.call_count += 1
        print(f"{self.func.__name__} called {self.call_count} times")
        return self.func(*args, **kwargs)

@CountCalls
def add(a, b):
    return a + b

add(2, 3)  # Output: add called 1 times → returns 5
add(5, 5)  # Output: add called 2 times → returns 10
print(add.call_count)  # Output: 2

Class decorators are useful for stateful decorators (e.g., tracking counts or caching results).

2.7 Practical Use Cases for Decorators

  • Logging/Monitoring: Track function calls, inputs, or outputs (e.g., debugging).
  • Timing: Measure execution time for performance optimization.
  • Caching: Store results of expensive functions (e.g., functools.lru_cache).
  • Authentication: Restrict access to functions (e.g., in web frameworks like Flask/Django).
  • Validation: Check inputs before executing a function (e.g., ensuring arguments are numeric).

Generators: Efficient Iteration with Yield

3.1 What Are Generators?

Generators are specialized iterators that generate values on-the-fly (lazily) instead of storing them all in memory at once. They are defined using the yield keyword and are ideal for working with large datasets, infinite sequences, or streaming data.

3.2 The Yield Keyword: How It Works

A generator function uses yield to return a value and pause execution, resuming from where it left off when the next value is requested.

Example: A Simple Number Generator

def count_up_to(n):
    current = 1
    while current <= n:
        yield current  # Pause and return current; resume here next time
        current += 1

# Create a generator object
counter = count_up_to(3)

# Iterate over the generator
print(next(counter))  # Output: 1 (resumes, increments current to 2)
print(next(counter))  # Output: 2 (resumes, increments to 3)
print(next(counter))  # Output: 3 (resumes, loop ends)
print(next(counter))  # Raises StopIteration (no more values)

Generators are iterators, so you can loop over them directly:

for num in count_up_to(3):
    print(num)  # Output: 1, 2, 3

3.3 Generator Expressions: Compact Iterators

Generator expressions are a concise way to create generators, similar to list comprehensions but with parentheses ().

# List comprehension (creates a list in memory)
squares_list = [x**2 for x in range(5)]  # [0, 1, 4, 9, 16]

# Generator expression (creates a generator)
squares_gen = (x**2 for x in range(5))

print(next(squares_gen))  # 0
print(next(squares_gen))  # 1

Generator expressions are memory-efficient for large ranges:

# List: Consumes ~4GB for 100 million integers (on 64-bit Python)
large_list = [x for x in range(10**8)]

# Generator: Consumes ~0 memory (values generated on-demand)
large_gen = (x for x in range(10**8))

3.4 Benefits of Generators Over Lists

  1. Memory Efficiency: Generators do not store all values in memory—they generate one value at a time.
  2. Lazy Evaluation: Values are computed only when needed, which improves performance for large/ infinite sequences.
  3. Simplicity: Generators simplify code for iteration (e.g., no need to manage indices or temporary lists).

3.5 Advanced Generator Features: send(), throw(), and close()

Generators support bidirectional communication using send(), error handling with throw(), and explicit termination with close().

Example: Sending Values to a Generator

def echo():
    while True:
        received = yield  # Pause and wait for input
        if received is None:
            break
        print(f"Echo: {received}")

# Create and prime the generator (run until first yield)
printer = echo()
next(printer)  # Required to start the generator

# Send values to the generator
printer.send("Hello")  # Output: Echo: Hello
printer.send("Python")  # Output: Echo: Python
printer.send(None)      # Exit the loop
printer.close()         # Clean up

3.6 Generators as Coroutines

Before async/await (Python 3.5+), generators were used as coroutines for asynchronous programming. They could pause and resume, making them useful for I/O-bound tasks (e.g., network requests).

def fetch_data():
    while True:
        url = yield  # Wait for a URL to fetch
        print(f"Fetching {url}...")  # Simulate I/O (e.g., HTTP request)

# Usage
fetcher = fetch_data()
next(fetcher)
fetcher.send("https://api.example.com/data")  # Output: Fetching https://api.example.com/data...

3.7 Practical Use Cases for Generators

  • Processing Large Files: Read a 10GB log file line-by-line without loading it all into memory.
  • Infinite Sequences: Generate Fibonacci numbers, prime numbers, or real-time sensor data.
  • Pipelines: Chain generators to process data in stages (e.g., filter → transform → aggregate).
  • Streaming APIs: Consume data from streaming services (e.g., Twitter API, Kafka) in real time.

When to Use Decorators vs. Generators

  • Decorators modify function behavior (e.g., add logging, caching). Use them for cross-cutting concerns or enhancing existing functions.
  • Generators handle iteration (e.g., large datasets, infinite sequences). Use them for memory-efficient or lazy evaluation.

They often complement each other: for example, a decorator could time a generator function processing a large file!

Conclusion

Decorators and generators are powerful tools in Python’s arsenal. Decorators enable clean, reusable code by wrapping functions with additional behavior, while generators revolutionize iteration with lazy evaluation and memory efficiency. Mastering these techniques will help you write more Pythonic, performant, and maintainable code—whether you’re building web apps, data pipelines, or automation scripts.

References