Multithreading in Python | Set 2 (Synchronization)

Table of Contents#

Recap: Multithreading Basics & the GIL
The Core Problem: Race Conditions
- What is a Race Condition?
- A Practical Example of a Race Condition
Synchronization Primitives in Python
Best Practices for Synchronization
Common Pitfalls to Avoid
Real-World Example: Thread-Safe Web Scraper
Conclusion
References

1. Recap: Multithreading Basics & the GIL#

Before we dive into synchronization, let’s refresh key concepts from Set 1:

Threads: Lightweight execution units within a process. Threads share the process’s memory space (e.g., variables, files).
GIL (Global Interpreter Lock): A Python-specific lock that allows only one thread to execute Python bytecode at a time. This limits parallelism for CPU-bound tasks (e.g., math calculations) but has no impact on I/O-bound tasks (e.g., network calls, file I/O), where threads spend most of their time waiting.

Critical Note: The GIL does not eliminate race conditions. Even with the GIL, threads can still interleave when accessing shared data (e.g., during I/O waits or context switches). Synchronization is still required to protect shared state.

2. The Core Problem: Race Conditions#

2.1 What is a Race Condition?#

A race condition occurs when the correctness of a program depends on the relative timing of thread execution. For example:

Two threads increment a shared counter.
One thread reads a variable while another writes to it.

The outcome is unpredictable because threads can be paused (context-switched) at any time—even in the middle of a "simple" operation like counter += 1.

2.2 A Practical Example of a Race Condition#

Let’s demonstrate a race condition with a shared counter. We’ll create two threads that each increment a counter 100,000 times:

import threading
 
counter = 0
 
def increment():
    global counter
    for _ in range(100_000):
        counter += 1  # Not atomic!
 
# Create two threads
threads = [threading.Thread(target=increment) for _ in range(2)]
 
# Start and wait for threads
for t in threads:
    t.start()
for t in threads:
    t.join()
 
print(f"Final counter value: {counter}")  # Expected: 200,000. Actual: ~150,000–200,000.

Why This Fails#

The operation counter += 1 is not atomic (indivisible). It breaks down into three steps:

Load the current value of counter into a register.
Add 1 to the register.
Store the new value back to counter.

If a context switch occurs between steps 1 and 3, two threads can overwrite each other’s changes. For example:

Thread A loads counter = 5.
Thread B loads counter = 5.
Thread A adds 1 (register = 6) and stores counter = 6.
Thread B adds 1 (register = 6) and stores counter = 6.

Instead of incrementing twice, the counter only increases by 1—race condition!

3. Synchronization Primitives in Python#

Python’s threading module provides several synchronization tools to eliminate race conditions. We’ll cover each with use cases, examples, and best practices.

3.1 Locks (`threading.Lock`)#

A Lock (or mutex, short for mutual exclusion) is the simplest synchronization primitive. It enforces that only one thread can acquire the lock at a time—preventing concurrent access to shared resources.

3.1.1 How Locks Work#

lock.acquire(): Blocks until the lock is free, then acquires it.
lock.release(): Releases the lock (must be called after acquire()).
Rule: A lock must be released exactly once for each acquire. Forgetting to release causes deadlocks (see Section 5.2).

3.1.2 Using Locks with Context Managers#

Python’s with statement automatically calls acquire() and release(), even if an exception occurs. This is the safe and recommended way to use locks:

from threading import Lock
 
lock = Lock()
 
with lock:
    # Critical section: access shared resources here
    counter += 1

3.1.3 Fixing the Race Condition with a Lock#

Let’s modify our counter example to use a lock:

import threading
 
counter = 0
lock = threading.Lock()
 
def increment():
    global counter
    for _ in range(100_000):
        with lock:
            counter += 1  # Critical section: protected by lock
 
threads = [threading.Thread(target=increment) for _ in range(2)]
 
for t in threads:
    t.start()
for t in threads:
    t.join()
 
print(f"Final counter value: {counter}")  # Output: 200000 (correct!)

The lock ensures only one thread can execute counter += 1 at a time—eliminating the race condition.

3.2 Reentrant Locks (`threading.RLock`)#

A reentrant lock (RLock) is a lock that can be acquired multiple times by the same thread. It keeps track of the number of acquire() calls and requires an equal number of release() calls to unlock.

3.2.1 When to Use RLock#

Use RLock for nested functions where the same thread needs to acquire the lock multiple times. For example:

A function that calls another function, both of which need to access the same shared resource.

Without RLock, this would cause a deadlock (the thread blocks on its own lock).

3.2.2 RLock Example#

from threading import RLock
 
rlock = RLock()
 
def outer_function():
    with rlock:
        print("Outer function acquired lock")
        inner_function()
 
def inner_function():
    with rlock:
        print("Inner function acquired lock (reentrant!)")
 
# Create a thread to run outer_function
thread = threading.Thread(target=outer_function)
thread.start()
thread.join()

Output:

Outer function acquired lock
Inner function acquired lock (reentrant!)

The same thread acquires the RLock twice—no deadlock!

3.3 Semaphores (`threading.Semaphore`)#

A semaphore is a synchronization primitive that controls access to a resource pool. Unlike locks (which allow 1 thread), semaphores allow up to n threads to access a resource concurrently.

3.3.1 Semaphores vs. Locks#

Feature	Lock (Mutex)	Semaphore
Maximum concurrent threads	1	`n`
Use case	Exclusive access	Resource pooling

3.3.2 Semaphore Use Case: Resource Pooling#

Semaphores are ideal for limiting concurrent access to resources like:

Network connections (e.g., 5 concurrent API calls)
File handles
Database connections

3.3.3 Semaphore Example#

Let’s limit 3 concurrent threads to simulate a resource pool:

import threading
import time
 
semaphore = threading.Semaphore(3)  # Allow 3 concurrent threads
 
def worker(thread_id):
    with semaphore:
        print(f"Thread {thread_id} is working")
        time.sleep(1)  # Simulate work
        print(f"Thread {thread_id} finished")
 
# Create 5 threads
threads = [threading.Thread(target=worker, args=(i,)) for i in range(5)]
 
for t in threads:
    t.start()
for t in threads:
    t.join()

Output:

Thread 0 is working
Thread 1 is working
Thread 2 is working
Thread 0 finished
Thread 3 is working
Thread 1 finished
Thread 4 is working
Thread 2 finished
Thread 3 finished
Thread 4 finished

Only 3 threads run at a time—semaphores work!

3.4 Events (`threading.Event`)#

An event is a signaling mechanism that allows one thread to notify others when a condition is met. Events are useful for:

Coordinating startup/shutdown of worker threads
Signaling status changes (e.g., "data is ready")

3.4.1 Event Methods & Workflow#

event = Event(): Creates an event (initially unset).
event.wait(): Blocks until the event is set.
event.set(): Sets the event (wakes up all waiting threads).
event.clear(): Resets the event to unset.
event.is_set(): Returns True if the event is set.

3.4.2 Event Example: Coordinating Worker Threads#

Let’s have 3 worker threads wait for a "start" signal from the main thread:

import threading
import time
 
event = threading.Event()
 
def worker(worker_id):
    print(f"Worker {worker_id} waiting for start signal...")
    event.wait()  # Block until event is set
    print(f"Worker {worker_id} started working!")
 
# Create 3 workers
workers = [threading.Thread(target=worker, args=(i,)) for i in range(3)]
 
for w in workers:
    w.start()
 
# Main thread waits 2 seconds, then signals workers
time.sleep(2)
print("Main thread sending start signal...")
event.set()
 
for w in workers:
    w.join()

Output:

Worker 0 waiting for start signal...
Worker 1 waiting for start signal...
Worker 2 waiting for start signal...
Main thread sending start signal...
Worker 0 started working!
Worker 1 started working!
Worker 2 started working!

3.5 Conditions (`threading.Condition`)#

A condition variable (Condition) is a more flexible tool for thread coordination. It combines a lock (for mutual exclusion) with signaling (like events) to wait for specific conditions (e.g., "queue is not empty").

3.5.1 Condition Variables Explained#

Conditions are used for the producer-consumer problem:

Producers: Add items to a queue.
Consumers: Remove items from the queue.
Condition: Consumers wait until the queue is not empty; producers wait until the queue is not full.

Key methods:

condition.acquire() / condition.release(): Manages the underlying lock.
condition.wait(): Releases the lock and blocks until notify() is called. Reacquires the lock before returning.
condition.notify(): Wakes up one waiting thread.
condition.notify_all(): Wakes up all waiting threads.

3.5.2 Condition Example: Producer-Consumer#

Let’s implement a thread-safe queue with producers and consumers:

import threading
import time
import random
 
queue = []
MAX_QUEUE_SIZE = 5
condition = threading.Condition()
 
def producer():
    for _ in range(10):
        time.sleep(random.randint(1, 3))  # Simulate production time
        with condition:
            # Wait until queue has space
            while len(queue) == MAX_QUEUE_SIZE:
                print("Queue is full—producer waiting...")
                condition.wait()
            # Add item to queue
            item = random.randint(1, 100)
            queue.append(item)
            print(f"Produced {item} | Queue: {queue}")
            # Notify consumers that queue is not empty
            condition.notify()
 
def consumer():
    for _ in range(10):
        time.sleep(random.randint(1, 3))  # Simulate consumption time
        with condition:
            # Wait until queue has items
            while len(queue) == 0:
                print("Queue is empty—consumer waiting...")
                condition.wait()
            # Remove item from queue
            item = queue.pop(0)
            print(f"Consumed {item} | Queue: {queue}")
            # Notify producers that queue has space
            condition.notify()
 
# Start producer and consumer threads
producer_thread = threading.Thread(target=producer)
consumer_thread = threading.Thread(target=consumer)
 
producer_thread.start()
consumer_thread.start()
 
producer_thread.join()
consumer_thread.join()

Output:

Produced 42 | Queue: [42]
Consumed 42 | Queue: []
Produced 17 | Queue: [17]
Produced 89 | Queue: [17, 89]
Consumed 17 | Queue: [89]
...

The condition variable ensures:

Producers don’t add items to a full queue.
Consumers don’t remove items from an empty queue.
Threads are notified only when the condition changes.

3.6 Barriers (`threading.Barrier`)#

A barrier blocks threads until a fixed number of threads (the parties) have reached the barrier. This is useful for phase synchronization—ensuring all threads complete one phase before starting the next.

3.6.1 Barrier Use Case: Phase Synchronization#

Barriers are ideal for:

Parallel algorithms (e.g., matrix multiplication) where threads must sync after each step.
Testing (e.g., ensuring all test threads are ready before running a test).

3.6.2 Barrier Example#

Let’s have 3 threads complete Phase 1 before starting Phase 2:

import threading
import time
import random
 
barrier = threading.Barrier(3)  # Wait for 3 threads
 
def worker(worker_id):
    # Phase 1: Do work
    print(f"Worker {worker_id} starting Phase 1")
    time.sleep(random.randint(1, 3))  # Simulate work
    print(f"Worker {worker_id} finished Phase 1")
    
    # Wait for all workers to finish Phase 1
    barrier.wait()
    
    # Phase 2: Do more work
    print(f"Worker {worker_id} starting Phase 2")
    time.sleep(random.randint(1, 3))
    print(f"Worker {worker_id} finished Phase 2")
 
# Create 3 workers
threads = [threading.Thread(target=worker, args=(i,)) for i in range(3)]
 
for t in threads:
    t.start()
for t in threads:
    t.join()

Output:

Worker 0 starting Phase 1
Worker 1 starting Phase 1
Worker 2 starting Phase 1
Worker 0 finished Phase 1
Worker 1 finished Phase 1
Worker 2 finished Phase 1
Worker 0 starting Phase 2
Worker 1 starting Phase 2
Worker 2 starting Phase 2
...

All threads wait at the barrier until Phase 1 is complete—no thread starts Phase 2 early!

4. Best Practices for Synchronization#

Follow these rules to write safe, efficient multithreaded code:

4.1 Minimize Lock Duration#

Only lock the critical section—the code that accesses shared resources. Long locks reduce parallelism:

❌ Bad: Locking the entire loop

with lock:
    for _ in range(100_000):
        counter += 1  # Too much work inside lock!

✅ Good: Lock only the critical operation

for _ in range(100_000):
    with lock:
        counter += 1  # Minimal work inside lock

4.2 Avoid Deadlocks#

A deadlock occurs when two or more threads block forever, waiting for each other’s locks. To prevent deadlocks:

Order Locks: Acquire locks in a fixed order (e.g., Lock A before Lock B).

Use Timeouts: Add a timeout to acquire() to detect deadlocks:

if lock.acquire(timeout=5):
    try:
        # Critical section
    finally:
        lock.release()
else:
    print("Lock acquisition timed out—deadlock detected!")

Avoid Nested Locks: Use RLock if nested locks are unavoidable.

4.3 Use Context Managers#

Always use with for locks, semaphores, and conditions. This ensures resources are released even if an exception occurs.

4.4 Prefer Higher-Level Primitives#

Use conditions, semaphores, or barriers instead of raw locks when possible. Higher-level primitives are safer and more expressive.

4.5 Be GIL-Aware#

CPU-Bound Tasks: Multithreading won’t speed up CPU-bound code (due to the GIL). Use multiprocessing instead (separate processes with their own GIL).
I/O-Bound Tasks: Multithreading works well—synchronization is still required for shared data.

5. Common Pitfalls to Avoid#

Forgetting to Release Locks: Fixed by using with.
Nested Locks Without RLock: Causes deadlocks.
Ignoring the GIL: Expecting CPU-bound threads to run in parallel (they won’t).
Over-Synchronization: Locking too much code—slows down the program.
Using Events for Complex Conditions: Use Condition instead (events can’t check state).

6. Real-World Example: Thread-Safe Web Scraper#

Let’s build a thread-safe web scraper that:

Limits 3 concurrent API calls (using a semaphore).
Tracks successful scrapes (using a lock).

import requests
import threading
import time
 
# Configuration
URLS = ["https://example.com"] * 10  # 10 identical URLs
MAX_CONCURRENT_REQUESTS = 3  # Avoid rate limits
success_count = 0
lock = threading.Lock()
semaphore = threading.Semaphore(MAX_CONCURRENT_REQUESTS)
 
def scrape(url):
    global success_count
    try:
        with semaphore:
            print(f"Scraping {url}")
            response = requests.get(url, timeout=5)
            response.raise_for_status()  # Raise error for HTTP status codes >=400
            # Update success count (thread-safe)
            with lock:
                success_count += 1
    except Exception as e:
        print(f"Failed to scrape {url}: {e}")
 
# Create and start threads
threads = [threading.Thread(target=scrape, args=(url,)) for url in URLS]
 
start_time = time.time()
 
for t in threads:
    t.start()
for t in threads:
    t.join()
 
end_time = time.time()
 
# Print results
print(f"\nScraped {success_count}/{len(URLS)} URLs successfully.")
print(f"Total time: {end_time - start_time:.2f} seconds.")

Output:

Scraping https://example.com
Scraping https://example.com
Scraping https://example.com
Scraping https://example.com
...
Scraped 10/10 URLs successfully.
Total time: 3.12 seconds.

This scraper is:

Thread-safe: The lock protects success_count.
Rate-limited: The semaphore prevents hitting API rate limits.
Efficient: I/O-bound tasks (network calls) benefit from multithreading.

7. Conclusion#

Synchronization is the backbone of safe multithreading in Python. By understanding:

Race Conditions: The problem synchronization solves.
Primitives: Locks (exclusive access), semaphores (resource pooling), conditions (producer-consumer), barriers (phase sync).
Best Practices: Minimize lock duration, avoid deadlocks, use context managers.

You can write robust, efficient multithreaded code for I/O-bound tasks. For CPU-bound tasks, remember to use multiprocessing instead of threads.

8. References#

Python Threading Documentation: https://docs.python.org/3/library/threading.html
Fluent Python (Chapter 17): A deep dive into concurrency in Python.
Real Python: Multithreading in Python: https://realpython.com/intro-to-python-threading/
GIL Explained: https://wiki.python.org/moin/GlobalInterpreterLock

Let me know if you have any questions—happy coding! 🚀

Table of Contents#

1. Recap: Multithreading Basics & the GIL#

2. The Core Problem: Race Conditions#

2.1 What is a Race Condition?#

2.2 A Practical Example of a Race Condition#

Why This Fails#

3. Synchronization Primitives in Python#

3.1 Locks (threading.Lock)#

3.1.1 How Locks Work#

3.1.2 Using Locks with Context Managers#

3.1.3 Fixing the Race Condition with a Lock#

3.2 Reentrant Locks (threading.RLock)#

3.2.1 When to Use RLock#

3.2.2 RLock Example#

3.3 Semaphores (threading.Semaphore)#

3.3.1 Semaphores vs. Locks#

3.3.2 Semaphore Use Case: Resource Pooling#

3.3.3 Semaphore Example#

3.4 Events (threading.Event)#

3.4.1 Event Methods & Workflow#

3.4.2 Event Example: Coordinating Worker Threads#

3.5 Conditions (threading.Condition)#

3.5.1 Condition Variables Explained#

3.5.2 Condition Example: Producer-Consumer#

3.6 Barriers (threading.Barrier)#

3.6.1 Barrier Use Case: Phase Synchronization#

3.6.2 Barrier Example#

4. Best Practices for Synchronization#

4.1 Minimize Lock Duration#

4.2 Avoid Deadlocks#

4.3 Use Context Managers#

4.4 Prefer Higher-Level Primitives#

4.5 Be GIL-Aware#

5. Common Pitfalls to Avoid#

6. Real-World Example: Thread-Safe Web Scraper#

7. Conclusion#

8. References#

3.1 Locks (`threading.Lock`)#

3.2 Reentrant Locks (`threading.RLock`)#

3.3 Semaphores (`threading.Semaphore`)#

3.4 Events (`threading.Event`)#

3.5 Conditions (`threading.Condition`)#

3.6 Barriers (`threading.Barrier`)#