Extracting Key-Value Pairs with Substring Matching in Python Dictionaries

Table of Contents#

Understanding the Problem
Basic Approach: Using a For Loop
Pythonic Solution: Dictionary Comprehension
Advanced Techniques
Performance Comparison
Real-World Examples
Best Practices
Conclusion
References

Understanding the Problem#

The challenge is straightforward: given a dictionary and a substring, we want to retrieve all key-value pairs where the key contains the specified substring.

Example Input:

data = {
    "username": "john_doe",
    "user_email": "[email protected]",
    "user_age": 30,
    "admin_name": "alice",
    "admin_email": "[email protected]",
    "created_at": "2023-01-01"
}
 
substring = "user"

Expected Output:

{
    "username": "john_doe",
    "user_email": "[email protected]",
    "user_age": 30
}

Basic Approach: Using a For Loop#

The most straightforward method uses a simple for loop to iterate through dictionary items:

def extract_pairs_loop(dictionary, substring):
    """
    Extract key-value pairs using a for loop
    
    Args:
        dictionary (dict): The dictionary to search
        substring (str): The substring to match in keys
    
    Returns:
        dict: Dictionary containing matching key-value pairs
    """
    result = {}
    for key, value in dictionary.items():
        if substring in key:
            result[key] = value
    return result
 
# Example usage
data = {
    "username": "john_doe",
    "user_email": "[email protected]",
    "user_age": 30,
    "admin_name": "alice"
}
 
matching_pairs = extract_pairs_loop(data, "user")
print(matching_pairs)
# Output: {'username': 'john_doe', 'user_email': '[email protected]', 'user_age': 30}

Pros:

Easy to understand and read
Simple to debug
Explicit control flow

Cons:

More verbose than other approaches
Slightly slower for large datasets

Pythonic Solution: Dictionary Comprehension#

Dictionary comprehensions offer a more concise and Pythonic way to achieve the same result:

def extract_pairs_comprehension(dictionary, substring):
    """
    Extract key-value pairs using dictionary comprehension
    
    Args:
        dictionary (dict): The dictionary to search
        substring (str): The substring to match in keys
    
    Returns:
        dict: Dictionary containing matching key-value pairs
    """
    return {key: value for key, value in dictionary.items() 
            if substring in key}
 
# Example usage
data = {
    "product_name": "Laptop",
    "product_price": 999.99,
    "product_category": "Electronics",
    "supplier_name": "TechCorp"
}
 
matching_pairs = extract_pairs_comprehension(data, "product")
print(matching_pairs)
# Output: {'product_name': 'Laptop', 'product_price': 999.99, 'product_category': 'Electronics'}

Pros:

Concise and readable
Generally faster than explicit loops
Considered more Pythonic

Cons:

Can become less readable with complex conditions
Limited debugging capabilities within the comprehension

Advanced Techniques#

Using filter() with Lambda#

For functional programming enthusiasts, the filter() function combined with dict() constructor provides an alternative approach:

def extract_pairs_filter(dictionary, substring):
    """
    Extract key-value pairs using filter() and lambda
    
    Args:
        dictionary (dict): The dictionary to search
        substring (str): The substring to match in keys
    
    Returns:
        dict: Dictionary containing matching key-value pairs
    """
    filtered_items = filter(lambda item: substring in item[0], 
                           dictionary.items())
    return dict(filtered_items)
 
# Example usage
data = {
    "config_host": "localhost",
    "config_port": 8080,
    "database_name": "mydb",
    "config_timeout": 30
}
 
matching_pairs = extract_pairs_filter(data, "config")
print(matching_pairs)
# Output: {'config_host': 'localhost', 'config_port': 8080, 'config_timeout': 30}

Case-Insensitive Matching#

In many real-world scenarios, you might need case-insensitive matching:

def extract_pairs_case_insensitive(dictionary, substring):
    """
    Extract key-value pairs with case-insensitive matching
    
    Args:
        dictionary (dict): The dictionary to search
        substring (str): The substring to match in keys (case-insensitive)
    
    Returns:
        dict: Dictionary containing matching key-value pairs
    """
    substring_lower = substring.lower()
    return {key: value for key, value in dictionary.items() 
            if substring_lower in key.lower()}
 
# Example usage
data = {
    "UserName": "john_doe",
    "user_email": "[email protected]",
    "USER_AGE": 30,
    "AdminName": "alice"
}
 
matching_pairs = extract_pairs_case_insensitive(data, "user")
print(matching_pairs)
# Output: {'UserName': 'john_doe', 'user_email': '[email protected]', 'USER_AGE': 30}

Multiple Substring Search#

Sometimes you need to match keys against multiple substrings:

def extract_pairs_multiple_substrings(dictionary, substrings):
    """
    Extract key-value pairs matching any of multiple substrings
    
    Args:
        dictionary (dict): The dictionary to search
        substrings (list): List of substrings to match in keys
    
    Returns:
        dict: Dictionary containing matching key-value pairs
    """
    return {key: value for key, value in dictionary.items() 
            if any(sub in key for sub in substrings)}
 
# Example usage
data = {
    "customer_name": "Alice",
    "customer_email": "[email protected]",
    "order_id": "12345",
    "order_date": "2023-01-01",
    "product_sku": "ABC123"
}
 
matching_pairs = extract_pairs_multiple_substrings(data, ["customer", "order"])
print(matching_pairs)
# Output: {'customer_name': 'Alice', 'customer_email': '[email protected]', 
#          'order_id': '12345', 'order_date': '2023-01-01'}

Performance Comparison#

Let's compare the performance of different approaches:

import timeit
 
# Create a large dictionary for testing
large_dict = {f"key_{i}_suffix_{j}": f"value_{i}_{j}" 
              for i in range(1000) for j in range(10)}
 
# Performance test
def performance_test():
    methods = {
        "Loop": "extract_pairs_loop(large_dict, 'suffix_5')",
        "Comprehension": "extract_pairs_comprehension(large_dict, 'suffix_5')",
        "Filter": "extract_pairs_filter(large_dict, 'suffix_5')"
    }
    
    for name, code in methods.items():
        time_taken = timeit.timeit(code, 
                                  setup="from __main__ import large_dict, "
                                        "extract_pairs_loop, "
                                        "extract_pairs_comprehension, "
                                        "extract_pairs_filter",
                                  number=100)
        print(f"{name}: {time_taken:.4f} seconds")
 
# Typical results (may vary by system):
# Loop: 0.3456 seconds
# Comprehension: 0.2987 seconds
# Filter: 0.3678 seconds

Performance Insights:

Dictionary comprehensions are generally the fastest
Simple loops are moderately efficient
filter() approach tends to be slightly slower due to function call overhead

Real-World Examples#

Example 1: Configuration Management#

def extract_config_sections(config_dict, section_prefix):
    """
    Extract configuration settings for a specific section
    
    Args:
        config_dict (dict): Full configuration dictionary
        section_prefix (str): Section prefix (e.g., 'database.', 'api.')
    
    Returns:
        dict: Filtered configuration for the specified section
    """
    return {key.replace(section_prefix, ""): value 
            for key, value in config_dict.items() 
            if key.startswith(section_prefix)}
 
# Usage example
config = {
    "database.host": "localhost",
    "database.port": 5432,
    "api.endpoint": "https://api.example.com",
    "api.timeout": 30,
    "logging.level": "INFO"
}
 
db_config = extract_config_sections(config, "database.")
print(db_config)
# Output: {'host': 'localhost', 'port': 5432}

Example 2: Data Filtering for Analytics#

def filter_metrics_data(metrics_dict, metric_patterns):
    """
    Filter metrics data based on multiple patterns
    
    Args:
        metrics_dict (dict): Dictionary containing various metrics
        metric_patterns (list): List of patterns to include
    
    Returns:
        dict: Filtered metrics dictionary
    """
    return {key: value for key, value in metrics_dict.items() 
            if any(pattern in key for pattern in metric_patterns)}
 
# Usage example
metrics = {
    "cpu_usage": 45.6,
    "memory_usage": 67.8,
    "disk_io_read": 1024,
    "disk_io_write": 2048,
    "network_rx": 512,
    "network_tx": 768
}
 
io_metrics = filter_metrics_data(metrics, ["disk", "network"])
print(io_metrics)
# Output: {'disk_io_read': 1024, 'disk_io_write': 2048, 
#          'network_rx': 512, 'network_tx': 768}

Best Practices#

1. Error Handling and Edge Cases#

def extract_pairs_safe(dictionary, substring):
    """
    Safe extraction with error handling
    
    Args:
        dictionary (dict): The dictionary to search
        substring (str): The substring to match in keys
    
    Returns:
        dict: Dictionary containing matching key-value pairs
    """
    if not isinstance(dictionary, dict):
        raise TypeError("First argument must be a dictionary")
    
    if not isinstance(substring, str):
        raise TypeError("Substring must be a string")
    
    # Handle empty dictionary or substring
    if not dictionary:
        return {}
    
    if not substring:
        return dictionary.copy()  # Return copy to avoid modifying original
    
    return {key: value for key, value in dictionary.items() 
            if substring in key}

2. Memory Efficiency for Large Dictionaries#

For very large dictionaries, consider using generators:

def extract_pairs_generator(dictionary, substring):
    """
    Generator-based approach for memory efficiency
    
    Args:
        dictionary (dict): The dictionary to search
        substring (str): The substring to match in keys
    
    Yields:
        tuple: Key-value pairs that match the substring
    """
    for key, value in dictionary.items():
        if substring in key:
            yield (key, value)
 
# Usage
large_data = {f"key_{i}": f"value_{i}" for i in range(1000000)}
 
# Process matches without storing all in memory
for key, value in extract_pairs_generator(large_data, "key_123"):
    print(f"{key}: {value}")

3. Using Regular Expressions for Complex Patterns#

import re
 
def extract_pairs_regex(dictionary, pattern):
    """
    Extract key-value pairs using regular expressions
    
    Args:
        dictionary (dict): The dictionary to search
        pattern (str): Regular expression pattern
    
    Returns:
        dict: Dictionary containing matching key-value pairs
    """
    regex = re.compile(pattern)
    return {key: value for key, value in dictionary.items() 
            if regex.search(key)}
 
# Example: Find keys containing digits
data = {"item1": "A", "item2": "B", "product": "C", "test": "D"}
result = extract_pairs_regex(data, r"\d")
print(result)  # Output: {'item1': 'A', 'item2': 'B'}

Conclusion#

Extracting key-value pairs based on substring matching is a common task in Python programming with applications ranging from configuration management to data analysis. The choice of method depends on your specific needs:

For simplicity and readability: Use dictionary comprehensions
For explicit control flow: Use traditional loops
For functional programming style: Use filter() with lambda
For complex pattern matching: Use regular expressions
For memory efficiency with large datasets: Use generators

Remember to consider factors like performance requirements, code readability, and error handling when choosing your approach. The dictionary comprehension method generally offers the best balance of performance, readability, and conciseness for most use cases.