py4u blog

Python urllib2 urlopen Response: How to Retrieve Expected JSON Token Output like {"token":"mYWmzpunvasAT795niiR"}

In modern web development and API interactions, retrieving authentication tokens (e.g., JWT, session tokens) is a common task. These tokens often come packaged in JSON responses, such as {"token":"mYWmzpunvasAT795niiR"}. If you’re working with Python 2.x, the urllib2 library is a built-in tool for making HTTP requests and handling responses.

This blog will guide you through the entire process of using urllib2.urlopen to send a request, handle the response, parse the JSON output, and extract the desired token. We’ll cover prerequisites, core concepts, error handling, and provide a complete working example. By the end, you’ll be able to reliably retrieve tokens from JSON responses using Python 2.x.

2026-01

Table of Contents#

Prerequisites#

Before diving in, ensure you have:

  • Python 2.x installed (urllib2 is a built-in library in Python 2; for Python 3, see the Python 3 Note).
  • Basic knowledge of Python syntax and HTTP requests (e.g., GET/POST methods).
  • Familiarity with JSON (JavaScript Object Notation), as we’ll parse JSON responses.
  • Access to an API endpoint that returns a JSON token (we’ll use a placeholder endpoint for examples).

Understanding urllib2 and urlopen#

urllib2 is a Python 2 library for opening URLs (e.g., HTTP, HTTPS) and interacting with web resources. Its core function, urllib2.urlopen(), sends a request to a URL and returns a response object. This object contains the server’s response data, status code, and metadata.

Key features of urlopen():

  • Supports HTTP methods like GET (default) and POST (with urllib2.Request).
  • Returns a response object with methods to access content (read()), status code (getcode()), and headers (info()).

Step 1: Making a Basic HTTP Request#

To retrieve a JSON token, start by sending an HTTP request to the target API endpoint. For example, let’s assume the endpoint https://api.example.com/auth/token returns a JSON response with a token field.

Code Example: Basic Request#

import urllib2
 
# Define the API endpoint URL
url = "https://api.example.com/auth/token"
 
# Send a GET request and get the response object
response = urllib2.urlopen(url)
 
# Print the response object (for debugging)
print("Response object:", response)

Output:

Response object: <addinfourl at 1402345678901234 whose fp = <socket._fileobject object at 0x7f8a1b2c3d4e>>

This confirms the request was sent successfully, but we need to extract the actual JSON data from the response.

Step 2: Handling the Response Object#

The response object returned by urlopen() has several useful methods:

  • read(): Returns the response content as a string (bytes in Python 3, but in Python 2, it’s a str).
  • getcode(): Returns the HTTP status code (e.g., 200 for success, 404 for “Not Found”).
  • info(): Returns response headers (e.g., Content-Type).

Extracting Response Content#

Use response.read() to get the JSON string:

# Read the response content as a string
response_data = response.read()
 
# Print the raw response data (JSON string)
print("Raw JSON response:", response_data)

Example Output:

Raw JSON response: {"token":"mYWmzpunvasAT795niiR", "expires_in":3600}

Now we have the JSON data as a string, which we can parse into a Python dictionary.

Step 3: Parsing the JSON Response#

To work with the JSON data, use Python’s built-in json module. The json.loads() function converts a JSON string into a Python dictionary, making it easy to access nested fields like token.

Code Example: Parse JSON#

import json
 
# Parse the JSON string into a Python dictionary
try:
    parsed_data = json.loads(response_data)
except json.JSONDecodeError as e:
    print("Failed to parse JSON:", e)
else:
    print("Parsed data (Python dict):", parsed_data)

Output:

Parsed data (Python dict): {'token': 'mYWmzpunvasAT795niiR', 'expires_in': 3600}

Now parsed_data is a dictionary, so we can directly access the token field.

Step 4: Extracting the Token#

With the parsed dictionary, extract the token using the token key:

# Extract the token from the parsed dictionary
token = parsed_data.get("token")  # Using .get() to avoid KeyError if "token" is missing
 
if token:
    print("Extracted token:", token)
else:
    print("Error: 'token' field not found in response.")

Output:

Extracted token: mYWmzpunvasAT795niiR

Using .get("token") (instead of parsed_data["token"]) is safer: it returns None if the token key is missing, avoiding a KeyError.

Error Handling: Common Pitfalls and Solutions#

APIs can fail for many reasons (e.g., network issues, invalid URLs, server errors, or malformed JSON). Always handle errors to make your code robust.

Common Errors and Fixes#

1. HTTP Errors (e.g., 404, 500)#

urllib2 raises urllib2.HTTPError for HTTP status codes like 404 (Not Found) or 500 (Server Error).

2. Network/URL Errors#

urllib2.URLError is raised for issues like no internet, invalid domain, or connection timeout.

3. Invalid JSON#

json.JSONDecodeError occurs if the response isn’t valid JSON (e.g., HTML error pages instead of JSON).

4. Missing token Key#

Even if the response is valid JSON, the token field might be missing, causing a KeyError (or None with .get()).

Error Handling Code Example#

import urllib2
import json
 
url = "https://api.example.com/auth/token"
 
try:
    # Step 1: Send request
    response = urllib2.urlopen(url)
    
    # Step 2: Check HTTP status code (optional but useful)
    status_code = response.getcode()
    if status_code != 200:
        raise Exception(f"HTTP Error: {status_code}")
    
    # Step 3: Read and parse JSON
    response_data = response.read()
    parsed_data = json.loads(response_data)
    
    # Step 4: Extract token
    token = parsed_data.get("token")
    if not token:
        raise KeyError("'token' field missing in response")
 
except urllib2.HTTPError as e:
    print(f"HTTP Error: {e.code} - {e.reason}")
except urllib2.URLError as e:
    print(f"URL Error: {e.reason} (Check network or URL)")
except json.JSONDecodeError as e:
    print(f"JSON Parse Error: {e} (Response: {response_data[:50]}...)")  # Print first 50 chars
except KeyError as e:
    print(f"Token Error: {e}")
except Exception as e:
    print(f"Unexpected Error: {e}")
else:
    print("Success! Token:", token)

Example Output (Success):

Success! Token: mYWmzpunvasAT795niiR

Example Output (404 Error):

HTTP Error: 404 - Not Found

Complete Example: Retrieve Token with Error Handling#

Combine all steps into a reusable function:

import urllib2
import json
 
def get_auth_token(url):
    """Retrieve a token from a JSON API endpoint with error handling."""
    try:
        # Send request
        response = urllib2.urlopen(url)
        
        # Check status code (200 = OK)
        if response.getcode() != 200:
            return None, f"HTTP Error: {response.getcode()}"
        
        # Read and parse JSON
        response_data = response.read()
        parsed_data = json.loads(response_data)
        
        # Extract token
        token = parsed_data.get("token")
        if not token:
            return None, "'token' field missing in response"
        
        return token, None  # (token, error)
    
    except urllib2.HTTPError as e:
        return None, f"HTTP Error: {e.code} - {e.reason}"
    except urllib2.URLError as e:
        return None, f"URL Error: {e.reason} (Check network/URL)"
    except json.JSONDecodeError as e:
        return None, f"Invalid JSON: {e}"
    except Exception as e:
        return None, f"Unexpected error: {e}"
 
# Usage
url = "https://api.example.com/auth/token"
token, error = get_auth_token(url)
 
if error:
    print("Failed to retrieve token:", error)
else:
    print("Token retrieved successfully:", token)

Note for Python 3 Users#

urllib2 is not available in Python 3. Instead, Python 3 uses urllib.request (for requests) and urllib.error (for errors). Here’s the Python 3 equivalent of the get_auth_token function:

import urllib.request
import urllib.error
import json
 
def get_auth_token_python3(url):
    try:
        with urllib.request.urlopen(url) as response:
            # Check status code
            if response.getcode() != 200:
                return None, f"HTTP Error: {response.getcode()}"
            
            # Read and parse JSON
            response_data = response.read().decode("utf-8")  # Decode bytes to string
            parsed_data = json.loads(response_data)
            
            # Extract token
            token = parsed_data.get("token")
            if not token:
                return None, "'token' field missing"
            
            return token, None
    
    except urllib.error.HTTPError as e:
        return None, f"HTTP Error: {e.code} - {e.reason}"
    except urllib.error.URLError as e:
        return None, f"URL Error: {e.reason}"
    except json.JSONDecodeError as e:
        return None, f"Invalid JSON: {e}"
    except Exception as e:
        return None, f"Unexpected error: {e}"
 
# Usage
url = "https://api.example.com/auth/token"
token, error = get_auth_token_python3(url)
print(token if not error else error)

References#

By following these steps, you can reliably retrieve and extract JSON tokens using urllib2 in Python 2 (or urllib.request in Python 3). Always include error handling to make your code resilient to real-world API failures!