py4u blog

Newspaper: Article Scraping & Curation with Python

In the age of information overload, newspaper article scraping and curation empower users to gather, organize, and distill news content from diverse sources. Python, with libraries like newspaper3k, simplifies this process—enabling automated extraction of text, metadata, and summaries, while curation tools help structure and enrich the data for analysis, aggregation, or reporting.

This blog explores:

  • Scraping news articles (single/bulk) with newspaper3k.
  • Best practices for ethical, efficient scraping.
  • Content curation (summarization, categorization, and formatting).
  • Solving common challenges (paywalls, dynamic content, anti-scraping measures).
2026-10

Table of Contents#

  1. Setting Up the Environment
  2. Scraping a Single Article
  3. Bulk Scraping (Multiple Articles)
  4. Content Curation: Organizing Scraped Data
  5. Best Practices for Ethical Scraping
  6. Common Challenges & Solutions
  7. Example Project: Build a News Aggregator
  8. References

1. Setting Up the Environment#

Installing newspaper3k#

newspaper3k is a Python library tailored for news extraction. Install it via pip:

pip install newspaper3k

Optional: NLP Dependencies#

For language processing (summarization, keyword extraction), newspaper3k relies on NLTK’s punkt tokenizer. Download it (once) with:

import nltk
nltk.download('punkt')

2. Scraping a Single Article#

Step 1: Import & Initialize#

The Article class handles downloading, parsing, and NLP:

from newspaper import Article
 
# Target article URL
url = "https://www.nytimes.com/2023/01/01/technology/ai-breakthrough.html"
article = Article(url)

Step 2: Download & Parse#

Fetch the article’s HTML, then extract text/metadata:

# Download the article
article.download()
 
# Parse content (text, title, authors, publish date, images, etc.)
article.parse()

Step 3: Extract Metadata & NLP#

Use article.nlp() for summarization and keyword extraction:

article.nlp()  # Generates summary, keywords, etc.
 
# Access extracted data
print("Title:", article.title)
print("Authors:", article.authors)
print("Publish Date:", article.publish_date)
print("Top Image:", article.top_image)
print("Text (first 100 chars):", article.text[:100] + "...")
print("Summary:", article.summary)
print("Keywords:", article.keywords)

3. Bulk Scraping (Multiple Articles)#

To scrape multiple articles, loop over URLs, handle errors, and store data (e.g., JSON/CSV).

Example: Scrape 5 Tech News Articles#

import time
import json
from newspaper import Article
 
# List of tech news URLs
tech_urls = [
    "https://techcrunch.com/2023/01/01/ai-breakthrough/",
    "https://www.theverge.com/2023/01/02/new-laptop-release/",
    "https://www.wired.com/story/quantum-computing-update/",
    "https://arstechnica.com/2023/01/spacex-starship-test-flight/",
    "https://www.engadget.com/new-smartphone-features-2023.html",
]
 
scraped_articles = []
 
for url in tech_urls:
    try:
        article = Article(url)
        article.download()
        article.parse()
        article.nlp()
        
        # Extract relevant data
        data = {
            "title": article.title,
            "summary": article.summary,
            "publish_date": str(article.publish_date),  # Convert to string
            "url": url,
            "keywords": article.keywords,
            "top_image": article.top_image,
        }
        scraped_articles.append(data)
        
        # Rate limit: Wait 2-5 seconds between requests to avoid overloading servers
        time.sleep(3)  
    except Exception as e:
        print(f"Error scraping {url}: {e}")
 
# Save to JSON
with open("tech_news.json", "w") as f:
    json.dump(scraped_articles, f, indent=2)

4. Content Curation: Organizing Scraped Data#

Curation transforms raw scraped data into actionable content (e.g., newsletters, reports, dashboards).

Example: Categorize by Keywords & Add Sentiment#

Use spaCy for entity recognition (people, organizations, locations) and nltk for sentiment analysis:

import spacy
from nltk.sentiment import SentimentIntensityAnalyzer
import nltk
 
# Initialize NLP tools
nlp = spacy.load("en_core_web_sm")  # Install: python -m spacy download en_core_web_sm
nltk.download('vader_lexicon')
sia = SentimentIntensityAnalyzer()
 
curated_articles = []
for article in scraped_articles:
    # Entity Recognition with spaCy
    doc = nlp(article["summary"])
    entities = {
        "people": [ent.text for ent in doc.ents if ent.label_ == "PERSON"],
        "organizations": [ent.text for ent in doc.ents if ent.label_ == "ORG"],
        "locations": [ent.text for ent in doc.ents if ent.label_ == "GPE"],
    }
    
    # Sentiment Analysis with NLTK
    sentiment = sia.polarity_scores(article["summary"])
    sentiment_label = "positive" if sentiment["compound"] > 0.1 else "negative" if sentiment["compound"] < -0.1 else "neutral"
    
    # Update article data
    curated = {**article, "entities": entities, "sentiment": sentiment_label}
    curated_articles.append(curated)
 
# Save curated data
with open("curated_tech_news.json", "w") as f:
    json.dump(curated_articles, f, indent=2)

5. Best Practices for Ethical Scraping#

To avoid legal/ethical issues and ensure reliability:

1. Respect robots.txt#

Check if a site permits scraping via its robots.txt:

from urllib.robotparser import RobotFileParser
 
def is_allowed(url):
    domain = url.split("/")[2]
    rp = RobotFileParser()
    rp.set_url(f"https://{domain}/robots.txt")
    try:
        rp.read()
        return rp.can_fetch("*", url)
    except:
        return True  # Assume allowed if robots.txt is unavailable (use cautiously)

2. Rate Limiting#

Add delays (time.sleep()) between requests to avoid overloading servers (e.g., 2–5 seconds per request).

3. User-Agent Rotation#

Use a legitimate User-Agent (mimic a browser) to avoid being blocked:

article = Article(url, user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36")

4. Error Handling#

Wrap scraping logic in try-except to handle network errors, 404s, or malformed HTML.

5. Ethical Boundaries#

  • Avoid scraping paywalled/private content (unless permitted).
  • Use scraped data for non-commercial, fair use (e.g., research, personal aggregation).

6. Common Challenges & Solutions#

1. Paywalls & Restricted Content#

  • Solution: Scrape only free/preview content. For metered paywalls (e.g., 5 free articles/month), respect the limit.

2. Dynamic Content (JavaScript-Rendered Pages)#

newspaper3k cannot parse JS-generated content. Use Selenium or Playwright to render pages:

from selenium import webdriver
from newspaper import Article
import time
 
# Initialize Selenium (Chrome)
driver = webdriver.Chrome()
url = "https://example.com/js-based-article"
driver.get(url)
time.sleep(3)  # Wait for JS to load
html = driver.page_source
driver.quit()
 
# Parse the rendered HTML with newspaper3k
article = Article(url)
article.set_html(html)
article.parse()
article.nlp()

3. Inconsistent HTML Structures#

If newspaper3k fails to extract data, fall back to BeautifulSoup for manual extraction:

from bs4 import BeautifulSoup
 
# After downloading the article
soup = BeautifulSoup(article.html, "html.parser")
custom_text = soup.find("div", class_="article-body").get_text()

7. Example Project: Build a News Aggregator#

Let’s create a web-based news aggregator that scrapes tech articles, curates them, and displays a formatted report.

Step 1: Scrape & Curate (As Before)#

Use the bulk scraping and curation code from Sections 3 and 4.

Step 2: Generate an HTML Report#

# Generate an HTML file with curated articles
html = """
<!DOCTYPE html>
<html>
<head>
    <title>Tech News Aggregator</title>
    <style>
        .article-card { border: 1px solid #eee; padding: 15px; margin: 10px; border-radius: 5px; }
        img { max-width: 300px; }
    </style>
</head>
<body>
    <h1>Curated Tech News</h1>
"""
 
for article in curated_articles:
    html += f"""
    <div class="article-card">
        <h2>{article["title"]}</h2>
        <p><em>Published: {article["publish_date"]}</em></p>
        <p><strong>Summary:</strong> {article["summary"]}</p>
        <p><strong>Keywords:</strong> {", ".join(article["keywords"])}</p>
        <p><strong>Sentiment:</strong> {article["sentiment"]}</p>
        <p><strong>People:</strong> {", ".join(article["entities"]["people"])}</p>
        <p><strong>Organizations:</strong> {", ".join(article["entities"]["organizations"])}</p>
        <p><strong>Locations:</strong> {", ".join(article["entities"]["locations"])}</p>
        {f'<img src="{article["top_image"]}" alt="Article Image">' if article["top_image"] else ""}
        <p><a href="{article["url"]}" target="_blank">Read Full Article</a></p>
    </div>
    """
 
html += "</body></html>"
 
# Save the HTML file
with open("tech_news_aggregator.html", "w") as f:
    f.write(html)

References#

By combining newspaper3k for scraping, NLP tools for curation, and ethical practices, you can build powerful news aggregation or analysis pipelines. Experiment with different sources, curation techniques, and output formats to meet your needs!