py4u blog

Pagination using Scrapy - Web Scraping with Python

Web scraping is a powerful technique to extract data from websites. When dealing with large datasets spread across multiple pages (pagination), Scrapy, a popular Python framework for web scraping, provides efficient ways to handle this. In this blog, we'll explore how to implement pagination in Scrapy, understand common and best practices, and see example usage.

2026-06

Table of Contents#

  1. What is Pagination in Web Scraping?
  2. Scrapy Basics Review
  3. Implementing Pagination in Scrapy
  4. Common Practices
  5. Best Practices
  6. Example Usage
  7. Conclusion
  8. References

What is Pagination in Web Scraping?#

Pagination is the process of navigating through multiple pages of a website to access all the relevant data. For example, an e-commerce site might have products listed on multiple pages. Instead of scraping just the first page, we need to programmatically move to subsequent pages to gather all the product information.

Scrapy Basics Review#

Scrapy has several key components:

  • Spiders: Classes that define how to scrape a particular site.
  • Items: Containers for the data we want to extract.
  • Item Loaders: Help in populating items.
  • Request and Response: Requests are made to URLs, and responses are received with the page content.

Implementing Pagination in Scrapy#

Using Response.urljoin#

When you find a link to the next page in the response, you can use Response.urljoin to create an absolute URL. For example:

import scrapy
 
class MySpider(scrapy.Spider):
    name = "my_spider"
    start_urls = ["https://example.com/page1"]
 
    def parse(self, response):
        # Extract data from the current page
        #...
 
        next_page = response.css('a.next-page::attr(href)').get()
        if next_page:
            absolute_next_page = response.urljoin(next_page)
            yield scrapy.Request(absolute_next_page, callback=self.parse)

If the next page URLs follow a pattern (e.g., https://example.com/page/1, https://example.com/page/2), you can use string formatting.

import scrapy
 
class MySpider(scrapy.Spider):
    name = "my_spider"
    start_urls = ["https://example.com/page/1"]
 
    def parse(self, response):
        # Extract data
        #...
 
        current_page = int(response.url.split('/')[-1])
        next_page_number = current_page + 1
        next_page_url = f"https://example.com/page/{next_page_number}"
        yield scrapy.Request(next_page_url, callback=self.parse)

Common Practices#

Error Handling#

When making requests for subsequent pages, there could be network errors or the page might not exist. Use try-except blocks around requests.

import scrapy
 
class MySpider(scrapy.Spider):
    name = "my_spider"
    start_urls = ["https://example.com/page1"]
 
    def parse(self, response):
        try:
            next_page = response.css('a.next-page::attr(href)').get()
            if next_page:
                absolute_next_page = response.urljoin(next_page)
                yield scrapy.Request(absolute_next_page, callback=self.parse)
        except Exception as e:
            self.logger.error(f"Error in pagination: {e}")

Respecting Robots.txt#

Before scraping, check the website's robots.txt file. Scrapy has built-in support for respecting it. Just make sure your spider's ROBOTSTXT_OBEY setting is True (default in some configurations).

Best Practices#

Throttling Requests#

To avoid overloading the website, use Scrapy's DOWNLOAD_DELAY setting. For example, in settings.py:

DOWNLOAD_DELAY = 2  # Wait 2 seconds between requests

Using Selectors Efficiently#

Use CSS or XPath selectors that are as specific as possible. For example, instead of using a broad * selector, target the exact element.

# Good
response.css('div.product-item h2::text').get()
 
# Bad (less efficient)
response.css('*').get()

Example Usage#

Let's say we want to scrape a blog with pagination. The blog has articles on each page.

import scrapy
 
class BlogSpider(scrapy.Spider):
    name = "blog_spider"
    start_urls = ["https://exampleblog.com/page/1"]
 
    def parse(self, response):
        for article in response.css('article'):
            yield {
                'title': article.css('h2::text').get(),
                'content': article.css('p::text').get()
            }
 
        next_page = response.css('a.next::attr(href)').get()
        if next_page:
            absolute_next_page = response.urljoin(next_page)
            yield scrapy.Request(absolute_next_page, callback=self.parse)

Conclusion#

Pagination in Scrapy allows us to scrape large datasets spread across multiple pages. By using techniques like Response.urljoin, following link patterns, and adhering to common and best practices (error handling, respecting robots.txt, throttling, etc.), we can build robust web scrapers.

References#