How to Get All Pages of a Website Efficiently

You've probably been there. A crawler finishes cleanly, the export looks complete, and then someone spots entire clusters of listings, articles, or category pages that never made it into the queue. The usual culprit is a mix of JavaScript-rendered content, hidden pagination, and sitemap coverage that wasn't as complete as it looked at first glance.

A reliable way to fix that is to combine discovery paths instead of betting on one source. That means parsing sitemaps, crawling internal links, and rendering pages that only reveal URLs after the browser runs the page logic. If you want a broader mental model for this trade-off, master web data extraction methods is a useful companion read, and the practical API-first alternative is documented in RealtyAPI's introduction.

Introduction to Getting All Pages of a Website

A crawler can finish cleanly and still miss the pages that matter most. That usually happens when a site serves part of its inventory through scripts, hides more URLs behind AJAX requests, or leaves older sections out of the sitemap. In production, those misses surface later as gaps in analytics, incomplete SEO audits, and broken downstream datasets.

The fix is a layered discovery process. Start with machine-readable URL sources, then crawl internal links, then render the parts a normal HTTP client cannot see. The same approach is why teams often compare crawling with structured extraction platforms like RealtyAPI's data layer, because sometimes the better choice is a different retrieval path instead of more scraping. If you want a broader frame for that trade-off, master web data extraction methods is a useful companion read.

Complete coverage comes from combining discovery methods, not from trusting one crawl mode to catch everything.

Setting Up Crawling Tools and Environment

The environment matters because page discovery fails in messy ways. A small site with static HTML can be handled with requests, wget, or a basic Scrapy spider, while a JavaScript-heavy catalog usually needs a browser runtime such as Puppeteer or Playwright. For orchestration workflows, teams often wire crawls into n8n integrations so discovery jobs, retries, and downstream exports stay coordinated.

Pick the tool to match the page structure

Crawler Tools Overview	Language	Use Case
Scrapy	Python	Structured crawling, link queues, deduplication
Requests	Python	Lightweight fetches for sitemap and HTML parsing
wget	CLI	Recursive site mirroring on simple sites
Puppeteer	Node.js	Rendering JavaScript and intercepting requests
Playwright	Node.js	Browser automation for dynamic pages and selectors

A practical setup is to keep discovery code separate from parsing code. That way, sitemap fetching, HTML crawling, and browser rendering each have their own logs and failure handling. If you wire everything into one script, debugging becomes guesswork.

# Python
import requests
from bs4 import BeautifulSoup

resp = requests.get("https://example.com/robots.txt", timeout=30)
print(resp.text)

// Node.js
import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
await page.goto('https://example.com', { waitUntil: 'networkidle' });

# CLI
wget --recursive --level=inf --no-parent https://example.com

Practical rule: keep one environment for discovery, one for rendering, and one for storage. Mixing them makes resume logic and error recovery much harder.

Discovering URLs with Sitemaps and Robots Directives

The first place to look is robots.txt. Sites often declare sitemap locations there, and the sitemap protocol allows a sitemap file to hold up to 50,000 URLs and a maximum uncompressed size of 50 MB. Large sites often split discovery across multiple files and point to them through a sitemap index, which is why a crawler should treat robots.txt, sitemap.xml, and sitemap indexes as a linked chain rather than isolated files. Google's Search Central documentation notes that sitemaps help search engines discover pages that might otherwise be missed during crawling, which is exactly why they belong at the front of any full-site discovery workflow. The sitemap protocol and discovery model are summarized here.

A process flow chart illustrating how web crawlers discover URLs using sitemaps and robots.txt directives.

Parse robots.txt first

Start by fetching robots.txt and scanning for lines that mention sitemap locations. Then fetch each sitemap URL, including sitemap indexes if present. A sitemap index is just another XML file that points to more sitemap files, and some of them are gzipped, so your fetcher needs to handle compression cleanly.

import gzip
import requests
import xml.etree.ElementTree as ET

def fetch_xml(url):
    resp = requests.get(url, timeout=30)
    resp.raise_for_status()
    if url.endswith(".gz"):
        return gzip.decompress(resp.content)
    return resp.content

robots = requests.get("https://example.com/robots.txt", timeout=30).text
sitemaps = [line.split(":", 1)[1].strip() for line in robots.splitlines()
            if line.lower().startswith("sitemap:")]

Walk sitemap indexes recursively

Once you load XML, namespaces matter. Many parsers miss URLs because they ignore the sitemap namespace, so extract both <sitemap> entries and <url> entries before placing them into your queue.

If a sitemap index points to more sitemaps, keep walking until you've flattened the full tree. Don't assume one file means one complete inventory.

Validate every extracted URL before enqueueing it. Deduplicate aggressively, ignore malformed values, and keep a stable list of discovered pages so downstream crawls don't chase the same URL twice.

Extracting Links and Handling Pagination

When the sitemap is incomplete, internal links become your second discovery path. A breadth-first crawl from the homepage is usually the safest default because it spreads out through the site in a predictable way, which makes it easier to detect missing sections, loop traps, and off-site drift. This is also where good link extraction discipline matters, because one sloppy crawler can wander onto external domains and pollute the queue.

Follow internal links with a queue

Scrapy's CrawlSpider rules fit this pattern well, and wget can help on simpler sites where recursive traversal is enough. Queue each new internal link, mark pages as seen, and deduplicate before scheduling new requests. The same general logic is described in strategies for web data extraction pipelines, especially when link harvesting feeds a larger ETL system.

from collections import deque
from bs4 import BeautifulSoup
import requests
from urllib.parse import urljoin, urlparse

seen = set()
queue = deque(["https://example.com/"])

while queue:
    url = queue.popleft()
    if url in seen:
        continue
    seen.add(url)

    html = requests.get(url, timeout=30).text
    soup = BeautifulSoup(html, "html.parser")

    for a in soup.select("a[href]"):
        link = urljoin(url, a["href"])
        if urlparse(link).netloc == "example.com" and link not in seen:
            queue.append(link)

Handle pagination like a pattern, not a guess

Pagination-heavy sites need a different rhythm. The practical method is to identify the page parameter or next-page token, then loop sequentially until no new results appear; monitoring HTML structure, using randomized delays, and switching to AJAX requests when needed helps avoid missed pages as described in pagination guidance. On some sites, a page count derived from total items and items per page is more reliable than trying to infer the endpoint shape.

Useful habit: watch for page-shifts where a paginated section suddenly becomes a single-page listing. That change is easy to miss if you only test the first few pages.

For operational discipline, keep status handling separate from URL discovery, and make sure your crawler knows when a response is a block, a redirect loop, or just an empty page. A clear status model prevents false positives from slipping into the queue. A status-code reference helps when you wire that logic into production crawls.

Rendering JavaScript and Dynamic Content

Static HTTP fetches miss anything created after the browser boots. That includes product grids, filtered listings, infinite scroll feeds, and route changes that only appear after client-side code runs. For those pages, use a real browser engine and capture the DOM after the page settles.

A robot arm using a brush to render a website from raw HTML and JavaScript code files.

A browser crawler works best when it waits for a specific signal instead of a fixed delay. In Puppeteer or Playwright, that usually means waiting for network quiescence or for a DOM selector that confirms the content appeared. If the site fetches URLs through XHR, intercept those requests and inspect the JSON payloads, because the hidden endpoint often exposes the same data more directly than the rendered page.

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();

page.on('response', async (response) => {
  const url = response.url();
  if (url.includes('/api/')) {
    console.log('API response:', url);
  }
});

await page.goto('https://example.com/listings', { waitUntil: 'networkidle' });
await page.waitForSelector('.listing-card');
const html = await page.content();

Parallelism helps, but only if you close pages and browsers cleanly. Idle tabs leak memory fast, so keep a strict open-and-close lifecycle and avoid leaving sessions hanging after a failed navigation. If a site exposes the same data through a hidden request, use that route instead of hammering the full rendering stack.

Later in the run, extract URLs from the rendered DOM and from any captured JSON responses. That combination usually catches the routes pure HTML crawling misses.

Scaling Crawls with Rate Control and Deduplication

A crawler that works on a laptop can still fail in production if it floods the target or stores the same page over and over. The cleanest scaling pattern is to separate request pacing, retry handling, and page identity. That way, the crawl stays stable even when individual pages fail or the site changes layout.

Control request pressure before you scale out

Best-practice guidance consistently recommends pausing between requests, caching HTTP responses, and using retries with exponential backoff to reduce blocking and data gaps as summarized here. Randomized delays are useful on pagination-heavy sections because they make your access pattern less bursty and reduce the chance of repeated failures.

If proxy rotation is part of your setup, optimize proxies for scraping before you expand concurrency. Proxy pools don't fix bad crawl logic, but they do help when you need isolation across workers or want to separate fetch lanes by domain and content type.

Deduplicate by identity, not by accident

Normalize URLs before storing them. Strip junk query parameters when they don't change content, canonicalize where appropriate, and track seen fingerprints so the same page doesn't re-enter the queue from multiple routes. Large crawls also need durable storage, so use a database, a message queue, or a file-backed set if you want restartability.

Practical rule: the moment duplicates start appearing in logs, stop and fix identity logic before adding more workers.

For distributed crawls, shard by domain or URL pattern, then persist crawl state frequently so interrupted runs resume cleanly. If a site has many repeated templates, deduplication often matters more than raw crawl speed because it saves storage, review time, and downstream parsing work. Rate controls and retries belong in the same failure model as any production crawler.

Conclusion and Best Practices

The reliable path to get all pages of a website is to combine sitemap parsing, internal-link crawling, and JavaScript rendering, then run the whole system with strict pacing and deduplication. Respect robots.txt, honor site terms, and keep logging detailed enough to show where coverage stopped. If the crawl keeps missing pages, fix discovery before adding more concurrency.

If you need this kind of coverage in a production workflow, start with a pilot crawl, compare discovered URLs against your expected page families, and then harden the queue, retry, and rendering layers. A CTA for RealtyAPI.io.