10 Web Scraping Libraries for Real Projects
Python is used in 72% of scraping projects, while JavaScript and Node.js account for 18%, but there's no universally best web scraping library. Choose based on whether the target is static or JavaScript-rendered, the crawl scale, your language stack, and the maintenance burden.
The Best Library Depends on the Site
The most popular advice about web scraping libraries is often too simple. Developers are told to pick Beautiful Soup for Python, Selenium for dynamic pages, or Scrapy for serious work. Those recommendations can be useful starting points, but they confuse tools that solve different problems.
A parser reads HTML that you already have. An HTTP client fetches that HTML. A browser automation library executes JavaScript and interacts with a rendered page. A crawling framework manages queues, retries, concurrency, pipelines, and exports around those lower-level components. Treating them as interchangeable usually creates unnecessary complexity.
The practical split is straightforward:
- Static HTML: Use a lightweight parser such as Beautiful Soup, Cheerio, jsoup, or Html Agility Pack.
- JavaScript rendering: Use Playwright, Puppeteer, or Selenium when the required data appears only after the page executes client-side code.
- Recurring crawling: Use Scrapy, Colly, or Apify SDK when queues, persistence, retries, throttling, and throughput matter.
- Language-specific projects: Prefer a library that fits the existing Python, JavaScript, Go, Java, or .NET stack.
Python remains the default ecosystem for many teams. A 2025 developer survey reported Python usage at 69.6%, compared with 34.8% for JavaScript, while Beautiful Soup appeared in 43.5% of responses and Scrapy in 13%. Those figures show adoption, not suitability. The right choice still depends on rendering needs, debugging, browser overhead, anti-bot pressure, legal review, and how much maintenance your team can absorb.
Listing aggregation makes the trade-off concrete. You might collect listing URLs with a fast HTTP crawler, render only pages that need JavaScript, parse the resulting markup, normalize fields, and monitor selector failures. At some point, a commercial API such as RealtyAPI.io may be more efficient than owning that entire system.
1. Scrapy
Scrapy is the strongest choice here when the job is full crawling rather than one-page extraction. It gives Python teams project scaffolding, spiders, item pipelines, request scheduling, retry behavior, caching, exports, and middleware in one structured framework. Its architecture is designed for crawls that need repeatable operations instead of a script someone runs manually.
The built-in shell is especially useful during development. You can test selectors against a response before embedding them in a spider, which shortens the feedback loop when listing pages contain several similar card layouts. Auto-throttle, concurrency controls, and backpressure also make it easier to avoid overwhelming a target while keeping a crawl productive. The Scrapy documentation and project site are the appropriate starting points for its architecture and extensions.
Where Scrapy earns its place
Scrapy performs best when pages return useful HTML without requiring a full browser. Its middleware ecosystem lets you add custom headers, retry policies, caching, authentication handling, and JavaScript support through integrations such as scrapy-playwright. That separation is valuable because you can keep the fast path lightweight and invoke a browser only for selected requests.
The cost is a steeper learning curve. Scrapy's asynchronous Twisted model, item lifecycle, and settings system feel less familiar than a short requests-and-parser script. Teams also need to design validation, deduplication, persistence, and monitoring rather than assuming the framework has solved the data quality problem.
Practical rule: Use Scrapy when the crawl itself is a product capability. Don't introduce it merely to parse one response.
For a listing aggregator, Scrapy can discover search pages, follow property URLs, retry transient failures, and send normalized items to a database. If maintaining source-specific crawlers becomes the larger problem, review the RealtyAPI.io introduction before expanding the spider fleet.
2. Playwright
Playwright is the practical default when a target behaves like a modern application rather than a document. It drives Chromium, Firefox, and WebKit through one API, executes JavaScript, waits for page conditions, and supports interactions such as clicking filters, scrolling through results, and completing multi-step flows.
That reliability matters on listing sites where the initial response contains only a shell and the cards arrive after client-side requests. Playwright's auto-waiting and resilient locator model reduce the need for arbitrary sleep calls. Traces, screenshots, videos, headed mode, and headless mode also give maintainers evidence when a selector fails in production. The Playwright website documents its browser and language integrations.
The browser tax is real
A rendered browser consumes more resources than an HTTP request followed by an HTML parser. It also takes longer to reach the point where content can be extracted, especially when a page loads analytics, advertisements, maps, images, and third-party scripts that the scraper doesn't need.
Use browser contexts carefully. Block unnecessary resources where appropriate, limit concurrency to what the host can support, and close pages reliably after each job. Playwright supports Node, Python, .NET, and Java, so it can fit more stacks than a Python-only parser, but language support doesn't remove the operational cost of running browsers.
For listing aggregation, a useful pattern is to use HTTP requests for search pages that expose links in the response, then send only JavaScript-dependent detail pages to Playwright. That hybrid approach keeps browser rendering focused on pages that need it.
Browser rendering should be a deliberate exception, not the default transport for every URL.
Playwright also fits teams that need strong diagnostics. When a page changes, a trace can show whether the failure came from navigation, an absent element, a consent dialog, or a changed data flow. If your data source already has a supported integration path, compare that operational burden with the RealtyAPI.io integrations documentation.
3. Puppeteer
Puppeteer is a strong fit for teams already invested in Node.js and Chromium. It controls Chrome and Chromium through the Chrome DevTools Protocol, lets you move through pages, evaluate JavaScript, intercept network traffic, capture screenshots, generate PDFs, and inspect browser behavior programmatically.
That Chromium focus is both its strength and its boundary. If the target application is tested and operated primarily in Chrome, Puppeteer provides close access to browser capabilities without requiring a broader abstraction. The official Puppeteer documentation covers the standard package and puppeteer-core, which is useful when the browser installation is managed separately.
Good for Chromium-first pipelines
Puppeteer works well when extraction depends on client-side rendering, network responses, or page evaluation. For example, a listing workflow can open a search route, wait for property cards, inspect links created by the application, and capture the rendered DOM for downstream parsing. Network interception can also help identify data requests that are more stable than visual selectors, provided the endpoint is accessible and its use is permitted.
The main trade-off is browser scope. Puppeteer is primarily Chromium-focused, while Playwright offers a single API across Chromium, Firefox, and WebKit. That difference matters when your quality assurance process must cover more than one browser engine or when a target behaves differently outside Chromium.
Puppeteer is also heavier than a parser. Every page still carries browser startup, navigation, JavaScript execution, and resource-management costs. Reusing browser processes and contexts can reduce overhead, but it makes isolation and cleanup more important. A failed page should not leave an open tab, pending request, or corrupted session in a long-running worker.
Use Puppeteer when your JavaScript team wants direct Chromium control and the target doesn't require broader browser coverage. Choose Playwright when cross-browser support, built-in diagnostics, and resilient locators matter more than tight Chromium alignment.
4. Selenium
Selenium remains useful because it combines broad browser coverage, many language bindings, a large community, and the standardized WebDriver approach. It can drive Chrome, Edge, Firefox, and Safari, making it a practical option for organizations with established browser automation infrastructure. The Selenium project also provides a mature ecosystem around drivers, grids, and distributed execution.
For scraping, Selenium is most useful when a real browser is unavoidable and the team already has Selenium expertise. It can open a page, click controls, submit forms, wait for content, and expose the resulting DOM for extraction. It also pairs naturally with Beautiful Soup, since the browser can render the page while the parser handles the resulting HTML. A University of Virginia scraping guide describes this division between browser interaction and HTML parsing.
A capable but maintenance-heavy option
Selenium's maturity doesn't mean it's always the easiest choice for a new scraper. Historically, setup and driver management have required more attention than newer browser libraries, and verbose selectors can become fragile as interfaces change. WebDriver BiDi is expanding bidirectional browser events, but teams still need to evaluate the maturity of the specific features they depend on.
Selenium Grid can help distribute work, but parallel browser execution introduces infrastructure concerns. You'll need clear session cleanup, timeouts, browser version management, and evidence for diagnosing failures. A listing aggregator that uses Selenium for every property page may spend more effort maintaining browsers than extracting fields.
Use explicit waits tied to meaningful page conditions. Avoid fixed delays as the main synchronization strategy. Capture page source and screenshots when extraction fails, then test selectors against representative layouts rather than only the current page.
For teams maintaining an older enterprise automation stack, Selenium can be the least disruptive choice. For a new browser-heavy collector, compare its operational model with Playwright before committing.
The RealtyAPI.io API Playground can also help determine whether a listing workflow needs browser automation at all.
5. Beautiful Soup
Beautiful Soup is the best-known lightweight option for parsing and navigating HTML or XML in Python. It doesn't fetch pages, execute JavaScript, schedule requests, retry failures, or manage a crawl. That narrow scope is a feature when you already have HTML and need a forgiving, readable way to extract fields.
A common workflow uses an HTTP client to fetch the response, inspects the page structure, and passes the content to Beautiful Soup for extraction. The OpenSource.com workflow describes this sequence of fetching HTML with requests, identifying relevant tags, and placing extracted values into a Python data structure. The parser can use backends such as Python's built-in parser or lxml, and its tree navigation and CSS selector APIs are approachable for prototypes.
Excellent parser, incomplete scraper
Beautiful Soup handles imperfect markup gracefully, which helps when public pages contain inconsistent nesting or malformed HTML. It's easy to combine with requests or httpx, and it's also useful after Selenium or Playwright has rendered a page.
The limitation appears when the target's content isn't in the initial response. Beautiful Soup won't execute the JavaScript that creates listing cards, opens a menu, or loads additional results. You'll need a separate browser or an accessible data request before parsing can begin.
It also leaves operational work to you. Add request timeouts, retries, rate control, caching, logging, deduplication, and schema validation around it. Without those controls, a script can collect partial data or send repeated requests after a source changes.
For a small static listing catalog, Beautiful Soup with an HTTP client is often the right starting point. For a recurring multi-source product, use it as the extraction layer inside a larger system rather than treating it as the complete crawler.
6. Cheerio
Cheerio gives Node.js developers a fast, jQuery-style way to parse and manipulate HTML and XML without launching a browser. Its familiar selectors make it comfortable for JavaScript teams that already know the DOM and want server-side extraction with low overhead. The Cheerio documentation explains its parser and manipulation APIs.
Cheerio is a parser, not a crawler. Pair it with fetch, axios, got, or another HTTP client to obtain the response, then load the HTML into Cheerio and select the fields you need. That separation keeps the code small and makes it easy to process many static responses in a Node service.
The right layer for static Node workloads
Cheerio works particularly well for catalog pages, feeds, server-rendered pages, and responses where the required fields are present in the HTML. It can extract listing titles, addresses, links, prices, and metadata without paying browser costs. For a listing aggregator, a Node worker can fetch result pages, use Cheerio to discover detail URLs, and pass structured records to a queue or database.
It can't execute client-side JavaScript. If the listing cards appear only after React, Vue, or another front-end application runs, Cheerio will see the original markup, not the final browser state. Adding more selectors won't fix that problem. Switch the fetch layer to a browser or identify an authorized structured endpoint.
Cheerio also doesn't provide built-in retries, rate limiting, scheduling, proxy handling, or persistence. Those responsibilities belong in your HTTP client and orchestration layer. That can be an advantage for teams that want to assemble a focused Node pipeline, but it means production reliability depends on the surrounding code.
Choose Cheerio when static HTML and a JavaScript ecosystem are the main requirements. Don't choose it because the target merely looks simple in a browser.
7. Apify SDK
Apify SDK is aimed at the orchestration layer. It provides primitives for building scrapers, crawlers, and automations called Actors, with request queues, autoscaled concurrency, datasets, key-value stores, and integrations for Playwright and Puppeteer. The Apify SDK documentation covers the JavaScript workflow and its local-to-cloud path.
That distinction matters. Apify SDK isn't merely an HTML parser or browser driver. It helps coordinate a job that must remember visited URLs, persist results, scale workers, resume after interruptions, and keep state between runs. Those capabilities reduce the amount of queue and storage code a team needs to write around its extractor.
Useful when the crawl becomes a service
A listing aggregator can use request queues for search pages and detail URLs, datasets for extracted records, and key-value stores for run metadata or diagnostic output. Playwright or Puppeteer can handle pages that need rendering while cheaper request-based handlers process static pages. This mixed strategy is more practical than forcing every URL through a browser.
The trade-off is coupling. The JavaScript SDK is the most mature part of the ecosystem, while teams using Python need to assess the current feature coverage for their project. A design that depends heavily on Apify platform features may also require migration work if you later move to different infrastructure.
Apify SDK reduces boilerplate, but it doesn't remove source maintenance, schema design, compliance review, or the cost of browser execution. It also doesn't guarantee that selectors remain valid after a site redesign.
If your team already uses n8n, the RealtyAPI.io n8n integration documentation provides another way to connect structured real estate data to an orchestration workflow without building every source adapter yourself.
8. Colly
Colly is the standout choice for teams that want a fast, compiled crawler in Go. It provides callbacks, concurrency controls, visit limits, rate limiting, caching, cookies, robots.txt support, and request and response hooks. The Colly project site documents its API and crawling model.
Go's concurrency model makes Colly attractive for high-throughput HTTP crawling, especially when the target exposes predictable HTML and the surrounding service is already written in Go. Its callback structure is clear: respond to a page, select relevant elements, collect links, and schedule the next requests. That makes it possible to build a focused crawler without adopting a large browser automation stack.
Fast crawling with a hard rendering boundary
Colly doesn't provide a built-in headless browser. If listing data is inserted by JavaScript, the crawler won't see it in the original response. You'll need to pair Colly with a browser or another rendering mechanism, then decide how to pass rendered content back into the Go pipeline.
That split can be useful. Let Colly handle discovery, static pages, caching, and rate control, and reserve a renderer for the minority of pages that require it. It can also become awkward if most pages depend on complex interactions, authentication states, infinite scrolling, or browser-only behavior.
Colly's support for caching and visit controls helps reduce duplicate work. Still, you'll need to implement field normalization, validation, persistence, alerting, and source-specific recovery. A fast crawler that stores inconsistent addresses or silently misses prices is still a poor data system.
Choose Colly when Go is already your team's operational language and the target is primarily HTTP-friendly. Choose Playwright or Selenium when the browser is the central requirement, not an occasional adapter.
9. jsoup
jsoup is a practical Java library for fetching, cleaning, and extracting HTML. It supports CSS selectors, DOM traversal, HTML cleaning, sanitization, and a built-in HTTP client for simple fetch-and-parse flows. The jsoup website is the main reference for embedding it in JVM services.
Its value comes from fit. A Java team can add jsoup to an existing service without introducing a separate scraping language or browser runtime for straightforward pages. The API is small and consistent, and the DOM model makes extraction readable. It's suitable for server-rendered listing pages, HTML cleanup, and data that can be obtained through ordinary HTTP requests.
A dependable JVM parser
jsoup can fetch a page, select listing cards with CSS selectors, traverse child elements, clean markup, and pass normalized values to the rest of a Java application. It's also useful when the application needs to sanitize HTML before storing or displaying content.
The boundary is JavaScript. jsoup doesn't execute client-side code, so it can't render a single-page application or perform the interactions needed to reveal hidden listing data. A team can pair it with Selenium or Playwright, but that adds a browser layer, its resource demands, and another failure surface.
jsoup also isn't a complete crawling framework. Scheduling, retries, distributed queues, proxy decisions, persistence, and monitoring belong elsewhere. That's acceptable for a service with a modest, controlled workflow, but recurring multi-source aggregation needs a deliberate orchestration design around the parser.
Use jsoup when your JVM stack needs solid HTML handling and the source exposes usable markup. Don't mistake its convenient HTTP client for a complete crawler.
10. Html Agility Pack
Html Agility Pack is a long-standing .NET option for loading, querying, modifying, and cleaning HTML. It tolerates imperfect markup and fits naturally into C# utilities that already use HttpClient, databases, and background workers. The Html Agility Pack site provides NuGet distribution details and examples.
Its core strength is parsing. XPath selection and the DOM read/write API give .NET developers precise control over extraction, while CSS-style selection can be added through tools such as Fizzler. For a static listing page, a C# worker can fetch the response with HttpClient, load the document, select cards, normalize text, and write records to an existing service.
Useful inside a .NET stack
Html Agility Pack handles malformed HTML more reliably than a parser that expects perfect markup. That matters when different sources use inconsistent nesting, missing closing tags, or presentation markup that changes between templates.
It doesn't include a JavaScript engine or a full HTTP workflow. You must configure HttpClient behavior, timeouts, retries, rate limits, and persistence separately. When a source renders data only after JavaScript runs, pair the parser with Selenium or Playwright and pass the rendered HTML into the extraction step.
CSS selector support can also require an additional package, which may make XPath the more consistent choice for teams that want to keep dependencies limited. XPath can be precise, but overly rigid paths tend to break after layout changes, so selectors should target stable attributes and be covered by fixture tests.
Choose Html Agility Pack when the application is already .NET-based and the target serves parseable HTML. It's a sensible component, not an end-to-end scraping platform.
Top 10 Web Scraping Libraries, Feature Comparison
| Tool | Core Capabilities | Best for 👥 | Performance & Reliability ★ | Unique strengths ✨ / 🏆 | Price / Value 💰 |
|---|---|---|---|---|---|
| Scrapy | Project scaffolding, pipelines, auto-throttle, middlewares | 👥 Python devs & data engineers building scalable crawlers | ★★★★☆, high-throughput, built-in retries | 🏆 Robust crawler architecture; rich middleware ecosystem ✨ | 💰 Free, OSS |
| Playwright | Cross‑browser automation, JS rendering, tracing & emulation | 👥 Teams needing reliable JS rendering across browsers | ★★★★★, very resilient for dynamic/SPAs | 🏆 Multi‑browser + deep diagnostics (traces/screenshots) ✨ | 💰 Free, OSS |
| Puppeteer | Chromium control, page eval, screenshots, DevTools protocol | 👥 Node teams focused on Chromium-based scraping | ★★★★☆, strong Chromium performance | ✨ Tight Chromium integration; puppeteer‑core for custom browsers | 💰 Free, OSS |
| Selenium | WebDriver-based real-browser automation, Grid support | 👥 Enterprise teams needing broad language/browser coverage | ★★★★☆, mature, stable; driver setup can be fiddly | ✨ Wide browser/language support; Selenium Grid for parallelization | 💰 Free, OSS |
| Beautiful Soup | HTML/XML parsing, tolerant of malformed markup | 👥 Python devs prototyping static-page extraction | ★★★☆☆, excellent for static/dirty HTML | ✨ Forgiving parsing and simple API for quick prototypes | 💰 Free, OSS |
| Cheerio | jQuery-like HTML parser/manipulator for Node | 👥 JavaScript devs doing server-side static parsing | ★★★☆☆, lightweight and very fast for static HTML | ✨ Familiar jQuery API; low overhead for bulk parsing | 💰 Free, OSS |
| Apify SDK | Actors, request queues, storage, proxy hooks, autoscale | 👥 Teams orchestrating large crawls or using cloud actors | ★★★★☆, production primitives, autoscaling support | 🏆 Built-in queues/datasets, proxy integration, local→cloud workflow ✨ | 💰 SDK free; Apify platform paid tiers |
| Colly | Go crawler with concurrency, rate limiting, caching | 👥 Go developers needing high-performance crawlers | ★★★★☆, very fast; Go concurrency advantages | ✨ Parallel crawling, robots/caching, low runtime overhead | 💰 Free, OSS |
| jsoup | Java HTML parser + simple HTTP client, sanitization | 👥 JVM services needing robust parsing & cleaning | ★★★☆☆, reliable for fetch+parse flows | ✨ HTML cleaning, CSS selectors, easy JVM integration | 💰 Free, OSS |
| Html Agility Pack | .NET HTML parser with XPath/CSS via add-ons | 👥 C#/.NET developers parsing/modifying HTML | ★★★☆☆, tolerant DOM parsing for .NET stacks | ✨ XPath support, NuGet distribution, integrates with HttpClient | 💰 Free, OSS |
Choose the Smallest Reliable Stack
Start with the data path, not the library's popularity. Fetch a representative page and inspect whether the required fields exist in the initial HTML. If they do, a parser may be enough. If they appear only after JavaScript runs, you need a browser or an accessible structured endpoint. If the job includes discovery, queues, retries, persistence, and recurring execution, add a crawling or orchestration layer.
For straightforward static HTML, choose the parser that matches your language ecosystem:
- Beautiful Soup: A forgiving Python parser for prototypes and moderate extraction work.
- Cheerio: A lightweight option for Node.js services that already use JavaScript.
- jsoup: A compact choice for JVM applications that need CSS selectors and sanitization.
- Html Agility Pack: A familiar .NET component for malformed HTML and XPath-based extraction.
Use Scrapy, Colly, or Apify SDK when the workload is a real crawl. Scrapy provides a mature Python framework with spiders, pipelines, middleware, retries, and scheduling. Colly brings Go concurrency and clear crawling callbacks. Apify SDK adds queues, storage, autoscaling, and browser integrations for teams that want orchestration primitives rather than a collection of disconnected scripts.
Use Playwright, Puppeteer, or Selenium when browser rendering is unavoidable. Playwright is a strong general-purpose choice for dynamic pages and cross-browser workflows. Puppeteer suits Chromium-first Node projects. Selenium remains practical for broad browser support, established enterprise stacks, and teams already operating WebDriver infrastructure.
The library is only one part of a reliable system. Apply respectful rate limits, review robots.txt and the target's terms, and obtain legal guidance where the data, jurisdiction, or intended use creates uncertainty. Minimize collected personal data, store only fields the product needs, and define retention rules before the first crawl runs.
Monitor the output, not just the process. Track empty fields, unexpected record counts, duplicate URLs, HTTP failures, and selector misses. Keep saved HTML fixtures for representative layouts, test parser changes against those fixtures, and alert when a source changes enough to invalidate the schema. A scraper that returns HTTP success while dropping addresses or prices can go unnoticed for a long time.
For a listing aggregation product, the architecture becomes more demanding when each source needs its own browser behavior, proxy strategy, retry rules, normalization logic, and ongoing maintenance. RealtyAPI.io presents a unified data layer for publicly available real estate and travel-stay information, with structured access to listing and market data through an API. Its documentation describes REST, GraphQL, webhooks, search by destinations or coordinates, and integrations that can reduce custom source-by-source extraction work.
That doesn't make a commercial API the right answer for every project. Custom scraping remains appropriate when you need a source that isn't covered, a highly specialized field, or complete control over collection and transformation. Evaluate RealtyAPI.io when the product's value lies in the normalized listing experience rather than in owning browser automation, proxy operations, retries, source adapters, and selector maintenance. Review the available coverage, fields, terms, and cost against the engineering time required to operate your own pipeline.
A good decision path is simple. Prototype with the smallest parser that can extract the required fields. Add a browser only for pages that need rendering. Introduce Scrapy, Colly, or Apify SDK when crawl coordination becomes a recurring operational concern. Replace custom collection with a structured API when maintaining source infrastructure costs more attention than the product can justify.
RealtyAPI.io provides a unified API for publicly available real estate and travel-stay data, returning structured information for listing search, property details, availability, reviews, amenities, and related market workflows. If you'd rather spend your engineering time on the aggregation product instead of browser automation and source maintenance, visit RealtyAPI.io to review the API and integrations.