Modern websites load critical data via JavaScript after the initial HTML shell. React, Vue, and Angular SPAs often return empty divs to simple HTTP clients. Headless browsers execute JS like a real user.
Puppeteer (Google, Node.js, Chrome/Chromium only) pioneered the category. Mature ecosystem, good documentation, tight Chrome DevTools integration. Weakness: single-browser, Node-only.
Playwright (Microsoft, 2020+) supports Chromium, Firefox, and WebKit from one API. Better auto-wait, tracing, and cross-browser testing. Our pick for new Node/TypeScript projects. scrapy-playwright bridges it into Python pipelines.
Scrapy alone handles ~80% of targets with static HTML — faster and cheaper. Add Splash (deprecated) or Playwright when you hit JS walls.
Our internal benchmark on 50 e-commerce product pages: Scrapy static — 12% success, 50ms/page. Playwright — 96% success, 2.1s/page. The trade-off is clear: use browsers only when necessary, but don't skip them when you must.
Scrapy Ninja automatically selects the right tool per target in our managed pipelines. You describe the data; we choose the stack.
