Zyte, Apify, and ScrapingBee now promote AI-assisted extraction. The promise: send HTML, receive structured JSON without writing site-specific selectors. The reality: powerful but requires guardrails.
Our hybrid pipeline: Scrapy collects raw HTML at scale → chunk and clean → LLM prompt with JSON schema → validate output against schema → reject/human-review failures → store validated records.
Cost management: GPT-4o-mini for high-volume fields, GPT-4o for complex nested structures. Cache by page hash to avoid re-processing unchanged pages. Typical cost: $0.001–0.01 per page depending on size.
Hallucination mitigation: require citations (text spans from source HTML), schema validation, cross-field consistency checks (price > 0, dates parseable), and sample human audit batches.
When LLM wins: aggregator sites with inconsistent templates, long-tail e-commerce, forum threads, PDF-adjacent content. When rules win: stable product pages with JSON-LD, API endpoints, repeated list formats.
Jean Cailleux leads our AI extraction R&D — from unstructured data to deep learning, scraping is a marvelous playground. Contact info@scrapy.ninja to pilot LLM extraction on your targets.
