ScrapingDuck v1.1 has landed! 🦆
We founded ScrapingDuck to make getting data from the web simple. Whether you run a price comparison site or a marketing dashboard, we wanted to handle the proxies and headless browsers so you didn’t have to.
With version 1.1, we are shifting gears slightly. We noticed that a huge chunk of our users are now building AI applications. You aren’t just storing data in a spreadsheet. You are feeding it into Large Language Models (LLMs) like GPT-4 or Claude.
That brought up a new problem we wanted to solve, specifically regarding scraping for LLMs and AI agents.
The problem with raw HTML
If you have ever tried to feed a raw scraped webpage into an LLM, you know the pain.
You get a massive wall of HTML code. There are <div> tags everywhere, massive chunks of JavaScript, CSS classes, and navigation menus that have nothing to do with the actual article.
This causes two major headaches:
- It costs too much. LLMs charge by the “token” (basically by the word). When you feed in raw HTML, you are paying to process code that doesn’t matter.
- It confuses the AI. Too much noise in the context window can make the model hallucinate or miss the specific data you asked for.
Meet the “Article Extractor”
In v1.1, we are introducing a new feature designed specifically for this issue. We call it the Article Extractor.
Instead of returning the full HTML source code, you can now hit an endpoint that does the cleaning for you. It acts as a powerful HTML to text API that strips away the ads, the sidebars, the pop-ups, and the code.
You get back a clean, text-only version of the content.
Why this matters: If you are dealing with web scraping for RAG (Retrieval-Augmented Generation), this is a game changer. You can stop writing complicated regex scripts to clean up your data. We deliver the content ready for your vector database.
Faster and smoother
While the AI data pipelines are the highlight, we also tightened up the core engine of our web scraping API.
- Better speed: We optimized how we route requests through our proxy network. You should see lower latency on standard requests.
- Smarter blocking: We improved our ability to handle sites that use heavy JavaScript or anti-bot measures.
Give it a try
We built ScrapingDuck v1.1 to be the most developer-friendly text extraction API out there. If you are tired of debugging headless browsers or wasting your API budget on useless HTML tags, give the new version a spin.
Get your API Key at ScrapingDuck.com
Tags: Web Scraping API, Scraping for LLMs, Clean web data extraction, Article Extractor, Convert HTML to markdown, ScrapingDuck v1.1
