{"id":9628,"date":"2025-07-20T04:54:19","date_gmt":"2025-07-20T04:54:19","guid":{"rendered":"https:\/\/www.newsbeep.com\/uk\/9628\/"},"modified":"2025-07-20T04:54:19","modified_gmt":"2025-07-20T04:54:19","slug":"scrape-smarter-not-harder-let-mcp-and-ai-write-your-next-scraper-for-you","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/uk\/9628\/","title":{"rendered":"Scrape Smarter, Not Harder: Let MCP and AI Write Your Next Scraper for You"},"content":{"rendered":"

In past episodes of\u00a0The Lab series<\/a>\u00a0on\u00a0The Web Scraping Club<\/a>, we used\u00a0Firecrawl\u00a0to scrape the content of this newsletter, a vector DB like\u00a0Pinecone\u00a0to store the articles’ markdown, and the OpenAI API to retrieve information and append the context to our prompt to get better answers.<\/p>\n

This approach has limitations. The context window size (the amount of information we can include in the prompt) is limited, so the answers were not always great.<\/p>\n

Since the AI landscape will change in a few months, it seems that using RAG and Pinecone is old school. Today, the new buzzword is\u00a0Model Context Protocol (MCP)<\/a>, so I\u2019ve conducted this small experiment to see how MCP can be used for web scraping.<\/p>\n

What is a Model Context Protocol?<\/p>\n

The\u00a0Model Context Protocol (MCP)\u00a0is an open standard, initially developed by Anthropic, that enables large language models (LLMs) to interact with external tools and data through a standardized interface. In essence, MCP provides a\u00a0universal connector\u00a0between AI models and the systems where data lives. Instead of building custom integrations for every data source or scraping tool, developers can expose their data or functionality via an MCP server. AI assistants (the MCP clients) can consistently request those resources, and this standardized, two-way connection enriches raw content with metadata, context, and instructions so that AI models can more effectively interpret information.<\/p>\n

But how does it work?<\/p>\n

The MCP follows a client-server architecture: The host application (e.g., an IDE or AI assistant) runs an MCP\u00a0client\u00a0(Claude, Cursor, etc.) that connects via the MCP protocol to one or more MCP\u00a0servers. Each server interfaces with a specific data source or tool (e.g., a database, file system, or web scraper) on your local machine or network, providing structured context or actions to the LLM.\u200b<\/p>\n

An MCP server can expose\u00a0resources\u00a0(read-only context data analogous to GET endpoints) and\u00a0tools\u00a0(actions or functions the model can invoke analogous to POST endpoints) to the LLM.<\/p>\n

Anthropic open-sourced MCP in early 2024 to encourage industry-wide adoption. The goal was to establish a\u00a0common framework for AI-to-tool communication, reducing reliance on proprietary APIs and making AI integrations more modular and interoperable.<\/p>\n

This structured context handling improves on plain prompts. The model no longer just receives a blob of text; instead, it can load data via resources and call tools with well-defined inputs\/outputs. In summary, MCP provides a standardized, secure way to feed context into AI models and receive actions\/results, which is especially powerful in complex tasks like web scraping.<\/p>\n

Why Use MCP?<\/p>\n

As we\u2019ve seen in previous posts using RAG, one challenge was adding the downloaded article to the prompt since the context size is limited. We tried different solutions, like splitting the articles into chunks on Pinecone and then saving only a summary made with OpenAI to reduce its length.<\/p>\n

Using MCP, especially in this case, where we\u2019re using it inside Cursor IDE, we can create a tool that extracts the HTML and saves it to a file that can be read and chunked by Cursor and used to generate the XPaths.<\/p>\n

Another interesting aspect of using MCP is that we mix programming and prompt engineering. In fact, creating an MCP tool is like creating a function in our code: once we crafted the prompt in the IDE, this is evaluated and calls our tools with the needed argument (like the URL to fetch) guessed from our prompt. The tool itself, instead, is a programmed function, so there\u2019s not the fuzziness of a prompt, but given an input, we know the output we can expect. This is great because it reduces the uncertainty of the whole process and, at the same time, opens up a myriad of use cases for enriching the context of your chat with the LLM.<\/p>\n

Last but not least, MCP is a standard, and theoretically, once we have a server up and running, we could plug it into any model and tool that supports it. However, the fact that OpenAI does not currently support it is not a good sign for the protocol’s diffusion.<\/p>\n

What we\u2019re going to use MCP<\/p>\n

We\u2019ll use two key technologies to implement our solution: the\u00a0MCP Python SDK\u00a0and\u00a0Camoufox.<\/p>\n

MCP Python SDK.\u00a0The MCP Python SDK is the official library for building MCP servers (and clients) in Python. It implements the full MCP specification, handling all the underlying protocol messaging so you can focus on defining the resources and tools you need.<\/p>\n

With this SDK, you can create an MCP server in just a few lines of code. It provides a high-level\u00a0FastMCP\u00a0server class that manages connections and lets you register functions as tools or resources via decorators. For example, you can annotate a function\u00a0@mcp.tool()\u00a0to expose it to the model as an actionable tool.<\/p>\n

The SDK packages the function\u2019s result and sends it back to the AI client in a consistent format. In summary, setting up an MCP server that can feed context to LLMs or perform tasks on their behalf is straightforward.<\/p>\n

Camoufox for HTML retrieval.\u00a0In web scraping, getting the raw HTML from target pages (especially those with anti-scraping measures or heavy JavaScript) is half the battle. I decided to use Camoufox, which is an open-source stealth browsing tool designed for such scenarios, to be almost sure to be able to get the HTML from every page. This is especially true because the MCP runs locally on my machine, so I won\u2019t need any proxy. On top of its stealth capabilities, I wanted to use Camoufox to build an MCP logic from 0. If you want to save time, you can use\u00a0the BrowserBase MCP server<\/a>\u00a0or\u00a0the Hyperbrowser one<\/a>. They have some prebuilt tools like extracting data and interacting with the page, making life easier for us.<\/p>\n

Technical Implementation of an MCP Server for writing a Camoufox scraper<\/p>\n

Now, we\u2019ll build the MCP server that helps us write a Camoufox scraper in three steps. Each of the steps of the process has its own tool:<\/p>\n

fetch_page_content\u00a0will be a tool that opens Camoufox and stores the HTML of the target page on a file.<\/p>\n

generate_xpaths\u00a0will be a tool that reads the HTML file and, given a template of output data, creates the selectors, again saving them to a file. That\u2019s why we want it flexible enough to handle different page types (for example, a product listing page vs. a product detail page in e-commerce).<\/p>\n

write_camoufox_scraper\u00a0will be a tool that reads the selectors and a template of Camoufox spider (Camoufox_template.py) and creates a new one based on that and the selectors just created.<\/p>\n

The server code (xpath_server.py) will be saved in the repository under folder 79.MCP\/MCPFiles.<\/p>\n

Step 1: Set Up the Environment<\/p>\n

First, make sure you have\u00a0Python 3.10+\u00a0installed and install the necessary packages. We\u2019ll need the MCP SDK and Camouf, which you can install via pip.<\/p>\n

pip install mcp camoufox <\/p>\n

The\u00a0mcp\u00a0package includes the MCP Python SDK and CLI tool.\u00a0Camoufox\u00a0may require an additional step to fetch its browser binary (for example, running\u00a0python -m camoufox fetch\u00a0to download the stealth Firefox \u2013 refer to Camoufox docs for details). Once these are installed, you\u2019re ready to write the server code.<\/p>\n

Step 2: Initialize the MCP Server<\/p>\n

Create a new Python script (in the repository, it will be\u00a0xpath_server.py). In this script, we’ll initialize an MCP server and define our tools. Using the MCP SDK is straightforward:<\/p>\n

from mcp.server.fastmcp import FastMCP
\nimport asyncio
\nfrom camoufox.async_api import AsyncCamoufox
\nimport time
\nimport os<\/p>\n

mcp = FastMCP(“Scrapy XPath Generator”)<\/p>\n

…..
\ncoded tools
\n….<\/p>\n

if __name__ == “__main__”:
\n\t# Initialize and run the server
\n\tmcp.run(transport=’stdio’)<\/p>\n

That\u2019s enough for setting up an empty MCP server.<\/p>\n

Step 3: Implement HTML Retrieval via Camoufox<\/p>\n

We want a tool that, given a URL, fetches the page\u2019s HTML content. We\u2019ll use Camoufox to do this in a stealthy way. Let\u2019s define an MCP\u00a0tool\u00a0called\u00a0fetch_page_content:<\/p>\n

HTML_FILE_PATH = “\/Users\/pierluigivinciguerra\/TWSC\/Code\/TheWebScrapingClub\/79.MCP\/downloaded_page.html” # File to store the downloaded HTML
\nCAMOUFOX_FILE_PATH = “\/Users\/pierluigivinciguerra\/TWSC\/Code\/TheWebScrapingClub\/79.MCP\/camoufox_template.py”<\/p>\n

@mcp.tool()
\nasync def fetch_page_content(url: str) -> str:
\n\tglobal latest_html<\/p>\n

\t“””Fetch page HTML using Camoufox stealth browser.”””
\n\tprint(f”[DEBUG] Fetching URL: {url}”)
\n\ttry:
\n\t\tasync with AsyncCamoufox(humanize=True) as browser:
\n\t\t\tpage = await browser.new_page()
\n\t\t\tawait page.goto(url)
\n\t\t\ttime.sleep(10)
\n\t\t\tlatest_html = await page.content()
\n\t\t\twith open(HTML_FILE_PATH, “w”, encoding=”utf-8″) as f:
\n\t\t\t\tf.write(latest_html)
\n\t\t\tprint(“[DEBUG] HTML stored for later use”)
\n\t\t\treturn “HTML fetched and stored successfully.”
\n\texcept Exception as e:
\n\t\tprint(f”[ERROR] {e}”)
\n\t\treturn f”Error fetching page: {str(e)}”<\/p>\n

Except for the decorator, it\u2019s a simple Camoufox scraper that opens the URL we\u2019ll pass in the prompt inside Cursor and stores the HTML code in a file.<\/p>\n

Step 4: Generate XPath Selectors Based on the template<\/p>\n

The second tool we need analyzes the HTML and produces XPath selectors for the data we want to scrape. We\u2019ll create a tool,\u00a0generate_xpaths,\u00a0that takes a template identifier (to indicate what kind of page or extraction pattern to use) as an argument.<\/p>\n

Based on the template, this function will output the appropriate XPath expressions. For example, we might have one template for a Product Listing Page (PLP, a page with multiple products listed) and another for a Product Detail Page (PDP, a page with a single product\u2019s details).<\/p>\n

@mcp.tool()
\ndef generate_xpaths(template: str) -> dict:
\n\t“””Write XPATH selectors for the requested fields using the downloaded HTML file.”””<\/p>\n

\tif not os.path.exists(HTML_FILE_PATH):
\n\t\treturn {“error”: f”No HTML file found. Run fetch_page_content() first.”}<\/p>\n

\tif template.lower() == “plp”:
\n\t\tfields = “product title, product link, product price, product image, product code”
\n\telif template.lower() == “pdp”:
\n\t\tfields = “product title, product price, product description, product image, product color, product size, product code”
\n\telse:
\n\t\treturn {“error”: “Unknown template type”}<\/p>\n

\t# Return the HTML and requested fields so Cursor can analyze them
\n\treturn {
\n\t\t“message”: “Print the XPath expressions for the requested fields using the variable latest_html.”,
\n\t\t“requested_fields”: fields
\n\t}<\/p>\n

In this case, we\u2019re simply defining the desired output structure and the fields for which we want to create the selectors.<\/p>\n

Step 5: Generate the new Camoufox scraper<\/p>\n

Now that we have the selectors, we want to combine a Camoufox Scraper template (Camoufox_template.py) with the selectors to write a scraper with the desired output structure. I learned that using a Camoufox Scraper template is necessary because, without it, the code generated is usually a hallucination and does not work.<\/p>\n

@mcp.tool()
\ndef write_camoufox_scraper(template: str, url: str) -> dict:
\n\tprint(f”[DEBUG] Writing scraper for template: {template} and URL: {url}”)
\n\t“””Reads file Camoufox_template.py and uses it to write a new Camoufox scraper with the requested fields and starting from the url”””
\n\twith open(CAMOUFOX_FILE_PATH, “r”, encoding=”utf-8″) as f:
\n\t\tlatest_html = f.read()
\n\treturn{“message”: “Using this template, write a working scraper with the requested fields and starting URL”}<\/p>\n

That\u2019s it; now we just need to configure Cursor to run this MCP server and write a prompt that uses its tools.<\/p>\n

Step 6. Using the MCP Server in Cursor<\/p>\n

One of the great things about Cursor (an AI-powered IDE) is its built-in support for MCP servers as extensions of the AI\u2019s capabilities. We will integrate our newly created MCP server into Cursor so that the AI assistant can use it to automate web scraping tasks (like generating XPath selectors for us).<\/p>\n

To connect the server in Cursor, follow these steps:<\/p>\n

Open Cursor Settings\u00a0\u2013 In Cursor IDE, go to the settings menu.<\/p>\n

Navigate to MCP Servers\u00a0\u2013 Under the Features section, find\u00a0MCP Servers. (This requires Cursor v0.45.6+ or later, which supports custom MCP server integration.)<\/p>\n

Add a New MCP Server\u00a0\u2013 Click the \u201c+ Add New MCP Server\u201d button. This will prompt you to enter details for the server.<\/p>\n

Configure the server\u00a0\u2013 Give it a name like \u201cScrapy XPath Server\u201d (any friendly name). For\u00a0Type, select\u00a0“command”\u00a0(since we will run a local command to start the server). For\u00a0Command, enter the command to launch your server. For example, if your script is saved at \/path\/to\/xpath_server.py, you might put it in the box.<\/p>\n

Save and verify\u00a0\u2013 After entering the details, save the configuration. You should see your MCP server listed in Cursor\u2019s MCP Servers section. Cursor will now launch this server (via the given command) and maintain a connection to it. You might see a green light next to the MCP server\u2019s name.<\/p>\n

\"\"\"\"<\/p>\n

Once the server is added,\u00a0Cursor\u2019s AI assistant (the \u201cComposer\u201d agent)\u00a0can use the new tools automatically when appropriate.<\/p>\n

In practice, when you ask in the composer, \u201cGet the HTML from this URL and then write the XPath selectors using the template PLP\u201d, the assistant will ask you for confirmation to run the\u00a0fetch_page_content\u00a0tool and then the\u00a0generate_xpath\u00a0one.<\/p>\n

\"\"\"\"<\/p>\n

Our tests<\/p>\n

After a few hours of interactions with the prompt to understand how to get the data needed in a reliable way, I\u2019ve found that this one does most of the tasks I asked for when used with Cloud 3.7 Sonnet (confirming that is one of the best LLMs for coding\u00a0as Marco Vinciguerra from ScrapeGraphAI said in our latest interviews<\/a>):<\/p>\n

\u201cFetch the URL @https:\/\/www.gucci.com\/it\/it\/ca\/women\/shoes-for-women-c-women-shoes
Then, write the XPath selectors using the template PLP. Use the downloaded HTML (downloaded_page.html file) to write working XPATH selectors.
Write the selectors in a file called selectors.txt.
Then, given the selectors previously created, Write a Camoufox scraper using the template PLP and the initial URL. If in doubt, read the selectors.txt file to understand how many fields and which selectors you have to use. Use the file Camoufox_template.py for building the scraper and customize it with the fields in output read by the PLP template.\u201d<\/p>\n

Cursor opens a Camoufox tab, browses to the URL I passed, saves the HTML in a file, reads it, and then generates XPATH selectors and saves them in another file. I opted to save the results of each step inside a file so that the following steps could read them and eventually chunk them automatically rather than passing their content in the context window, which could be too small.<\/p>\n

The scraper created is executable the first time, so it\u2019s technically correct, and the results are quite good.<\/p>\n

\"\"\"\"<\/p>\n

The scraper, without saying anything about the website, correctly populated all the fields, which is excellent and more than I expected from this test.<\/p>\n

I still have some concerns about the solution’s reliability. I don\u2019t get the same results every time I run the same prompt, which can be frustrating because you don\u2019t know how to fix the process if you don\u2019t know what to expect from it.<\/p>\n

The second issue is probably more of a prompt engineering one. The output data structure is not the one I expected, but the one that the assistant found on the Camoufox scraper template. I still haven\u2019t figured out how to solve this, but I\u2019m sure, in some way, it can be done.<\/p>\n

I think we’re done for today. In the next episode, we\u2019ll continue on our journey to make Cursor our web-scraping AI assistant.<\/p>\n","protected":false},"excerpt":{"rendered":"In past episodes of\u00a0The Lab series\u00a0on\u00a0The Web Scraping Club, we used\u00a0Firecrawl\u00a0to scrape the content of this newsletter, a…\n","protected":false},"author":2,"featured_media":9629,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[554,733,4308,86,56,54,55],"class_list":["post-9628","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-technology","tag-uk","tag-united-kingdom","tag-unitedkingdom"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/9628","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/comments?post=9628"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/9628\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media\/9629"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media?parent=9628"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/categories?post=9628"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/tags?post=9628"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}