This n8n workflow automates the process of scraping web data, cleaning HTML, generating custom extraction code, and extracting structured information. It is designed for users who need to extract specific data from web pages without manual intervention, leveraging AI models to generate custom scraper code dynamically.
The workflow begins with a manual trigger, signaling the start of the scraping process. It uses the ‘ScrapeNinja’ node to fetch HTML content from a target URL, such as Hacker News. Once the data is retrieved, it passes through a cleanup process (‘Cleanup HTML’ node) to strip unnecessary elements and prepare the HTML for parsing.
Next, the AI language model (‘Google Gemini Chat Model’) generates custom JavaScript code using Cheerio for extracting relevant data fields from the cleaned HTML, like article titles, URLs, scores, and comments. This code is dynamically created based on a prompt specifying the extraction logic.
Finally, the workflow evaluates the generated JavaScript code (‘Eval generated code to extract data’ node) to run the extraction on the cleaned HTML, extracting structured data that can be further processed or stored.
This workflow is particularly useful for web scraping projects where the structure of the target site may change frequently or when custom, AI-generated extraction logic is preferred over static scrapers. It simplifies the maintenance of web scrapers by automating code generation and execution.
Reviews
There are no reviews yet.