Automated Generation of AI-Ready Content Files from Crawled Websites

This n8n workflow is designed to streamline the creation of ‘llms.txt’ files from Screaming Frog website crawl exports. It helps SEO and content teams generate structured, AI-friendly content lists that can be used for training language models or enhancing content discovery. The process begins with a user-submitted form, where you provide your website details and upload a CSV export from Screaming Frog. The workflow then extracts, filters, and processes URL data, selecting only the most relevant pages based on status, indexability, and content type. Optional AI-based content evaluation further refines the URL selection, classifying pages as useful or low-value. Finally, it formats the gathered data into a structured ‘llms.txt’ file, ready for download or upload to cloud storage. This workflow is particularly useful for SEO professionals, content strategists, or webmasters seeking to leverage AI for content optimization, site analysis, and training purposes.