Automated Web Crawling & AI Insights with LinkedIn Integration

This n8n workflow automates the process of crawling web pages and extracting valuable insights using AI. It starts by fetching HTML content from specified URLs, including LinkedIn profiles, and extracts relevant information like company details and body content. The workflow intelligently handles missing URLs by performing DuckDuckGo searches, and manages crawl depth to prevent excessive crawling. It queues links for continuous crawling while avoiding duplicates, and stores the page data for analysis. The collected content is then combined, chunked, and used to generate intelligent insights via OpenAI’s language models. Finally, it integrates with Salesforce to associate the insights with accounts, making it ideal for maintaining an enriched CRM with updated company data and social profiles. Practical scenarios include monitoring competitor websites, enriching CRM data with LinkedIn info, or automating market research on targeted online content.