Automated Collection and Interaction with Paul Graham’s Essays

This n8n workflow automates the process of scraping, storing, and interacting with Paul Graham’s essays using AI and a vector database. It begins with a manual trigger to fetch the list of essays from Paul Graham’s website, extracts essay URLs, and retrieves their full texts. The workflow limits fetching to the first three essays for efficiency.

The essay texts are cleaned to extract only relevant content and then split into manageable chunks using a recursive text splitter. These chunks are embedded into vectors with OpenAI embeddings and stored in a Milvus vector database, filtering or updating the collection as needed.

Once the data is stored, the workflow is set up to listen for incoming chat messages via a webhook. It uses a Langchain AI agent that interacts with users, retrieving relevant essay chunks from Milvus, embedding new query data, and generating responses with OpenAI’s GPT-4 model. This setup enables a conversational AI that can answer questions based on the stored essays.

Practical use cases include research, education, content analysis, and AI-powered Q&A with a specific corpus of essays or documents. The workflow can be activated to fetch and index new essays, then interact with users to provide insightful responses based on the stored knowledge.