Automated Hacker News Hiring Scraper and Data Structuring

This n8n workflow automates the process of scraping, extracting, and structuring ‘Who is Hiring’ posts from Hacker News using Algolia Search and the Hacker News API. It begins with a manual trigger aligned with testing purposes. The workflow guides users through setting up the data source by visiting the HN Algolia search and copying API call cURL commands. It then fetches relevant posts related to hiring, filters recent posts from the last 30 days, and processes nested comments and job posts. Using OpenAI’s GPT-4 mini model, it transforms raw text into a structured data format that includes details like company name, job title, location, employment type, salary, description, apply URL, and company URL. The data is cleaned by removing HTML tags, special characters, and extraneous whitespace, ensuring clarity and consistency.

Finally, the structured data is stored into Airtable for easy viewing and management. The workflow is ideal for recruiters, developers, or startup teams who want to automate monitoring and extraction of hiring posts from Hacker News, thereby saving time and improving data accuracy for talent sourcing or market analysis.