Automated GitHub Trending Projects Data Collection to Supabase

This n8n workflow automates the process of scraping GitHub trending repositories based on different timeframes (daily, weekly, monthly) and stores the extracted data into a Supabase database.

The workflow starts with three schedule triggers that run at specified intervals for daily, weekly, and monthly updates. Each trigger sets a variable ‘type’ indicating the timeframe.

For each scheduled run, the workflow uses Firecrawl to scrape the GitHub trending page corresponding to the timeframe (determined by the ‘type’ variable). The scraping excludes non-essential content to focus on main project data.

The ‘AI Agent’ node then processes the scraped Markdown content, extracting key project details such as project name, URL, description, main programming language, and star count. It also tags each project with its respective ‘type’ (daily, weekly, monthly).

Finally, the workflow inserts this structured data into the ‘githubtrending’ table in Supabase, creating a comprehensive and up-to-date repository of popular GitHub projects for analysis or display purposes.

This setup is particularly useful for developers, data analysts, or teams who want to track trending repositories automatically for research, dashboards, or reporting.