Auto-Scraping YouTube & Reddit for Obsidian: My N8N MCP Experiments
- 03/08/2025
Description
This video explores a powerful automation stack designed to enhance Obsidian workflows by intelligently scraping YouTube and Reddit. The creator demonstrates how N8N, AI tools like Claude Desktop/Sonnet, and Fresh RSS are used to automatically gather, summarize, and organize content related to Obsidian, providing insights into setup, costs, and the benefits of staying focused and organized.
Best Ideas
1. Automated Content Discovery to Combat Distraction: Set up daily or weekly scrapes for specific topics (e.g., Obsidian videos, top Reddit posts) to get a curated digest, effectively preventing “rabbit holing” and “doom scrolling” inherent in manual searches on platforms like YouTube and Reddit.
2. AI-Assisted Workflow Generation: Leverage AI (like Claude MCP) to generate complex N8N workflows from natural language prompts, significantly reducing the manual effort of building automations from scratch and lowering the barrier to entry for non-programmers.
3. Intelligent Content Summarization: Use AI (e.g., Google Gemini) to process transcripts and posts, generating concise “AI card summaries” that provide quick overviews, allowing for rapid assessment of content relevance before diving deeper.
4. Centralized Information Hubs: Direct scraped and summarized content to dedicated channels in tools like Discord or integrated directly into Obsidian via web clippers, creating easily accessible and organized repositories for review.
5. Cost-Benefit Analysis of AI APIs: Understand the different pricing models (free tiers, pro subscriptions, pay-as-you-go APIs) for AI services (Claude, Google Gemini) to optimize usage and manage costs effectively based on workflow frequency and complexity.
6. Curated RSS Feeds for Focused Consumption: Utilize RSS readers like Fresh RSS to subscribe to specific subreddits, forum categories, or individual YouTube channels, providing a clean, distraction-free feed of desired content without platform algorithms interfering.
7. Self-Hosted Automation Infrastructure: Consider self-hosting tools like N8N and Fresh RSS on platforms like Docker or Proxmox to gain full control over data, potentially reduce long-term costs, and build a robust personal automation lab.
Tools and Resources
- Obsidian: Note-taking and knowledge management application.
- N8N: Workflow automation tool (self-hosted via Docker container on Proxmox).
- Claude Desktop / Claude Sonnet 4 API: AI model used for generating N8N workflows and summaries.
- Modal Context Protocol (MCP): A protocol enabling two-way connection between data sources and AI tools.
- N8N MCP tool (by ZinnLoskey): An AI assistant providing comprehensive access to N8N node documentation for workflow creation.
- Google Gemini AI Studio API Key: Used for AI-powered summarization in the YouTube scraping workflow.
- VS Code: Code editor used for testing Claude Sonnet 4 API and managing JSON outputs.
- Discord: Communication platform used as a destination for AI-generated summaries.
- Fresh RSS: Self-hosted RSS feed reader (via Docker container on Proxmox).
- Proxmox: Virtualization platform for hosting Docker and Linux containers.
- Docker: Containerization platform for deploying N8N and Fresh RSS.
- Node.js: Prerequisite for running the N8N MCP server.
- `npx @n8n/mcp`: Command to launch the N8N MCP server.
- Obsidian Web Clipper: Browser extension or tool to quickly save web content into Obsidian.
- Recall AI: Mentioned as an alternative service for YouTube summaries.
Key Learning Points
- Automation is a Time-Saver and Focus-Enhancer: Proactively setting up automated content gathering can save significant time and prevent distractions, leading to a more focused and efficient information diet.
- AI as an Automation Builder: AI’s utility extends beyond data processing; it can be a powerful tool for *creating* the automation workflows themselves, making complex setups more accessible.
- Cost-Conscious AI Usage is Crucial: Be aware of API costs and free tier limitations for AI services. Tailor your usage strategy (e.g., free tier for occasional use, API for frequent tasks, paid plans for higher volume) to optimize expenses.
- Integration Creates Power: Combining a robust automation engine (N8N) with AI, a knowledge management system (Obsidian), and curated information sources (RSS, Discord) builds a highly effective, personalized information workflow.
- Self-Hosting Benefits: Hosting your own automation and content aggregation tools (like N8N and Fresh RSS) offers greater control, privacy, and potentially more flexible usage compared to relying solely on commercial cloud services.
- Custom Feeds for Precision: Learn how to create custom RSS feeds for specific YouTube channels or subreddits to ensure you only receive the content that is most relevant to your interests, cutting through general noise.