Second Brain

A full-stack personal knowledge workspace featuring automated ingestion workers, hybrid semantic search, context-grounded AI synthesis (RAG), and a subscription-based upgrade funnel.

ReactExpressMongoDBAI Search

Visit Live Source Code

AI-assisted summaries

Semantic retrieval with fallback

Background queue processing

Google OAuth

Overview

Second Brain is a decentralized cognitive extension for modern professionals, developers, and researchers. Built on a modular monorepo architecture, the application aggregates scattered bookmarks, articles, tweets, documents, and transcripts into a unified portal.

By utilizing vector database indexes, it creates a personal semantic web of knowledge—allowing users to search their memories, generate automated AI summaries, and converse with their database through a context-grounded AI assistant.

Problem Statement

With the explosion of daily digital consumption, users suffer from information fragmentation. Bookmarks get lost in browser folders, articles sit unread, and videos lack text searchability.

Moreover, engineering a unified knowledge capture system poses severe technical challenges:

Scraper Blockades: Major platforms (like YouTube, Google, and Notion) block standard data center IP addresses (Render, AWS, Vercel) with captchas and 429 errors.
Credit Consumption & Quotas: Synthesizing deep AI insights is expensive; without strict conversion funnels, manual extraction costs can scale unsustainably.
Account Duplication: Users signing up via email/password often create duplicate accounts when later attempting to authenticate via Google OAuth.

The Goal

To build a resilient, secure personal knowledge system with zero capture friction. The platform must automate ingestion (scraping pages, downloading video transcripts), support fast semantic vector query lookups, and feature a robust conversion funnel that restricts free users to a monthly quota while providing a seamless upgrade route to Pro.

Key Features

Second Brain is packed with robust capabilities designed to minimize friction while keeping the architecture modular and clean:

Resilient Multi-Source Ingestion: Automatic parsing of articles, Notion pages, Reddit posts, Twitter feeds, and YouTube transcripts. Features an automatic public oEmbed API fallback to bypass data center IP blocks.
Descriptive RAG Chat & Cited Sources: A sidebar assistant that answers questions using only stored content, rendering descriptive inline badges with truncated card titles (e.g. [1] Ram Singh Verma...) and interactive source cards showing document details.
SaaS Pro Upgrade Funnel: Restricts free users to 5 manual neural extractions. Exceeding the quota displays a premium, glassmorphic conversion modal directing the user to a tab-linked billing settings panel (/settings?tab=billing) backed by a Razorpay payment gateway and an animated success completion page.
Dynamic Settings & Privacy Panel: Allows users to manage custom profiles, set security passwords, toggle integrations, delete accounts, and update AI preferences (such as auto-tagging default states).

Tech Stack

The architecture consists of standard web layers coupled with background processors and queue workers:

Frontend: React, TypeScript, Vite, React Router, Axios, Tailwind CSS, TanStack Query
Backend: Node.js, Express, TypeScript, MongoDB, Mongoose, Redis (BullMQ queues), Razorpay
Shared Layer: @secondbrain/contracts (Shared TypeScript DTO definitions and interfaces)
AI & Ingestion: Groq & OpenAI APIs, standard Vector Embeddings, Puppeteer & JSDOM, youtube-transcript

Architecture & Workflow

The monorepo separates ingestion services and public API routes to isolate heavy scraping tasks from user interactions:

[User Action] ──> [React App] ──> [Express API]
                                     │ (Check Quota & Deduct)
                                     ▼
[MongoDB Atlas] <── [AI Services] <── [Queue Workers] <── [Ingestion Pipeline]
      │                                                         │
      │ (Create Vector Embeddings)                              ▼
      └───────────────────────────────────────────> [oEmbed API / Puppeteer]

Capture: The user saves a link. The Express API checks the user's plan and deducts credits (if on a free tier).
Ingestion: The link is queued. Extraction workers retrieve transcripts or fetch page content.
Synthesis: The AI summarizes the text, extracts key ideas, identifies difficulty tags, and generates vector embeddings stored in MongoDB.
Interactive Retrieval (RAG): The user queries the chat panel. The system generates query embeddings, performs a hybrid (vector + text) lookup, and feeds the top grounded context to the LLM to write a cited response.

Key Challenges & Resolutions

Resolving complex operational and integration issues was essential to make the application reliable in production:

Challenge 1: Data Center IP Blocks on YouTube Scraper • The Problem: Standard watch page scrapes and transcript downloads failed with 429 Too Many Requests or captchas when executed from deployed server environments (Render/Vercel) due to IP address flagging by Google. • The Resolution: Built an automatic fallback using YouTube's public, unauthenticated oEmbed API. If the standard watch page HTML fetch fails, the worker queries the oEmbed endpoint to retrieve the video title, author, and description, saving the bookmark successfully as processed rather than failing.
Challenge 2: Duplicate Account Creation with OAuth • The Problem: Users who registered with username/password would experience duplicate account creation when later selecting "Sign in with Google", even if they used the exact same email address. • The Resolution: Redesigned the Google callback controller lookup to query both the username and primary email fields. If a matched record is found, it merges the Google integration details into the existing profile rather than writing a duplicate user document.
Challenge 3: Stuck-State Cron Recovery Loops • The Problem: A background cron recovery job meant to re-evaluate failed or stuck extraction items query-targeted all "pending" records, which caused newly saved free bookmarks to automatically process, bypassing the free tier limits. • The Resolution: Restructured the MongoDB query filter to target strictly queued or processing ingestion states, ensuring newly created free bookmarks remain untouched until manual user extraction is initiated.

Performance & UI/UX Optimizations

User engagement relies heavily on latency and design polish:

Hybrid Context Caching: Integrated a 5-minute memory cache map for vector searches on repetitive chat queries to minimize database lookups and speed up response times.
Conversion Modal UX: Replaced native browser window.alert messages with a modern glassmorphic conversion modal containing Sparks icons, checklists of unlocked Pro features, and direct billing tab redirection.
De-congested Navigation: Refactored the dashboard sidebar to compress the profile settings and logout options, maximizing the space available for user folders and saved nodes.

Results & Key Learnings

Synthesizing multiple service APIs taught us the importance of system design beyond simple endpoints:

Resilient API Design: Learned that relying purely on HTML parsing is fragile for web-scale scraping; integrating public metadata API fallbacks (like oEmbed) is critical for ingestion pipeline uptime.
Idempotent Queuing: Designed strict worker queues and job status workflows to prevent credit consumption race conditions when tasks execute asynchronously.
SaaS Funnels: Gained experience in creating elegant conversion loops that elevate standard tool utility into a monetization-ready product.