Second Brain

Overview

Problem Statement

Goal

Solution

Features

• Content capture pipeline for posts, videos, and documents with URL normalization and metadata extraction.
• Semantic search using vector search with text-search fallback for meaning-based retrieval.
• AI synthesis workflow that generates summaries, tags, and topic intelligence for saved content.
• Background ingestion via queue worker to avoid blocking user actions.
• Google OAuth support for sign-in and integration flows.
• Share controls with private, link-based, and public visibility modes.
• AI chat over the personal knowledge base with source-aware responses.
• Manual fallback path for protected or unreadable sources so users can paste content and continue processing.

Tech Stack

• Frontend: React 19, TypeScript, Vite, React Router, Tailwind CSS, React Query.
• Backend: Node.js, Express 5, TypeScript, Mongoose, Zod, JWT, bcrypt.
• AI / Processing: OpenAI or Groq providers, Xenova/all-MiniLM-L6-v2 embeddings, BullMQ, Redis.
• Database: MongoDB with text indexes and vector-search compatible flow.
• Shared package: @secondbrain/contracts for DTO and enum consistency.
• Deployment: Vercel for the frontend, Render for the backend and worker, MongoDB Atlas, and Redis.

Architecture / Workflow

• Routing: React Router handles /, /signin, /signup, /share/:id, OAuth callbacks, and integration callbacks.
• State management: React Query handles server state, while local component state and localStorage hold UI preferences.
• API flow: Axios interceptors normalize errors and inject bearer tokens automatically.
• Authentication: Username and password auth uses bcrypt plus JWT, and Google OAuth uses a callback and code exchange.
• Database interaction: Mongoose models store users, content, and share links with performance-oriented indexes.
• Data flow: content is saved immediately, then background AI and embedding jobs enrich it asynchronously.

Challenges Faced

• Reliable AI retrieval across environments was solved with a three-tier retrieval strategy: vector search, text search, then in-memory cosine and recency fallback.
• Latency versus UX quality was solved by decoupling ingestion into background workers and showing progressive AI states in the UI.
• OAuth complexity across login and integration use cases was solved by separating login scopes from account-connect scopes.
• Protected-source ingestion limitations were solved with a manual-content fallback in the UI.
• Consistency between frontend and backend contracts was solved with a shared contracts package in the monorepo.

Performance & Optimization

• Debounced semantic search to reduce noisy API traffic.
• React Query stale-time tuning and selective invalidation to minimize unnecessary refetches.
• Optimistic UI updates for edit and delete actions.
• Targeted polling only for currently selected content under AI processing.
• Background queue processing for heavy AI work.
• Puppeteer scraping optimization by blocking non-critical resources like images, fonts, and styles.
• Progressive rendering patterns and responsive layout behavior for desktop and mobile flows.

UI/UX Decisions

• Single-dashboard workflow so creation, search, filtering, and AI insights happen together.
• Clear processing states such as queued, processing, and failed to improve trust during asynchronous AI work.
• Mobile-first sidebar controls and adaptable layout for smaller screens.
• Direct source access from cards to preserve transparency and verification.
• Manual synthesis entry when automated extraction fails so the user never hits a dead end.

Security Considerations

• Passwords are hashed with bcrypt.
• Protected endpoints use JWT middleware and bearer-token verification.
• Request payloads are validated with Zod to reduce malformed-input risk.
• OAuth integration stores encrypted token fields where available.
• CORS is configured using allowed frontend origins.
• Token handling is centralized through API client interceptors and backend middleware checks.

Results / Outcomes

• Reduced friction from save now, forget later by enabling semantic recall and contextual insights.
• Established a maintainable monorepo architecture with shared contracts and clean frontend/backend boundaries.
• Created a flexible foundation for adding more integrations and advanced retrieval features.

What I Learned

• Graceful fallback paths matter more than optimistic model assumptions.
• Queue-based processing is essential when balancing user-perceived speed with heavy compute.
• Shared type contracts significantly reduce integration bugs across frontend and backend.
• Good AI product UX depends on communicating state, confidence, and failure modes clearly.

Future Improvements

• Add per-user analytics on retrieval quality and insight usefulness.
• Introduce stronger token-hardening policies with encryption-required mode.
• Add role-based access and team or shared-workspace support.
• Expand connectors beyond Google with Notion, Slack, and GitHub.
• Improve AI ranking with reranking and hybrid retrieval tuning.
• Add end-to-end observability dashboards for queue latency, provider errors, and search success rates.

Deployment

• Frontend: deployed on Vercel with SPA rewrite rules for client-side routing.
• Backend API and worker: deployed on Render as separate services.
• Data and infrastructure: MongoDB Atlas for persistence and Redis for queue orchestration.
• Environment management: separate configs for API URLs, OAuth, JWT secrets, AI providers, and queue settings.