Building a Personal AI Research OS: From Notes to Knowledge

The Problem with Passive Knowledge

Most developers suffer from 'note hoarding'—collecting thousands of articles, GitHub repos, and meeting transcripts that remain inaccessible when needed. Standard RAG pipelines or browser-based tools like NotebookLM are either too complex to maintain or lack the agentic flexibility required for deep, multi-step research. The goal is to move from a passive archive to a 'Research OS' where an agent can autonomously navigate your personal data to synthesize new insights.

The Three-Layer Architecture

The system evolves from simple scraping to a structured, persistent knowledge base.

1. Raw Layer: Immutable storage of all source material (Obsidian notes, PDFs, YouTube transcripts, GitHub repos).
2. Index Layer: A centralized index.yaml file that acts as the agent's map. It contains metadata, summaries, and pointers to raw files, allowing the agent to reason about what data is available without needing a heavy vector database.
3. Wiki Layer: A synthesized output layer where the agent creates 'derivative' notes—comparisons, concept deep-dives, and entity mappings—that compound over time.

The Deep Research Algorithm

Instead of a single-shot query, the system uses an iterative loop. An orchestrator agent breaks a topic into sub-questions, gathers context from both the public web and the user's local 'Second Brain,' and performs multiple rounds of research. Crucially, it uses a ranking algorithm to filter high-signal information, scraping only the most relevant content to keep token usage efficient and context windows clean.

Why File-Based Over Vector Databases

For personal use, vector databases introduce unnecessary infrastructure overhead. By using a YAML-based index and standard Markdown files, the system remains human-readable, easily version-controlled via Git, and natively compatible with local LLM agents (like Claude Code or Codex). This ensures the 'memory' is portable and inspectable by the user at any time.