Claude Agent Dual Memory: Milvus + Filesystem Tools

The Breakthrough

The video demonstrates a dual memory system for Claude agents that layers Milvus vector search over filesystem tools to enable persistent retrieval from complex documents with tables and images.

What Actually Worked

• Authors parse documents using LlamaParse, which extracts text from tables, detects images, and screenshots pages with visuals for separate storage.
• Pipeline chunks parsed text, embeds with Gemini embeddings (high-dimensional), and stores in Milvus with schema including source document, extracted text, computed embedding, image file path, and metadata for filtering.
• Agent accesses Milvus tools for semantic similarity search, image retrieval, and filesystem batch tools to scan folders, preview/parse/read documents, and pattern-search contents.
• Retrieval starts with query embedding to Milvus cosine similarity for top chunks; agent reasons over chunks, reads full source documents via tools, and backtracks to missed files.
• Setup uses Claude Agent SDK; adaptable to Claude code or CodeAx via MCP server or skill.

Context

The author addresses limitations of standard RAG on complex PDFs with mixed text, tables, and images by building agentic retrieval. The system pre-filters via vector search to cut costs, then uses tools for precise extraction. Demos on medical documents show handling simple side-effect lookups, FDA vs. ADA guide comparisons, open-ended food interactions, and forced scans for blood thinners across 16 steps. This provides long-horizon reasoning over unstructured data sources.

Notable Quotes

• "The agent initially makes a plan reads the folder then based on that it identifies the documents that it thinks are going to be interesting. It parses those files look at those individually. Then it does semantic search."
• "We use the semantic similarity search to reduce the search space to identify the most relevant chunks. And then we use these file system based tools to read through the documents."
• "Milvus... supports fully distributed Kubernetes native architecture that enables it to scale horizontally which means you can run tens of thousands of search queries on billions of vectors."

Content References

Wait, no: this is in JSON array.