Building Graph-Based Context for AI Agents on Lakehouse Data

The Problem: Contextual Blindness in Agents

Modern AI agents often fail when interacting with large-scale data because they lack a holistic view of how information connects. While vector search retrieves similar snippets and Text2SQL queries structured tables, both methods treat data as isolated slices. This leads to 'confident but wrong' answers, especially when the agent needs to perform estate-level reasoning, identify missing information, or traverse complex join paths across massive schemas.

The Solution: Context as Shapes

Instead of relying on raw queries, the workshop introduces the concept of using 'graph shapes' to provide an agent with a semantic map of the data. By modeling metadata and relationships in a graph database (Neo4j), developers can give agents a structural understanding of the data environment. This approach is agnostic to the underlying storage, working effectively with BigQuery, Databricks, or Snowflake.

Three Essential Graph Shapes

1. Trees (Table of Contents): Used primarily for unstructured data, this shape provides a hierarchical structure. It allows the agent to navigate the 'what' and 'where' of the data, helping it understand the scope of the available documentation before attempting to query it.
2. Communities (Thematic Surfacing): Using algorithms like Leiden, this shape identifies latent patterns and groupings within the data. It allows the agent to answer questions about global trends or recurring issues that aren't explicitly labeled in the database.
3. Paths and Cycles (Semantic Layer): This shape acts as a semantic layer over structured data. By mapping table schemas and join paths as a graph, the agent can correctly navigate complex SQL relationships without needing to guess how tables connect.

Implementation Workflow

1. Environment Setup: Use a GitHub Codespace pre-configured with Neo4j, Anthropic API keys, and BigQuery credentials.
2. Semantic Layering: Utilize tools like NeoCarta to ingest metadata from your warehouse into a graph. This creates a 'metadata graph' that the agent can query to understand the schema before executing SQL.
3. Agentic Coding: Instead of manual Cypher coding, leverage an agent (e.g., Claude Code) equipped with specific skills (Neo4j CLI, GDS library) to generate queries based on the graph structure.
4. MCP Integration: Expose the graph as a Model Context Protocol (MCP) server, allowing the agent to 'see' the graph structure as a native tool for reasoning.