Zero-Shot Predictive Modeling on Relational Databases

Relational Data as Graph Context

Instead of flattening relational databases into single tables, which loses critical foreign key and primary key relationships, this approach represents the database as a graph. To generate predictions for a specific entity, the model samples the surrounding neighborhood graph using depth-first or breadth-first search methods. This context is serialized into a sequence of cells, which are then passed to a transformer architecture.

Specialized Attention Mechanisms

The core innovation lies in replacing standard full attention with targeted, masked attention mechanisms that process relational data more efficiently:

• Column Attention: The model attends to specific columns across sampled entities to learn feature distributions, such as gender or salary ratios, within the neighborhood.
• Feature Attention: This mechanism aggregates all relevant information about the specific entity targeted for prediction.
• Neighborhood Attention: The model extracts predictive signals from connected entities, capturing relationships that are typically lost in traditional tabular pipelines.

Full, non-masked attention was tested but discarded after showing no significant performance gains, allowing the model to remain lightweight at 22 million parameters.

Performance Comparison

In an item-churn classification task, the model achieved an AUC of 73, compared to an AUC of 62 for a 4-billion parameter language model. The authors suggest this architecture is optimized for integration into agentic workflows, where a smaller, specialized model handles predictive tasks while a larger LLM manages high-level query generation and reasoning.