Preventing AI-Driven Cloud Cost Spikes
Your Average Tech Brogo watch the original →
the gist
The author incurred a $20,000 cloud bill by allowing an agentic AI to recursively scrape and classify every video from hundreds of social media accounts without cost-based monitoring or architectural guardrails.
The Breakthrough
The author discovered that delegating end-to-end feature implementation to agentic AI without strict architectural oversight led to an unmonitored recursive inference loop, resulting in a $14,000 monthly Google Cloud bill.
What Actually Worked
- Implement LLM-specific monitoring: The author integrated PostHog to track LLM usage and configured alerts for anomalous spikes, specifically setting thresholds at over 3,000 requests per hour or costs exceeding $100 per hour.
- Adopt a 'Grill Me' requirement phase: Before implementation, the author forces the AI to act as a reviewer using a 'Grill Me' prompt skill, which requires the AI to challenge the user on the product and technical requirements of the proposed feature.
- Enforce granular phased planning: The author uses a custom 'Phased Plan' skill to break complex features into extremely small, manageable work streams, ensuring that each pull request is small enough for human verification.
- Shift from exhaustive to outlier processing: The engineering team modified the viral content database logic to ignore high-volume, low-engagement posts, opting to only perform expensive inference on high-performing content outliers.
Before / After
- Monthly Cloud Spend: The author incurred a $6,000 bill in March and a $13,999 bill in April 2026, totaling nearly $20,000 due to unmonitored Gemini 3.1 Pro inference costs.
- Monitoring: Previously, the author relied on a $200 monthly budget alert that only triggered on credit card charges, failing to account for startup credits. The new system uses real-time product-level alerts based on request volume and cost metrics.
Context
The author built a social media marketing tool that automatically scraped and classified hundreds of videos per account using Gemini 3.1 Pro. Because the system was built via 'vibecoding'—letting the AI handle architecture and implementation without human review—the recursive nature of the scraping process went unnoticed until the credits were exhausted. The author now prioritizes engineering quality over velocity by keeping human-in-the-loop oversight for all system architecture decisions.