12 Lessons from 5,000 Hours of AI Development
Nate Herk | AI Automationgo watch the original →
the gist
Moving from AI hobbyist to professional requires shifting from building demos to solving business constraints, implementing rigorous verification, and treating AI as an employee rather than a chat partner.
Shift from Demos to Business Outcomes
To stand out in a saturated market, stop collecting portfolio builds and start collecting receipts. A portfolio proves you can build, but a receipt proves you can solve a business problem. Document the before and after metrics for every project, such as hours saved or leads captured. When engaging with stakeholders, identify the actual business constraints—the clogs or leaks in the process—rather than simply building the specific chatbot or automation they request. Define a single North Star metric for every project before writing code to ensure the value is measurable and clear.
Agent Management and Verification
Stop treating AI as a chat partner and start managing it like an employee. Instead of giving direct instructions, define the problem and require the AI to ask clarifying questions before proceeding. To mitigate sycophancy, force the model to play devil's advocate by adopting personas like a skeptical customer or a maintainer. Crucially, implement a verification layer for all agents. If a human would manually test or review a task, build an automated test suite or a screenshot loop to verify the output. Never assume a prompt is a security boundary; use scoped API keys and tool-level permissions to restrict what an agent can physically execute, such as preventing an agent from sending emails even if it can draft them.
Engineering for Reliability and Cost
AI models are non-deterministic, meaning a successful run does not guarantee future reliability. Build a golden dataset of 20 to 500 examples of known good outputs and run automated evaluations (evals) to score system changes. Use an LLM as a judge to grade outputs that require reasoning. For production systems, implement model routing to optimize costs. Match the model to the task complexity—use cheap, fast models like Haiku for grunt work and reserve high-reasoning models like Claude 3.5 Sonnet only for strategic decision-making. This approach can reduce operational costs by 10x or more.