Evaluating LLM Code Quality and the ACDC Framework
AI Engineergo watch the original →
the gist
Sonar evaluated 53 LLMs against 4,444 Java assignments, finding high variability in security and verbosity. They introduced the ACDC framework (Guide, Verify, Solve) to automate code analysis and remediation before and after commits.
LLM Code Quality Analysis
Sonar conducted an evaluation of 53 LLMs using a dataset of 4,444 Java programming assignments to measure functional correctness, code bloat, and security vulnerabilities. The findings indicate that while some models achieve high functional pass rates, they often produce significant technical debt and security risks. For instance, Claude Sonnet 4.6 generated 627,000 lines of code with a security issue rate of 300 per million lines, while GPT 5.4 produced 1.2 million lines for the same task. The research suggests that models frequently inherit security flaws and subtle logic errors from their training data, leading to inconsistent code quality that fails to meet enterprise engineering standards.
The ACDC Framework
To address these reliability issues, Sonar introduced the ACDC (Guide, Verify, Solve) framework designed to integrate into agentic development workflows:
- Guide: Uses context augmentation and Sonar Sweep to ensure the model is trained or prompted with high-quality, secure data and relevant codebase context.
- Verify: Implements agentic analysis that runs SonarQube checks in 1 to 5 seconds before code is committed, significantly faster than traditional 1 to 5-minute CI pipelines.
- Solve: Employs a remediation agent that generates fixes for identified issues, automatically runs them through compilation and analysis to prevent regressions, and only presents the fix to the developer if it passes all quality gates.
This framework allows developers to address technical debt by selecting issues from the SonarQube dashboard and assigning them to the agent, which then generates individual pull requests for each fix.