For code smells, which relate primarily to maintainability, most issues were distributed across the ‘MAJOR’ and ‘MINOR’ categories for all models. In contrast, Claude Sonnet 4 and Llama 3.2 90B presented the highest proportion of ‘BLOCKER’ bugs (approximately 14%), which represent defects that can prevent application functionality. This section analyzes the proportional distribution of severity levels (Blocker, Critical, Major, Minor) within each issue type to provide insight into the potential impact of the defects generated by the models. This study argues that the common error patterns in LLM-generated code make static analysis an essential tool for ensuring its quality and security. This research investigates the gap between the output of LLMs and the quality standards for production-ready software. The prevailing discourse on LLMs has largely emphasized their generating capabilities, such as measuring functional correctness , sometimes overshadowing the critical need to evaluate the broader quality and security of their output 12, 13.
Automatic analysis that finds critical vulnerabilities (e.g., injection flaws) and dependency risks that AI models are known to replicate from their training data AI models can introduce subtle security vulnerabilities or hard-to-detect errors that expose your organization to risk. Static analysis may help bridge the gap between the output of LLMs and the quality and security standards often required in professional software engineering. The nature of these observed bugs suggests that LLMs may face challenges with the semantic correctness and runtime implications of the code they generate, potentially prioritizing syntactic plausibility over comprehensive functional integrity. LLMs might generate code that catches overly broad exceptions without https://indianhelpline.in/business-contact/24257-yokogawa-india-limited-yil/index.html reacting appropriately or that disregards checked exceptions, potentially due to challenges in inferring the specific purpose of library-thrown exceptions from training data.
According to Qodo’s research, 25% of developers estimate that 1 in 5 AI-generated suggestions contain factual or functional errors. Our suite of AI agents are guided by quality-focused workflows, underpinned by a platform for context-awareness and codebase intelligence. We’ll uncover in this report, how trust in AI output is directly tied to how accurate, contextual, and reviewable AI generated code is. This is a call for both toolmakers and engineering teams to prioritize trust alongside productivity by investing in continuous review, automated https://www.softcourier.com/68418/details-code-to-flowchart-converter.html testing, and integrated context awareness.
3 Methodology: Code Generation and Static Analysis
Run the same prompt ten times and check how many outputs pass your tests. The next phase of AI adoption won’t be decided by how much code agents can produce. But without traceability, they cannot reliably determine whether AI is improving cycle time, increasing review burden, contributing to defects, or making the codebase easier or harder to maintain.
The comprehensive rule sets of such tools are often designed to cover a wide spectrum of issues—including bugs, vulnerabilities, and code smells—which aligns well with the types of potential flaws identified in this study. As discussed in Section 7, this appears to be a common pitfall for LLMs, possibly related to challenges in discerning the sensitive nature of certain constants. The value of a tool like SonarQube can be seen in its capacity to detect specific, and often critical, flaws of the types observed in the LLM-generated code. An increase in benchmark scores may paradoxically accompany more severe bugs and vulnerabilities, which underscores the critical role of rigorous static code analysis when applying LLMs to software development.
While not representing direct functional errors, they can impede maintainability, comprehensibility, and evolvability, often contributing to the accumulation of technical debt 10, 11 or the introduction of bugs over time. When developers see both fewer errors and higher-quality output, they’re much more likely to trust the AI and use it in production. Context gaps aren’t limited to edge cases—they emerge in the most common and critical development tasks. It outranked trusting AI output, integration with existing workflows, security and compliance concerns, context retrieval, and cost in both samples. The organisations that will thrive are the ones that measure the right things, build continuous feedback loops, and invest in testing infrastructure that matches the velocity of AI generated code. The organisations that maintain low defect escape rates despite high AI adoption are the ones with continuous testing embedded in their CI/CD pipelines.
Analysis of Code Smells in LLM-Generated Java
A closer look at these categories, presented in Table 10, reveals common challenges LLMs face in ensuring the semantic correctness and runtime integrity of generated code. This section provides an analysis of the specific sub-categories of code smells observed, as detailed in Table 9, and explores potential factors that may contribute to these challenges for LLMs. In addressing the most common categories of issues (RQ2), this section provides a detailed analysis of code smells.
Table of Contents
- LLMs may face challenges in this area, as distinguishing between security-critical and routine operational errors can be a subtle distinction that requires deeper contextual understanding.
- Formulating specific exception handling can require a detailed analysis of dependencies across a codebase, which is a challenge for common use of LLMs that operate primarily within a localized context.
- As a critical component of understanding the common issue categories (RQ2), this section analyzes the security vulnerabilities found in the generated code.
- The organisations winning the AI code quality challenge are the ones that have built continuous feedback loops where testing is not a gate at the end of the pipeline but a signal embedded throughout it.
- Aikido catches logic bugs, incorrect conditional checks, edge cases such as null or undefined dereferences, potential runtime errors, and other common code quality pitfalls that are often missed in standard code reviews.
What developers describe instead is reduced trust in their peers’ pull requests, because authorship is unclear, and more difficulty spotting hidden bugs, security issues, and gaps between code and intent. Developers cite incorrect code suggestions and time spent debugging AI generated defects as their top frustrations. When testing is written in Natural Language Programming, every stakeholder, from developers to business analysts to product managers, can read, understand, and validate what is being tested. AerCap, testing complex Salesforce environments, moved from under 20% first time pass rates to 83% by implementing continuous behavioural validation. When AI suggests code blocks, it often reproduces patterns from its training data rather than reusing existing functions within the repository. When developers need to modify duplicated code, they must manually update multiple instances, increasing the risk of inconsistency and error propagation across the entire codebase.
Aikido tells you where the codebase is hard to maintain, hard to read, slow, or likely to break. Critically, smaller can be just as good or better, with smaller models sometimes producing cleaner code for the tasks they successfully passed. This quantitative analysis of five prominent LLMs over 4,442 Java tasks provides clear insights into the quality and security of AI-generated code. As LLMs become more integral to software development, the function of static analysis tools could evolve from a quality and security assurance measure to a component of responsible AI adoption.
- It outranked trusting AI output, integration with existing workflows, security and compliance concerns, context retrieval, and cost in both samples.
- When developers see both fewer errors and higher-quality output, they’re much more likely to trust the AI and use it in production.
- The organisations that maintain low defect escape rates despite high AI adoption are the ones with continuous testing embedded in their CI/CD pipelines.
- While not representing direct functional errors, they can impede maintainability, comprehensibility, and evolvability, often contributing to the accumulation of technical debt 10, 11 or the introduction of bugs over time.
- Context gaps aren’t limited to edge cases—they emerge in the most common and critical development tasks.
Aikido auto maintains your code quality
LLMs may face challenges in this area, as distinguishing between security-critical and routine operational errors can be a subtle distinction that requires deeper contextual understanding. The organisations winning the AI code quality challenge are the ones that have built continuous feedback loops where testing is not a gate at the end of the pipeline but a signal embedded throughout it. Aikido catches logic bugs, incorrect conditional checks, edge cases such as null or undefined dereferences, potential runtime errors, and other common code quality pitfalls that are often missed in standard code reviews. As a critical component of understanding the common issue categories (RQ2), this section analyzes the security vulnerabilities found in the generated code. Continuing the investigation into the most common categories and severity levels of defects (RQ2), this section examines the bugs identified in the LLM-generated https://www.softarmy.com/24113/download-text-file-workshop.html Java code. Formulating specific exception handling can require a detailed analysis of dependencies across a codebase, which is a challenge for common use of LLMs that operate primarily within a localized context.
