Study finds that Github Copilot lowers code quality and maintainability:
"We find disconcerting trends for maintainability. Code churn -- the percentage of lines that are reverted or updated less than two weeks after being authored -- is projected to double in 2024 compared to its 2021, pre-AI baseline. We further find that the percentage of 'added code' and 'copy/pasted code' is increasing in proportion to 'updated,' 'deleted,' and 'moved 'code. In this regard, AI-generated code resembles an itinerant contributor, prone to violate the DRY-ness [don't repeat yourself] of the repos visited."
The paper concludes: "How will Copilot transform what it means to be a developer? There's no question that, as AI has surged in popularity, we have entered an era where code lines are being added faster than ever before. The better question for 2024: who's on the hook to clean up the mess afterward?"
@tante Turns out that 'garbage in, garbage out' also applies to LLMs and their training/RAG.
And this example is not the worst quality. When using a language other than English or a poorly documented/understood niche topic they all fail miserably.
Rubbing AI on a data landfill (no, it's not a lake) just exposes the garbage more quickly. Though I guess that does have value if you want to clean up. ;)
@tante What they're not taking into account is that those things are intentional.
When a human churns code, it's a waste of payroll. When a machine does it, it's scaffolding that holds up an undeveloped feature while the human works on developing a more important one. Of course it's going to need to be replaced later, but that doesn't mean it's slowing down development or making the codebase worse.
Like any tool, AI can be used properly or improperly. Used properly, it's a great efficiency boost.
Add comment