New development policy: code generated by a large language model or similar technology (e.g. ChatGPT, GitHub Copilot) is presumed to be tainted (i.e. of unclear copyright, not fitting NetBSD's licensing goals) and cannot be committed to NetBSD.
@netbsd I don't think this was a necessary policy. I think the code should be reviewed on a case-by-case basis. AI in its current state is mostly an advanced completion tool, and I believe it could improve the productivity of developers significantly.
@andrei@netbsd I have a hard time believing that it would help much. In my experience, actually typing the code and getting the syntax right and whatnot is hardly what takes time, you can type pretty much as fast as an LLM can generate tokens once you know what you want.
The part that actually takes time is understanding the needs correctly, reasoning about a possible solution and its impacts, reflecting on all possible edge cases, etc. No LLM can replace humans at that part.
@iwein This is a hiring policy - it's part of the developer contract that all new members of the Foundation are required to sign. Foundation membership is required for commit access.
@netbsd Figuring out code is tainted by use of copyrighted code from another source is as straightforward as string-matching, maybe some fuzzy matching.
How would one identify code generated with the assistance of an LLM if the contributor doesn't admit to doing that?
@mark This is one of the sets of rules that every person with commit access has to follow. Becoming a committer is not easy, it requires joining the Foundation and signing various contracts that place the burden of responsibility on the member. It's a fairly reasonable assumption that we should be able to trust our members, and if not they shouldn't be members.
@ParadeGrotesque@asmodai You would have to automate it in that case. I doubt reviewers want to sit and prove that some code was generated by an LLM. Also it could be just a few lines that has been generated, no way to prove that and given the enough context of the code the LLM and a human may come up with exactly the same result. Would the code by denied in that case then?
@ParadeGrotesque@asmodai If you did something like telling ChatGPT to "generate me a bubble sort in C" and did a copy/paste of the code is likely easy to spot, but for more subtle cases it will be quite hard imo.
True but on the other hand, something tells me NetBSD developpers are unlikely to rely too much on ChatGPT - a portable system is not something I see LLM as being able to produce code for.
The NetBSD code in general is reportedly of a high standard as dev place an emphasis on correctness. I think anything generated by ChatGPT would stand out like a sore thumb.
To be clear: NONE of my code will ever make it into NetBSD either! 🤓
@asmodai Still think it would be hard. Sure if you can "backwards" prove that some specific input generates exactly some code that someone commits without any changes then maybe, but usually you don't write code that way.
You implement something and then you change stuff to what you want it to do. In general using LLMs to do algorithms is a bad idea.
But using it for generating boilerplate (i.e repeating code patterns) and test code is very useful and doesn't affect the "real" code.
@daniel_collin@asmodai This is one of the sets of rules that every person with commit access has to follow. Becoming a committer is not easy, it requires joining the Foundation and signing various contracts that place the burden of responsibility on the member. It's a fairly reasonable assumption that we should be able to trust our members, and if not they shouldn't be members.
Add comment