This is a most excellent place for technology news and articles.
From the article:
The policy summarizes itself this way:
It's fine to use LLMs to answer questions, analyze, distill, refine, check, suggest, review. But not to create.
Uses in the first category are allowed, sometimes requiring disclosure. Uses in the second category are heavily restricted.
Seems pretty level-headed to me.
Interesting policy.. I suspect it falls apart in the details though. Like, there are plenty of creative elements in responding to questions (and any of the other categories). That mean there will be lots of weird edge cases that are hard to adjudicate. And those GenAI components of "non-creative" categories will have ripple effects into the broader project and community (as they are everywhere, so maybe not that important for a.policy to grapple with, IDK).
It's not like it is a legally binding agreement that needs a high level of quality in adjucation. If things seem off then it can be cited and you don't have to prove it. Conversely, if something passes human review despite being codegen, well then I guess the codegen isn't that bad
I've seen a pull request that was ultimately nonsensical but plausible and short, and purported to be needed in a configuration none of the maintainers had access to. It was a pretty harmless change either way so they were inclined to accept it. But they sought a little clarification and the clearly LLM response in comment got it shut down. No human stepped up to actually explain an understanding and the LLM sounded like a TV show writer writing tech, sentences that sounded credible but to anyone with understanding was nonsensical bullshit.
LLM code can sometimes be hard to tell in a short code change, but the prose when things are anything vaguely needing explanation is just so very very obvious.
"refine" vs "create" is already a super grey area
That mean there will be lots of weird edge cases that are hard to adjudicate.
Humans a pretty decent at that. Source: Of all things, moderation, where even if you have nice rules written up, it feels like half the reports are edge cases and you just have to make a call.
I suspect they'll be able to handle making those calls. Get some wrong, get most right, that's probably good enough.
Yeah in theory I could have a model suggest code through a PR and then review said PR and just be the one to hit squash and merge and it technically only suggested and reviewed and I “created” the PR
But if you actually described this methodology in your PR, then it'd no doubt be closed immediately, based on the spirit of the rules rather than the exact letter of the rules.
Rules like these ultimately presuppose that the contributor is acting in good faith. Because otherwise the contributor could just lie about using an LLM. They don't need to engage in rules-lawyering or jump through hoops to circumvent the rules
Rules like these ultimately presuppose that the contributor is acting in good faith.
All of civilization is based on that.
Yes, and even the less honest contributor would need to manually review and modify most generated blocks to pass a sniff test. The same has always been true of “found” source code.
That is, even contributors who read the policy as “don’t get caught” still have to meet the same de facto standards, which to a maintainer is a satisfactory result.
Example: I block illegal websites on my network, not because I care whether my roommates or their friends access illegal websites, but to require they practice proper hygiene (i.e., they need a VPN and encrypted DNS to circumvent) so we don’t amass love letters from our ISP.
Projects that don’t prohibit LLM-generated code expose themselves to greater liabilities than mere low-quality contributions.
Agreed… although this still leaves the horrible environmental effects and losing any kind of ownership ofethe code
But solely in terms of using "ai" as a tool, it is a sensible approach
What counts as creating? How can one refine without creating?
Well a light refactor of deduplicating code would be a refinement that is generally easy to review and likely comfortably in the capability of the codegen.
Versus creating, which generally is harder to review and requires the submitter to explain the rationale behind the change.
Volume of code and nature of change.
With "refine", I would guess they're talking about refining a task, so breaking it down into smaller, more concrete/actionable tasks.
Because that's usually meant in dev lingo, and because the other possible meaning of improving already-written code/docs is kind of already covered by "review".
