As a contributor who had to deal with 2 automated reviews in the past month, it's a bad experience. I really wanted to upstream my changes, but these bots made me question if i should at all.
Bots don't understand how their words can hurt people's feelings, and can even less understand that talking to a LLM is even more sad/infuriating for actual humans. I can handle CI/tests just fine, but LLMs playing on my human emotions will have me swearing in no time.
I contribute to free software projects because i want user control over the machine (that was the idea, long before LLMs). Slopware might technically be FLOSS, but it's certainly not in the spirit of FLOSS.
I haven't been exposed AI reviews in OSS but I like them for work. Quality of PR review feedback is the most revolutionary bit of AI I think. More so than automated coding. I guess one issue with the AI reviews in OSS is that your dialog (I assume) is public. You know the interaction is with a bot but you also sense it may be observed by humans.
If I get a review at work from a bot that is annoying, misunderstands, etc I just ignore their reviews, or tell the bot to leave me alone. The bot is there to help me, and if it doesn't - that's not my problem. There's also always a human reviewer. For any automated feedback for PR's I think there should be "manual review" requests. If you disagree and the bot can decide that there is real human effort involved, then just let real humans also make the effort to review.
This is the key idea: to spend the human effort from maintainers, where there has been human effort from contributors. The AI should be able to determine the amount of human effort in a contribution. The bot should be an aid to make sure the human maintainer effort is spent only where there is human contributor effort.
So i'm critical of LLMs even if you consent to their use (for ecological/political/cognitive reasons). But in that case at least it's your tool and your expectations. I'm not even sure the maintainers had a say in the random LLM interactions i had on their projects, but i certainly didn't consent.
If the maintainer wants a LLM to review my PRs, let the maintainer use a LLM. Don't make me deal with abrasive chatbots when you want to deal with abrasive chatbots. I don't want an LLM to pre-screen my work to decide if a human should review it; a human should decide if my contribution is worth it. Otherwise, that project is not worthy of my modest contribution.
> a human should decide if my contribution is worth it. Otherwise, that project is not worthy of my modest contribution.
But that's the problem at hand: if there are 999 low-effort automated AI contributions for every human contribution, either all 1000 contributions must be pre-screened to gauge "human effort" and the human efforts get human review, or all PR's get 1/1000 the human attention.
I agree it's still a problem, but closing obviously-LLM PRs and banning the "contributors" would help i guess. What would help even more is hosting your project on a network where that's bad etiquette, eg. Codeberg.
> Bots don't understand how their words can hurt people's feeling
That's a super weird way of putting it, which makes me hesitate to agree with you.
buuuut I agree that the LLM bot review experience is usually miserable, as the persona these things use is insufferable. It's a know-it-all only-speaks-in-imperatives thing, where the speaker lacks the actual merit that would grant them the right to that style.
LLMs can be very useful for code review, but the existing product implementations took a bit too many pages out of Kafka's works and are a bit too real in simulating a dysfunctional bureaucratic org.
> That's a super weird way of putting it, which makes me hesitate to agree with you.
Yes that's weird sorry. What i meant is it belongs in the uncanny valley. I don't have any good/bad feelings about test suites and CI feedback, it's very binary. Receiving a mail about a comment on my PR raises my hopes that it reached the maintainers (or interest from other contributors); receiving LLM text destroys my expectations.
Then, about the content itself, it has a very patronizing tone. Saying out loud every single thing that's in my +25/-3 PR is really infantilizing. I would probably have slightly less negative reactions if it only commented to suggest improvements, instead of patting me on the back. See this example: https://github.com/TwiN/gatus/pull/1712#pullrequestreview-46...
Bots don't understand how their words can hurt people's feelings, and can even less understand that talking to a LLM is even more sad/infuriating for actual humans. I can handle CI/tests just fine, but LLMs playing on my human emotions will have me swearing in no time.
I contribute to free software projects because i want user control over the machine (that was the idea, long before LLMs). Slopware might technically be FLOSS, but it's certainly not in the spirit of FLOSS.