Discussion: consider using `coderabbit.ai`

[error-four-o-four]

Topic

coderabbit provides a free tier which summarizes PRs. This could make it easier to review PRs.

[ksen0]

Hi @error-four-o-four ! Is this a tool you have experience with / how do you think it compares to other tools (like GH copilot)?

I'm curious what others think about this topic, and as we have this discussion I want to keep mindful our code of conduct that applies to online community spaces, and the guidelines on answering questions. The purpose of code reviews on online community spaces is as much to support contributors - particularly new contributors and folks just getting started in open source - as it is to ensure technical correctness. Which is to say, that any tool to help with code reviews has to also support that goal of GitHub as an online community space.

[SableRaf]

Thanks for the suggestion @error-four-o-four :)

I agree with @ksen0, and I’ll just add that PR reviews can often be a good way for new contributors to get involved. Replacing that process with AI risks cutting off a key way people support each other and learn together.

That goes for PR reviews, but also for smaller contributions like fixing “good first issues,” which are often how people take their first steps into a project.

I'm not against using LLMs on principle, I believe we should not use them to replace human interactions and contributor onboarding.

[error-four-o-four]

Honestly I do not have a lot of experience with this tool. I've just seen it being used in another public repository and thought that it might be helpful. This is due to my pragmatic approach and I realize that it's not the best approach.

I'm not against using LLMs on principle, I believe we should not use them to replace human interactions and contributor onboarding.

I totally agree and in case that such a tool would be used it should only act in a supportive manner which means being disabled by default and only give reviews if requested

[GregStanton]

Hi all!

I've been thinking about this issue myself recently, but in relation to API design. While the process I'm envisioning would start in a GitHub Issue, it'd end in a PR review. So I thought it might be helpful to share a few of my ideas about this here.

Although I've been using AI tools myself to help me test and refine my own designs, I only recently started thinking about how such tools could support a larger community and a vibrant design culture. For now, I'll just share a few brief thoughts about the fundamental problem, and how one AI tool might help with both API and code reviews.

Problem

Crafting a thoughtful API is really important for accessibility. It's also really difficult. A concise set of guidelines and examples is a great start, but it’s not enough by itself. In addition to requiring a good understanding of our users, it requires broad contextual knowledge. This includes not just the immediate feature set, but also the wider p5 API, and even familiarity with much broader contexts (the JavaScript ecosystem, graphics literature and tutorials, etc.). It’s difficult for a contributor, or even a set of contributors, to consider all this context every time they make a design decision.

Solution

I've been finding various LLMs to be helpful as design collaborators, but it's often necessary to provide a lot of context, and doing that repeatedly takes a lot of time. So, what's needed is the ability to provide the AI assistant with a persistent context window, ideally including the full p5 API, as well as community-determined API-design guidelines.

As it turns out, this is actually really simple to implement. One option is to create a small, plain-language Markdown file (.github/copilot-instructions.md) based on our existing guidelines and API. That's it. GitHub Copilot will automatically make use of it.

I still need to think through how we might incorporate the use of such a tool as a community, but I suspect we could integrate it into some process improvements that I've already started discussing with others. With the support of API stewards, we could create a simple process that doesn't require contributors to learn new workflows. It'd be based around the following components:

• An API guide in the contributor docs:
  • Guidelines and examples
  • A sentence or two about accessing the API Support Bot, and a short list of tips on when and how to use it
• Small additions to the GitHub Issue and PR templates

If we want to set a policy around using AI to assist with code reviews too, then that's built in as well—we can actually provide custom review instructions in the same file as the API design instructions. I haven't tried out this specific set of features or thoroughly researched alternatives, but so far, I'm finding that even a general chatbot can be really helpful if certain basic principles are followed.

Thoughts?

I'd be very interested to hear any thoughts about this! Like many in this community, I'm generally cautious about the potential pitfalls of AI, but I also want to consider the possible benefits. The first thing I'd like to consider is how an API review assistant could support community interaction rather than taking away from it. A starting point may be to think about when contributors are most likely to need support, and how AI might spark more discussions in those situations.

[limzykenneth]

For me I'd want to see how other repos are using it and whether it adds value for them before I can see us using it. Personally I probably will ignore it for the most part if it's there, partly because I will still want to review the code and run it myself, partly because it's quite opaque what the AI is going to look for. Even more personally, I have no way of building trust with this AI reviewer like I can with all of you here and other contributors as well and that made me nervous 😓