Exploring AI issue resolution with Sentry Seer

As part of our ongoing efforts to explore technology that improves how we work, we recently trialled Sentry Seer, an AI-powered assistant designed to help developers identify and resolve issues in their codebase.

We’ve been using Sentry for years to monitor and debug our applications, so trialling Seer felt like a natural next step. While our team is highly technical, we know not everyone in our wider audience is, so here is a high-level overview of our experience and why we believe this kind of technology is worth paying attention to.

What is Seer?

Seer works within Sentry and scans open issues in your code, assigning each one an "Actionability" score. This score indicates how likely it is that the issue can be fixed automatically. It is clearly displayed on the main issues page, and you can filter issues based on this score. For example, we were able to view all issues with a "High" actionability rating, which are the ones we’re most interested in. 

As part of the scan, Seer also makes an initial guess at the root cause of the issue using the context it has from your codebase. In every case we tested, this initial attempt at working out the issue, was always correct.

How the process worked

From that point, we could click into an issue and ask Seer to generate a solution with a single click. It used a wide range of information to do this, including issue details, tracing data, logs, performance metrics and the structure of our codebase. Sentry calls this information “Context”. The more context you give it, the better results you will get. Like any AI, the better your prompt is, the better the response will be. Seer is no different in this regard. 

Seer also asked for input when needed. In one example, it requested details about the structure of a particular database table. Once we provided the information, it used it to improve its understanding and restarted the solution because it knew it had to tackle the problem in a different way, based on the additional context we gave. These interactions were clear and straightforward, and made it feel like we were genuinely collaborating with the tool.

One moment that stood out to us was when one of our developers began explaining what he thought the root cause of a particular issue might be. At the same time, Seer was already analysing the same section of the code and had drawn the same conclusion. It was a good demonstration of how effective Seer can be when it has sufficient context,  and how closely its thought process can align with a human’s.

If you’re not happy with the fix Seer comes up with, you also have the option to restart the process from scratch. This allows you to give it a fresh attempt, potentially with new or refined context, which can be a useful fallback if it doesn’t get it right the first time. Remember, this is AI we’re dealing with and it doesn’t get everything correct 100% of the time so a human still needs to be involved at some point in the process.

Once a fix has been generated, Seer explains the solution in detail so you can review it before proceeding. It then offers to create the solution and open a pull request in your codebase. After a human verified the fix, we were able to merge several of these pull requests into our codebase, ready for deployment.

Does AI have a place in our lives?

Despite controversies such as AI-generated art using creators' work without consent and fears about job displacement, there is no denying that AI has an established place in modern life.

Julie Zhuo offers a compelling analogy:

‘AI is basically your new insanely productive, cheap yet junior talent that must be aggressively micromanaged, and it can do the work of half the team. This only works if the other half – the human half – is made of classically trained experts (and maybe a few generalists) with the intuition and taste to guide these productive-yet-ignorant underlings toward good work.’

This perspective captures the essence of AI's role. It is not a replacement for human creativity, insight or expertise, but a tool to amplify them. Used thoughtfully, AI can improve productivity and efficiency, but it still requires oversight and input from knowledgeable people.

Feedback & Suggestions

Although our overall experience was very positive, we did have a couple of suggestions, which we have shared with the team at Sentry. One of the developers on the machine learning team replied quite soon after to let us know they are already looking into our feedback:

• Pull request customisation: We would like the ability to tailor the pull requests that Seer creates to fit our existing pull requests templates. This would help us better align fixes with our internal release process.
  
  

• Greater automation: Ideally, we would like Seer to handle high-actionability issues without any manual input. For example, when a highly actionable issue appears, Seer could automatically begin analysing the root cause, generate a fix, and open a pull request. A human could then step in only to review and approve the change. This would make Seer feel more like an autonomous teammate working quietly in the background.

Our final thoughts

Overall, Seer is a fantastic product. As long-time users of Sentry, we were excited to explore how Seer could extend the platform’s capabilities, and it didn’t disappoint. The fact that we successfully merged multiple Seer-generated fixes into our codebase shows that this is more than just a concept. It’s already delivering real results.

Looking to enhance your eCommerce store with practical AI solutions? Discover what Visualsoft can do for you.