Why is bug reporting so difficult?

Finding a bug and reporting it are two different tasks. A recent experience on a motorbike trip gave me a new perspective on the gap between encountering a bug and reporting it — and got me thinking about how we can make the journey to a bug fix a little smoother. 

When I took a motorbike trip to Agile on The Beach, I had a pretty nerve-wracking afternoon the day before departure. I had taken my bike to the garage for maintenance and a new rear tire the previous week and decided to take it out for a quick test run.

Something weird happened: after moving from a 60km/h to a 30km/h zone, my bike started stalling and I couldn’t get it to accelerate properly. I had to stop by the garage for an unrelated reason anyway, so while I was there I told my mechanic about it. I told him it seemed weird, but I also felt a bit silly. I had just gotten the bike checked out the previous week, surely nothing was wrong with it! We agreed it was probably driver error, maybe I was in a higher gear than I’d thought.

On the way home, shortly after leaving the garage, it happened again. This time I was slightly more prepared so I tried to pay closer attention to what was happening. I knew what gear I was in this time. Even in first gear, I couldn’t get the bike to run smoothly. I feared it could be a brake issue, so I decided to go back to the garage to have the mechanic take a second look. He took the bike for a test run but didn’t feel anything wrong with it. Weird, he agreed, but maybe I’d gotten the gears mixed up again?

At this point I had a gut feeling that it was definitely the bike, but I felt a bit silly for insisting, so I left.

When the issue happened again, I was ready for it. I had taken the quieter backroads so I felt safer experimenting and lo and behold: the bike showed the same behavior. But this time, I had the chance to record a video of what was happening so I could show the mechanic. 

I went back to the garage and finally had some evidence. The result: my mechanic ended up investigating and fixing the rear brake, which he discovered was getting stuck. So why was it so hard for me to get my mechanic to realize there actually was a problem? 

Bug reporting is trickier than you might think

Reporting a bug is actually really difficult. It’s hard to describe exactly what went wrong because we often don't know — we only know that we encountered an error. As a user, we know it felt off, but that doesn’t necessarily tell us what exactly went wrong on a system level. 

I felt my bike stall after moving from a 60km/h zone to a 30km/h zone, but I couldn’t tell what had caused the problem. Was it braking? Switching gears? Perhaps something else I wasn’t aware of?

Encountering a bug can be stressful

When my bike started acting up while I was driving it, my first instinct was to scream “What the **** is going on?” rather than “let me gather some data about this error.”  When you’re stressed, it’s hard to think about all the steps that lead you to this frustrating moment. 

While finding a bug in a controlled environment is not quite as nerve wracking as being on a malfunctioning motorbike in traffic, the situation can still be stressful. Think about trying to buy concert tickets. If you've spent time registering for an account only to encounter an error that blocks you from completing the purchase, you're going to feel stressed out. You're not going to think about what is causing the error, only that the bug could cost you those perfect seats.

When you’re facing a bug, it’s tricky to remember every single step you took and the result of each step. You might have to repeat the process to find the bug again, but the exact process might not be easy to remember. Your mind fills with questions: What factors were at play? What was the exact sequence of your steps? Did you click “save” or did you hit enter? Are you sure you even clicked save? Maybe you hit cancel! Were you even in the correct environment?

Various factors can result in a lack of trust in bug reports

Pride-based doubt

The expert does not necessarily expect the product to malfunction. My mechanic likely believed that since he had just done maintenance on my bike, user error was more likely than equipment malfunction. And yet, the equipment ended up being the problem. 

Often, software developers don’t expect their software to malfunction. They worked hard on that code and have worked through the logic of the functionality, so hearing that it needs improvement is never fun. It can feel like criticism — and it’s coming from someone who doesn’t even write code themselves! This perspective can result in bug reports being doubted. Developers may question if you’re sure about what you saw. 

This is not to say that their work is subpar or that they shouldn’t be proud of it — only that being the person whose work is put into question can cloud perceptions. 

Addressing doubt

One way to make the whole process smoother is to change the perception. Rather than immediately telling a developer “hey, I found a bug,” I start by asking them to walk me through a flow instead. This approach leaves space for any possible mistakes on my part, and if we do encounter the bug, we’ve discovered it together. 

I ask the developer to work with me in real time (in person or via screen share), so I can go through the steps I’ve taken, explain what I’m expecting to see, and show what I’m actually seeing. 

By taking this approach, we can first clear up if there were any misunderstandings or mistakes on my part. If it turns out there truly is a problem, we've established that together and it feels less like a confrontation. Rediscovering the bug together is also helpful in the long term because now someone else has witnessed it first-hand, leaving less room for misunderstandings when it comes to addressing a fix. 

Another thing I do is anticipate the questions I know I'm going to be asked. Rather than waiting for a back and forth about whether I’m in the right environment, using the correct login, etc, I provide the information upfront. I list all the things I've already tried and explain how/why it didn’t work so we can save time and energy. 

Bias

Doubt in a bug report can also be exacerbated by some type of bias against the reporter.

In my experience, women often aren’t taken as seriously in male-dominated fields, whether it be in software development or a motorbike garage. Society tends to believe that men know more about technology and machinery than women do. As a result, people assume that women’s lack of knowledge or skill is the issue rather than the product. 

Unfair bias affects other underrepresented groups as well — different cultures, levels of education, past experience, etc. can all influence whether or not the bug report is trusted. 

Unfortunately, testers sometimes internalize this lack of confidence. Years and years of not being taken seriously makes testers like myself question our own findings. Deep down, we know what we saw, but it’s hard to make ourselves (and thus, others), believe us. 

How to address bias

This is a tough problem to solve, partially because the lack of confidence is coming from both sides. On the tester side, I try to be as objective as possible. I gather all the data, examine it myself, and gain confidence in my report.  When my bike was malfunctioning, it took a third encounter of the issue before I was certain I could go back to the garage and insist there was a problem.

One of the challenges underrepresented minorities face in tech is that we're not always given the benefit of the doubt. If we mess up, it's seen as evidence of our relative incompetence, rather than just a one-off mistake. Meanwhile, our more privileged colleagues are not under the same level of scrutiny.

On a somewhat grander, societal level, we can try to improve the situation for everyone by showing solidarity. When we see a colleague being dismissed more often than others — perhaps because of their skin color, gender, culture, level of experience, or some other reason — we need to call it out. 

While I know it’s scary (and not always feasible in terms of job security), it’s important that we call out bias when we see it happening. And of course this is where our allies come in — people in more privileged positions who raise their voice in these situations can make a real difference.

Assumptions of normal use

Lastly, there’s assumptions about what normal use is. My mechanic asked if I used my rear brake, which I found odd because, yes, of course I do. As it turns out, he does not use his. When he took my bike for a test run to see if he could observe the problem, he didn’t notice anything wrong, because in his typical use case, it simply didn’t occur. 

Now imagine how this transfers to testing. If I’m working with a developer to reproduce an issue on their machine and their setup is slightly different, they use a different CLI, they access the database via their IDE instead of the tool I use, or any other variance, it could affect the test. We can’t assume what’s “normal” use. 

How to address assumptions of normal use

To get ahead of assumptions, I provide as much detail as I can about the steps leading up to discovering the error. Ideally, I can pair with someone to reproduce the bug on their system so we can avoid the “it works on my machine” issue. If we can’t reproduce it on their machine, then we know there is likely a difference in setup. 

If I can’t work with the developer in real time, screen recordings can be very helpful as well, especially when they include a voiceover describing each step of the process. When I recorded the video of my motorbike, I went through the steps and made them explicit. I noted that the bike was in neutral, I was holding the clutch, and that I was on a stretch of road that goes slightly downhill but the bike still struggled to move, all while ensuring the camera captured each element I was naming. 

When I do screen recordings of an error, I follow a similar process so the developer reviewing it can easily follow along. I also find that narrating my steps makes it easier for me to spot mistakes as I make them — when I say things out loud, I might notice I’ve skipped a step, copied the wrong test data, or made some other mistake. 

Bug reporting takes practice and patience

Reporting a bug is tricky because there is usually more to report than we think there is. But with practice we can get better at it. Next time you encounter a bug, take a moment to consider how you can proactively avoid any lack of confidence in your report and focus on clear communication with your team.