My name is Alena Lutsik and I've been working as a QA engineer for seven years now. Before joining Qase, I worked at a company that runs a platform for creating resumes and job vacancies.
In 2018, shortly after the release of a new feature, my team and I noticed something unusual: residents of a small village created 45,000 new resumes on the platform in a single week — a number nearly 7 times larger than the total number of people living in the small village.
Prompted by this anomaly, we urgently initiated an investigation to identify the issue. And we found it in the middle of an empty field in Iran. Here’s how this happened.
Background: Feature development and a rush to the finish line
The new feature aimed to make it easier for users to apply for a vacancy with a resume and expand our resume database in the process.
Previously, users could apply for vacancies without including a resume. As a result, many users chose to skip the time-consuming resume form that required candidates to write about their previous experience, education, knowledge of languages, and their strengths.
But we wanted to change this. Basically, the the idea was: if a person clicks the “respond” button on the vacancy card, a modal window would immediately open in front of them, prompting them fill out a form to create a resume. To speed up the process, this modal window would contain only the main fields, and we would pull the candidate’s name and city from the vacancy card.
For example, if a person responds to an offer to work as a driver in Moscow, the platform creates their resume in Moscow with the title “Driver.” When that person clicks “create,” they are automatically returned to the job card and can continue to apply, while the resume is created in the background.
As a result, the applicant quickly receives a basic resume that other employers can find and we expand our database. Win-win.
The idea was given the green light, and it reached our quarterly goals.
Hitting the deadline
We aimed to release this feature across all platforms by the end of the quarter. In order to roll out our feature to users, it was necessary to supplement the backend and configure its integration with three clients — native applications for iOS and Android as well as the mobile version of the site.
The backend, iOS, and the version for mobile browsers were completed normally, but Android took longer than expected. We had to hurry — according to the company’s release policy, updates were rolled out once every two weeks, and if we didn’t finish quickly, we would not make it into the last release of the quarter.
Despite the challenges, we successfully met our deadline, fulfilling our quarterly objectives.
Unraveling the mystery: a sudden surge in resumes
Initially, we were elated by the growing number of resumes, but our excitement was short-lived. About a week after the release, I observed a peculiar trend in our statistics: an overwhelming majority of new resumes seemed to be from candidates in Magaramkent, a small village near the Russia-Azerbaijan border.
At the time, less than 7,000 people lived in Magaramkent. But somehow, 45,000 resumes had already appeared, and there were no complaints to technical support. It was very unlikely that every resident of Magaramkent had created six or seven resumes for different vacancies.
Investigating the cause
I frantically started double-checking everything and by the end of the day I found an explanation for the sudden surge in resumes from Magaramkent.
Finding 1: Latitude and longitude swap in Android client
In our haste to complete the Android platform, we overlooked a critical error. When the Android application sent a request to the backend, it inadvertently swapped the latitude and longitude values. As a result, a resume intended for Moscow with coordinates of 55 degrees north latitude and 37 degrees east longitude was mirrored in the system to 37 degrees latitude and 55 degrees longitude.
Finding 2: The coordinates of Moscow were not Magaramkent, but a field in Iran
Given the sheer volume of 45,000 resumes in a week, I reasoned they were more likely from a large city than a small village. Upon investigating the actual location corresponding to Moscow's mirrored coordinates, I discovered they pinpointed a remote field in Iran.
Finding 3: Geolocation degradation misinterprets foreign coordinates
The platform, designed to operate exclusively within Russia, employed a graceful degradation logic for geolocation detection. When a resume was posted from outside Russia, this logic didn’t show an error. Instead, it defaulted to the nearest point within Russian borders. In this case, that point was Magaramkent. This degradation logic was probably introduced to eliminate user errors and inaccuracies, but it played a cruel joke on us.
Resolving the issue: implementing a quick fix
We implemented a hot-fix the following day, effectively correcting the misplaced resumes. The case was solved late in the evening. Fortunately, I managed to find an Android developer who was still in the office. We urgently fixed it and went home with peace of mind. The bug fix rolled out to all users the next day.
Key lessons from the bug investigation
Although this investigation was challenging, it provided invaluable professional lessons including:
- Integration must be carefully tested with each client. If we weren't in such a hurry, we would have noticed the problem on Android.
- In autotests, it is necessary to check not only the fact of data creation, but also data correctness.
- You need to be well aware of the graceful degradation logic algorithms. If we had researched how ads are assigned to geography in advance, we might have noticed the risks.
Now, whenever I conduct client integration testing, I approach it with heightened vigilance. But we learn and grow from mistakes, so it’s important to talk about them. And I still cherish the thought of someday visiting Magaramket and erecting a monument there to one of the most ridiculous mistakes in my professional experience.
