Today’s ‘making video games is hard’ news: Respawn’s journey to track down the bug that caused audio issues in Apex Legends for months. Months of painful research – all of this seems to have been caused by a single line of code added in Apex Legends’ Season 16 update.
as outlined in Full Reddit Post (opens in new tab) According to Respawn community manager Amy Thiessen, the troubles started at the start of Season 16 in February. The studio was starting to get reports of “disappearing Nade” on Apex. Respawn quickly determined that grenades didn’t exactly “disappear”, but sometimes they didn’t explode when they damaged the player.
“This did not occur during Season 16 playtesting, and we were not able to reproduce it internally after initial reporting. Also, the root cause was not always shown from the player’s perspective, so It was extremely difficult to identify using the live gameplay video,” the post reads.
Respawn now handles the issue better after receiving similar reports of missing gun sound FX and particle effects. “After preliminary investigation, the primary suspect is that the server dispatches ‘start’/’stop’ commands for various effects (certain sounds, particle systems, physical effects, bullet trackers, explosions, etc.). It turned out to be the system to use. “
Essentially, something happened during the match that could overload the server limit for sound FX or particles, causing some sounds and FX to be dropped.
“From there it was theorized that something might be pushing the limits of this engine and demanding thousands of effects per second!” the post says. “But was this a systemic problem, or was a single entity working? Each season update contains thousands of changes to assets, code, scripts, and levels. It meant finding needles in haystacks.”
Respawn turned to metrics to investigate the issue, but telemetry showed no obvious problem. This suggested to Respawn that this bug was a unique situation their system had never seen before.
“This left us with a complex issue that we knew was impacting our community, but which was difficult to reproduce despite detailed reporting and had minimal internal leads. , there were no indicators that definitively demonstrated that this limit had been reached at all.”
Where do you go from there? Respawn decided to test the theory of the overload effect by deliberately destroying an Apex Legends server. The team spun up a test build and spawned 50 characters. All of these characters fired their guns at the same time and used their abilities indefinitely to push the server’s load of effects to the limit. The team was eventually able to reproduce the audio he drop similar to the bug report, but how it happened to real players still remained a mystery.
“This gave us evidence of FX dropping, but only in completely unrealistic test cases. We investigated various aspects of server performance, but found nothing obvious. I did.”
Respawn has been keeping a close eye on the issue as Season 16 rages on. The team eventually realized that dropped audio reports tended to come from high-level play. This gave rise to the idea of rolling out an update to his server that would allow Respawn to track the new metric in a smaller subset of matches, leading to an immediate breakthrough.
“When the server update entered its final stages, we found it. One line of code was identified as the root cause of the problem. New weapons for Season 16.”
That weapon is the Nemesis, Apex’s newest burst-fire energy assault rifle. Nemesis has a unique mechanic where dealing damage “recharges” the gun, allowing it to fire faster ( Indicated (opens in new tab) by YouTuber Dazs above). This charging effect is visually represented on the gun by the arc electricity in the barrel. According to Respawn, the line of code that tells it to “stop” this effect when the gun isn’t loaded or holstered actually works for any player holding Nemesis in their inventory. was repeated indefinitely.
“This means that every player with an uncharged Nemesis will create a ‘stopping particle’ effect on the server every frame, and this line of code will work even if the weapon is holstered. had been called. “
Funny enough, this also explains why audio drops were happening more often at higher levels of play. “14 clients with Nemesis running at 180 fps is enough to start dropping FX.” Disadvantage.
Respawn says this also explains why internal testing didn’t reveal any bugs.
“The build used for testing may not have Nemesis holsters played well enough, may have rare correlation with missing FX, or may not have enough clients at that fps. This is something we need to keep in mind and improve for future testing.”
A patch rolled out last week finally eliminated the bug for good. In the long run, it’s a nasty audio his bug with a complicated root cause that helps Respawn catch similar bugs before they reach the player. Respawn ended the post with a digression about testing, telling players, “One minute of his player playing Apex is the equivalent of 10 testers playing his year of the game!” reminded me.
