Explaining how fighting games use delay-based and rollback netcode (original) (raw)

How to design your game for optimal play over a network.

Credit: Aurich Lawson / Capcom / Getty Images

Ricky "Infil" Pusch is a long-time fighting game fan and content creator. He wrote The Complete Killer Instinct Guide, an interactive and comprehensive website for learning about Killer Instinct. This article was originally published there.

Hang around the fighting game community for any period of time, and you'll hear discussion about why playing fighting games online can be frustrating. A genre built on twitch reflexes and player reactions, fighting games can struggle at times to translate their offline success to online environments. Good online play is possible, though, and nothing is more important for realizing this goal than choosing the right approach to netcode.

At its core, netcode is simply a method for two or more computers, each trying to play the same game, to talk to each other over the Internet. While local play always ensures that all player inputs arrive and are processed at the same time, networks are constantly unstable in ways the game cannot control or predict. Information sent to your opponent may be delayed, arrive out of order, or become lost entirely depending on dozens of factors, including the physical distance to your opponent, whether you’re on a Wi-Fi connection, and whether your roommate is watching Netflix.

Online play in games is nothing new, but fighting games have their own set of unique challenges. They tend to involve direct connections to other players, unlike many other popular game genres, and low, consistent latency is extremely important because muscle memory and reactions are at the core of virtually every fighting game. As a result, two prominent strategies have emerged for playing fighting games online: delay-based netcode and rollback netcode.

There’s been a renewed sense in the fighting game community that rollback is the best choice, and fighting game developers who choose to use delay-based netcode are preventing the growth of the genre. While people have been passionate about this topic for many years, frustrations continue to rise as new, otherwise excellent games repeatedly have bad online experiences.

There are relatively few easy-to-follow explanations for what exactly rollback netcode is, how it works, and why it is so good at hiding the effects of bad connections (though there are some). Because I feel this topic is extremely important for the future health of the fighting game community, I want to help squash some misconceptions about netcode and explain both netcode strategies thoroughly so everyone can be informed as they discuss. If you stick around to the end, I’ll even interview some industry experts and community leaders on the topic!

Before we dig into the details, though, let’s get one thing straight.

Why should I care?

Both companies and players should care about good netcode because playing online is no longer the future—it's the present.

While most other video game genres have been this way for a decade or longer, fighting game developers seem to be resistant to embracing online play, perhaps because of the genre’s roots in offline settings such as arcades and tournaments. Playing offline is great, and it will always have considerable value in fighting games, but it’s simply the reality that a large percentage of the player base will never play offline. For many fighting game fans, playing online is the game, and a bad online experience prevents them from getting better, playing or recommending the game to their friends, and ultimately causes them to simply go do something else.

Even if you think you have a good connection, or live in an area of the world with robust Internet infrastructure, good netcode is still mandatory. Plus, lost or delayed information happens regularly even on the best networks, and poor netcode can actively hamper matches no matter how smooth the conditions may be. Good netcode also has the benefit of connecting regions across greater distances, effectively uniting the global player base as much as possible.

Bad netcode can ruin matches. This match, played online between two Japanese players, impacted who gets to attend the Capcom Pro Tour finals. (source)

What about those who never play online because they prefer playing offline with their friends? The healthy ecosystem that good netcode creates around a game benefits everyone. There will be more active players, more chances to consume content for your favorite game—from tech videos to spectating online tournaments to expanding the strategy of lesser-used characters—and more excitement surrounding your game in the fighting game community (FGC). Despite _Killer Instinct_’s pedigree as an excellent game, there’s no doubt that its superb rollback netcode has played a huge part in the sustained growth of its community.

Good netcode matters, period. So let’s talk about it.

The basics

Before we get into the specifics of how the two netcode strategies work, we first have to set a few ground rules that govern fighting games and introduce a few terms.

In fighting games, time is measured in a unit called a frame. Just to make the discussion easier, we’ll assume that all fighting games operate at 60 frames per second, which means that one frame is around 16 milliseconds (ms) of real time.

Importantly, this isn’t just how fast the game renders new images to your screen; every frame, the game executes its game loop, which (among other things) asks the players’ controllers for inputs, checks the network for new information, runs AI for any CPU players, animates the moves each character is doing, and checks if someone is now getting hit. After it has done all these things, it draws the results of all its calculations to the screen, then does it all again 16 milliseconds later. In fighting games, this loop needs to be tight and consistent for all players who play your game, regardless of how fast or slow their computer is.

When playing a fighting game offline against your friend, you connect two controllers to one computer or console. If you both happen to press a button within the same 16 millisecond window, the game will receive and process the inputs on the same frame and apply the logic as expected. You will both see the same output, because there is only one computer doing the calculations.

The inputs for each player are shown at the bottom. When playing offline, there is no trouble processing all inputs for both players as soon as they are pressed.

This changes when two players are playing over the Internet.

First of all, information always takes time to send through a network. This is measured in ping, the amount of time it takes for information to be sent to the other player and then back to you. Over a connection with 90ms ping, for example, it takes 45ms (on average) for information to reach the other side, which is about three in-game frames. This means games now need to be clever about how they handle the input part of their game loop, as they can no longer guarantee button presses for the remote player will line up with the local player.

When playing online, your own inputs are still processed immediately, but the remote player's inputs now take time to travel over the network. The game needs to decide how to handle this so both games remain in sync.

Secondly, two different computers are now trying to run two copies of the game at the same time but still produce identical results for both players. That’s why it’s a great idea for fighting games to be deterministic—given identical inputs, every machine that runs the game must produce identical results. This is cool for non-networking features like replays because you can simply save the inputs from each player and always reconstruct the match perfectly, but it also means the game only needs to send player inputs over the network to play online matches. We get to avoid sending complicated information about the game state and can save a lot of bandwidth.

When games send information to each other and then rely on the computers to independently run the simulations in sync, it’s said they are using lockstep networking. They don’t talk to a central authority that keeps track of the game for them and tells them what to do, like a server. Instead, they police themselves by asking each other periodically if they have the same game state. If the games start to disagree about the state of the game, they are desynced and will probably just have to abandon the match entirely. Games talking to each other directly can often be faster than being forced to talk through an intermediary, and lockstep solutions are particularly good at preventing many types of cheating. For example, even if you hacked your game so Ryu can throw faster fireballs, my simulation of the game won’t agree with yours and we will quickly desync.

Now, after all this setup, we can finally get to the meat of the problem. If we have a deterministic fighting game that uses lockstep networking, the only thing we need to play online matches is the input from both players. Let’s talk about these "clever ways" a fighting game handles player inputs that are never received on time.

Delay-based netcode

As a reminder of the problem, fighting games are used to process inputs from both players at the same time. When playing online, your own inputs are received immediately, but your opponent’s inputs for the same frame need to be sent over the network and will arrive late. The game needs a strategy for how it will deal with these late inputs while keeping the game feeling as close to offline play as possible.

(As a side note, we won’t even talk about the naive solution where both clients simply wait for remote input before completing the game loop for each frame. Here, you could wait 100ms or more, instead of the tight 16ms we require. This would slow the simulation down to a crawl and make your fighting game instantly unplayable.)

The first strategy, called delay-based or input delay, is the simplest and most common implementation for games using lockstep networking. If a remote player’s inputs are arriving late because they need to be sent over the network, delay-based strategies simply artificially delay the local player’s inputs by the same amount of time. Then, in theory, both inputs will “arrive" at the same time and can be executed on the same frame as expected.

Let’s illustrate this more clearly with a video example. Here, two players are playing UNIST, a fighting game that uses delay-based netcode, over a connection that has 90ms ping. This means it takes half of that time on average (45 ms, or about three in-game frames at 16ms each) for your opponent’s input to reach you. The local player presses a button on frame three. In offline play, we would see this move begin to animate immediately, but instead, the game will delay the input by three frames and begin executing the move on frame six.

In delay-based netcode, when the local player presses a button, it is artificially delayed (here, by 3 frames).

These extra three frames give the necessary time for your opponent’s input from frame three to reach you over the Internet. On frame six, we’ll have input from both players and the game can proceed. As long as every input from your opponent can travel the network in these three frames, the game remains stable and consistent.

Both players press a button on the same frame. The artificial delay for the local player gives time for the remote player's input to reach them, so they can still be executed on the same frame.

Unfortunately, this is not the reality of the Internet.

The problems with delay-based netcode

You might suspect the main problem is that, if your inputs are delayed any amount, it will hugely impact your reactions and game plan and make the game unplayably different from offline matches. While delays are not ideal, there are many players who will not be able to notice small delays on the order of two or three frames. For those who can notice it, or are training to do well at offline tournaments where there will be no delay, clever game design and the delay remaining consistent can mitigate the effects of the delay surprisingly well and still allow online play to be a useful method of practice.

But no, the main problem is consistency. Because networks are notoriously inconsistent, delay-based strategies struggle because they are terrible at handling network fluctuations. Suppose there is a spike in the connection and some information from your opponent takes a little longer than the estimated three frames to reach you. Because the game simply cannot proceed without information from both players, it has no choice but to stop and wait for the input to arrive, causing the game to chug and lurch along during moments of prolonged network trouble.

At 30% speed, you can see the game having to pause during network trouble. A frame-by-frame view shows the remote player's input getting lost for 6 frames, and the game has no choice but to sit there and wait for the input to arrive before it can continue.

Since pausing the game even briefly is pretty detrimental to a player’s experience, most delay-based implementations will try to prevent it from happening as much as they can. By keeping an eye on the network conditions, they can change the amount of input delay on the fly to match the current connection health. But because network behavior is very difficult to predict, it will often change too late to avoid having a few game pauses, and it will often keep the input delay inflated for longer than is needed. For example, before switching to a rollback approach, _Mortal Kombat X_’s delay-based netcode would fluctuate between five and 20 frames of delay, and connection spikes would inflate the delay for many seconds, even if the spike only affected a few frames. It can lead to instances where players feel the game is lagging a lot more than it is reporting.

Guilty Gear Xrd changes the input delay on the fly to hopefully avoid needing to pause the game and wait for inputs. Sometimes the fluctuations are small, and sometimes they can get out of control.

Delay-based netcode is also hyper-sensitive to the distance between you and your opponent, because a greater distance increases the travel time across the network and therefore the game must also increase the artificial input delay. Distance plays a factor in all online connections regardless of netcode choice, but it’s particularly difficult to play delay-based games even just a single time zone away, let alone trying to unite the player base globally.

And finally, one last drawback that may not be obvious at first is that delay-based solutions do not care about the game state. Whether the announcer is calling “Round 2, Fight," your opponent is knocked down, or both players are walking around in neutral, a delay-based game treats connection issues identically in all cases. This means there is no place to hide from a bad connection, even if your opponent’s inputs would not impact the match in any way.

Ultimately, delay-based solutions fail to provide a good user experience because, unlike offline play, player inputs are not consistent or responsive. Fighting game players thrive on reproducible and trainable situations that are tied heavily to reactions and reflexes. If the opponent jumps, I can input my anti-air. If I land this combo starter, I can continue the combo with muscle memory. When I knock my opponent down, I can meaty them with this specific setup. When input delay fluctuates wildly, whether it’s between two different matches or even during the same match, fighting game players lose confidence that anything they do will work as expected. And when they are unable to enact the game plan they know would help them win, online play becomes extremely frustrating and often useless.

But what if I have a good connection?

People drastically overrate their connections. While it’s possible you may have a fast, stable connection to some people near you, it’s impossible to control the quality of the connections to every other potential player. Some players may be far away from you or might have a higher tendency to drop information over the network. Even two people near each other with normally fast connections may struggle to maintain a stable connection to each other, for reasons uncontrollable by either party.

But yes, it’s possible that, in some subset of connections with suitably small input delay, some matches in delay-based games will perform well. However, netcode needs to be judged based on how it handles bad connections, not good ones. If given a perfect connection with inputs that always sync up, there is no need for any clever strategy to approximate offline play—you’re already there! It’s the netcode’s job to cleverly hide latency, and if your strategy falls apart at even the slightest sign of network trouble, then it’s not an effective method after all.

Why is delay-based still used?

The main reason so many fighting games implement delay-based netcode is because it is relatively simple. Your gameplay logic is largely unchanged from offline play, since you can always assume both players’ inputs are provided at the start of your game loop. There is also no extra game state processing that needs to occur, which means your game takes roughly the same amount of resources from the computer to run both offline and online matches. It is likely the cheapest, easiest option for developers seeking to provide a barebones online mode.

Unfortunately, it has been mostly Japanese developers who continue to use delay-based approaches, as virtually every fighting game made in America has given up on delay-based netcode and made the switch to rollback. This includes titles from big publishers, like Killer Instinct and Mortal Kombat 11, to smaller indie games like Skullgirls, Punch Planet, Pocket Rumble, and Them’s Fightin’ Herds. It’s possible the larger geographic area of America and Europe has encouraged these developers to make netcode a bigger priority, but it’s not like people playing in Japan are safe from bad connections and the frustration of bad netcode, even though they live closer together and have strong online infrastructure.

Due to the insistence of fans and trends in the industry, it’s also increasingly unlikely that Japanese developers are simply unaware of rollback as an option for their games. Capcom, one of the premier Japanese fighting game developers, has attempted rollback netcode in three different titles (Street Fighter x Tekken, Street Fighter V, Marvel vs. Capcom: Infinite), with varying degrees of success. Daisuke Ishiwatari, chief creative officer at Arc System Works, admitted that fans have been bugging him about rollback netcode, but they still chose to use in-house delay-based netcode for 2018’s Blazblue: Cross Tag Battle. Some fans (and even some developers) have theorized that this may be due to some companies’ preference to write their own solutions to solve problems, an issue that seems to especially impact Japanese studios. But because so many popular fighting games in our community are made by Japanese developers, delay-based netcode continues to be prevalent. More than a decade after rollback netcode was first pioneered, only two of the nine games at EVO 2019 used rollback for their online multiplayer, and only one of these two games has had good fan reception to their netcode.

With all this said, it’s time to talk about exactly how rollback works.

Rollback netcode

Since a game’s choice of netcode can never magically change the distance between a player and their opponent or prevent networks from dropping or delaying information, you may wonder how one netcode strategy could be drastically better than any other. The key lies in how the netcode handles uncertainty.

When there is no information from the remote player, delay-based netcode needs to pause and wait, as described in detail on the previous page. Rollback’s main strength is that it never waits for missing input from the opponent. Instead, rollback netcode continues to run the game normally. All inputs from the local player are processed immediately, as if it was offline. Then, when input from the remote player comes in a few frames later, rollback fixes its mistakes by correcting the past. It does this in such a clever way that the local player may not even notice a large percentage of network instability, and they can play through any remaining instances with confidence that their inputs are always handled consistently.

But let’s start by taking a look at what a “rollback" even is.

Act now, apologize later

When remote inputs are missing, rollback netcode will continue to simulate the game. But, when those inputs eventually come in, the game will have advanced past the time when the opponent pressed that button, and the game will have already shown a different result to the screen. To fix this, rollback netcode will rewind the simulation, apply the correct input, and show the new result to the player immediately.

In the example below, both the local player and the remote player press medium punch on frame 1, but let’s say the opponent’s input is delayed over the network and reaches us three frames later on frame 4. Like offline play, the local player’s move immediately begins animating on their screen, and the game happily continues along assuming the opponent has not done anything. On frame 4, input from the opponent, marked as being pressed on frame 1, finally arrives. This means what we’ve shown to the player in the last 3 frames is not what really happened, and the game must fix the past by performing some calculations in the background.

First, the game state gets reloaded to what it was before frame 1—that is to say, it “rolls back" to the frame where the input needs to be applied, a few frames in the past. Then, the inputs from the remote player (as well as any inputs the local player had pressed during past frames) are applied, and the game re-simulates multiple frames forward to reach the present frame once again.

The remote Jago player presses MP on frame 1. When it reaches us, we're already on frame 4, so we have to rewind to frame 1, get Jago to press MP, then re-simulate back to frame 4 very quickly. The orange filter and "ghosted" version of Jago indicate background calculations that the player never sees. At 30% speed, the local player just sees Jago's MP appear a few frames into its startup.

All of this happens instantly, in one game frame; the local player does not ever see the game perform these steps individually. Instead, all they see is the game state they thought was correct (but was false) get immediately replaced with the actually correct game state. Depending on what’s happening in the game, the characters might suddenly jump around a little bit, and in general, animations for the remote player’s moves will tend to appear a few frames already into their startup when they are shown to the local player.

Jago attacks with his overhead. The video shows the move with 0, 1, 2, or 3 frames of rollback right when Jago attacks, which shaves that same number of frames off the beginning of the move's startup. Even at 30% speed, it's hard to tell the difference.

To put it another way, rollback allows each client to temporarily break the lockstep model—the game may be showing slightly different things to each player, depending on the connection quality and what’s happening at the time of the rollbacks. However, the game will always correct itself to be the same for both players whenever the inputs have been received a few frames later.

It’s worth reiterating this important property of rollback netcode; the local player’s inputs are always shown immediately and can never be undone. If you press a button on frame four, that information is immediately processed by your game, and your move will instantly come out on your screen. If any rollback occurs when you pressed that button, it is always correctly re-applied during the rollback calculation. There is also no way your input can be invalidated or “eaten" by network lag, which can occur in delay-based frameworks when the game is waiting for remote input and is unresponsive for both players. Therefore, a player can feel confident that the buttons they press will be executed regardless of the network quality, greatly enhancing the consistency of online play.

As an added bonus, when a game encounters network trouble, there will only ever be rollbacks in the immediate time around the spike, making it a very localized approach. In delay-based solutions, the game may choose to inflate the input delay for many seconds, causing long-lasting effects far away from the time of trouble, even though the network may have smoothed out its issues by then. Rollback is particularly great for connections with high likelihood for packet loss, like Wi-Fi, for this reason.

This is the core concept of rollback. But we can do better.

The basics of prediction

In the example above, when input was missing for the remote player, we simply assumed they were inputting nothing so that the game had something to simulate. This is a pretty bad assumption in general because players are rarely doing nothing, and the visual states shown to the player right before a rollback would almost always be wildly incorrect. Turns out, we can predict what the opponent is doing with an uncanny degree of accuracy.

When input is missing, what rollback solutions actually do is duplicate the last known input from the remote player for the current missing frame. If they were holding down-back, the game predicts they will continue to hold down-back. If they were holding a button, the game assumes they are probably still holding that button. The game then runs with these predicted inputs for the remote player instead.

(Note: for the purposes of keeping these example videos as easy to understand as possible, we’ll temporarily assume that network transmission time is instant and whenever there are missing frames, it’s because the network got briefly overloaded. Don’t worry, the concept still works perfectly fine with network delay, but the videos would have been a little too confusing.)

Inputs for remote Fulgore are lost after frame 3, but in a rollback model, the game duplicates the last known input to keep the game moving forward. Predicted frames are colored orange, and the last known input is indicated by the orange caret under frame 3's input box.

When the actual, correct remote inputs arrive a few frames later, the game needs to decide how to use them. If the inputs were incorrect, we perform a rollback, as previously described—the game rewinds to a previous game state and uses the new inputs to simulate (and re-predict) forward. But if these inputs are exactly as predicted (for example, the player did actually continue to hold down-back), no rollback needs to happen; the game that was shown to the player during the prediction period was correct! The game simply updates what it knows to be the last correct input from the remote player, and the local player doesn’t know anything went wrong. When the game correctly predicts the inputs, rollback ensures the game feels perfect, even if there was network trouble.

Inputs for remote Fulgore are, again, lost after frame 3. The inputs eventually come flooding in on frame 8, and Fulgore did indeed hold forward, the same as our prediction. The game quickly verifies this is true and keeps going without even needing to roll back.

Prediction is a relatively basic addition to the paradigm, but it has a huge impact. Rolling back and re-simulating the game only if the remote player did something unexpected has a lot of nuanced advantages that we need to discuss further. It does a lot more to hide the effects of bad connections than you might initially think.

Why prediction is so good

Rollback’s version of “prediction" is not as advanced as you might expect if you play other genres, like first-person shooters. As stated, all it really does is assume the remote player has not changed their inputs from the last time it received information from the network.

As it turns out, this is an extremely reasonable assumption if you break down how often players actually change their inputs. Consider a match in a Street Fighter-like game where players are walking back and forth; let’s assume a player changes their walking direction around five times per second, which is a good ballpark even for especially active players. This means we actually only need inputs from the remote player on five of the 60 frames in a given second (a staggeringly low 8% of the time). The player’s inputs for the other 55 frames can be predicted with a high degree of accuracy just by assuming they are the same as the previous frame.

And this is typically during the most active part of a match. If you have knocked your opponent down and are trying to pressure them, it’s not uncommon for a defensive player to simply hold down-back for 30, 60, or 120 consecutive frames. If your opponent jumps, they are airborne for around 45 frames and are likely to only press one button during that entire time frame. Why do we need to wait for the network to tell us this, if the frame-by-frame prediction that your opponent will continue to hold down-back, or continue to not press a button, is correct 95% of the time?

What this means is, if there is a network spike during any of these moments where remote input was unchanged, it is completely invisible to both players. Even in dodgy connections where information may arrive quite late many times per second, a pretty large percentage of these delays will randomly coincide with moments where the remote player was not changing their inputs. A delay-based game would be forced to pause and wait here, but because a rollback game verifies the inputs were correct when the information arrives and doesn’t need to change anything, the connection appears completely perfect.

Rollback cares about game state

There are even more ways rollback hides bad connections. Unlike delay-based solutions, which could have problems any time during a match, rollback approaches only show a visual change if the new inputs from the remote player have changed the game state.

Let’s explain with a video example. Here, your remote opponent playing Jago whiffs a heavy punch on frame two, and your game receives this input over the network just fine and begins animating the move. Right after pressing heavy punch, there is a huge network spike, and all information for the next five frames does not arrive on time. As explained, rollback doesn’t care; it will continue to simulate the game anyway—in this case, it will assume Jago inputs nothing, as they had released their finger off the heavy punch button on the last known frame.

Six frames in the future on frame eight, all the information for these missing five frames floods in, and as it turns out, the opponent was mashing buttons. This was not what we predicted, so we have to rewind the game five frames to the start of the heavy punch, re-simulate these five frames with the correct input, and show the player the real game state.

Jago presses HP, but after that, inputs are lost for a bit. Rollback simulates forward anyway. When they come in, turns out Jago was mashing buttons; it doesn't match our prediction, so we must rollback and re-simulate. But the end result is identical, so the players don't know there was any network trouble at all.

But because the opponent was whiffing a move, the game’s rules say they aren’t allowed to control their character right now. These new inputs have changed nothing about the game state! Even though the game rolled back and re-simulated multiple frames, the end result was the same as what was shown to the screen, and once again, the lag is completely invisible to both players.

As in the previous section, think about all the times during a match that your opponent’s inputs can’t change the game state. This varies by game but usually includes times when they are knocked down, being comboed, in block stun, when any move is starting up, recovering or whiffing, and many others. The number of frames these actions take to complete is not short, either; hard knockdowns in Street Fighter games can take 60 frames, heavy attacks whiffing could take 30 frames or more, and combos can last for 10 or more seconds! As long as the game has correctly registered the beginning of these actions, you are completely immune to lag until the opponent can change the game state again, even though the game might be rolling back furiously during these periods.

Putting this together with the prediction model, you can begin to see just how often lag gets “absorbed" during a fight. The connection problems haven’t gone away, but rollback solutions do the best job of just simply ignoring them as often as they can, and they can make even the laggiest connections feel extremely playable. In _Killer Instinct_’s case, it even allows online sets to be played between Singapore and the Eastern US over connections that approach 200ms ping, which would be unbearable with any delay-based netcode.

Can we split the difference?

In real-world situations, inputs always take time to send over the network, which means every input will always arrive after the frame it was intended for. You may have correctly deduced that this means every single button press or change of direction your opponent does will always cause at least a slight rollback, as long as the input changes the state.

Turns out, we can eliminate this by combining delay-based and rollback solutions. In fact, every modern fighting game that uses rollback is built on a delay-based framework, with the important caveat that the delay is fixed at the start and never changes. If you pick a suitably small delay and keep it fixed, it should give enough time for “many" of your opponent’s inputs to reach you without needing to trigger a rollback on every state change. If there are network spikes, rather than increasing input delay or pausing and waiting, instead rollback takes over for any input that takes longer than the chosen delay. As long as the delay is small and consistent, players will quickly adapt, and many will not even notice a difference from offline play.

What number to set this delay at is up to the discretion of the developer. Some games, like 3rd Strike Online Edition, Darkstalkers Resurrection, Skullgirls and Samurai Shodown V Special, allow the user to set their own delay, usually between zero and eight frames. Other games, like Killer Instinct and Injustice 2, universally choose the number for all players (three frames, in the case of these games) and provide no option to change it.

Skullgirls lets each user manually choose how many frames of input delay they would like. Rollback only kicks in if the network delay is bigger than this number.

Both methods have their strengths and weaknesses. Allowing the user to choose their own delay means they can factor in the strength of the connection and adjust the delay to match. But it also means that players who don’t understand how rollback works, or what this setting controls, are much more likely to choose a setting poorly suited for the connection and experience. Forcing the delay to always be consistent may mean some matches have slightly more delay than is theoretically necessary, but it gains a measure of consistency across all connections. Some developers may feel that asking the player to adjust their reactions to different (although stable) delays across different connections has more drawbacks than just always using a well-chosen setting that will work for the vast majority of connections.

A summary of rollback's benefits

Rollback largely solves the consistency issue that plagues pure delay-based solutions. The local player is not affected by network trouble, which leads to their inputs always being treated the same way; if you try to perform a specific knockdown setup, or you have a combo that needs very tight timing, rollback always lets you execute it the same way regardless of lag. If a network spike occurs, it only impacts the present moment of the match, whereas a delay-based game might inflate the input delay for many seconds.

Large portions of network instability or delay are simply made invisible to both players when using rollback netcode. Every time the remote player cannot change what’s happening to their character, or every time they choose to repeat the action (or non-action) of their previous frame, the rollback system will produce no visual change, and any networking hiccup during that time vanishes. This also impacts things like fireballs thrown by your opponent; once they have been thrown, the opponent can do nothing to change their trajectory, so their path across the screen is immune to rollbacks and network issues, letting you block or jump over them with offline timing.

Rollback is the best solution we have for hiding lag in fighting games. But now it’s time to talk about what it takes to get rollback netcode into a game.

The technical side of rollback netcode

On the previous page, we talked about how rollback works and why it’s good at hiding bad connections. But now, we need to talk about how rollback netcode gets made, what a game that implements rollback must be able to do, and why some games have trouble creating good rollback.

As much as fighting game fans wished creating good netcode was simple, implementing rollback takes work and requires commitment and a strong belief from the studio that high-quality online play is a priority.

How a game must be built

As a brief reminder of how rollback works, when remote inputs are unknown, the game simulates forward anyway with predicted inputs. If those predictions are incorrect when the real input is received, the game must roll back by loading a previous game state and simulating several in-game logic frames.

Therefore, the primary requirement of a rollback game is that it can save and load different game states and simulate game logic in the background, and it must be able to do this all very rapidly. There is already a lot to unpack with just this seemingly simple statement.

The act of converting objects in computer memory to a format that can be saved and loaded is called serialization, and the game state for every single frame must be serialized and saved in case we need to roll back to it later. This means your game state serialization, which can be a time-consuming operation depending on the data, must be lightning fast and very well optimized. Depending on how the game has been built, you may even need to change how the data is stored and how the various systems of your game use it to gain the necessary performance. If you build your game with rollback in mind from the start, it can lessen the burden, but adding rollback to an existing game can mean a lot of system-wide changes to the very core of your game. Michael Stallone, engineer at NetherRealm Studios, estimates that it took his team two man-years to build and optimize serialization systems in Mortal Kombat X when they retrofitted the game with rollback netcode.

If your game needs to roll back a large number of frames, once the previous state has been deserialized and loaded, the game must re-simulate all these frames back to the present (and then also serialize all of them!), and this must be done quickly and in the background. This means your game logic must be independent from everything else in your gameplay loop; you need to be able to run many frames of logic without rendering to the screen, or waiting on the network, or using any number of other systems that normally operate every frame. Depending on the engine you chose, or how much customization to the engine-provided game loop systems you’ve done, separating and turning off these subsystems may be surprisingly hard.

Because all these things need to be performed in one game frame, in addition to all your normal game loop operations, optimization in general is another key concern. Perhaps those costly CPU cycles that your fancy particle system or cloth simulation uses are now putting you over your performance budget. Or maybe you’ve been playing fast and loose with your console’s power, because the need to be extremely efficient wasn’t there when you initially built the game. Stallone says that MKX, under its old delay-based solution, took only an average of 10ms of its 16ms budget to run a complete frame due to the extra horsepower of the PS4 and Xbox One. When initially switching to a nonoptimal, naive rollback solution, he says the cost to run a frame more than tripled to 32ms, which forced his team to re-evaluate almost every single in-game system and look for performance gains.

A small sampling of the things NetherRealm Studios needed to optimize to get MKX running fast enough to use rollback. It all sounds very complicated. (source)

Note that any deterministic game can use rollback netcode as long as it is able to save and load game states and perform multiple frames of game logic quickly enough to fit inside its game loop. To dispel a common myth about rollback, there are no limitations based on the genre of game. Rollback could be (and has been) used for 2D or 3D fighting games alike, including Street Fighter III 3rd Strike: Online Edition, Killer Instinct, Brawlhalla, and For Honor. It’s also used outside of fighting games; Iron Galaxy incorporated rollback netcode into the four-player action game Dungeons & Dragons: Chronicles of Mystara. Even games like Rocket League incorporate rollback to make online play just as responsive as offline play.

Keeping the world in sync

All netcode solutions, whether driven by input delay or rollbacks, must keep both systems running completely in sync. This means both systems are showing each player the same frame at the same time, or as close to this as possible. Even though most fighting games try to lock the framerate at 60fps, it is not simply enough to sync at the beginning of a match and assume the players will remain in sync for the whole fight. Many issues outside of networking, such as consoles overheating, computers performing operating system functions, or someone temporarily overloading their CPU, can cause performance drops which will put them out of sync with their opponent.

The consequences of not correcting this sync are catastrophic. Consider a rollback system where player A is being shown frame 20, but player B is being shown frame 23, across a network delay of three frames. Player B will send input for their frame 23 to player A. Because they are on frame 20, the network delay means they will receive the input on their own frame 23, which is just in time to execute it and will not have to roll back at all. Meanwhile, player A sends their input for frame 20 to player B. With the network delay, player B will receive this input on their own frame 26, which will repeatedly cause lots of very uncomfortable six-frame rollbacks.

An example of what extremely one-sided rollback might look like for an unlucky player. The other side sees a much more normal version of the same match. Playing in these conditions is impossible.

Note that this rollback is entirely one-sided; only the player who is ahead of the other will experience the effects of the bad connection. Tony Cannon, in many ways the founder of modern rollback with his invention of GGPO, theorizes that this is one of the flaws of _Street Fighter V_’s rollback system, and proper clock syncing might go a long way to addressing SFV’s poorly received online play. Some fans have even run their own tests and come to similar conclusions. To help illustrate the point even further, Zinac shows the one-sided rollback effect in action using a very simple project designed to teach about the benefits of rollback.

Zinac's networking example has P1 controlling red and P2 controlling blue; he made it so P2's game is several frames ahead of P1 with no way to resync. When blue moves, P1, who is behind on the sync, sees the blue box move smoothly. But when red moves, P2, who is ahead on the sync, sees tons of huge rollbacks. (source)

Solving the problem is relatively straightforward. Most rollback systems will briefly pause the player who is ahead for one or two frames so the player who is behind can catch up. If the game keeps on top of this system and never lets the games drift very far apart, the correction will be quick and painless. Note that these pauses are not due to missing input or other networking difficulties; they are designed to fix the fact that different computers will run software at slightly different rates in unpredictable ways, and losing one frame every 10 seconds to maintain the sync is unnoticeable to even the most astute players.

What exactly is GGPO?

If you spend any time around the fighting game community, you have undoubtedly heard about GGPO, the rollback library (that is now free and open source!) written by EVO co-founder Tony Cannon. You’ll often hear “GGPO" and “rollback" used interchangeably among fighting game fans. GGPO’s first release in late 2006 was designed to test the concept of rollback with popular legacy fighting games running on emulators, and shortly after, Cannon released a version of the library that could be licensed and incorporated into modern games.

The main function of GGPO is to provide a robust solution for syncing machines and game states. For example, GGPO can tell your game when two machines are out of sync, and by how much. It can also keep track of the inputs from all connected parties and tell your game when to roll back, by how much, and what new inputs to apply during your rollbacks.

Because GGPO is well-tested and proven as a solution to handling these parts of the rollback framework, it is an excellent asset toward making games that want to use rollback netcode. However, the act of actually performing the rollbacks still falls on the developers. They have to make sure their game is performant, their game logic is properly split away from other parts of their game loop, and all edge cases are handled intelligently. GGPO simply tells them when and why to roll back, and what new inputs you should simulate your game with. It does not concern itself with the particular implementation of how the game achieves these goals, which makes it a flexible tool but not the complete suite of functionality needed for rollback to work. It also means that some games, including Killer Instinct, implement rollback netcode without directly using the GGPO library by coding their own in-house solution for tracking the game state and syncing the clients.

If you are further interested in the exact ways GGPO handles rollback and talks to games, I recommend reading this excellent article in Game Developer Magazine written by Tony Cannon. It has a bit more of a technical focus, but may interest you if you’re a programmer or curious about the exact things GGPO provides your game. You can also peruse the source code and read Tony’s own description of the theory behind rollback.

Be careful of edge cases

As if rollback wasn’t complicated enough, it’s worth briefly mentioning just a few of the edge cases you will run into when you support a game loading up the past.

Creating and destroying objects becomes more difficult. Let’s pretend the local player throws a fireball, and the opponent holds down-back in front of the fireball, ready to block it. Rollback uses predicted inputs to play out the opponent blocking this fireball on the local player’s screen. The fireball is destroyed when it is blocked, and particle effects around the fireball are shown. But then remote inputs come in and it turns out the opponent did an invincible move through the fireball right before it was blocked, which means it was actually not destroyed and must continue to travel on the screen. In many games, recreating the fireball after it has been destroyed is expensive and could lead to odd visual popping. A solution many rollback games take is to simply save objects for many frames after they’re destroyed, in case rollback needs to un-destroy them, which may require some changes to your game logic. Particle systems also require a lot of special attention.

Audio is also a particularly difficult problem in rollback games. Sounds that were played in the predicted version often have to be cut short and stopped in the rolled back version, and perhaps sounds that were never played on the local version must now be played several milliseconds in when the rollback changes the game state. This is especially problematic when you remember that game logic needs to be divorced from all other aspects of the game loop. In early versions of rollback in modern games, like Street Fighter x Tekken, there were horrific sound bugs where the sound would cut out for seconds at a time, or sound that was played would sharply pop or play out of sync. Tony Cannon gives some advice for how to handle audio in a rollback engine in his Game Developer Magazine article, specifically the “Separating Game State from Rendering" section.

There may also be times you never want to roll back. Stallone mentions how rollbacks are very undesirable during drastic camera shifting events, like brutalities in MKX or stage transitions in Injustice 2, which means their game must prevent those events from executing until the local player is 100% positive the event occurred and cannot be rolled back. This leads to extra engineering and debugging efforts across all the gameplay system teams, and therefore extra expense.

You also have to be careful to correctly copy the last known frame during your prediction algorithm. This includes what directions the remote player was holding, as well as what buttons they were holding (and not holding). If you mess up this prediction, you may cause a remote player to do something they never actually did, which will be incredibly disorienting when it gets rolled back.

The remote Cody player does EX Zonk Knuckle, a move which is triggered by holding and then releasing buttons, which is then somehow rolled back. My best guess for why this happens is because SFV did not correctly remember what buttons Cody was holding during the prediction phase, which made him do a button-release move. (source)

Is rollback worth the effort?

While there’s no denying that adding rollback to a game takes effort and expense, I don’t want the takeaway from this section to feel like implementing rollback is too difficult to be feasible. The reality is somewhere in the middle between “copy and paste GGPO into your game in an afternoon" and “invest your entire budget on rollback systems."

It largely depends on when you decide to implement rollback. Adding rollback to a game after its gameplay systems have been built is often very challenging. When NetherRealm Studios decided to patch in rollback netcode to Mortal Kombat X, a game that had already shipped, it took approximately eight man-years of time split between many programmers over 10 months. The very small studio behind Rivals of Aether spent a lot of development time over a few years exploring how to convert their finished game from delay-based to rollback, but eventually had to move on when it was proving too costly. On the other side of the spectrum, Mike Zaimont, lead programmer on Skullgirls, added GGPO to his game in about two weeks early in development using his custom engine, likely well-planned-out for rollbacks. Therefore, deciding that rollback is a priority early in development will greatly reduce the amount of investment required.

Implementing rollback also gives you some added benefits in other areas of your game. For example, if your game has a robust method for saving and loading game states, you can use it to provide better training mode options for your players. And when you’ve implemented rollback once, you can reap the benefits of the knowledge in future games. After putting in the hard work for MKX, NetherRealm and its fans now enjoy smooth netplay in all their future titles that were built off modifications of the same engine, including Injustice 2 and Mortal Kombat 11.

But in reality, rollback netcode is far from the first worthwhile advancement in technology to be more work than the previous method. Animation techniques such as inverse kinematics are now standard in many games despite being a higher initial investment for many studios to implement effectively, and after years of working on the problem, developers now have better tools in place to improve it even further. The value for rollback is there, and developers should not be wary of prioritizing that benefit in their game. After all, is there anything more important for a modern fighting game than smooth, reliable, consistent online play that can connect communities worldwide?

Iron Galaxy interview

In addition to explaining how delay-based and rollback solutions work, I wanted to get opinions and perspective from game developers who have worked with rollback on multiple projects for many years. I’m very fortunate to be joined by Ramón "krazhier" Franco, Iron Galaxy software engineer and networking mastermind, and Adam "Keits" Heart, Iron Galaxy designer. They’ve generously taken time out of their schedules to let me ask a few questions about rollback and generally discuss the landscape of netcode a bit.

Thanks a lot for your time!

Keits: Thanks for having us! We’re big believers in rollback, so we want to do what we can to help.

krazhier: Yeah, agreed. I could talk about this stuff all day if you let me.

Most people will know Keits from his work as combat designer on S2 and S3 of Killer Instinct_, but others may not know that Ramon did hugely important work, too, working on and improving KI’s implementation of rollback behind the scenes. But maybe it’s best to get a bit of history first. What was your first experience with rollback in online fighting games?_

krazhier: As an online engineer on all of Iron Galaxy’s fighting games, I first experienced rollback in Third Strike Online Edition. I didn’t really know it was a thing until then. Coincidentally, that’s when I started playing fighting games in earnest. Before that, working on fighting games was just something you’d do as a job, or to button mash with friends. I have to give credit to my co-workers at IG, particularly Floe, for showing me how to like fighting games.

Keits: For me, it was back in the old-school days of the #capcom IRC channel. Tony Cannon, founder of GGPO, was a co-owner of the channel and he offered some people the chance to try out the closed beta, using Alpha 2 as the test game. The experience was so good. After years of tolerating bad online experiences in the early days of Xbox Live, I thought for the first time that the FGC was saved and an online future was possible.

You guys went from playing rollback games to designing and programming them. There’s so many different questions I could ask. Let’s start with perhaps the most obvious question. How would you assess how difficult it is to include rollback in fighting games?

krazhier: It’s definitely more work than delay-based, because to implement rollback, you have to solve the delay-based part first. I can take any rollback game I’ve ever worked on and turn off the prediction frames and rollback algorithm, and I now have a delay-based game. It’s also quite a bit more work if you choose to do it the way Mortal Kombat X did, retrofitting an existing game with rollback. Those guys did a fantastic job, but I imagine it was a big investment for them. If you plan for it in advance, it’s more manageable, but it’s still more work than delay-based.

Also, any time a new element is added to the game, it has the possibility of affecting your game state and making it so rollbacks could cause a desync. There’s no getting around that. When Keits comes to me and says "hey, I want these projectiles to be random and sometimes four of them come out and they loop around in odd patterns," these are now elements I have to make sure can be rolled back, including adding new elements to the code. I can train programmers to make sure their variables are registered with our rollback system, but it’s unavoidably more work. Even non-programmers like the QA team now need to incorporate rollbacks into their testing. For example, with Killer Instinct, our QA team tested with 20% packet loss and 150ms of latency always turned on. You should throw your router into the nearest garbage if you’re having that kind of packet loss, but we made sure KI could play acceptably well at that rate.

Rollbacks can also add some small handcuffs to the programming team, and you have to be creative how you work around them. For instance, let’s say your game has a projectile that can be fired and then they disappear a short distance in front of you. If you need to roll back to a point when those projectiles existed, getting rid of them will add a huge performance debt. So instead, you should probably save the projectiles in a clever way and not actually destroy them until long enough in the future. Performance of these things (and many others) is a big deal. That thing you could have done in 16ms without even thinking twice now needs to fit into 1ms if you want to simulate it 10 times in a single frame.

I’m not trying to make it sound like the end of the world, though. Despite it being more work, all these things have solutions and good rollback games do them. My opinion is that it’s a super worthwhile investment.

There are some games that let you choose the input delay before rollbacks kick in, while other games hide that option from the user and always choose the same delay for each player. What’s your take on these two options? Is one better than the other?

krazhier: Killer Instinct is one of those games that doesn’t let the user choose, it always adds three frames of delay. It gives us 45ms for the input to get to the other side. The reason this is nice is because it means pings can be 90ms before you start having any rollbacks occurring at all. And even if you have one or two frames into the rollback threshold, it’s pretty invisible to players. I personally can’t tell when one frame of rollback is happening. So that’s why I personally like the three frame delay approach.

In some of the early rollback games Iron Galaxy did, we let the users set the delay. And this would of course mean that people would automatically set the delay to 0, because 0 is a nice number and there’s no reason their supposedly good Internet couldn’t handle it, right? But then that means every time he pressed a button, the other user would experience a rollback, because, at 0 frames with a proper timesync, when the other user gets your input, it will always be in the past. That said, I know people who preferred the Darkstalkers Resurrection version we did, which let the user set the input delay, because they always just played in lobbies with local friends to practice for tournaments. And that’s a fine preference to have. But as a whole, the studio has to think about the connection quality for the average player. Three frames of rollback every time you press a button can be a little challenging for the non-rollback aware consumer.

Keits: I agree that letting users set it manually is difficult, because they will always choose 0 and things will look worse, especially since most players do not understand what the setting really means. A delay of three frames gives you 90ms with basically nothing noticeable, which means for players in North America, most of the continent is a possible opponent. 150ms ping is then possible with just a couple frames of rollback occurring, which increases the range to a pretty large chunk of the world. That same 150ms ping connection would be hilariously unplayable in delay-based netcode.

If you’re going to not let the user choose the delay, would it ever be desirable or possible to get the game to detect the connection quality, then automatically lower the delay to one or two frames?

krazhier: Having something that decides one frame of delay at the beginning of the match means that, halfway through the match, you can all of a sudden be playing at a higher latency and the delay won’t adjust. Then you’d end up with significantly more rollbacks, and changing the input latency on the fly is what we’re trying to avoid from delay-based approaches. I also think it’s not entirely clear whether playing 10% of your matches at two frames of delay, but 90% of your matches at 3f of delay, is even that valuable. I think consistency across all games is possibly more important.

Keits: You can also do things to make the game quite a bit more playable even with one or two extra frames of delay, like increase the size of the buffers to make reversals easier and combos easier to link. I think the number of opponents you can play at one to two frames of delay and have it be steady is very tiny. Even people who live 30 minutes away from you often can’t maintain that good of a connection with you. Just think about how rarely the delay stays at a steady one frame when you play Guilty Gear online. So I think the people who say "in delay-based games, I get to play at one frame of delay but I’ll never get below three in these rollback games" are being a bit dishonest with how things truly work in real conditions.

Seems like a good time to branch into how the designer can make smart design choices, knowing you are building for a rollback game. I’m sure you’ve thought about this quite a bit Keits.

Keits: Maybe the first point to talk about would be the startup of moves. Because rollbacks happen only around your opponent’s button presses, it means when a rollback occurs, it always cuts off some part of the startup. This means having moves with slower startup makes it easier to hide rollbacks. For example, three frames missing off the front of a 10-frame jab is way less noticeable than three frames missing off the front of a three-frame jab. I think it’s a decent argument against having three-frame normals in a modern fighting game, actually. _Killer Instinct_’s fastest normal hits on the fifth frame, and the speed of the game doesn’t suffer for it. The typical Tekken jab is 10 frame startup, which I think makes Tekken a great candidate for rollback.

This goes beyond just normals, too. If you take a game like Marvel vs Capcom Infinite, which uses rollback, you’ve got characters like Chun-Li and Spider-Man using air dashes that travel extremely far forward during their first few frames, so when a rollback occurs, these characters just warp all over the place and the player’s frustration goes way up. You don’t have to sacrifice the speed of Marvel, either, you just have to consider how you animate these startup frames. What if the airdash had two or three frames of stationary or slower moving startup with the same total duration? Your game plays almost identically but the visual artifacts of rollback are way reduced.

Divekick is another example of this. We implemented GGPO very well in that game, but I didn’t know how to design the game with rollbacks in mind, so the startup of all jumps and all divekicks is instant. There’s no prejump or anything, so the teleporting can get a little wild in that game with the faster characters. If I had added just two or three frames to the startup of jumps and divekicks, the game would feel and play mostly the same but look vastly better online.

In general, though, designers just need to think about how rollbacks will look in their game. One of the things people don’t like about rollback is when a move looks like it hit, but then rolls back into being blocked and you get yourself killed. Firstly, I should say that a game with extremely tight hit confirms like this would play extremely poorly in delay-based netcode, so the answer is not to avoid rollbacks. But you can design your game such that hit confirms right on the edge of offline human reaction into a -20 on block move aren’t required to play the game. This could manifest itself in all sorts of ways, like slightly bigger cancel windows, how safe certain special moves are, how much damage you can get off these confirms, and how good your defensive options are. If you design the game such that moments around rollback are pivotal every five seconds, you’re not playing smart with the system, and you’re increasing player frustration.

Why do you think fighting games are always peer-to-peer and never a client-server architecture like so many other genres? Does rollback not play well with a server-based system? Should fighting games switch?

krazhier: Well first of all, there’s no reason a server can’t act as an intermediary just to send inputs to each player. I believe Brawlhalla uses such a system, and Street Fighter V will sometimes fall back on using a server between players if they can’t connect directly to each other. But I think the main two reasons are latency and cost. Direct peer-to-peer connections means your message gets to the other client directly rather than routing through a server, and you also don’t have to pay costs to keep a server up and running.

As for why we do lockstep as opposed to server-client topology, I think either is possible in theory. But it’s been my experience that lockstep works nicely for fighting games because the only variable is player input. So if you build a deterministic simulation, you can derive game state off input alone rather than needing to do a lot of work on client-side prediction the way that shooters do. Some older fighting games actually did host-client architecture instead of lockstep, and those games had differences in online vs offline play.

Keits: The important thing about lockstep is that everybody is simulating the exact same thing. If one person’s PC runs the result of a hitbox collision slightly faster, or differently, due to floating point error for example, then you aren’t lockstep and you’d need a server to track the "real" game state. In a server-client system, when you see my character, you are seeing a ghost of my character’s past, instead of waiting for inputs to be sure we are synced. I do something, then the server finds out and verifies, and then sends it to you and then you see it—it’s not actually synced.

In Overwatch for example, if I see you running towards a corner as I aim at you, I shoot you just before you make it around the corner. But actually on your side, you had already passed the corner. Now, should you take damage? My shot technically went to a place you were not, and you were behind a wall that should stop shots, but on my screen it was a clean hit. There are dozens of tricks games use to make that feel not as bad in shooters, but shooters don’t have the same type of hit reactions that fighting games do. If you take a bullet behind a corner for some small damage, you don’t lose agency or control over your character and it’s more tolerable. But if you take a Ryu heavy punch "behind a corner," you can clearly see it didn’t hit and then you are being comboed in a normally impossible situation. This would feel really frustrating to play. If you wanted to make server-client work for a fighting game, you’d have to do a lot of weird stuff to built the whole game around those limitations.

The prediction model in fighting games simply duplicates the last-known frame until input is received. Should we try to improve our prediction model?

krazhier: I’ve thought about this a bunch. Fighting games in general are pretty great in that, more often than not, you are pressing (or not pressing) the same button you pressed the frame before. When you do a quarter circle forwards, you might press down for two frames, then down-right for two frames, then maybe right for around four frames. Now, there’s a world where we could maybe guess "hey, he’s going to do a quarter circle." But what about those times where we only sit on each input for one frame in a row? It just makes it so that predicting to this degree is not something that makes a ton of sense.

I imagine it’s considerably worse to predict he will throw a fireball, then have it rollback when your guess was wrong, instead of just waiting until the fireball is actually thrown and then rollback the startup by one or two frames?

krazhier: I would imagine you’d be wrong considerably more often than right if you went the approach of actually trying to guess the end result before it happens. And I think a solution that mispredicts with a rate of even 10%, and then has to rollback buttons that never happened and the player wasn’t even thinking of pressing, would feel so much worse than the current prediction model that it’s probably not worth it.

How does rollback work with spectators or more than two players? Is the model robust to adding more players?

krazhier: Rollback can work fine with more than two players. Iron Galaxy actually shipped a port of a four-player dungeon crawler named Dungeons & Dragons: Chronicles of Mystara that uses GGPO, and it works fantastic. As you would expect, with more players you’ll have more opportunities to roll back, and you’ll have to send more data across the network. Eventually you might hit a limit where there are too many players and the bandwidth hit is too big, but the limit is definitely not two players.

Keits: As for spectators, they don’t have to see the game truly "live," they can see it 10, 20, or 30 frames in the past and it’s still close enough to live. This means that their delay isn’t hyper sensitive, and it doesn’t put the burden on the players to have to send their inputs not only to each other, but also to eight spectators with the same type of urgency.

krazhier: You could, for example, bundle and then compress all the inputs for the last 30 frames and designate one player to send it to all the spectators. Because it only happens every 30 frames and the inputs can be compressed, the bandwidth hit isn’t that high. You could also designate a server to do this, if you’re willing to incur the cost. But in theory, spectators shouldn’t have to roll back. They are just running on a delay-based system with very large delay, since they should always have the input they need to run the simulation. And if a packet loss happens, the game doesn’t have to kill itself to get the inputs there immediately. Because you’re running 30 or more frames behind, you can just resend the packets 10 frames later or whatnot and it usually works out just fine.

I think all of us could talk about these topics all day, but perhaps it’s best to start summing up some final thoughts. Do you guys have any general thoughts on rollback and its use in games in general?

krazhier: I can tell you that if it wasn’t for GGPO, I would not be the proponent of rollback that I am today. It’s funny, because when I first saw rollbacks in action, my reaction was "duh, why isn’t everyone doing it??" And then when Iron Galaxy first did it on 3rd Strike, it was like "wow, this is just magic and it works." So I think GGPO is incredibly obvious in hindsight, and that’s maybe the coolest thing about it. I worked on lockstep games before coming to Iron Galaxy, like Blitz the League and NBA Ballers, and I would have never thought to do what GGPO does. That’s why I love GGPO and rollback solutions, not only as a gamer but also what it’s changed in my career.

We’ve been doing delay-based lockstep networking architecture for 25 years. Quake came out and we switched to a server-client architecture where the server tells everyone what they should be seeing and everyone interpolates it in order to make it look smooth. When this happened, lockstep networking kind of stopped evolving. It’s my opinion that rollback is the next step for lockstep networking and, just like every other discipline in video game development, it’s time for lockstep implementations to evolve. The time investment is just super worth it to me.

Keits: I don’t think there’s anything more valuable to online fighting games than good netcode, so to me it’s a no brainer that devs should prioritize rollback in their development. You need to change a bit about how your game is built, since you now need to manage your game state differently and be able to both load it and simulate it very fast, but the end result is being able to confidently tell your fans that they can enjoy your game coast to coast and even be able to play most matches overseas. It’s the confidence that says "you can run online tournaments and they won’t be a joke, the results will be valid."

Can rollback improve at all? What possible advancements in rollback should developers be experimenting with over the next decade?

krazhier: I think the main two ways to improve are implementing ways that can reduce the number of rollbacks needed, and improving the per-frame simulation time. There are still unexplored tricks for these two things. Exploring other genres besides fighting games is another possibility. It’s cool that games like For Honor, a third-person action game, have implemented rollbacks, but we can do more here. For example, I’d love to do a rollback sports game.

It would also be cool to see rollbacks go beyond the small scope and enter more of a hybrid approach where we can use frame-lock rollback simulation for "significant" actions (that is, actions that are close to you), and use more traditional server authority architecture for events further away from you. Can you imagine an MMO action game that used rollback deterministic solutions for close combat gameplay, but could use the advantage of having a server to do the heavy lifting for the rest of the world? These would be fun problems to explore.

Sajam interview

I also thought it would be extremely valuable to get some perspective on netcode from a community member who lives and breathes fighting games every day. Stephen "Sajam" Lyon has commentated tons of top 8s for the fighting game community's biggest tournaments, plays and studies a dozen different fighting games, and loves to chat about important fighting game topics on his Twitch and YouTube channels.

Thanks for taking some time to chat!

Sajam: Thanks for having me! I spend a lot of time chatting about networking for fighting games, and I’m always happy to chat about rollback and delay based solutions.

I think it's a good idea to learn a bit about people first experienced online fighting games. What was your first experience with rollback in online fighting games?

Sajam: The first time I ever played on rollback netcode was for Street Fighter x Tekken back in 2012. At the time, I remember feeling like the netcode was really different from Street Fighter 4 and Ultimate Marvel vs Capcom 3, but I didn’t think too much of it. Then the following year, I got to try Killer Instinct and realized just how important rollback netcode was. I remember playing Southern California to New York on Killer Instinct, and thinking that this was the future of next-gen fighting games online. Even a few years later, I was amazed at Injustice 2's online experience when I had really fun online sessions with players from the East Coast US, and even the UK!

It’s no secret that we both enjoy Killer Instinct and I think a lot of that has to do with the online experience. For me, coming from years of bad online fighting games, it was so nice to see a big studio finally do rollback well, and to find out that it wasn’t just reserved for retro games played on the GGPO beta on PC. On that topic, why do you think so few games have rollback netcode?

Sajam: In most cases, I expect it’s because by the time developers consider what to do with online play, they are super deep into development and use the fastest/easiest solution. Which is unfortunate, because 95% of fighting games are about playing another opponent, and most of the time you do that online. So without a good online solution, the majority of the fun of the game can be tough to find. These days though, it seems crazy for developers not to consider the online portion of the game important, so hopefully that’s something that can change!

I agree, but the interesting thing is that GGPO is not all that new anymore. It was developed a decade ago and it’s only been in a few mainstream games since then. Why do you think adopting rollback is taking so long? If it was the best solution, shouldn’t it be in more games by now?

Sajam: I think the demand for rollback has been especially loud lately, despite it being a really great solution for a long time. Western developers seem to have embraced rollback, but it’s struggling to be common in Japanese games. That’s especially unfortunate because Japan creates so many beloved fighting game franchises like Street Fighter, Tekken, King of Fighters, Soul Calibur, Marvel vs Capcom, Guilty Gear, BlazBlue, Under Night In-Birth, and uh, I think you get the point.

How do you think the ecosystem around these games would change if they implemented rollback netcode? How would the players, commentators, and fans of the game benefit?

Sajam: Well for starters, the amount of possible opponents for somebody to play with online expands by an incredible amount. That alone is a huge deal for communities with smaller player pools, or for regions who want to train together but can’t because of a large distance. We see this with SonicFox training consistently with players from all over the US, and Europe in Mortal Kombat 11 every week on their stream.

Besides casual and competitive players gaining possible opponents, it gives credibility to online events/tournaments. Events like the Capcom Pro Tour Online gain respect as a viable method for determining who should be handed out points on their competitive circuits. It also lets a developer run ranked competitions or leagues knowing there is validity to it. For commentators, all that online match footage from tournaments becomes usable, and we as commentators would have more access to online tournaments without the need to travel. I’ll stop here, because I think I have an endless list of reasons why it benefits tons of different types of fighting game fans and players.

A lot of those benefits seem targeted towards competitive players or those who are super invested in the game. Do you think the casual player would notice rollback over a basic delay-based solution? You often hear comments like “when I play the game, there’s no lag” coming from casuals who might not even think twice if rollback was in the game. Should the devs only put rollback in because of the hardcore community, or are there benefits for the casual fans too?

Sajam: Oh there are definitely benefits for casual fans. For starters, lots of casual fans who play games on console use Wi-Fi, and that is really brutal for delay based solutions. Rollback is much better at handling inconsistent connection types, and making it seem like there was no lag there at all. In the case of casual fans, expanding the amount of friends they are able to play with it based on distance, and making online play feel great is such a huge win. It’s hard to explain how often I hear people say that bad netcode, and lack of good online features really hurt somebody’s enjoyment of fighting games. As far as noticing delay based solutions vs rollback goes, a big reason rollback is great is actually how well it masks bad lag.

A point of view that I’ve heard from some players is that when a game has good netcode, people would rather sit at home and play online than travel to locals or tournaments for a game. Do you think there’s any merit to this idea that good netcode may actually, in some ways, be a “bad thing” or have a negative impact on local scenes?

Sajam: I’ve heard the same thing, and I feel like it’s a pretty interesting thing to consider. As good as netcode can be, it won’t replace the feeling of playing fighting games offline, and it can never replace the community aspect that exists in our grassroots events and tournaments. That being said, I can definitely see the ability to play lots of people online with a smooth connection be tempting over attending a local occasionally. In that case, I think it opens up the door to play with those some local friends online, or even host local/regional events weekly online. In either case, I don’t expect it steps on the toes of larger regionals or majors very much at all. Those are worth attending, regardless of how good online play is.

Often when consumers are unhappy with something, they choose to “vote with their wallet”. In the case of fighting games, there’s been some sentiment that maybe the community at large should stop supporting games that don’t implement rollback so the developers get some tangible feedback. On the other hand, it’s very hard to not be excited about some fighting games on the horizon, even if it’s unlikely that they’ll use rollback. What would you say to someone who’s caught in the middle of wanting to play cool games, but also wanting to try and vote with their wallet about the rollback issue?

Sajam: I mean that’s a tough spot to be, and I say that as somebody who is in that exact position all the time. I feel that exact way about the upcoming Guilty Gear, but then I have hope for stuff like the Riot fighting game announced recently as well. I think you can vote with your wallet if netcode is a deal breaker to you, and at the very least we should all demand that developers consider netcode when creating games in the modern era.

That being said, it can be tough to resist with how many great fighting games come from Japan with delay based netcode. Being a fighting game fan is full of these moments of, “Man this is so close to being awesome, but it’s too bad they didn’t do this or that.” And feeling that all the time gets pretty tough I think, especially when you look at other competitive games out there. Especially when many of the things we want in fighting games, like good netcode, have existed for over a decade now. It’s really a shame, I think.

Do you think SFV’s implementation of rollback has done damage to rollback’s reputation as a good online solution for fighting games?

Sajam: I think it undoubtedly has, yeah. Unfortunately, Capcom tried to do right by implementing rollback netcode, but had some issues, some of which were mentioned in this article, that lead to the current state of SFV’s online play. That being said, it’s important for articles like this, and creators like myself to use our voices and let people know that rollback is still the best option available.

Some people have even cited that they enjoy the netcode in SFV more than SF4, which is really fascinating to me. The idea that somebody would enjoy a rollback solution that doesn’t function correctly, versus a pretty standard delay based solution, is a good sign that they were on the right path. Still, if SFV’s version of rollback has soured you, I can’t blame you. I do encourage you to look at games like Killer Instinct, Punch Planet, Skullgirls, Mortal Kombat 11, etc that have done rollback correctly!

Any closing thoughts you want to share with people who play lots of online fighting games?

Sajam: Yeah! Mostly that adding rollback netcode is tough if developers don’t decide that’s the route for them from the start, so any time a developer uses it make sure to thank them and appreciate how awesome fighting games with good netcode are! Besides that, thank you for writing this awesome article, and thanks to everybody who took the time to read this interview!

I would like to thank krazhier and Keits for taking hours out of their busy schedules to discuss technical aspects of netcode with me, and Sajam for taking time to answer interview questions and being supportive throughout the writing process. I would also like to especially thank MagicMoste for making all the wonderful videos you see in this article. All their help was offered for free and I am thankful for their friendship.

Also thank you, the reader, for sticking with this article to the end! My sincere hope is that you learned something new about the challenges of making fighting games play well online, and if conversations come up about rollback netcode with your friends or other members of the FGC, I hope it's easier to have informed discussions with them. Thanks for reading, and until next time!

Listing image: Aurich Lawson / Capcom / Getty Images

139 Comments