Here's an interesting write up by Rami Ismail talking about the patch process.
There’s two things that are relevant here:
1.Consoles are platforms and devices that come with an expectation of quality, and as such have systems in place to ensure that quality.
2.Developers are creatives working on a commercial schedule, leading to the ancient and never-broken rule that a developer will always be two weeks late for their deadline – no matter how big or small the deadline is.
When you make a game for console – no matter which one – you have to realize the systems developers and platform holders are dealing with weren’t built for 2016. These systems are legacy systems, built upon legacy systems, built upon legacy systems. It’s like the system you are or were forced to use at your day job, if you’ve ever worked in retail, stock or booking systems, or at any checkout. Many of these systems interface closely with bureaucracy and console technology, and instead of radically changing how systems work between generations and breaking everything at a console launch, console platforms tend to try and not break things that work. As such, many of these systems are unwieldy, complicated, intricate and really built for teams that can afford to hire more people to read the manual and figure out the systems. These systems come from an age before indie, and some of the manual pages still refer to mailing copies by postal mail. Despite that, the console creators and their teams all work super hard to ensure developers have a smooth process, improving their systems where possible without touching the legacy foundation, and ensuring players get a functional game.
The most egregious example of this is called ‘certification’. On computer platforms, stores like Steam, Humble, GOG and itch.iohave decided that developers just have to deal with the fallout of releasing a broken product themselves, and thus allow you to push a product or patch at any point whatsoever (they often do a pre-release check of your store page, though!). Consoles on the other hand, come from the ‘Seal of Quality’ mindset. To ensure that quality, they use a system called ‘certification’, or FQA, or TRC, or TCR, or some other random acronym that refers to something technical and a checklist. Devs like to call this quality assurance process ‘cert’, and no matter what developer you ask, you’ll find most developers understand why it exists, and we all really appreciate all the people working super hard to ensure our games are working right, but we tend to all hate ‘cert’. What you have to imagine when it comes to cert, is a giant book of checkboxes. There’s an absurd amount of them, and they could be different not only per platform, but per territory (for example, a European build has different certification rules than a US one, requiring differences between the two), and sometimes even between a patch, a DLC, and a release version.
Some of these make a lot of sense (don’t crash), and some of these are reasonable (if you leave the main menu open for 24 hours, is the game still smooth?), and some of them sound obscene (if you rapidly plug and unplug the controller, does the game know what to do?). Some of these are enlightening (your game needs to figure out what controller the player is assigned to, thus requiring the ‘Press [button] to start’ screen only console games still have), and some of them are just headaches (don’t put UI in the outer 10% of the screen, unless you use one of those ‘how big is your screen’ interfaces). Some are legal (is any form of parental control activated or is the profile under the allowed age for gameplay? if so, did you disable the required functionality?), and some can make you desperate (the console can not have had firmware updates between your release build and the patch). Did you know consoles don’t necessarily pause your game for you when players switch to other interfaces? You have to do that yourself.
Anyway, certification is a big thing. The process for it is also a big thing. In most cases, you have to fill out a weirdly complex form for your submission, and then ‘book a slot’ of sorts, or wait until you get an OK to submit your game. Since the people testing games aren’t infinite, you need to let people know when you’re submitting your build. So you check which dates are available, and usually there’s a free slot a few days from today. If your build isn’t in a certain amount of time ahead of that, your slot can be lost, and you’ll have to ‘book’ a new one. When the slot starts, and your game goes into certification, there’s a variety of reasons your submission can be rejected, in which case your slot can be lost too. Then, the teams start testing, and they report certification violations to you. In many cases, your violations are graded along a scale from ‘Must Fix’ or ‘Fix This’ to ‘Suggestion’ or ‘Whatever really’. If any of them exceed a certain level, your submission fails, and you have to start from square one and fill out the form, find a free day to book a slot or wait until certification starts, and submit a new build.
Some of the certification problems are impossible to avoid. Developers can’t control when consoles update their firmware, and when engines shift their support for firmware. In those cases, developers can request an exception to a rule. This is a bureaucratic process, and it can require negotiating, a formal request, and formal permission. It takes time. Then, when you get an exception, for most platforms you often need to fill out on the submission form how, where, and from who you got that exception when you submit again.
Did I mention all of this is poorly documented? One console has a field that says ‘assets file’. It doesn’t mention what the assets file is, nor what it does, or what these assets are. If you don’t add the file, it can’t process your submission. If you add it, but it isn’t ‘right’, your build can fail. You lose a week. If there’s a checkbox somewhere in the hundreds or thousands of obscure rules that you missed, you lose a week. If there’s something that’s subtly different between Europe and America, you lose a week. What I’m trying to say is that certification could take a week, and in the worst cases, it could take months. From personal experiences, I can say that it can make developers cry. It could delay your game. At the end, though, the game that launches checks every checkbox. You’ve got your proverbial “Seal of Quality”. Your game is allowed to launch.
Now, I’m not saying No Man’s Sky did this, but in most cases, developers with a launch date need to make sure they can hit that launch date. We start submitting certification builds somewhat early, in the hopes that one of them gets the check mark that says ‘you’re good to negotiate a launch day’. Certification is technical – it doesn’t bother with what the game is, it just concerns itself with whether it works technically. It checks whether the boxes are checked. You can market a dark gritty murder game titled Dark Horses, and submit a pony farm tycoon game, and as long as the name on the form matches the name of the game, they would not object.
So – since development is messy and unpredictable – if you’ve got a build that’s ‘pretty much done’ and it works, and you can get it through certification, that’s a good sign for your final build. So you submit it, it goes through cert, and you stage it for download and launch. For disc games, the game needs to go through certification in time to be printed, boxed and distributed. At that point, developers are usually still one to three months from release, which tends to mean you need one and a half to three and a half months to get the game done, and then you still need to keep in mind that unpredictable amount of time you’ll spend in certification. A day 1 patch is technically still submitted at least one week from launch, but until it actually goes through certification, it can’t be made available to the public.
Knowing that, it should be easy to see why Day 1 patches are often “huge”. For a game that goes on disc, the ‘gold’ build that went through certification is one to three months old by the time the game launches. That gives developers half a month to two and a half a month to do a month and a half to three and a half months’ worth of work to make the game ‘perfect’while still hitting the release date with the patch. If your studio is huge, you probably have an internal QA department that (for good reason) slows things down internally, but if your studio is nimble and small, you can change enormous portions of the game in that span of time.
So in the hypothetical example of No Man’s Sky, when No Man’s Sky launches, for most people, it’ll launch into the intended experience thanks to the Day 1 Patch. That build is as close to what the developers envisioned as they worked, learned and improved upon that vision. That’s No Man’s Sky. The version that is on the disc, however, is months old. The only way to avoid that kind of thing is to not launch on disc.
One could argue that developers then, should make certain that a game is perfect when they submit it to disc, which is not an invalid stance. It’s just an impractical stance. If you’ve got months to improve upon a game that went through cert, do you think you would leave those months? Do you think audiences would appreciate a developer just kind of doing nothing for three months? Can you imagine the Kickstarter outrage if a developer, three months from launch, posted ‘we’re done, it’s good, we’re not touching it again until you get to play in three months’? Anybody arguing that a game should be done when it goes ‘gold’ is living in the 90s.
Developers care about the games they make, and we’re trying to make the best game we can for our players. We’ll take every opportunity we can get for that. If consoles operated like Steam did, No Man’s Sky wouldn’t have a Day 1 Patch, because the build you’d download and play when it comes out would’ve been submitted comfortably a few days before launch. Day 1 Patches aren’t necessarily a failure on the developers or the platforms side, they the result of people that care about what they make, trying to deliver the game the audience expects by the date they expect it, while everybody involved is struggling with outdated systems on cutting-edge technology. Everyone is trying their hardest. Nobody is doing anything wrong. The developer isn’t lazy. The platforms aren’t malicious. Day 1 patches are simply a patch to a submission system that’s old and convoluted.
There’s two things that are relevant here:
1.Consoles are platforms and devices that come with an expectation of quality, and as such have systems in place to ensure that quality.
2.Developers are creatives working on a commercial schedule, leading to the ancient and never-broken rule that a developer will always be two weeks late for their deadline – no matter how big or small the deadline is.
When you make a game for console – no matter which one – you have to realize the systems developers and platform holders are dealing with weren’t built for 2016. These systems are legacy systems, built upon legacy systems, built upon legacy systems. It’s like the system you are or were forced to use at your day job, if you’ve ever worked in retail, stock or booking systems, or at any checkout. Many of these systems interface closely with bureaucracy and console technology, and instead of radically changing how systems work between generations and breaking everything at a console launch, console platforms tend to try and not break things that work. As such, many of these systems are unwieldy, complicated, intricate and really built for teams that can afford to hire more people to read the manual and figure out the systems. These systems come from an age before indie, and some of the manual pages still refer to mailing copies by postal mail. Despite that, the console creators and their teams all work super hard to ensure developers have a smooth process, improving their systems where possible without touching the legacy foundation, and ensuring players get a functional game.
The most egregious example of this is called ‘certification’. On computer platforms, stores like Steam, Humble, GOG and itch.iohave decided that developers just have to deal with the fallout of releasing a broken product themselves, and thus allow you to push a product or patch at any point whatsoever (they often do a pre-release check of your store page, though!). Consoles on the other hand, come from the ‘Seal of Quality’ mindset. To ensure that quality, they use a system called ‘certification’, or FQA, or TRC, or TCR, or some other random acronym that refers to something technical and a checklist. Devs like to call this quality assurance process ‘cert’, and no matter what developer you ask, you’ll find most developers understand why it exists, and we all really appreciate all the people working super hard to ensure our games are working right, but we tend to all hate ‘cert’. What you have to imagine when it comes to cert, is a giant book of checkboxes. There’s an absurd amount of them, and they could be different not only per platform, but per territory (for example, a European build has different certification rules than a US one, requiring differences between the two), and sometimes even between a patch, a DLC, and a release version.
Some of these make a lot of sense (don’t crash), and some of these are reasonable (if you leave the main menu open for 24 hours, is the game still smooth?), and some of them sound obscene (if you rapidly plug and unplug the controller, does the game know what to do?). Some of these are enlightening (your game needs to figure out what controller the player is assigned to, thus requiring the ‘Press [button] to start’ screen only console games still have), and some of them are just headaches (don’t put UI in the outer 10% of the screen, unless you use one of those ‘how big is your screen’ interfaces). Some are legal (is any form of parental control activated or is the profile under the allowed age for gameplay? if so, did you disable the required functionality?), and some can make you desperate (the console can not have had firmware updates between your release build and the patch). Did you know consoles don’t necessarily pause your game for you when players switch to other interfaces? You have to do that yourself.
Anyway, certification is a big thing. The process for it is also a big thing. In most cases, you have to fill out a weirdly complex form for your submission, and then ‘book a slot’ of sorts, or wait until you get an OK to submit your game. Since the people testing games aren’t infinite, you need to let people know when you’re submitting your build. So you check which dates are available, and usually there’s a free slot a few days from today. If your build isn’t in a certain amount of time ahead of that, your slot can be lost, and you’ll have to ‘book’ a new one. When the slot starts, and your game goes into certification, there’s a variety of reasons your submission can be rejected, in which case your slot can be lost too. Then, the teams start testing, and they report certification violations to you. In many cases, your violations are graded along a scale from ‘Must Fix’ or ‘Fix This’ to ‘Suggestion’ or ‘Whatever really’. If any of them exceed a certain level, your submission fails, and you have to start from square one and fill out the form, find a free day to book a slot or wait until certification starts, and submit a new build.
Some of the certification problems are impossible to avoid. Developers can’t control when consoles update their firmware, and when engines shift their support for firmware. In those cases, developers can request an exception to a rule. This is a bureaucratic process, and it can require negotiating, a formal request, and formal permission. It takes time. Then, when you get an exception, for most platforms you often need to fill out on the submission form how, where, and from who you got that exception when you submit again.
Did I mention all of this is poorly documented? One console has a field that says ‘assets file’. It doesn’t mention what the assets file is, nor what it does, or what these assets are. If you don’t add the file, it can’t process your submission. If you add it, but it isn’t ‘right’, your build can fail. You lose a week. If there’s a checkbox somewhere in the hundreds or thousands of obscure rules that you missed, you lose a week. If there’s something that’s subtly different between Europe and America, you lose a week. What I’m trying to say is that certification could take a week, and in the worst cases, it could take months. From personal experiences, I can say that it can make developers cry. It could delay your game. At the end, though, the game that launches checks every checkbox. You’ve got your proverbial “Seal of Quality”. Your game is allowed to launch.
Now, I’m not saying No Man’s Sky did this, but in most cases, developers with a launch date need to make sure they can hit that launch date. We start submitting certification builds somewhat early, in the hopes that one of them gets the check mark that says ‘you’re good to negotiate a launch day’. Certification is technical – it doesn’t bother with what the game is, it just concerns itself with whether it works technically. It checks whether the boxes are checked. You can market a dark gritty murder game titled Dark Horses, and submit a pony farm tycoon game, and as long as the name on the form matches the name of the game, they would not object.
So – since development is messy and unpredictable – if you’ve got a build that’s ‘pretty much done’ and it works, and you can get it through certification, that’s a good sign for your final build. So you submit it, it goes through cert, and you stage it for download and launch. For disc games, the game needs to go through certification in time to be printed, boxed and distributed. At that point, developers are usually still one to three months from release, which tends to mean you need one and a half to three and a half months to get the game done, and then you still need to keep in mind that unpredictable amount of time you’ll spend in certification. A day 1 patch is technically still submitted at least one week from launch, but until it actually goes through certification, it can’t be made available to the public.
Knowing that, it should be easy to see why Day 1 patches are often “huge”. For a game that goes on disc, the ‘gold’ build that went through certification is one to three months old by the time the game launches. That gives developers half a month to two and a half a month to do a month and a half to three and a half months’ worth of work to make the game ‘perfect’while still hitting the release date with the patch. If your studio is huge, you probably have an internal QA department that (for good reason) slows things down internally, but if your studio is nimble and small, you can change enormous portions of the game in that span of time.
So in the hypothetical example of No Man’s Sky, when No Man’s Sky launches, for most people, it’ll launch into the intended experience thanks to the Day 1 Patch. That build is as close to what the developers envisioned as they worked, learned and improved upon that vision. That’s No Man’s Sky. The version that is on the disc, however, is months old. The only way to avoid that kind of thing is to not launch on disc.
One could argue that developers then, should make certain that a game is perfect when they submit it to disc, which is not an invalid stance. It’s just an impractical stance. If you’ve got months to improve upon a game that went through cert, do you think you would leave those months? Do you think audiences would appreciate a developer just kind of doing nothing for three months? Can you imagine the Kickstarter outrage if a developer, three months from launch, posted ‘we’re done, it’s good, we’re not touching it again until you get to play in three months’? Anybody arguing that a game should be done when it goes ‘gold’ is living in the 90s.
Developers care about the games they make, and we’re trying to make the best game we can for our players. We’ll take every opportunity we can get for that. If consoles operated like Steam did, No Man’s Sky wouldn’t have a Day 1 Patch, because the build you’d download and play when it comes out would’ve been submitted comfortably a few days before launch. Day 1 Patches aren’t necessarily a failure on the developers or the platforms side, they the result of people that care about what they make, trying to deliver the game the audience expects by the date they expect it, while everybody involved is struggling with outdated systems on cutting-edge technology. Everyone is trying their hardest. Nobody is doing anything wrong. The developer isn’t lazy. The platforms aren’t malicious. Day 1 patches are simply a patch to a submission system that’s old and convoluted.
