- Joined
- Jan 28, 2011
- Messages
- 97,442
Tags: Owlcat Games; Pathfinder: Wrath of the Righteous
Pathfinder: Kingmaker was in a disastrously buggy state when it launched and Owlcat have pledged to do better with Pathfinder: Wrath of the Righteous. One way they aim to achieve that goal is by using automated QA testing. Today's rather interesting Kickstarter update describes some of their QA tools, which include a bot that automatically plays through the game as well as various smart logging and script inspection mechanisms to assist in fixing bugs more quickly.
Pathfinder: Kingmaker was in a disastrously buggy state when it launched and Owlcat have pledged to do better with Pathfinder: Wrath of the Righteous. One way they aim to achieve that goal is by using automated QA testing. Today's rather interesting Kickstarter update describes some of their QA tools, which include a bot that automatically plays through the game as well as various smart logging and script inspection mechanisms to assist in fixing bugs more quickly.
When development started on Wrath of the Righteous, we decided that quality was our number one priority. We carefully calculated that to achieve our goal, we would need 23 manual testers working all day, every day from now until launch. That’s an eye-watering number of man hours to spend on testing, and that doesn’t even include management, coordination, and everything else that goes into QA. We decided that the smarter approach would be to increase the number of manual testers by a sensible amount, and alongside this, to focus on strengthening our ranks with people who could build us the extra help we needed. Yes, you heard that right! In the QA department at Owlcat Games, all our in-house tools are considered part of the team, and we named them after creatures from Golarion.
Clockwork & Arbiter
The first team member we want to introduce is Clockwork, our bot that plays through the game for us. He can:
So, while Clockwork is busy checking the game’s mechanics, our second bot, Arbiter, is responsible for tracking visual changes in the game world. He rants indignantly whenever he detects a missing texture or model, or sees a character holding a weapon the wrong way round, or notices a drop in performance.
The best thing about our bots is that they find bugs super efficiently, allowing us to quickly figure out what’s causing the issue and resolve it. Making fixes during the dev process increases the overall quality of the game, and also saves time that would have been spent on bug reports and fixes further down the line, since this work will stop some bugs from cropping up entirely.
Aeon, Scrivenite & Bebilith
Besides direct testing, we’re also working on reducing the time spent on finding and fixing bugs. That’s where the next guys come in.
Aeon is our comms guardian. All the messages you send us through the in-game report function pass through his true-neutral hands. Since Pathfinder: Kingmaker, he’s learned how to gather more information to understand the context around a reported error. Now the developers can log all bugs directly from the game.
Aeon also notifies the devs about critical errors, holds all the necessary information about preparing the current release, and provides it on request.
The game data consists of interrelated resources. Imagine you’re waiting for the final line of dialogue in a romance storyline. You wait and wait, but it never comes. You need this line to appear before you can finally embark on a relationship with your paramour of choice. Why isn’t the line appearing? It could be there’s a problem with the line itself, or maybe there’s an issue with the conditions required to trigger it: 3 completed quests, 10 dialogue phrases spoken at just the right time, and 18 decisions that endear you to your beloved. During debugging, the time it takes to manually study all these connections is staggering.
But with Scrivenite and Bebilith, we make this process more transparent. Scrivenite contains the entire history of changes made to resources, and Bebilith records all the connections between them. They complement each other beautifully, and together they know everything about changes made to the game code.
This data-driven duo helps us quickly see how fundamental a change is, and what percentage of the game has been touched. They also hugely speed up the process of searching for an error, the author, and when the error occurred. They let us see a clear list of changes that have been made to a specific section of the game, and whether these changes may affect the rest of the game.
On top of all this, they also seriously cut down the time it takes to finalize a game build, meaning that we can get fixes out to players in record time.
This is actually the first time we've posted about a Kickstarter update for this game that wasn't about the campaign's next stretch goal. At the rate the campaign is going right now, it'll be a long time until it gets there. We'll see what other interesting updates Owlcat can come up with during this period.Clockwork & Arbiter
The first team member we want to introduce is Clockwork, our bot that plays through the game for us. He can:
- Play through a specific scene. This helps us to maintain the game at the quality level already achieved and, most importantly, save time on regression testing, the most labor-intensive part of the testing process. Playing through an 80-hour game in a gazillion different ways takes a long, long time. But it’s essential work, since indirect changes can cause parts of the game to break.
- Play according to simple behavior patterns, for example, “kill everything on sight.” These exploratory excursions will help us understand how difficult the game can be, depending on the actions players choose to take.
So, while Clockwork is busy checking the game’s mechanics, our second bot, Arbiter, is responsible for tracking visual changes in the game world. He rants indignantly whenever he detects a missing texture or model, or sees a character holding a weapon the wrong way round, or notices a drop in performance.
The best thing about our bots is that they find bugs super efficiently, allowing us to quickly figure out what’s causing the issue and resolve it. Making fixes during the dev process increases the overall quality of the game, and also saves time that would have been spent on bug reports and fixes further down the line, since this work will stop some bugs from cropping up entirely.
Aeon, Scrivenite & Bebilith
Besides direct testing, we’re also working on reducing the time spent on finding and fixing bugs. That’s where the next guys come in.
Aeon is our comms guardian. All the messages you send us through the in-game report function pass through his true-neutral hands. Since Pathfinder: Kingmaker, he’s learned how to gather more information to understand the context around a reported error. Now the developers can log all bugs directly from the game.
Aeon also notifies the devs about critical errors, holds all the necessary information about preparing the current release, and provides it on request.
The game data consists of interrelated resources. Imagine you’re waiting for the final line of dialogue in a romance storyline. You wait and wait, but it never comes. You need this line to appear before you can finally embark on a relationship with your paramour of choice. Why isn’t the line appearing? It could be there’s a problem with the line itself, or maybe there’s an issue with the conditions required to trigger it: 3 completed quests, 10 dialogue phrases spoken at just the right time, and 18 decisions that endear you to your beloved. During debugging, the time it takes to manually study all these connections is staggering.
But with Scrivenite and Bebilith, we make this process more transparent. Scrivenite contains the entire history of changes made to resources, and Bebilith records all the connections between them. They complement each other beautifully, and together they know everything about changes made to the game code.
This data-driven duo helps us quickly see how fundamental a change is, and what percentage of the game has been touched. They also hugely speed up the process of searching for an error, the author, and when the error occurred. They let us see a clear list of changes that have been made to a specific section of the game, and whether these changes may affect the rest of the game.
On top of all this, they also seriously cut down the time it takes to finalize a game build, meaning that we can get fixes out to players in record time.
