Official Codex Discord Server

  1. Welcome to rpgcodex.net, a site dedicated to discussing computer based role-playing games in a free and open fashion. We're less strict than other forums, but please refer to the rules.

    "This message is awaiting moderator approval": All new users must pass through our moderation queue before they will be able to post normally. Until your account has "passed" your posts will only be visible to yourself (and moderators) until they are approved. Give us a week to get around to approving / deleting / ignoring your mundane opinion on crap before hassling us about it. Once you have passed the moderation period (think of it as a test), you will be able to post normally, just like all the other retards.
    Dismiss Notice

Development Info Pathfinder: Wrath of the Righteous Kickstarter Update #19: QA Automation Tools

Discussion in 'News & Content Feedback' started by Infinitron, Feb 18, 2020.

  1. Infinitron I post news Patron

    Infinitron
    Joined:
    Jan 28, 2011
    Messages:
    85,726
    Grab the Codex by the pussy Dead State Divinity: Original Sin Project: Eternity Torment: Tides of Numenera Wasteland 2 Shadorwun: Hong Kong Divinity: Original Sin 2 A Beautifully Desolate Campaign Pillars of Eternity 2: Deadfire Pathfinder: Kingmaker Pathfinder: Kingmaker
    Tags: Owlcat Games; Pathfinder: Wrath of the Righteous

    Pathfinder: Kingmaker was in a disastrously buggy state when it launched and Owlcat have pledged to do better with Pathfinder: Wrath of the Righteous. One way they aim to achieve that goal is by using automated QA testing. Today's rather interesting Kickstarter update describes some of their QA tools, which include a bot that automatically plays through the game as well as various smart logging and script inspection mechanisms to assist in fixing bugs more quickly.

    When development started on Wrath of the Righteous, we decided that quality was our number one priority. We carefully calculated that to achieve our goal, we would need 23 manual testers working all day, every day from now until launch. That’s an eye-watering number of man hours to spend on testing, and that doesn’t even include management, coordination, and everything else that goes into QA. We decided that the smarter approach would be to increase the number of manual testers by a sensible amount, and alongside this, to focus on strengthening our ranks with people who could build us the extra help we needed. Yes, you heard that right! In the QA department at Owlcat Games, all our in-house tools are considered part of the team, and we named them after creatures from Golarion.

    Clockwork & Arbiter

    The first team member we want to introduce is Clockwork, our bot that plays through the game for us. He can:
    • Play through a specific scene. This helps us to maintain the game at the quality level already achieved and, most importantly, save time on regression testing, the most labor-intensive part of the testing process. Playing through an 80-hour game in a gazillion different ways takes a long, long time. But it’s essential work, since indirect changes can cause parts of the game to break.
    • Play according to simple behavior patterns, for example, “kill everything on sight.” These exploratory excursions will help us understand how difficult the game can be, depending on the actions players choose to take.
    We’ll be expanding Clockwork’s abilities even more, so that he can find the maximum number of bug combinations and regressions. But while our automation team is developing Clockwork’s new features, he’ll be spending night after night tirelessly slaying demons (current kill count: 83,452).

    So, while Clockwork is busy checking the game’s mechanics, our second bot, Arbiter, is responsible for tracking visual changes in the game world. He rants indignantly whenever he detects a missing texture or model, or sees a character holding a weapon the wrong way round, or notices a drop in performance.

    The best thing about our bots is that they find bugs super efficiently, allowing us to quickly figure out what’s causing the issue and resolve it. Making fixes during the dev process increases the overall quality of the game, and also saves time that would have been spent on bug reports and fixes further down the line, since this work will stop some bugs from cropping up entirely.

    Aeon, Scrivenite & Bebilith

    Besides direct testing, we’re also working on reducing the time spent on finding and fixing bugs. That’s where the next guys come in.

    Aeon is our comms guardian. All the messages you send us through the in-game report function pass through his true-neutral hands. Since Pathfinder: Kingmaker, he’s learned how to gather more information to understand the context around a reported error. Now the developers can log all bugs directly from the game.

    Aeon also notifies the devs about critical errors, holds all the necessary information about preparing the current release, and provides it on request.

    The game data consists of interrelated resources. Imagine you’re waiting for the final line of dialogue in a romance storyline. You wait and wait, but it never comes. You need this line to appear before you can finally embark on a relationship with your paramour of choice. Why isn’t the line appearing? It could be there’s a problem with the line itself, or maybe there’s an issue with the conditions required to trigger it: 3 completed quests, 10 dialogue phrases spoken at just the right time, and 18 decisions that endear you to your beloved. During debugging, the time it takes to manually study all these connections is staggering.

    But with Scrivenite and Bebilith, we make this process more transparent. Scrivenite contains the entire history of changes made to resources, and Bebilith records all the connections between them. They complement each other beautifully, and together they know everything about changes made to the game code.

    This data-driven duo helps us quickly see how fundamental a change is, and what percentage of the game has been touched. They also hugely speed up the process of searching for an error, the author, and when the error occurred. They let us see a clear list of changes that have been made to a specific section of the game, and whether these changes may affect the rest of the game.

    On top of all this, they also seriously cut down the time it takes to finalize a game build, meaning that we can get fixes out to players in record time.
    This is actually the first time we've posted about a Kickstarter update for this game that wasn't about the campaign's next stretch goal. At the rate the campaign is going right now, it'll be a long time until it gets there. We'll see what other interesting updates Owlcat can come up with during this period.
     
    • Informative Informative x 2
    • Brofist Brofist x 1
    • "It was Aliens" "It was Aliens" x 1
    ^ Top  
  2. Jaedar Arcane Patron

    Jaedar
    Joined:
    Aug 5, 2009
    Messages:
    6,540
    Project: Eternity Shadorwun: Hong Kong Divinity: Original Sin 2 Pathfinder: Kingmaker
    Pretty neat, although I don't know how common these kinds of bots are.
     
    ^ Top  
  3. Van-d-all Arbiter

    Van-d-all
    Joined:
    Jan 18, 2017
    Messages:
    1,308
    Location:
    Smallest Church In Saint-Saëns
    Smoke testing bots are certainly an improvement, but they also tend make new bugs just pop around the preset test paths. I just hope they won't give in into the pitfall of automation and feel like they can cut back on human QA.
     
    ^ Top  
  4. gellyo Barely Literate

    gellyo
    Joined:
    Jul 24, 2019
    Messages:
    2
    That looks like a really well-designed bot, but yeah human QA will still be necessary ultimately
     
    ^ Top  

As an Amazon Associate, rpgcodex.net earns from qualifying purchases.