Game News Bethesda: Making Games for the PC Gamers

[VentilatorOfDoom]

Tags: Bethesda Softworks; Elder Scrolls V: Skyrim

<p><a href="http://www.dsogaming.com/news/skyrim-acceleration-layer-offers-cpu-optimization-up-to-20fps-performance-boost/" target="_blank">Modders fix it.</a></p>
<p>&nbsp;</p>
<blockquote>
<p>God, I love modders. I&rsquo;m pretty sure that everyone has noticed slowdowns when entering Skyrim&rsquo;s cities and where there are a lot of NPC&rsquo;s. Have you ever wondered what was the culprit and caused that performance hit? Apparently, it was due to un-optimization as modder Arisu released a mod &ndash; that is currently compatible only with the 1.3.10 version of Skyrim &ndash; that offers up to 40% performance boost on both high-end and low-end machines. And get this right; it really does work. So yeah, this is a must-have mod for every Skyrim player.</p>
<p>&nbsp;</p>
<p>As the modder described:</p>
<p>&nbsp;</p>
<p>&ldquo;This patch will improve your frame rate by up to 40% in all CPU-dependent situations, i.e. especially in cities.It works mostly by rewriting some x87 FPU code and inlining a whole ton of useless getter functions along the critical paths because the developers at Bethesda, for some reason, compiled the game without using any of the optimization flags for release builds.&rdquo;</p>
</blockquote>
<p>So, they forgot to flag their compiled executable for those optimizations. Shit happens. Endearing, if you're thinking about it.</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p><em>Thanks, racofer.</em></p>

 

[Black]

For someone who doesn't care about scyrim, racofer-kun sure does follow it and its modding scene

 

[circ]

So they've apparently patched that piece of shit but actually making it run hasn't been a priority and neither has bucket abuse. Just like every other developer then.

 

[Breaking Axe]

Is there a way to make this work with say, the 4gb mod?

 

[kris]

I’m pretty sure that everyone has noticed slowdowns when entering Skyrim’s cities and where there are a lot of NPC’s.

Which city would that be?

 

[Zarniwoop]

So the reason why games with 2006 graphics run like shit on my high-end 2011 PC is because they're un-optimised console shit? As opposed to the commonly-held belief that consoles using mobile components from half a decade ago are just superiour to modern desktops?

And to make matters worse, game developers are lazy fucks who prefer to let modders do their jobs?

Mind = Blown.

 

[Turjan]

It's an old Bethesda tradition not to optimze their games. Morrowind was also a blatant example of complete failure on that front.

 

[Mozgoëbstvo]

I don't even care. That blight is finally gone from my computer. Finally!

AHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAAHA

 

[SCO]

It's an old Bethesda tradition not to optimze their games. Morrowind was also a blatant example of complete failure on that front.

Ah yes. The "exe optimizer", that to boot, prevented crashes from their dodgy floating point math.

And to make matters worse, game developers are lazy fucks who prefer to let modders do their jobs?

It's actually much worse: these optimizations are automatic if the compiler has them turned on. They are turned off by default mostly because the C linker is a piece of shit that can't be fixed for compatibility reasons and it takes forever to build something with optimizations.
Just incompetence.

 

[Tuco Benedicto Pacifico]

Is there a way to make this work with say, the 4gb mod?

Isn't that mod completely pointless and redundant after the third patch?

 

[villain of the story]

There will be future blights, son.

 

[Overweight Manatee]

I hope this doesn't stop Bethesda from getting their check from AMD/Intel for making people buy 2010 hardware for a 2005 game.


... I wonder if that actually happens anymore?

 

[Mozgoëbstvo]

I know, but the removal of this one from my hard drive beats the purifying sensation of the best bowel movements I had in all of 2011.

 

[Roguey]

I've read elsewhere on the internet that this thing breaks every scripted sequence. So much for "must-have mod" eh? Dumb modders.

 

[Overweight Manatee]

I've read elsewhere on the internet that this thing breaks every scripted sequence. So much for "must-have mod" eh? Dumb modders.

If so it still doesn't let Bethesda off the hook for not enabling optimizations like every developer who doesn't have their head up their ass. Applying optimizations in reverse is incredibly hacky, its no wonder that it breaks something. It should be applied when compiled from the source, but only Bethesda can do that.

If just these optimizations are gaining 40% performance I can only imagine that, correctly compiled, the Bethesda-fixed version would be significantly faster still and with no bugs. If Bethesda fixes it, I wouldn't put it past them to ignore it.

EDIT:

Things I'd like to note here:

Only a fraction of the speedup comes from using SSE2 code. The original exe also uses SSE2 code, just not in the right places where it is truly needed. This could've been prevented by using automated SSE2 vectorization and/or another math library. Interestingly, in this case, it's the dot product function that has been rewritten, which is somewhat ironically the #1 textbook example for automated vectorization in compiler demos.
Much of the speedup is gained by manually eliminating (only possible if the entire function can be reduced to 5 bytes or less), or at least simplifying calls along the critical code paths as far as possible. This doesn't even produce nearly as good results as an optimizing compiler could have because of many restrictions a compiler doesn't have to deal with in the first place, so every optimizing compiler can do and usually does an excellent job at this if told to do it. Skyrim would probably experience an execution speed gain of over 100% just by applying this single optimization, as it has drastic consequences to the amount of code that could be detected as being redundant and thus completely eliminated. I know that sounds exaggerated, and normally would be, but it isn't when you've read and profiled enough of the code to know just how bad the compiled code is.
Just 3 functions have truly been rewritten, everything else is either a variant form or an instruction-level simplification of functions consisting of things like "return *this;" which are at the very top in the profiled list because the compiler was obviously told not to inline it. So, every time a certain kind of pointer needs to be dereferenced, the game will call a lengthy function to do what can be (and is) replaced by a single instruction. Fixing this manually isn't feasible after a certain point, but the compiler can do this for the whole binary at the cost of just a few seconds extra compiling time and much better than ever possible by a human (at least at these code dimensions).
In general, the TESV code has pretty high register pressure. A huge part of this is simply due to the completely missing optimizations which would otherwise eliminate the unneeded allocations, but an x86_64 build would also definitely help improving this condition.
Jump targets are completely unaligned, including the so-called hot targets which are hit millions of times in short periods, leading to cache stress due to multiple fetches being required to execute a jump, whether correctly predicted or not. Optimizing compilers can automatically align them properly.
I guess I don't have to mention how bad the threading is; this isn't trivial to fix though. Just sad that it's almost 2012 and this thing can't even properly use two threads. Besides all the other obvious flaws, this is the main reason why the game is so strongly limited by the CPU. Single-core speed didn't grow nearly as much as the number of cores did. Everyone knew it 10 years ago, but back then they could still just wait for the hardware to provide the additional power needed to run the sloppy code - this trick doesn't work anymore.


Yeah, Skyrim is a nice game, but many obstacles we've got here have trivial fixes compared to the size of their respective payoffs (little to none extra coding required). Especially with over 10 million copies already sold, I somewhat expect that it will at least run on recent hardware without sub-30 framerates.


Question: Thanks for explaining (and of course for making the awesome mod). Interesting to read. Do you think you will be able to do more optimization fixes like this?
Theoretically, yes, of course. I'd currently estimate that an additional +10% could be achieved in the game executable itself using the same procedure as before. However, to fix a problematic function using only a normal person's tools, it has to be detected and isolated from the call graph, then its assembly code has to be reverse engineered, rewritten and tested until it can be replaced by a better performing variant.

Now that many of the biggest CPU hogs have been softened, there still is a lot of potential gains ahead, but acquiring them would require multiple times the work already invested for a smaller amount of additional performance. It all has to be hand-crafted - there are no tools to automate this. Worse yet, some of the work has to be redone when porting the changes to a different game version. This way, only very simple optimizations are even possible at all, due to the sheer amount of places where a call must be inlined. Most function calls suitable for inlining have a varying amount of setup and cleanup code for that function call and I just can't go through each of the (tens of!) thousands of references to figure out the optimal replacement at each point. A compiler does all this for free, within seconds - and not only for the first dozen of hottest targets, but for the whole game, which consists of millions of lines of assembly code.

My true hope is therefore that this little demo will create a demand that Bethesda must answer. Much like what happened with the LAA patch. This would be the best outcome for all and also the best base for any further optimizations that can only be hand-crafted.


Question: And did I understood you right that you said that if an optimization compiler would have been used by Bethesda from the start, we would have experienced over 100% better performance?
Yeah, pretty sure about that. The current patch doesn't modify even nearly 1% of the code, but still manages to cut the cycles per frame by 30-40%. The whole binary is full of redundant code, it's just way too much to ever do manually, but sums up quickly when eliminated by an optimizing compiler.

Jesus fucking christ.

To reiterate from the previous thread, turning on optimizations takes this much effort:

 

[Kz3r0]

Hey guyz, for what I can see all is fine from here.

 

[villain of the story]

It's amazing, really. They have shit programmers. Shit coders. Shit animators. Shit texture artists. Shit 3D artists. Shit writers. Shit game design. Literally, shit is oozing out of their entire catalogue. It's like a catalogue of sewage.

And yet, somehow, it just works when it all comes together.

 

[Omicron]

 

[Overweight Manatee]

Some of the Bethtards are hilarious on defense. Here's the best I've found:

I'd also like to put forth the following. These a facts, and not opinion, as seems to be the basis of most of the posts in this thread.

Fact 1. We don't know why Beth "left" these errors in.

Fact 2. Bethesda may or may not be incompetent. Stating that they either are, or aren't, is opinion and subjective.

Fact 3. Bethesda must have some talent and can't all be idiots, they're the only dev creating truly open world RPG games with the depth of the ES.

 

[DarkUnderlord]

What with all that money you'd have thought they could afford to hire people who actually knew what they were doing.

Hyuk.

 

[Saxon1974]

Interesting, I have played about 30 hours and been to most all the big cities and I haven't noticed any slowness whatsoever. I do have a pretty beefy PC though so that might be why but If the code is not optimized I would think it would still be slow on my pc.

That being said there aren't that many heavily populated places in Bethesda games. I think Whiterun probably has the most people walking around from what I have seen and thats what 10 people in an area at once?

 

[racofer]

Fact 3. Bethesda must have some talent and can't all be idiots, they're the only dev creating truly open world RPG games with the depth of the ES.

 

[Oarfish]

Jesus fucking christ.

To reiterate from the previous thread, turning on optimizations takes this much effort:

Not quite that simple with C++, far more fine grained control over optimization and some of them have pretty heavy caveats. The runtime does most of the optimisation in .NET, not the compiler.

That said, if it is the case that they screwed their code so badly they can't enable compiler optimisations, lol. Be interesting to see if the PS3 binaries suffer from the same issue, as that's a totally different toolchain to windows / xbox.

It can't be an oversight. Surely they can afford a build grid to take care of build overhead. Must have been explicitly disabled.

 

[SCO]

Compiler optimizations broke their broken code?

Wouldn't surprise me at at all.


"Let's cast a object to char*, it's a superset so it's ok"
C programmer on a C++ project

 

[Overweight Manatee]

Jesus fucking christ.

To reiterate from the previous thread, turning on optimizations takes this much effort:

Not quite that simple with C++, far more fine grained control over optimization and some of them have pretty heavy caveats. The runtime does most of the optimisation in .NET, not the compiler.

That said, if it is the case that they screwed their code so badly they can't enable compiler optimisations, lol. Be interesting to see if the PS3 binaries suffer from the same issue, as that's a totally different toolchain to windows / xbox.

It can't be an oversight. Surely they can afford a build grid to take care of build overhead. Must have been explicitly disabled.

I just pulled a screenshot off google images. Haven't worked in a compiled language for almost a year now.

Its true that you might require some time/effort to enable optimizations, but not nearly so much as to be worth leaving them off. A few days max assuming a single competent employee and a not completely broken code base. And keeping in mind that Oblivion/Fallout3/NV all are compiling to much faster code I can't imagine that things have degraded THAT much just for Fallout with Swords.

The probable cause is that they simply had their release build configured incorrectly at some point and never noticed because they are developing on super machines which still run fast even on horrible code, and all their effort is placed on manually optimizing for consoles. If PS3/Xbox were running code anywhere near as bad as the PC is you would never see a double digit FPS.