We began work on the project almost two years from ship. At the start we planned to "ninja in, make some minor changes, replace the graphics system and ninja out." As we began to work with the code base, we found severe performance and stability issues with the engine. We traced some issues back to the core threading model of the Infinity engine.
Infinity was multi-threaded before there were processors capable of parallel code execution. As such, it was designed around a concept of threading that never really emerged. The main issue was that all of the threads shared the same set of data. The problem is all these threads were created and they all hit the same memory and as such, blocked each other, stalling all threads until the current one completed.
The code was also heavily laced with critical sections, which caused (in some cases) 70 percent of execution time spent in no-ops. Our long-term solution was to machete in and remove the threading, removing a couple hundred thousand lines of code from the game. The end result was more stable and had fewer performance irregularities. The "ninja-in" approach was tried in other areas, as well, and every time the end result was consistent. We always found a ton of complexity solving an era-specific problem that no longer applied. We continue to find roadblocks in the code and we're improving it as we go.
2. Intricate Code-Data Dependency
I have a development joke I often drop: "When a programmer believes he/she is being clever is when they create the greatest atrocities." The Infinity Engine is chock full of clever. For example: to render a character to the screen, the correct frame and orientation of a character sprite is first loaded out of a resource file using a very heavy resource-management system. The sprite is then color-mapped using a 256-color palette swap to enable player colors. Following the re-mapping, any number of further palette-manipulation code (stoneskin, anyone?) can step in and further change the actual data. Then the sprite is rendered against whatever potentially covering elements are nearby.
The final result is sent to the screen in 64x64 pixel tiles to be rendered. The entire system runs under a dynamic update system that flags 64x64 tiles as updated and renders them or, with no change, leaves the tile from the previous buffer. The volume of clever shifts and tweaks along the way make it nearly impossible to track down all the ways in which a simple sprite can be manipulated. In some cases, the data can reference many different .2da data files on how it can be manipulated, from equipment-changing animation frames to a data redirection to render a dwarf sprite instead of the default human. Complexity is par for the course in an RPG of this magnitude, but the intricate linking of assets and code really limited our ability to make the architectural improvements we wanted to make.