Kingston
Arcane
What's the difference?
I think Jaesun wanted to output the midi into some hardware.
What's the difference?
Because playing it off an actual Roland module will sound a lot better than using soundfonts.Yeah but, what for?
Because playing it off an actual Roland module will sound a lot better than using soundfonts.Yeah but, what for?
So... can I actually use this to play Daggerfall or what?
Lucius have released a video demo of his XL Engine:
February 20, 2016 |Author luciusDXL
In order to facilitate improved modding support and faster iteration times when working on some of the tasks required for the Beta 1 release, I have added scripting support to the engine. Previous builds of DarkXL and DaggerXL already had scripting support but overall I was not happy with the results.
Considerations
DarkXL used “Angel Code” for scripting. Angel Code has some nice advantages over some scripting languages, such as Lua, in how nicely it integrates with C/C++ code and provides some nice syntax. However, due to the lack of JIT support, it is very slow when compared to other options such as LuaJIT, Javascript and C#.
I would like to be able to move some of the game code into scripts – such as AI and weapon code – in addition to using it for UI in both the engine and games. So this presents a few issues with the previous solution – I need the overhead of calling script functions and those functions calling back into the engine or game code to be very small, the scripts must execute quickly (ideally no virtual machine), the scripts should have very fast access to data shared with the engine, scripts must be hot-reloadable so no lengthy compilation or optimization steps, the language should have a common syntax that I don’t mind using (since I will be one of the biggest script writers) and it should be small.
Obviously the DarkXL script system, using Angel Code, does not fulfill these goals. LuaJIT is fast compared to other scripting languages but it can be difficult to find errors (syntax errors can stay hidden until code is executed) and garbage collection can be problematic. There are other reasons I would prefer not to use Lua in the this project that I won’t get into. That said, Lua has been successfully used in many projects and has many useful advantages. If you are looking for a scripting language for your own projects, you could do much worse than Lua. Finally C# and Javascript (using V8 or similar) are just too big for my taste, though C# in particular is a nice language.
XLC
XLC stands for XL Engine “C”, which uses JIT compiled “C99″ as the basis for the scripting system. In order to improve the scripting experience, the environment is sandboxed (only engine provided functions and services are available) and the API is written in a way that avoids or hides the use of pointers and avoids user memory management. The language is not hobbled, however, and advanced users can use the full power of C. The compiler is built into the engine and compiles code, as needed, directly into memory. This means that the engine can call script functions with very little overhead, memory and data can be shared between scripts and the engine and scripts can call into the engine with the same cost as calling any C or C++ function using a pointer.
The only tool required to write XLC scripts is a text editor. If the engine is running and you edit a script, the engine will automatically hot-reload and recompile the script – which takes a fraction of a second – and you can see the changes immediately. The compilation is fast enough that scripts can easily be included as part of the data – so levels and game areas can have their own scripts in mods. Scripts can also include other scripts in order to import their functionality, variables and structures – allowing scripts to be broken up into files and allowing people to provide encapsulated functionality that anyone can use in their scripts for any game (unless game specific functions are used).
Obviously extremely fast compilation times come at a cost, the generated code isn’t as fast as Visual Studio C++ or GCC optimized code. However performance is much better than debug code, interpreted languages and even a decent improvement over other JIT compiled scripting languages in most cases. In addition the overhead of passing the “script barrier” is much better then the alternatives – you would need to call many tens of thousands of script functions per frame before it starts to become a problem.
And honestly having the engine reload scripts that change automatically while running is pretty awesome for development. It’ll make finishing the UI work so much easier. Something doesn’t look right? Make a small tweak in the text editor, save and see the change instantly.So much better then shutting down the program, making the tweak, recompiling, launching, getting back to the same place again just to find out your tweak wasn’t quite right.
To be clear, all of these features have already been implemented and are currently working. I have already started to move the XL Engine UI over to scripts to gain the iteration time benefits I mentioned above. And being “C” at heart – any game code I want to move to scripts required very little modification to work correctly.
Below is a small test script that I have used to test various features. Lines starting with // are comments, they describe various features being shown. /**/ type comments are also valid. Fixed size types are also included as well as standard C types. Sized types are defined as u/s/f (unsigned/signed/float) + sizeInBits and include:s8,u8,s16,u16,s32,u32,s64,u64,f32,f64. bool is also defined as a type, meaning true or false.
//include any script files that you wish to pull functionality from. All script functions,
//constants/enums/defines, script global variables and script defined structures will be
//accessible.
#include "test2.xlc"
//structures defined in the script, this is shorthand for C structure typedefs which
//can also be used: typedef struct name { ... } name;
Struct(Test2)
{
int y;
};
Struct(TestStruct)
{
int x;
int y;
};
Struct(Vec3)
{
float x;
float y;
float z;
};
//script global variables using both built-in types, structures defined within this
//script and arrays.
Vec3 data0[1024];
Vec3 data1[1024];
Vec3 data2[1024];
int runCount;
//internal script functions - other scripts that include this one can use them but they
//will not be used by the engine.
f32 blend(f32 t, f32 x, f32 y)
{
return x + (y-x)*t;
}
void testFunc(string printMe)
{
xlDebugMessage(printMe);
}
int fib(int n)
{
if (n <= 2)
{
return 1;
}
else
{
return fib(n-1) + fib(n-2);
}
}
//"public" functions can be called by the engine. Future tools will be able to list
//all of these, for example you could be editing a level, load a level script and
//then select these public functions from a list to run them based on various events.
//This function was used to help test a certain kind of performance. If you can't
//figure out the point, don't worry its only meant to test floating point math
//performance and function call overhead.
public void perfTest(void)
{
f32 blendfactor = 0.7594f;
for (s32 k=0; k<1000; k++)
{
f32 value = 2.0f + (float)(k-50)*0.01f;
//step 1. fill the data with values.
for (s32 i=0; i<1024; i++)
{
data0.x = value; value *= 1.25987f;
data0.y = value; value *= 2.25987f;
data0.z = value; value /= 2.25987f;
data1.x = value; value *= 2.25987f;
data1.y = value; value *= 7.25987f;
data1.z = value; value /= 20.25987f;
}
//step 2. blend between the values.
for (s32 i=0; i<1024; i++)
{
data2.x = blend(blendfactor, data0.x, data1.x);
data2.y = blend(blendfactor, data0.y, data1.y);
data2.z = blend(blendfactor, data0.z, data1.z);
}
blendfactor *= 1.001f;
}
}
//Testing a public script function with a different number of arguments.
public void simpleInc(int a, int b, int c, int d)
{
runCount++;
}
//xl...() functions are provided by the engine and are available to all scripts.
//sqr() and someVar are both defined in "test2.xlc" and are available since that
//script is included.
public void simple_main(int arg0, int arg1, int arg2)
{
int r = fib(32);
xlDebugMessage("fib(32) = %d.", r);
runCount++;
//MAX_MAPPING_COUNT is a engine provided define.
xlDebugMessage("MAX_MAPPING_COUNT = %d", MAX_MAPPING_COUNT);
xlDebugMessage("Clock = %d", xlGetClock());
//Using a script defined structure.
Test2 test;
test.y = 3;
//Another script defined structure.
TestStruct test2;
test2.x = sqr(test.y) + someVar - 2;
xlDebugMessage("test2.x = %d.", test2.x);
//testing string passing.
testFunc("this is a string.");
xlDebugMessage("Test inputs: %d, %d, %d", arg0, arg1, arg2);
}
Setup Enhancements In 0.2
Posted on February 27, 2016 by Interkarma
There are a few areas where setting up Daggerfall Unity could be a lot easier:
Starting from 0.2, I will try to address the above problems with the following changes:
- Obtaining Daggerfall’s game files. There is no easy way of obtaining Daggerfall’s game files for non-Windows users. You basically need to install the game on a Windows PC then copy game files to your platform of choice.
- Using the right game files. Most setup issues boil down to the game files being a CD-based version (movies not copied into arena2), not patched to .213, or game files have been modified over the years and don’t work as expected.
- Settings not persistent. Every time you download a new version of Daggerfall Unity, you need to configure your settings.ini and keybinds all over again. There is no way of pushing out new settings without giving you whole new files each time with all default values.
- Unclear when something goes wrong. If you start the game without setting your Daggerfall path, or if files are missing, you just see an unhelpful black screen or an obscure message.
More news on the 0.2 release will be posted soon.
- Providing game files download. Starting from Daggerfall Unity 0.2, I will provide a download archive for a known-good set of game files. This archive can be unzipped and used on Windows/Linux/Mac. You can still point to your own installation as before, so this download is completely optional, but it will be the recommended source of Daggerfall’s game files moving forwards. As Daggerfall’s game files are static, you will generally only need to download this archive once and it can be used for all future versions of Daggerfall Unity.
- Settings will be persistent. The settings.ini and keybinds files are now deployed to Application.persistentDataPath, so you will keep your settings whenever upgrading Daggerfall Unity. New settings will be automatically synced without changing your other settings.
- Setup UI. The new Daggerfall Unity game setup UI will be a friendly starting point guiding you through first-time configuration. If there are problems with your game files, the setup UI will try to point you in the right direction. This will evolve over time based on user feedback.
- Options UI. Coming later in 0.2 cycle will be a Daggerfall Unity options UI. Most settings can be configured without opening settings.ini or keybinds files at all.
If I were you I'd replay whichever I feel like replaying now, and then once again in 10 years when XL is finally released, patched and stable.This is driving me crazy...should I delay playing Blood, Shadow Warrior, Dark Forces and Daggerfall, or should I wait for the superior XL version to be completed?
It feels like a wild mouse chase sometimes.
Truth, plus Dagger Unity will totally beat out XL Daggerfall in terms of release date.If I were you I'd replay whichever I feel like replaying now, and then once again in 10 years when XL is finally released, patched and stable.This is driving me crazy...should I delay playing Blood, Shadow Warrior, Dark Forces and Daggerfall, or should I wait for the superior XL version to be completed?
It feels like a wild mouse chase sometimes.
DU is a new project though. Wait 7 years until it's as old as XL and see how much he works on it then. Back in 2009-2010 Lucius was releasing updates literally every other day.
It's a sad fact that all these projects start with a ton of work and excitement, then the motivation dies out long before they actually have a finished product.
First Look At Setup UI
Posted on March 8, 2016 by Interkarma
The first time you run Daggerfall Unity 0.2 and later, a simple setup wizard helps you configure the game. This replaces the Unity resolution dialog, and for many users will entirely remove the need to edit an INI file to get up and running.
As discussed in my previous post the INI file and KeyBinds files are now stored in a persistent data path. This means you only need to setup Daggerfall Unity once and future updates will continue to use your custom settings. This should make things more convenient when downloading incremental releases in future.
There are a few areas where setting up Daggerfall Unity could be a lot easier:
Starting from 0.2, I will try to address the above problems with the following changes:
- Obtaining Daggerfall’s game files. There is no easy way of obtaining Daggerfall’s game files for non-Windows users. You basically need to install the game on a Windows PC then copy game files to your platform of choice.
- Using the right game files. Most setup issues boil down to the game files being a CD-based version (movies not copied into arena2), not patched to .213, or game files have been modified over the years and don’t work as expected.
- Settings not persistent. Every time you download a new version of Daggerfall Unity, you need to configure your settings.ini and keybinds all over again. There is no way of pushing out new settings without giving you whole new files each time with all default values.
- Unclear when something goes wrong. If you start the game without setting your Daggerfall path, or if files are missing, you just see an unhelpful black screen or an obscure message.
More news on the 0.2 release will be posted soon.
- Providing game files download. Starting from Daggerfall Unity 0.2, I will provide a download archive for a known-good set of game files. This archive can be unzipped and used on Windows/Linux/Mac. You can still point to your own installation as before, so this download is completely optional, but it will be the recommended source of Daggerfall’s game files moving forwards. As Daggerfall’s game files are static, you will generally only need to download this archive once and it can be used for all future versions of Daggerfall Unity.
- Settings will be persistent. The settings.ini and keybinds files are now deployed to Application.persistentDataPath, so you will keep your settings whenever upgrading Daggerfall Unity. New settings will be automatically synced without changing your other settings.
- Setup UI. The new Daggerfall Unity game setup UI will be a friendly starting point guiding you through first-time configuration. If there are problems with your game files, the setup UI will try to point you in the right direction. This will evolve over time based on user feedback.
- Options UI. Coming later in 0.2 cycle will be a Daggerfall Unity options UI. Most settings can be configured without opening settings.ini or keybinds files at all.
March 19, 2016 |Author luciusDXL
This has been posted on the forums a few weeks ago but I neglected to post it on the blog until now.
There have been concerns regarding XLC and supporting different platforms. To address those concerns and describe some changes that I am implement (or will implement in the future), I decided to make a new post. If you have not already, please read the Introducing XLC – XL Engine Scripting System post. Note that most of the features described here will become available after Beta 1.
Bytecode Compilation
By default, XLC code will be JIT compiled and executed as native code without using a VM. However byte code compilation and execution using a virtual machine will be available for platforms that the JIT compiler does not support. While executing scripts using a VM will be slower, it will still be as fast as I can make it.This will allow XLC to run on any platform that the XL Engine supports – though I will obviously like to add JIT support to those platforms to maximize script speed. Initially the JIT compiler will support common CPU/instruction sets, including x86, x64 and ARM. (Note, this does NOT mean that the XL Engine will support ARM based platforms for Beta 1 or even give any timelines for that – it merely means the JIT Compiler will support ARM)
Optional Bytecode Optimization
Whenever iterating on scripts, reloading and JIT compiling XLC scripts is the most efficient way to work. In many cases, just writing the code in a text editor and using it directly should be fast enough. However, in cases where the JIT compiled code isn’t fast enough (or you need to use the VM), it will be possible to pre-compile to a binary format – platform independent bytecode – and then spend much more time during the optimization process. This should allow the script execution speed to get even closer to full native performance. A tool will be available, tentatively called “xlcc” which will generate binary files for each script file “.xlo”. Since the bytecode is platform independent, it will still be JIT compiled on load – and no additional tools are required (except for xlcc of course).
Later Features
Language Extensions
In addition to the existing language extensions – such as every script being executed as a coroutine and cooperative threading incoporated as a core feature – I plan on adding a few more language extensions. Primary, I plan on adding HLSL/GLSL like vector and matrix math capabilities – such as swizzles and basic mathematical operations (code such as: float3 a, b, c; a.xy = b.zz * c.xz. This should support a similar set of types, including float2/3/4, int2/3/4, uint2/3/4, bool2/3/4, float2x2, float2x3, float2x4, and so on. Why? Multiple reasons – the first is to make standard 2D and 3D math operations simpler, which should be useful for the targeted games.
Kernels
The second reason for the language extensions is to support “kernels” – optimized JIT compiled code designed to operate on multiple values simultaneously (either 4, 8 or 16) that can be used by the software renderer (i.e. software renderer “shaders”). Kernels are still JIT compiled, but instead of generating scalar code – the compiler will generate data parallel vectorized code that will make the best use of the CPU specific vector instruction set such as SSE2, AVX, AVX2, AVX-512 and NEON. I also plan on using XLC as an API independent shader language, to allow even GPU shaders to be written once and available for OpenGL, OpenGL ES and later Vulcan and other APIs. This will replace the current GLSL shaders and make supporting different platforms very easy for the shader writer. Kernels will be accessible to normal scripts in the cases where the scripter needs to process larger data sets, where the per-element operations are homogenous and simple. Note that kernels will also process data in batches of a multiple of 16 elements.
Final Words
In conclusion, I have big plans for XLC – to both accelerate my own work flow (runtime updating when editing UI code for example), to allow modders with a platform independent scripting language with near native performance and to even allow CPU optimized vectorized kernels, which can take advantage of CPU specific features while allowing code to be written in XLC, to provide better then native performance in those cases.
Of course I need to stay focused on Beta 1 so some of these features will not come until later, though probably not as far off as it sounds.
Faction Support
Posted on April 21, 2016 by Interkarma
Just a quick post today. I have implemented the faction back-end for Daggerfall Unity, which is a key pillar of quests and NPC dialog. Here’s the data shown as a flat list in the Unity editor. All parent-child relationships are actually in place, list is just drawn flat for debugging.
Starting faction data is parsed from FACTION.TXT and your save games are supported too! Importing a classic save will now also import your standing with all factions.
While there isn’t much happening with factions yet in Daggerfall Unity, it’s impossible to implement many gameplay systems without them. I look forward to doing much more with this data in a future build.
---------------------------------------------------
Posted on April 11, 2016 by Interkarma
One of Daggerfall’s long-running puzzles is how to generate the correct building name for any given building in a location. Daggerfall’s binary data exposes this information only as a seed value with no obvious correlation to the final name. From today, I’m happy to say this has been solved and I will be able to generate proper building names in the future. This article is a summary of the technical journey, minus all the dead ends and frustration.
The seed value used to generate building names has been known about for some time. This can be found in the BuildingData structure (link to UESP). The first step along the way was to generate some known values by changing a known seed value in MAPS.BSA. I started at the location Ashfield Hall in the Daggerfall province, which has a single tavern and some residences. Taverns are a great place to start as they have a very obvious PartA + PartB structure. For example The Bat And Skull. In Ashfield Hall, our single tavern is the The Howling Stag with a name seed value of 27748.
The first thing I did was change the name seed value for The Howling Stag in MAPS.BSA then start up Daggerfall to see how the name changes. Here’s a small sample of names generated from seeds 0-3. Keep this list in mind as we’ll return to it later.
0 = The Dancing Chasm
1 = The Knave and Scorpian
2 = The Pig and Ogre
3 = The Thirsty Fairy
Now I have somewhere to begin. I know the building is a tavern and have a sample group of seeds that result in specific names. The next trick is to work out how Daggerfall derives these names from the seed value.
I open up FALL.EXE in a hex viewer and search through for strings like “The Dancing” and “Chasm”. These strings are easy enough to locate, but these are just resources packed into the executable. What I need is the actual PartA and PartB tables Daggerfall is selecting from at runtime.
To get this information, I first have to use the DOSBox debugger to dump out memory from Daggerfall while it’s running. I can then search not just for strings, but for memory offsets pointing to those strings. I write a small bit of code to do the searches for me, and it doesn’t take long to find the correct offset tables for Part A and Part B of tavern names. Just think of this as a pair of arrays. In this case, both arrays are 36 elements long. Here they are as captured from the spreadsheet I dumped them out to.
So how do we go from a seed of 0 to The Dancing Chasm? This is where most of the difficulty started. It was obvious Daggerfall used a random number generator to pick both parts, but the trick was to find the correct random number generator used by Daggerfall’s C compiler circa 1994-1996. Fortunately, I also needed this for correct texture table generation (still an open problem at time of writing) and had previously researched the correct random generator, known as a linear congruential generator, specific to Daggerfall. Here it is for completeness.
static ulong next;
public static void srand(int seed)
{
next = (uint)seed;
}
public static uint rand()
{
next = next * 1103515245 + 12345;
return ((uint)((next >> 16) & 0x7FFF));
}
There are two methods here, one to set the seed (srand) and another to generate the next random number from that seed (rand). This is pretty much the standard ANSI LCG but specific to Daggerfall’s needs. Implementing this manually ensures that critical random number generation will always work just like Daggerfall, regardless of platform.
Now that I have the right random number generator, let’s feed it our test seeds from earlier and see what comes out. Starting withseed=0 and generating two numbers (indices into Part A and Part B name tables above), I get the following results.
PartA = 0
PartB = 12
First obvious thing is the spreadsheet starts from 1, not from 0. Just need to +1 each number to match the tables above (although zero-based arrays will be used in actual code). Matching these numbers to the above name table we get: Chasm The Dancing. OK, so Daggerfall obviously generates PartB first then PartA. Let’s try that again with the +1 and order swapped.
Seed = 0
PartA = 13 (The Dancing)
PartB = 1 (Chasm)
Result: The Dancing Chasm
Using our handy table we can match row 13 with row 1 and we get The Dancing Chasm. Good! Let’s run some more tests and prove the concept.
Seed = 1
PartA = 35 (The Knave and)
PartB = 27 (Scorpion)
Result: The Knave and Scorpion
Seed = 2
PartA = 30 (The Pig and)
PartB = 9 (Ogre)
Result: The Pig and Ogre
Seed = 3
PartA = 16 (The Thirsty)
PartB = 36 (Fairy)
Result: The Thirsty Fairy
So far, so good! Building names are output just like Daggerfall given the same inputs. Let’s try the original, unmodified seed value of 27748 which should give us The Howling Stag.
Seed = 27748
PartA = 21 (The Howling)
PartB = 33 (Stag)
Result: The Howling Stag
And there we have it! Building name generation from initial seed value resulting in a string exactly matching Daggerfall.
From here, I still need to extract hard-coded name tables for other building types like armorers and weapon-smiths. This isn’t hard though, I just need to find the tables using the same methods as taverns. I also need to assign full building data from MAPS.BSA to the correct models in Unity scene and wire up API methods to query this data when inspecting or entering a building. One challenge at a time though.
For regular small updates on Daggerfall Unity, I can be found on Twitter @gav_clayton.
Daggerfall Unity 0.2.9 (Updated)
Posted on April 6, 2016 by Interkarma
I’ve released a small patch to version 0.2.9. Latest download is on the standalone download page.
Patch notes for recent versions below:
0.2.7
0.2.8
- Reverted minor change to terrain tilemap shader. This might fix black ground issue on older DX9 systems.
- Implemented floating origin for Y axis to correct flickering shadows at high elevations. This still requires full testing.
- Items imported from classic saves will have dye synced to material type at import time.
- Settings INI now saves floats with invariant culture.
- Update to Uncanny_Valley’s grass mod.
- Added restart button to options UI.
- Nystul fixed white interior textures when reflections mod enabled. Also fixes stalled fireplace animation.
- Arrows should now always display correct inventory icon.
- Can no longer equip arrows to hands.
- Disabled bows until archery is implemented.
- Can now open inventory from character window.
- Weapon manager now resets sheathe state and equipped hand on new game / load.
0.2.9
- Fixed floating origin issue that would start player high in the air when exiting a building at higher altitudes.
- Nystul fix for resetting dungeon map on new game.
For regular small updates on Daggerfall Unity, I can be found on Twitter @gav_clayton.
- Disabled floating origin Y implementation for now. This means lighting and shadow issues at high elevations will return, particularly when using distant enhanced mod (which has a much higher vertical scale than default terrain).
- Camera now clears background when inside a dungeon allowing you better see within the void.
- Lypyl: Fix for white film on travel map. New console commands to display FPS and trigger action objects.
- Nystul: Automap camera settings now preserved when opening automap.
Daggerfall Unity 0.2 Release
Posted on April 3, 2016 by Interkarma
This post is a mirror of the new standalone download page. Please refer to this page for the latest version.
Daggerfall Unity 0.2
Daggerfall Unity 0.2 is now available for general download. Key features of this build are:
- Nearly complete item back-end. Monster loot and treasure piles coming soon.
- Inventory UI.
- Setup helper UI.
- Persistent data for settings, keybinds, and saves.
- Hundreds of small bug fixes and enhancements.
- Travel map (Lypyl).
- Almost complete support of dungeon actions (Lypyl).
- Updates to enhanced sky (Lypyl)
- Dungeon and interior auto-map (Nystul).
- Realtime reflections (Nystul).
- Animated grass and birds (Uncanny_Valley).
Download
Current version: 0.2.9 (9 April 2016)
Windows
Download Daggerfall Unity Test (Windows)
Linux
Download Daggerfall Unity Test (Linux)
Mac
Download Daggerfall Unity Test (Mac)
Note: Mac build is experimental as I don’t own a Mac to test on. Please let me know how you go with it on forums (see below).
Game Files
For convenience, here is a universal archive of compatible Daggerfall game files. This is primarily for platforms where installing Daggerfall is more difficult (e.g. Linux) but can be used on any supported desktop platform.
Note: This download contains game data only for Daggerfall Unity. It is not a standalone version of Daggerfall.
Download Daggerfall Game Files
Controls
General
Weapons
- Mouse to look.
- W, S, A, D to move.
- C to toggle crouch.
- SHIFT (hold) to run.
- SPACE to jump.
- LEFT-CLICK mouse to open doors, enter dungeons, operate switches, etc.
- ESC to pause game or go back to previous window.
- F5 to open Character Sheet.
- F6 to open inventory.
- M to open interior automap (indoors only).
- V to open travel map (outdoors only).
- ` (backquote) to open console. Enter help to list console commands.
Save/Load
- Z to unsheathe / sheathe weapon.
- H to switch equipped hands.
- RIGHT-CLICK and drag mouse to swing weapon.
Note: Keys can be rebound by editing keybinds.txt. See manual for more details. A full key-binding UI will be implemented in a future release.
- F9 to quick-save.
- F12 to quick-load.
Manual
A PDF manual is included with the download, but you can also download a standalone copy.
Feedback
If you would like to offer feedback and bug reports, please use this thread on the forums or contact me directly.
April 12, 2016 |Author luciusDXL
As many of you may have guessed, I have been working long hours the last few weeks – delaying my plans for posting an update and temporarily slowing progress. Fortunately the pace has died down, so progress can continue more smoothly again. As you can guess, however, this won’t be the last time there are hiccups in the road. Unfortunately I’m way past the point I originally thought I was going to be done – a good reason to take any estimates I give with a giant grain of salt (to be fair though, this is true for most developers and most large scale projects – especially for hobby projects).
Progress
As you might imagine, most of my time lately on getting the game code ready. Most of this has been spent refining my tools, adjusting the resulting code – which then goes back to refining the tools again. The good news is that the game code is between 80 – 90% complete. There is a fair bit of work left but things have been progressing well. I expect to have the games starting to become playable within the next few weeks – “real life” permitting. There is a sizable amount of work to go from playable to the whole package being ready for release, including bug fixes – so this is not a release estimate, merely an indication regarding the state of the release and progress.
Process
As I talked about before, there are several stages that allow me to go from the original executable and allow me to start piecing the code together and moving towards a fully decompiled result that runs natively on the XL Engine. I thought it would be interesting, in this post, to give a “behind the scenes” look at the intermediate format, between the disassembled code and decompiled code and give a little insight on that process.
This skips the processing steps that discovers the 32 bit code (for now 16 bit code is ignored), disassembles the code, figures out how the functions are arranged, removes dead code, converts the assembly into an internal format for processing and other required work. For now we are going to focus on the processing of a single function. Function processing occurs in several global passes and each pass has several steps, such as figuring out function inputs and outputs but for now we focus on only a few steps.
Listed here is a simple function, at this point in the process the purpose of the function isn’t known.
The first step is to disassemble the function:
000B094C 55 push ebp
000B094D 89E5 mov ebp,esp
000B094F 53 push ebx
000B0950 51 push ecx
000B0951 52 push edx
000B0952 56 push esi
000B0953 57 push edi
000B0954 81EC00000000 sub esp,dword 00000000h
000B095A B8603F0300 mov eax,00033F60h
000B095F E83CF60200 call 0000DFFA0h
000B0964 A1783F0300 mov eax,[000033F78h]
000B0969 2B05743F0300 sub eax,[00033F74h]
000B096F C1E002 shl eax,byte 02h
000B0972 A3983F0300 mov [000033F98h],eax
000B0977 8B157C3F0300 mov edx,[00033F7Ch]
000B097D C1E202 shl edx,byte 02h
000B0980 A1983F0300 mov eax,[000033F98h]
000B0985 01C2 add edx,eax
000B0987 8915903F0300 mov [00033F90h],edx
000B098D A1783F0300 mov eax,[000033F78h]
000B0992 C1E002 shl eax,byte 02h
000B0995 A3943F0300 mov [000033F94h],eax
000B099A A1983F0300 mov eax,[000033F98h]
000B099F A39C3F0300 mov [000033F9Ch],eax
000B09A4 8D65EC lea esp,[ebp-14h]
000B09A7 5F pop edi
000B09A8 5E pop esi
000B09A9 5A pop edx
000B09AA 59 pop ecx
000B09AB 5B pop ebx
000B09AC 5D pop ebp
000B09AD C3 ret
Next the function is converted to an intermediate format. At this point variables are assigned as inputs (%i#), locals (%l#) or globals (%g#). At this stage the tool is only looking at a single function at a time, later variables will be matched up and merged as appropriate and renamed.
void func_000b094c()
{
/* variables
Locals (7)
EBX, ECX, EDX, ESI, EDI, EAX, [ebp-20]
Globals (7)
[33f78], [33f74], [33f98], [33f7c], [33f90], [33f94], [33f9c]
*/
%l5 = 0x00033f60;
func_000dffa0();
%l5 = %g0;
%l5 -= %g1;
%l5 <<= 2;
%g2 = %l5;
%l2 = %g3;
%l2 <<= 2;
%l5 = %g2;
%l2 += %l5;
%g4 = %l2;
%l5 = %g0;
%l5 <<= 2;
%g5 = %l5;
%l5 = %g2;
%g6 = %l5;
return;
}
Then the code is simplified by merging statements into blocks, as discussed in a previous blog post. Block boundaries are defined by certain instructions such as calling a function or writing to global memory. At this point the tool already knows what the function inputs and outputs are and can modify the intermediate format to reflect that.
void func_000b094c()
{
func_000dffa0( 0x00033f60 );
%g2 = (%g0 - %g1) << 2;
%g4 = (%g3<<2) + %g2;
%g5 = (%g0<<2);
%g6 = %g2;
}
Additional passes are then required to define all of the global variables and types, to try to match strings to variables, functions and/or files and then to finally assign better names to everything. The end result (when everything works) is compiling code which then must be debugged and cleaned up. Once that is done, parts are modified to use the XL Engine services or features (for example using the XL Engine for rendering, input and sound).