Putting the 'role' back in role-playing games since 2002.
Donate to Codex
Good Old Games
  • Welcome to rpgcodex.net, a site dedicated to discussing computer based role-playing games in a free and open fashion. We're less strict than other forums, but please refer to the rules.

    "This message is awaiting moderator approval": All new users must pass through our moderation queue before they will be able to post normally. Until your account has "passed" your posts will only be visible to yourself (and moderators) until they are approved. Give us a week to get around to approving / deleting / ignoring your mundane opinion on crap before hassling us about it. Once you have passed the moderation period (think of it as a test), you will be able to post normally, just like all the other retards.

improving on mersenne twister

jewboy

Arbiter
Joined
Mar 13, 2012
Messages
657
Location
Oumuamua
How about improving the random number generator to use genuinely random numbers when possible? This could mean using _rdseed64_step() or equivalent when a new enough Intel CPU is present or downloading a package of random bits from https://www.fourmilab.ch/hotbits/ or random.org if internet is available. I would do this myself at least for character generation if I could figure out how to access the relevant functions in the game code, but it would be nice to get it working for combat rolls as well. If a new enough Intel CPU is unavailable and no internet then it could just fall back to Mersenne Prime again.
 

Sitra Achara

Arcane
Joined
Sep 1, 2003
Messages
1,859
Codex 2012 Codex 2013 Codex 2014 PC RPG Website of the Year, 2015
I have no interest in the subject but I can provide guidance. The random integer function is called in templeplus/rng.cpp. Let me know if you need anything else.
 

jewboy

Arbiter
Joined
Mar 13, 2012
Messages
657
Location
Oumuamua
Thanks. I will take a look at rng.cpp. Do you think it is doable? This is something I have wanted to do for a long time, but had no idea how since afaik the ToEE source code has never been released.
 

Sitra Achara

Arcane
Joined
Sep 1, 2003
Messages
1,859
Codex 2012 Codex 2013 Codex 2014 PC RPG Website of the Year, 2015
I don't see why not.

There's not much in rng.cpp, it just provides an interface to the original RNG function. You can substitute it with whatever code you want and the game will use it.
 

jewboy

Arbiter
Joined
Mar 13, 2012
Messages
657
Location
Oumuamua
Well I have lots of questions. I hope you can help me out. There really is almost nothing in rng.cpp.

#include "stdafx.h"
#include "rng.h"
#include <temple/dll.h>

LegacyRNG rngSys;
int LegacyRNG::GetInt(int min, int max)
{
return temple::GetRef<int(__cdecl)(int, int)>(0x10038DF0)(min,max);
}


So LegacyRNG::GetInt() is your method? Where is the method itself? Where is the LegacyRNG class? It seems to limit the results of temple::GetRef with 'min' and 'max' which for combat rolls I assume would usually be 1 and 20. Is LegacyRNG::GetInt() what the game code will call when it needs to roll the dice? Is temple::GetRef a Troika function? Is there somewhere I can find more info on temple::GetRef? I don't even really understand that syntax with <int()()>()(); What sort of statement is that? Are those inputs to the temple::GetRef function? And what is the hex number for? So I would just replace temple::GetRef with my own hwrng() function? Does temple::GetRef return a single (signed) integer? Every attack roll and damage roll and saving throw roll and character stat roll just calls temple::GetRef to get an int?
 

Sitra Achara

Arcane
Joined
Sep 1, 2003
Messages
1,859
Codex 2012 Codex 2013 Codex 2014 PC RPG Website of the Year, 2015
Well I have lots of questions. I hope you can help me out. There really is almost nothing in rng.cpp.

#include "stdafx.h"
#include "rng.h"
#include <temple/dll.h>

LegacyRNG rngSys;
int LegacyRNG::GetInt(int min, int max)
{
return temple::GetRef<int(__cdecl)(int, int)>(0x10038DF0)(min,max);
}


So LegacyRNG::GetInt() is your method? Where is the method itself? Where is the LegacyRNG class? It seems to limit the results of temple::GetRef with 'min' and 'max' which for combat rolls I assume would usually be 1 and 20. Is LegacyRNG::GetInt() what the game code will call when it needs to roll the dice? Is temple::GetRef a Troika function? Is there somewhere I can find more info on temple::GetRef? I don't even really understand that syntax with <int()()>()(); What sort of statement is that? Are those inputs to the temple::GetRef function? And what is the hex number for? So I would just replace temple::GetRef with my own hwrng() function? Does temple::GetRef return a single (signed) integer? Every attack roll and damage roll and saving throw roll and character stat roll just calls temple::GetRef to get an int?

The hex number is the address of the original toee GetInt function.

GetRef returns a reference to that function. It generally returns a typed pointer. The stuff in the angular brackets tells it that it's a pointer to a function - in this case a func that takes two int args and returns an int. You can read some more on how we interface with temple.dll in the wiki.

GetInt is used elsewhere in the code, mainly in the dice roller which as you guessed just calls it N times for NdM dice. So you just need to fill in whatever rng code inside there and you're basically done. I imagine you'll also need some init , but that'll probably be independent of the rest of the toee systems so it shouldn't be an issue.

If for some reason you need the original mersenne code let me know.
 

jewboy

Arbiter
Joined
Mar 13, 2012
Messages
657
Location
Oumuamua
Just thought I should start my own thread about this rather than clutter the feature requests thread with this little project. Do you happened to know if the max-min interval for dice rolls is ever greater than 256? I am assuming it is usually 20 or less.

I am tempted to use byte addressing with a long long call to the hwrng. This may be premature optimization, but I hate the idea of wasting hwrng bits. If i get a parameter interval > 256 though there will be big trouble. Probably I should test for that and fallback to temple::GetRef if it happens. Maybe for debugging I could store each parameter interval in a static array and write the intervals to a text file in append mode every so often. Then I should be able to see what the range of dice roll intervals actually is.

I have given up on using hotbits for pre ivy bridge CPUs for now. Way too complicated to implement. Maybe I could add that later though as an experiment and refine it over time. In theory radioactive decay may produce better randomness than the ring oscillators in CPUs though. I think I would also need to add something like a 'use hotbits radioactive decay for dice rolls' checkbox somewhere in the config program though which requires more than just modifying rng.cpp. It would be weird and difficult because temple+ would have to download data from a remote server at regular intervals and it requires a valid API key that has to be applied for by email.

So for now the changes I make to rng.cpp should only affect Ivy Bridge or newer and AMD Ryzen or newer CPUs. Sandy Bridge or Core2 = Mersenne Twister. Will fall back to temple::GetRef if older CPU. Actually I may eventually want to fallback to a different prng like http://simul.iro.umontreal.ca/rng/MRG32k3a.c which is supposed to be better than Mersenne Twister, but may not be worth bothering with that since hwrng/trng is so much better than any prng.

https://cs.stackexchange.com/questions/50059/why-is-the-mersenne-twister-regarded-as-good
 

Sitra Achara

Arcane
Joined
Sep 1, 2003
Messages
1,859
Codex 2012 Codex 2013 Codex 2014 PC RPG Website of the Year, 2015
Dice rolls are all below 256. The game also generates random timing for various internal things though (e.g. some event delays in millisecs, and certain animations I think), and I think some scripts use random numbers too, but I don't remember the ranges. Anyhow using specialized method for <256 sounds fine to me.

On that note, T+ already logs combat rolls, heh. But i guess you'd have to play a very long time for it to be meaningful...

As for hotbits, I don't want another internet interface in Temple+, so that's a negative on those. Feel free to fork Temple+ for that kind of thing.
 

jewboy

Arbiter
Joined
Mar 13, 2012
Messages
657
Location
Oumuamua
Well for now this rdseed/rdrand version should be a huge step up in terms of randomness from mersenne twister if you have a new enough cpu and Ivy Bridge isn't exactly new anymore. Maybe at some point in the future I can upload an experimental/unofficial 'hotbits enabled' version of temple+ somewhere. Not reforking, but just trying to keep up with your updates. At first I was just thinking of people who have older CPUs, but now I am thinking of XORing the hotbits with the output of rdrand/rdseed() for a kind of auto-whitening that combines randomness from two different independent sources.

Will Temple+ compile with Intel Parallel Studio and VS 2017 or only with the VS 2017 Microsoft compiler? What do I actually do with the new code when I finish debugging it? Should I just email you the new rng.cpp?
 

Sitra Achara

Arcane
Joined
Sep 1, 2003
Messages
1,859
Codex 2012 Codex 2013 Codex 2014 PC RPG Website of the Year, 2015
Well for now this rdseed/rdrand version should be a huge step up in terms of randomness from mersenne twister if you have a new enough cpu and Ivy Bridge isn't exactly new anymore. Maybe at some point in the future I can upload an experimental/unofficial 'hotbits enabled' version of temple+ somewhere. Not reforking, but just trying to keep up with your updates. At first I was just thinking of people who have older CPUs, but now I am thinking of XORing the hotbits with the output of rdrand/rdseed() for a kind of auto-whitening that combines randomness from two different independent sources.

Will Temple+ compile with Intel Parallel Studio and VS 2017 or only with the VS 2017 Microsoft compiler? What do I actually do with the new code when I finish debugging it? Should I just email you the new rng.cpp?
Pull request via github.
 

jewboy

Arbiter
Joined
Mar 13, 2012
Messages
657
Location
Oumuamua
I just built TemplePlus.exe with a much fatter rng.cpp. Doesn't look empty anymore :). Well I mean GetInt() is big now. I am assuming TemplePlus.exe is what I use to run ToEE and not like an installer or utility of some kind. Searched for other TemplePlus.exe and found one in AppData\Local\TemplePlus\ (308kB) and one in AppData\Local\TemplePlus\app-1.0.67\ (8142kB). The one I just built in ..repos\TemplePlus\Debug\ is 30141 kB. Hmm. Maybe that is due to my installing so many different versions. I guess I could just overwrite both with the fat debug version.

Well it runs, but I am getting numbers below 8 and and then all zeros when I reroll a character. Ah well...that would have been too easy. No idea what I did wrong, but this isn't looking very good. Maybe the built in function does more than just return pseudo random numbers within an interval range? I was sort of assuming that other parts of the code would call GetInt with parameters and I wouldn't have to worry about ranges, but maybe it is something else. I am stumped for now. Is the GetRef code viewable somehow?
 
Last edited:

Sitra Achara

Arcane
Joined
Sep 1, 2003
Messages
1,859
Codex 2012 Codex 2013 Codex 2014 PC RPG Website of the Year, 2015
I just built TemplePlus.exe with a much fatter rng.cpp. Doesn't look empty anymore :). Well I mean GetInt() is big now. I am assuming TemplePlus.exe is what I use to run ToEE and not like an installer or utility of some kind. Searched for other TemplePlus.exe and found one in AppData\Local\TemplePlus\ (308kB) and one in AppData\Local\TemplePlus\app-1.0.67\ (8142kB). The one I just built in ..repos\TemplePlus\Debug\ is 30141 kB. Hmm. Maybe that is due to my installing so many different versions. I guess I could just overwrite both with the fat debug version.

Well it runs, but I am getting numbers below 8 and and then all zeros when I reroll a character. Ah well...that would have been too easy. No idea what I did wrong, but this isn't looking very good. Maybe the built in function does more than just return pseudo random numbers within an interval range? I was sort of assuming that other parts of the code would call GetInt with parameters and I wouldn't have to worry about ranges, but maybe it is something else. I am stumped for now. Is the GetRef code viewable somehow?

That's all it does. You can test by overwriting the return value.
 

jewboy

Arbiter
Joined
Mar 13, 2012
Messages
657
Location
Oumuamua
Notes: (updated)

I think the code is working properly now. I want to implement an "Enable TrueRandom Mode (experimental)" checkbox that is off by default in the config program. The game play will I think be too different to just enable it by default. In fact I think that would definitely be a bad idea. Mersenne Twister or another deterministic PRNG should always be enabled by default.

The only thing that might be reasonable as a default enabled substitute to Mersenne might be a single call to rdrand() because that isn't truly random. It's just a good PRNG that is seeded with true randomness occasionally and I think game play will only be subtly different instead of very different. In fact I am experimenting now with a hybrid, a frequently but not always seeded rdrand(). It is still much more random than mersenne() but is not always truly random.

If you have a Broadwell or newer CPU the code path just defaults to rdseed() which is truly random and so quite 'harsh'. It would be nice if there could be another option in the config program to allow people with newer cpus to also use the less random rdrand() for a softer/gentler but still more random experience. Instead of using the config program I suppose a file like truerand.ini could be read to control various options and in fact the absence of that file could skip straight to mersenne().

There is a very slight delay after hitting the 'roll' button. Not only is hardware rng slower, but in order to reseed the generator with genuine randomness for CPUs that only have rdrand() I had to loop rdrand() 32 times with a 10 microsecond delay per Intels spec. I could call rdrand() only once but it wouldn't be truly random. By default rdrand is only reseeded with a genuinely random number every 1022 iterations or 32 with a delay. Of course it might be easier to roll munchkins with fewer iterations. I am testing with only 16 instead of 32.

Now that I see that there are many calls to GetInt with intervals of < 16 I'd like to try introducing a code path that tests for that condition and only uses a nibble of random data. A nibble path.

I would also like to check for the existence of a hotbits.bin or randbits.bin file in the appdata\local\TemplePlus folder and mix in that data if and while it is available. The biggest problem with that is if the game exits before the file data is used up. Do I just reuse the data if the player exits the game and restarts? There is only so much I can do from within a small function inside of a vast machine without causing too many problems.

I plan to write a hotbitsfetch utility that downloads 12000 bytes per day (with a 24hr lockout) from the hotbits server and stores it in appdata\local\TemplePlus\hotbits.bin. I can also write a utility to alternatively store bits in a file from any machine with rdseed() or from random.org as well. 12000 bytes is enough for 12000 byte dice rolls or 24000 nibble dice rolls per day. I assume per 24 hour period.
...
Well I finished the nibble code path. Now there are 8 dice rolls per hwrng call for nibble sized ranges. I want to start working on fixed file hotbits support, but first I think I need to figure out how to enable rng customization. For instance how does TemplePlus.ini work? I was thinking of putting a few values in there like:

TrueRandom = 0/1 (disable/enable TRNG mode, if 0 then Mersenne Twister)
ForceRdrand = 0/1 (disable/enable the less random CSPRNG even if you have a more recent CPU that supports TRNG)
RdrandLevel = [1-32] (number of whitening xor iterations for rdrand(); larger numbers are more random but slower)
HotbitsMode=0/1 (disable/enable support for reading random data from a hotbits.bin file on the hard drive)

Would I have to parse the TemplePlus.ini file myself to read that data or does Temple+ automatically store everything in there into some kind of global variable? Another option is to have a second .ini file: rng.ini.

Also I think it would be better if I could read hotbits.bin into memory (a global) from outside of GetInt(), but I don't know where to put the code.
 
Last edited:

Sitra Achara

Arcane
Joined
Sep 1, 2003
Messages
1,859
Codex 2012 Codex 2013 Codex 2014 PC RPG Website of the Year, 2015
You can add stuff to the global config struct. You can search the code for the other existing params to see how it's done.

Once again I don't want downloading hotbits on the main Temple+, nor anything that causes noticeable delay (if I understand correctly you're saying 0.32ms per GetInt?).

I'm also not clear on the benefits, can you explain what changes in effect?
 

jewboy

Arbiter
Joined
Mar 13, 2012
Messages
657
Location
Oumuamua
Well I should really profile my version of GetInt vs the original and see how much slower it may be if it is slower. I can adjust that somewhat I think and I don't think it will be enough of a delay to be noticeable. Mersenne Twister and most other PRNGs are very very fast. So it is hard to compete with that, but it should not be necessary to compete with a a few XORs and bitshifts which don't produce any real randomness.

I am already finished with both the byte and the nibble paths for rdseed() and rdrand() rngs. I just need to playtest more before doing a pull request. Well I also want to add some adjustable settings either in the existing global .ini file or in a separate rng.ini file. At the very least I would like to give people control over the amount of TRNG mixing in the csprng rdrand(). It is actually pretty cool because rdrand() should start off as being slightly more random than Mersenne because it is reseeded with a TRNG every 1022 iterations or every 32 iterations with a 10 microsecond delay in between, but it isn't exact. The more you iterate the closer you get to a pure rdseed() function which is the actual ring oscillator output. Based on my testing TRNG dice rolls seem to make the game harder rather than easier and so a TrueRandom slider effect would also function like a difficulty slider.

As far as the hotbits code path which is what I am currently working on I understand you don't want any internet connecting in Temple+, but I came up with an alternative idea. First of all, my code is designed with the principles of progressive enhancement / graceful degradation. So the more sophisticated options should not interfere with the simpler ones. The plan I am coding for now involves two separate programs. One is just a batch file (hotbitsfetch.bat which I will probably recode in c++ or autoit) to do the actual hotbits downloading. It will save a 12kB file in the current directory called hotbits.bin according to the hotbits daily quota of 12000 bytes. That is enough for something like 12000 to 18000 dice rolls depending on how many calls to GetInt() have an interval of 15 or less. If that is not enough for an entire gaming session then someone can use that utility to download 2-3 daily quotas or more to be stored in a larger hotbits.bin file (up to 21 days for a 260kB file). So you might wait a few days or even a full week before starting a TOEE playthrough in order to get enough random bits stored.

The second program, Hotbits Manager, is designed to be a go-between between the hotbits data file and Temple+. You would run Hotbits Manager before starting Temple+ and GetInt() would detect it and take advantage of its more advanced code path. It makes use of memory mapped shared data to monitor GetInt(). I am still working on it but its main function is to trim/split the data file when Temple+ exits so that Temple+ won't just reuse the same bits after a restart. It also outputs 5 variables to the console, the min-max params, the diceroll, the number of times GetInt has been called so far, and an index into the hotbits data to keep track of how much has been used so far.

If you run Temple+ without this additional program it will simply search to see if a hotbits.bin file is available and mix in the data until there is no more left. If you restart it will just start reading hobits.bin from the beginning. So far all of my code is inside GetInt() and I don't think writing out the current hotbits index to disk is a good idea inside that function. The last time I tried to write data to a file from inside GetInt Temple+ crashed immediately. I don't know if it just took too much time or caused some other problem, but I am wary of doing that and anyway it would definitely add delay to GetInt() calls because disk writes are so slow. Remember this is all optional. Running Temple+ with hotbits enabled requires some additional work and planning and the benefits are well probably marginal anyway.

The benefit to running Temple+ with Hotbits is even greater randomness. It should better simulate real life dice rolls. Radioactive decay is probably a better source of randomness than a ring oscillator and mixing entropy from two TRNG outputs should simulate dice rolls about as well as is possible. In fact I wonder if radioactive decay is actually more random than dice rolling especially since dice are never perfectly manufactured and so probably slightly biased. Also if you have an Ivy Bridge or Haswell CPU instead of Broadwell/Ryzen+ you won't have direct access to the TRNG ring oscillator output and mixing in the hotbits data should really help a lot.

I can even mix the hotbits data with the output of Mersenne so that people with Sandy Bridge or Core 2 CPUs can try playing in TrueRandom mode. Not everyone will like it because it seems to be harder, but some people probably will. The group of people who will benefit the least from fully enabling the hotbits code path are people with Broadwell or newer CPUs because they can use the rdseed() code path which is the direct output of a TRNG, but even then I think mixing the output of two TRNGs will produce better randomness. Even if you don't agree at the very least it won't make the dice rolls less random and it is completely optional. As I said the case for Hotbits mode is more compelling with CPUs older than Broadwell.

Since ToEE combat feels different to me with some genuine randomness I think it is important to keep all of this disabled by default. Either the config program or the .ini file or both should have a TrueRandom = 0/1 setting which starts at '0'. So only people who want to experiment with this stuff will be running this new code. The output of rdrand() with a low number of mixing iterations (Rdrand = 1, Rlevel = 1-8 in the .ini file) is the fastest option compared to Mersenne and will play the most like the original game since it is only slightly to somewhat more random. I will well document all of this btw and upload the hotbitsfetch program and the hotbitsmgr program somewhere like Mega. I can even host the files myself if necessary.

If you really don't want the additional complexity of all of this hotbits stuff in your GetInt() function that is ok. I understand that. It does something like double the size of the GetInt() code even though almost all of the hotbits code won't run at all if you don't have a hotbits.bin file somewhere. I can do a pull request without any of the hotbits code as we previously discussed and just keep this yummy hotbits code path for my own compiles or (with your permission) upload my Temple+ with Hotbits exe somewhere for that rare Codexian who might want to experiment with MaxRandom ToEE combat rolls. Or you can look at my new GetInt() function for yourself and decide if the Hotbits code is just too much.
 

Sitra Achara

Arcane
Joined
Sep 1, 2003
Messages
1,859
Codex 2012 Codex 2013 Codex 2014 PC RPG Website of the Year, 2015
I still don't understand what's the difference in terms of actual measurable effect. And how is "true random" harder? More strings of bad rolls?

As for writing to file, you can use the vfs system. I don't mind having Temple+ access some static/externally updated hotbits.bin file. Also and as mentioned you are free to fork Temple+ and do whatever you want with it there.
 

Raghar

Arcane
Vatnik
Joined
Jul 16, 2009
Messages
22,383
Waste of time. When I tested mersene twister I found randomness hiccups. But this game is primitive and don't need to create a whole world including building shapes from one seed.

Thus using any unbiased decent RNG would work.
 

jewboy

Arbiter
Joined
Mar 13, 2012
Messages
657
Location
Oumuamua
Essentially yes. TrueRandom is harder because it is more like rolling actual dice. You get more low rolls I think or at least it feels that way. For a [1 6] interval you can get 1 1 2 3 1 2 2 2 5 6 2 2 for instance and then you could get 6 6 6 4. So called 'whitening' can smooth out some of those repeats of the same number that you get with true random processes, but it still seems like more of a thing than with PRNGs. I think PRNGs are a bit more forgiving because they are not really random at all. They are designed for cryptography, for resisting cryptographic attack and not for simulating actual random processes like dice rolling and those randomness tests are mostly designed to test vulnerabilities to cryptographic attack and do not seem relevant for rolling dice in a game. It is kind of hard to objectively judge any of this since there aren't any real tests for randomness. Not this kind of randomness.

You can't get randomness without some physical process like a ring oscillator or radioactive decay. I don't think anyone has used a TRNG in a game before. So it's sort of uncharted territory. It could turn out that PRNGs are more fun. I don't know. But using rdrand() with iteration/whiting adjustment should be a good compromise between PRNG and TRNG. You can go from slightly random to very random just by changing a number in the ini file. This could also be implemented for people with old CPUs by not mixing in as much hotbits data with Mersenne. That should be adjustable too.

It also makes me feel better in some way to know that the dice rolls are more realistic. A psychological thing. There is a kind of fairness involved with rolling real dice. If you are getting clobbered because the game keeps rolling 1s with Mersenne you can wonder if it's some glitch in the pseudorandom algorithm, but if it's based on real random processes then you can feel some assurance that the rolls are justified. If you are going to roll dice at all it seems like you may as well get as close as you can to the real thing. Although that may or may not end up being more fun. The original game was balanced with a PRNG.

If it makes the combat feel different and I am pretty sure it does then it is not a waste of time. I don't know if the combat will be 'better' in some way. That's subjective. I think it will be different though. Anyone interested can just try it and see if it feels different to them. It does to me. If you like Mersenne you can keep using it. Just don't "TrueRandom = 1" in the ini file. Gotta implement that soon. Mostly just having fun with all this. Gonna try to implement the hotbits helper program in ncurses and I've never tried to use ncurses before. Looks pretty easy though.

VFS = Virtual File System? Can you unpack that a bit? How would I do that? Even without my Hotbits Manager program running I would like to have a default hotbits mode that at least basically works and that kind of does require trimming/splitting/truncating the hotbits.bin file when the game exits. Where is main() anyway? Is there a good place for me to put a TrimHotbits() function in main() after all calls to GetInt() just before the program exits? I see a main.cpp and a temple_main.cpp and they both seem to have main() functions. main.cpp has WINAPI WinMain() and TempleMain() and temple_main.cpp has TempleMain(). Of course I would still define the function in rng.cpp. I would just call it in WinMain() or TempleMain() or whereever. How about if I insert a TrimHotbits() in temple_main.cpp at the end between loop.Run() and return 0? So:
loop.Run();
TrimHotbits();
return 0;

Is there no Cleanup() or BeforeExit() function before return 0?

I haven't looked in temple_main.cpp before. Kind of interesting. I notice there is an applyGameConfig() function with a choice of rngs. That might be relevant. Does that just choose which PRNG the GetRef<> call in rng.cpp uses?

I found config.cpp and static ConfigSetting configSettings[] = {}. Looks like an object. Can I just insert something like CONF_BOOL(TrueRandom) in there? How do I read the setting later? Looks like some kind of Getter() and Setter() member functions? Can I call ConfigSetting::Getter<> then?
 
Last edited:

Sitra Achara

Arcane
Joined
Sep 1, 2003
Messages
1,859
Codex 2012 Codex 2013 Codex 2014 PC RPG Website of the Year, 2015
Essentially yes. TrueRandom is harder because it is more like rolling actual dice. You get more low rolls I think or at least it feels that way. For a [1 6] interval you can get 1 1 2 3 1 2 2 2 5 6 2 2 for instance and then you could get 6 6 6 4. So called 'whitening' can smooth out some of those repeats of the same number that you get with true random processes, but it still seems like more of a thing than with PRNGs. I think PRNGs are a bit more forgiving because they are not really random at all. They are designed for cryptography, for resisting cryptographic attack and not for simulating actual random processes like dice rolling and those randomness tests are mostly designed to test vulnerabilities to cryptographic attack and do not seem relevant for rolling dice in a game. It is kind of hard to objectively judge any of this since there aren't any real tests for randomness. Not this kind of randomness.

Ok. It would still be interesting to see some metric, maybe e.g. probability distribution of 3-4 byte sequences.

It also makes me feel better in some way to know that the dice rolls are more realistic. A psychological thing. There is a kind of fairness involved with rolling real dice. If you are getting clobbered because the game keeps rolling 1s with Mersenne you can wonder if it's some glitch in the pseudorandom algorithm, but if it's based on real random processes then you can feel some assurance that the rolls are justified. If you are going to roll dice at all it seems like you may as well get as close as you can to the real thing. Although that may or may not end up being more fun. The original game was balanced with a PRNG.
As regards balance, I don't know if it'll have such a major impact - after all your enemies are subject to the same RNG.

VFS = Virtual File System? Can you unpack that a bit? How would I do that? Even without my Hotbits Manager program running I would like to have a default hotbits mode that at least basically works and that kind of does require trimming/splitting/truncating the hotbits.bin file when the game exits.
Yeah. Currently it's basically a wrapper around TIO (Troika's IO library). You can also use tio_* if anything is missing there.

Where is main() anyway? Is there a good place for me to put a TrimHotbits() function in main() after all calls to GetInt() just before the program exits? I see a main.cpp and a temple_main.cpp and they both seem to have main() functions. main.cpp has WINAPI WinMain() and TempleMain() and temple_main.cpp has TempleMain(). Of course I would still define the function in rng.cpp. I would just call it in WinMain() or TempleMain() or whereever. How about if I insert a TrimHotbits() in temple_main.cpp at the end between loop.Run() and return 0? So:
loop.Run();
TrimHotbits();
return 0;

Is there no Cleanup() or BeforeExit() function before return 0?
The proper place actually would be in RandomSystem, which is one of the basic game systems. See legacysystems.h/cpp. You can insert hotbits handling in the Constructor/Destructor (currently there's only a non-default C'tor, so you need to add the latter).
The current constructor just sets the RNG seed by the system time. (this is what's in 0x10039040 that's referenced there)

I haven't looked in temple_main.cpp before. Kind of interesting. I notice there is an applyGameConfig() function with a choice of rngs. That might be relevant. Does that just choose which PRNG the GetRef<> call in rng.cpp uses?
Yes. The options are 0 ('default' - which uses mersenne), 1 - 'arcanum', 2 - 'mt19937ar' (mersenne again, same as default).

I found config.cpp and static ConfigSetting configSettings[] = {}. Looks like an object. Can I just insert something like CONF_BOOL(TrueRandom) in there? How do I read the setting later? Looks like some kind of Getter() and Setter() member functions? Can I call ConfigSetting::Getter<> then?
Yes, just add CONF_BOOL and whatever you need. You can access the settings via the global config object, e.g.
Code:
if (config.useTrueRandom){
return TRNG(min,max);
}
It will also automatically read/write from Saved Games/TemplePlus/templeplus.ini on startup/exit.
 

jewboy

Arbiter
Joined
Mar 13, 2012
Messages
657
Location
Oumuamua
Ok so in legacysystems.cpp:
Code:
RandomSystem::RandomSystem(const GameSystemConf &config)
   {
   auto startup = temple::GetPointer<int(const GameSystemConf*)>(0x10039040);
   if (!startup(&config)) throw TempleException("Unable to initialize game system Random");
   //read from (memory map) hotbits.bin here?
   }
const std::string &RandomSystem::GetName() const {
   static std::string name("Random");
   return name;
}
RandomSystem::~RandomSystem(void)
   {
   //split/trim hotbits.bin here?
   }

And in legacysystems.h:
Code:
class RandomSystem : public GameSystem {
public:
   static constexpr auto Name = "Random";
   RandomSystem(const GameSystemConf &config);
   const std::string &GetName() const override;
   ~RandomSystem(void);
};

And I assume it would be best to leave all the code in rng.cpp. That is, define the functions in rng.cpp and just call them from RandomSystem::RandomSystem() and RandomSystem::~RandomSystem() in legacysystems.cpp.
 
Last edited:

Sitra Achara

Arcane
Joined
Sep 1, 2003
Messages
1,859
Codex 2012 Codex 2013 Codex 2014 PC RPG Website of the Year, 2015
Ok so in legacysystems.cpp:
Code:
RandomSystem::RandomSystem(const GameSystemConf &config)
   {
   auto startup = temple::GetPointer<int(const GameSystemConf*)>(0x10039040);
   if (!startup(&config)) throw TempleException("Unable to initialize game system Random");
   //read from (memory map) hotbits.bin here?
   }
const std::string &RandomSystem::GetName() const {
   static std::string name("Random");
   return name;
}
RandomSystem::~RandomSystem(void)
   {
   //split/trim hotbits.bin here?
   }

And in legacysystems.h:
Code:
class RandomSystem : public GameSystem {
public:
   static constexpr auto Name = "Random";
   RandomSystem(const GameSystemConf &config);
   const std::string &GetName() const override;
   ~RandomSystem(void);
};

And I assume it would be best to leave all the code in rng.cpp. That is, define the functions in rng.cpp and just call them from RandomSystem::RandomSystem() and RandomSystem::~RandomSystem() in legacysystems.cpp.

Actually it would be best to move all the functionality inside RandomSystem. Since it'd get fatter, it'd be best to move the class definition outside of legacysystems.cpp and into its own file.
Then the places where GetInt is called would be replaced with gameSystems->GetRandom().GetInt(a,b).
If it's too much of a hassle I can take care of it after you're done.
 

jewboy

Arbiter
Joined
Mar 13, 2012
Messages
657
Location
Oumuamua
Well I am KISS programming and not ooping. No new classes. I am even trying to avoid defining any new functions except for remotely located code like readhotbits() and trimhotbits(). This stuff shouldn't be complicated. An experienced programmer could probably have done what I did in an hour, but I am not very experienced. My code is just some simple conditional {} blocks, and I hope easy to quickly read and understand and fully commented. Well maybe the mutexed shared memory stuff for the hbmgr program is a bit more than that, but probably not for an experienced programmer. I wish there were a simple way to hide that hbmgr stuff, but I can't think of anything worth the trouble of implementing. My goal is good organization and readability more than anything else, but maybe a real c++ programmer would prefer that everything be classes and objects. I just want a more experienced programmer to be able to quickly scan my code and think hmmm I can see why he did that that way, but that is not optimal. Let me see if I can improve that. But there isn't much to this.

When you say 'all the functionality' are you just talking about the hotbits file reading/writing code or the new code inside the new GetInt() as well? I am trying to keep almost everything local to GetInt(), but are you saying you would like to rehome GetInt() itself? I don't really understand from where or how GetInt() is called. I assume it would be called from all over the place in the core game code, but indirectly since presumably GetInt() is not a Troika thing.

So GetInt() is not already a member function of the RandomSystem class? GetInt() appears to be a member function of the LegacyRNG class whatever that is. Hmm. I am looking for the class definition for LegacyRNG, but I am not finding much. I thought it might be a subclass nested within the RandomSystem class. Is that what you would like to see? I have like zero experience with OOP programming. I just have the most basic book knowledge about how it is supposed to work. I mostly have just done some procedural c-style programming to write small utilities that I need for something like once a year. I don't think you want me messing with your overall program organization. Hehe.

Having said that, if you would like to see GetInt as a direct member function of the RandomSystem class instead of the empty looking LegacyRNG class or making LegacyRNG a subclass of RandomSystem I could try that and you could always fix it if it is wrong. I think the important thing is the new code itself which is now basically written. If it weren't for this new hotbits code which is also pretty much finished I could probably have submitted a pull request already and you could have read my code. The only reference I could find to GetRandom() is some kind of declaration in gamesystems.h that I don't understand. Is GetRandom() a member function of RandomSystem? The problem that I have with OOP code is that it can be so hard to find the actual code to read. Everything seems so buried and hidden in different places. Does GetRandom() already call GetInt() or would that have to be implemented?

Right now I am just trying to pull out the hotbits file read/write code and place them inside functions that can be called from somewhere else at the start and end of a gaming session. The RandomSystem class constructor and destructor as you suggested seem like good places for that to me. I just hope the calls to win32 file operations don't crash the game. I am slightly worried about that. I am hoping I don't need to use the Troika virtual file system. Because I don't understand how to use that at all. I am just making calls to the win32 api ReadFile() and WriteFile() functions and storing the data with malloc(filesize) and a pointer to uints.

Well I was using file mapping before for [reasons], but it doesn't seem necessary now except for the shared memory variables for the hotbits manager program which will no longer do any file reading or writing, but just display useful hotbits stats like hotbits used per hour and something like a battery meter showing the percentage of hotbits data left and maybe an alternative listing of dice rolls with their intervals just because. The display stuff is of limited use because you have to alt-tab out of the program to see any of it, but monitoring the hotbits data seems important. At the very least I would like to know when I run out. The only other way seems to be writing stats to the cheat console or even directly to the display, but that seems overly ambitious and may cause unforeseen problems. Maybe the hbmgr window could be made very small and 'always-on-top'. Dunno if that would work in full screen mode.
 

redactir

Artist Formerly Known as Prosper
Joined
Jul 16, 2018
Messages
696
I could share my random code.

The function names are a little long but everything works as expected. I use PCG as a generator rather than boost.

The ranges are inclusive. Which means when selecting by index from a container you'll want to do size() - 1 in the max field.

Unity3D does the opposite and uses an exclusive range. Which is quite annoying when you're rolling for things other than indices to access containers. (it excludes the max not the min parameter. )

I have a function in my random code that seeds the rng. the first parameter is a bool, where true means use a random device. The second parameter, let's you set a specific seed. But it also has a default seed so you can just seedTheRng(false).

This allows a simple edit in source to change the random from using a fixed seed each startup to something that doesn't.

OOP is a bullshit choice for getting random numbers. It's better to have copies of the seeds and reseed the generator than to have multiple generator objects hanging around. Although I haven't looked that much under the hood of PCG, maybe it handles copying and references to a generator quite optimally.

Either way no one wants a generator object to take a shit on their beautiful code with it's ugly presence. So we create non-member functions to wrap-up the presence of what's being used.

As for all this hotbits nonsense I have a function that can test if the random output is working as expected.
(Windows for example will output 0,1,0,1,0,1 or just all 0s, if it cannot find a random device. But don't take my word for it.)

This isn't the same as knowing if the entropy collected is fresh or "there". But I suggest just doing a quick test if the values have gone to hell.

If entropy is being pissed away so fast that you need to constantly monitor it, you need to learn to use it better.

source for the PCG, they have a C implementation too. Reminder it's a generator not a way to access any random device.
http://www.pcg-random.org/
 

jewboy

Arbiter
Joined
Mar 13, 2012
Messages
657
Location
Oumuamua
Well I am more focused on the true random paths rather than improving on the pseudorandom part. But are you saying that PCG is better than Mersenne Twister for pseudo? Sitra I think expressed some interest in some kind of bias testing of this stuff to see if there is any measurable improvement and that is something I was thinking of putting into my little companion program just as something else to output: some kind of bias meter bell/whistle. Let me ask you this. Do you know what the absolute best measurement of randomness is supposed to be? Is Monte Carlo the gold standard?
 

Sitra Achara

Arcane
Joined
Sep 1, 2003
Messages
1,859
Codex 2012 Codex 2013 Codex 2014 PC RPG Website of the Year, 2015
Well I am KISS programming and not ooping. No new classes. I am even trying to avoid defining any new functions except for remotely located code like readhotbits() and trimhotbits(). This stuff shouldn't be complicated. An experienced programmer could probably have done what I did in an hour, but I am not very experienced. My code is just some simple conditional {} blocks, and I hope easy to quickly read and understand and fully commented. Well maybe the mutexed shared memory stuff for the hbmgr program is a bit more than that, but probably not for an experienced programmer. I wish there were a simple way to hide that hbmgr stuff, but I can't think of anything worth the trouble of implementing. My goal is good organization and readability more than anything else, but maybe a real c++ programmer would prefer that everything be classes and objects. I just want a more experienced programmer to be able to quickly scan my code and think hmmm I can see why he did that that way, but that is not optimal. Let me see if I can improve that. But there isn't much to this.

When you say 'all the functionality' are you just talking about the hotbits file reading/writing code or the new code inside the new GetInt() as well? I am trying to keep almost everything local to GetInt(), but are you saying you would like to rehome GetInt() itself? I don't really understand from where or how GetInt() is called. I assume it would be called from all over the place in the core game code, but indirectly since presumably GetInt() is not a Troika thing.

So GetInt() is not already a member function of the RandomSystem class? GetInt() appears to be a member function of the LegacyRNG class whatever that is. Hmm. I am looking for the class definition for LegacyRNG, but I am not finding much. I thought it might be a subclass nested within the RandomSystem class. Is that what you would like to see? I have like zero experience with OOP programming. I just have the most basic book knowledge about how it is supposed to work. I mostly have just done some procedural c-style programming to write small utilities that I need for something like once a year. I don't think you want me messing with your overall program organization. Hehe.

Having said that, if you would like to see GetInt as a direct member function of the RandomSystem class instead of the empty looking LegacyRNG class or making LegacyRNG a subclass of RandomSystem I could try that and you could always fix it if it is wrong. I think the important thing is the new code itself which is now basically written. If it weren't for this new hotbits code which is also pretty much finished I could probably have submitted a pull request already and you could have read my code. The only reference I could find to GetRandom() is some kind of declaration in gamesystems.h that I don't understand. Is GetRandom() a member function of RandomSystem? The problem that I have with OOP code is that it can be so hard to find the actual code to read. Everything seems so buried and hidden in different places. Does GetRandom() already call GetInt() or would that have to be implemented?

Right now I am just trying to pull out the hotbits file read/write code and place them inside functions that can be called from somewhere else at the start and end of a gaming session. The RandomSystem class constructor and destructor as you suggested seem like good places for that to me. I just hope the calls to win32 file operations don't crash the game. I am slightly worried about that. I am hoping I don't need to use the Troika virtual file system. Because I don't understand how to use that at all. I am just making calls to the win32 api ReadFile() and WriteFile() functions and storing the data with malloc(filesize) and a pointer to uints.

Well I was using file mapping before for [reasons], but it doesn't seem necessary now except for the shared memory variables for the hotbits manager program which will no longer do any file reading or writing, but just display useful hotbits stats like hotbits used per hour and something like a battery meter showing the percentage of hotbits data left and maybe an alternative listing of dice rolls with their intervals just because. The display stuff is of limited use because you have to alt-tab out of the program to see any of it, but monitoring the hotbits data seems important. At the very least I would like to know when I run out. The only other way seems to be writing stats to the cheat console or even directly to the display, but that seems overly ambitious and may cause unforeseen problems. Maybe the hbmgr window could be made very small and 'always-on-top'. Dunno if that would work in full screen mode.

Just submit your code and I'll make adjustment as I see fit.

It's possible to add a simple ingame display using the IMGUI infrastructure, should be fairly easy to add. See tig_console.cpp and ui_debug.cpp.
 

As an Amazon Associate, rpgcodex.net earns from qualifying purchases.
Back
Top Bottom