improving on mersenne twister

[jewboy]

I would really like to make a small in-game display instead of using an external program that you have to download separately and have to alt-tab out to check on. I was hoping you might have a suggestion like that. I will look at those files. I am finished with all of the code I think (aside from the in-game display if I try that). I tested the code in my dice rolling test program and have integrated the rng code into the game and it all compiles without errors. I decided to put the hotbits file reading/writing function definitions in legacysystems.cpp right before they are called in the constructor and destructor and the rest (GetInt()) is in rng.cpp. The function definitions could be moved of course.

I just have to spend some hours actually playing ToEE in TrueRandom mode with and without a hotbits file. I added 3 settings to config.cpp/h and the .ini file: useTrueRandom (defaults to 0), forceRdrand (defaults to 0), and randLevel (defaults to 8 and represents the number of whitening iterations when running rdrand()). I decided not to add a useHotbits option because if useTrueRandom = 1 the program searches for a hotbits.bin file anyway before enabling that code path. I also included a code path for people with old CPUs who want to use a hotbits file (until it runs out) instead of Mersenne. For now I decided not to mix/whiten the hotbits data in that scenario. It should just use the hotbits until they are gone and then switch back to Mersenne.

Right now I am re-coding my hotbits downloader .bat file as a c++ program (well trying to) and I will do some toee playtesting soon. It is amazing how much longer and more complex the c version is compared to a simple batch file, but it is also doing more.

I have never worked on any large projects with multiple source files etc before and I have had to learn how to deal with the added complexity. In general it is still possible to set breakpoints and single-step through the code, right? I can't get that to work though. I can't get VS2017 to 'start debugging' at all. It says "Unable to start program ..\repos\TemplePlus\Debug\Infrastructure.lib" and "..\repos\TemplePlus\Debug\Infrastructure.lib is not a valid Win32 application." I have a suspicion that if I did succeed in getting it to run the game might crash at a breakpoint anyway, but I was going to at least try. Is it not possible to do that sort of debugging with this project perhaps because a lot of the game code is actually compiled Troika code? There are some things I just can't test with an external test program.
...
Well I looked at those source and header files and that code is not 'self-documenting'. I could only make wild guesses about how to make a display. Maybe if there were comments, but that is not The Way of modern programming I guess. The modern way is 'It was hard to write. It should be hard to read.' Hehe.

Perhaps if you could give me a general outline of how I might do it. I really just need to display a single variable (a global pointer to uints) that represents the progress through the hotbits file as a number and a percentage. A battery meter style display would be good too if possible.
...
I decided that the hotbitsmon program could still be useful for crash protection. If the game crashes hotbitsmon can essentially run trimhotbits() itself so that the hotbits data status is kept updated. If I implement this then the only time the hotbits file won't stay updated is with a power loss. A game crash will be no problem. I thought I had to go back to storing the bits in a shared memory object, but then realized I could just use the hotbits file itself to reread the data if there was a crash. The important thing is that the index remains in hotbitsmon if the game crashes.

Anyway it's all looking good. My c/c++ downloading program (hotbitsfetch) is much more sophisticated than the simple batch file and seems to be working. It won't allow downloads until 24 hours (in seconds) has passed since the last download and will automatically append to previous downloads or to the main hotbits.bin file depending on whether or not templeplus.exe is currently running. It can be set to run from the windows task scheduler every 24 hours and just keep growing the size of the file every day. I am working on implementing the crash protection in hotbitsmon now.

 

[Dr. Bak]

RNG is usually what angry players blame the most in all sorts of games, I see why it'd make sense to optimize it.
The (allegedly biased) feeling of being cheated by the odds is somehow relatable, although it might just be black magic afaik.
Good luck with your code.

 

[Sitra Achara]

I would really like to make a small in-game display instead of using an external program that you have to download separately and have to alt-tab out to check on. I was hoping you might have a suggestion like that. I will look at those files. I am finished with all of the code I think (aside from the in-game display if I try that). I tested the code in my dice rolling test program and have integrated the rng code into the game and it all compiles without errors. I decided to put the hotbits file reading/writing function definitions in legacysystems.cpp right before they are called in the constructor and destructor and the rest (GetInt()) is in rng.cpp. The function definitions could be moved of course.

I just have to spend some hours actually playing ToEE in TrueRandom mode with and without a hotbits file. I added 3 settings to config.cpp/h and the .ini file: useTrueRandom (defaults to 0), forceRdrand (defaults to 0), and randLevel (defaults to 8 and represents the number of whitening iterations when running rdrand()). I decided not to add a useHotbits option because if useTrueRandom = 1 the program searches for a hotbits.bin file anyway before enabling that code path. I also included a code path for people with old CPUs who want to use a hotbits file (until it runs out) instead of Mersenne. For now I decided not to mix/whiten the hotbits data in that scenario. It should just use the hotbits until they are gone and then switch back to Mersenne.

Right now I am re-coding my hotbits downloader .bat file as a c++ program (well trying to) and I will do some toee playtesting soon. It is amazing how much longer and more complex the c version is compared to a simple batch file, but it is also doing more.

I have never worked on any large projects with multiple source files etc before and I have had to learn how to deal with the added complexity. In general it is still possible to set breakpoints and single-step through the code, right? I can't get that to work though. I can't get VS2017 to 'start debugging' at all. It says "Unable to start program ..\repos\TemplePlus\Debug\Infrastructure.lib" and "..\repos\TemplePlus\Debug\Infrastructure.lib is not a valid Win32 application." I have a suspicion that if I did succeed in getting it to run the game might crash at a breakpoint anyway, but I was going to at least try. Is it not possible to do that sort of debugging with this project perhaps because a lot of the game code is actually compiled Troika code? There are some things I just can't test with an external test program.
...
Well I looked at those source and header files and that code is not 'self-documenting'. I could only make wild guesses about how to make a display. Maybe if there were comments, but that is not The Way of modern programming I guess. The modern way is 'It was hard to write. It should be hard to read.' Hehe.

Perhaps if you could give me a general outline of how I might do it. I really just need to display a single variable (a global pointer to uints) that represents the progress through the hotbits file as a number and a percentage. A battery meter style display would be good too if possible.
...
I decided that the hotbitsmon program could still be useful for crash protection. If the game crashes hotbitsmon can essentially run trimhotbits() itself so that the hotbits data status is kept updated. If I implement this then the only time the hotbits file won't stay updated is with a power loss. A game crash will be no problem. I thought I had to go back to storing the bits in a shared memory object, but then realized I could just use the hotbits file itself to reread the data if there was a crash. The important thing is that the index remains in hotbitsmon if the game crashes.

Anyway it's all looking good. My c/c++ downloading program (hotbitsfetch) is much more sophisticated than the simple batch file and seems to be working. It won't allow downloads until 24 hours (in seconds) has passed since the last download and will automatically append to previous downloads or to the main hotbits.bin file depending on whether or not templeplus.exe is currently running. It can be set to run from the windows task scheduler every 24 hours and just keep growing the size of the file every day. I am working on implementing the crash protection in hotbitsmon now.

Have you set TemplePlus as the startup project? If not right click on it.

As for imgui there should be a demo in there (imguidemo.cpp i think).

 

[jewboy]

Ok. Will take a look. I am pretty sure I am finished with everything now except debugging actual gameplay. I just managed to finish my hotbitsfetch program recently. It has been getting really complicated with all of the different possible conditions including game crashes where the trim and cleanup code is not run. I need to start playtesting and maybe single stepping TemplePlus if I can ever get that to work without crashing.

Then my todo list includes adding http://qrng.anu.edu.au/FAQ.php# as another RNG source either in parallel or in series with hotbits and writing some sort of scatter plot app either as a separate app or as a part of the hotbits monitor program. Probably best to convert the monitoring program from ncurses to full GUI window to do a proper scatterplot. Also found another quantum RNG source: http://qrng.physik.hu-berlin.de/download.

Adding a qbits.bin to the mix at this point is a bit worrying, but it would be cool to add another true random source that is not limited to 2000 bytes per day and that doesn't require applying for a key by email like hotbits does. Mixing data from the outputs of 3 different physical processes, 3 different TRNGs, to simulate a dice roll? Seems cool to me. It could be nightmarish in terms of the amount of work to properly integrate it, but it's not like anyone is anxiously waiting for me to finish. Probably no one will ever run this code except me. So it's just more fun I guess.

Most of the complication of adding another TRNG is in the program logic changes involved with keeping them as two separate sources to be mixed or appended. If I just appended or mixed them at download time then the program logic should not change afaict. Really the biggest advantage of the quantum RNG source is the unlimited nature of it. If you run out of bits you can just exit the game, download more, and then restart. You can't do that with hotbits. So I guess the appending / mixing can be done either in game or externally and that is one of the first things I have to figure out.
...
Well I decided to just handle the multisource RNG logic externally at download time. I will have a single master RNG file called randbits.bin which will be modified as needed and loaded (copied to an array) at game startup. I have also decided that for now I will just append multiple sources by default possibly adding the ability to mix the sources instead later on.

I am planning to have 3 different data download programs:
1) hotbitsfetch (finished) fourmilab.ch radioactive decay RNG which is liimited to 12000 bytes per day and which requires applying for a key with a full explanation of why you want one. I gave a terrible reason and still received a key, but still a pita. Seems like a highly reliable source though because radioactive decay is considered one of the best RNGs and the guy is Swiss what's more. Seems like just the sort of thing a Swiss person would excel at doing well.
2) qanufetch (finished) Australia National University quantum laser RNG. Data is in json-text format which has to be parsed. Maximum request size is a megabyte and no official limits on how many times you can request that megabyte. Can specify a command line parameter with the number of kilobytes you want up to 1024 (1 MB).
3) qhufetch (finished) Humboldt University, Berlin quantum laser RNG and a non-stingy and very fast RNG with massive amounts of raw binary RNG data available, but which requires registering at the web site and getting a username and password. Has their own c\c++ library functions for downloading data. I would say this is the most elegant solution internally. Very slick. It just downloads straight to memory. No idea if there is a size limit. I am thinking about limiting each download to no more than 1 megabyte though. More than that is just so overkill for this applicaton. Accepts a command line parameter BYTES which lets you choose how many bytes you want. Like the Swiss, Germans seem like they would be particularly good at implementing something like this.

The Humboldt University source seems especially cool because one download could probably supply enough random data for millions of dice rolls or like a whole year of ToEE. ANU also allows large downloads, up to a megabyte per request and they have confirmed that it is unlimited too at least for now.

I am not sure how much data either of these new downloaders should download by default. I have to figure out how much random data ToEE uses per hour and ideally even for a full playthrough. I have noticed that the Humboldt University source was recently offline for a few days. So that can happen. Another reason to have multiple sources.

Ideally if all 3 sources are available though enough would be downloaded in one day to run a full playthrough of ToEE. With massive sources like that I also have to consider the memory requirements of loading all of that data into memory. Another reason to find out how much random data a single playthrough or at least a single day actually uses. I could always download to a separate intermediate stage file (randbitscache) and only load enough random data for a single playthrough no matter how big the randombits file so as not to waste RAM.
...
Finished downloading progs basically and the monitoring proggy was already done. So finally time to actually playtest I think.