TLDR: There’s no tldr; I spent weeks experimenting trying to understand what’s going on and finding a solution including many hours to write this up. Now that the title grabbed your attention, please read this post to understand fully what’s going on.
1. Exploding requirements, The big issue and the true Endgame Crisis.
Over the years many additions and enhancements were done to this game, none seems to be more controversial that the pop system change. Now we could be discussing aspects of the system for ages, but one thing is certain: late game performance took a crippling blow. The term late game certainly defies definition and it’s quite relative, but all players have experienced the problem. The games goes from a state where you don’t feel it as slow (or you don’t mind it being slow) to a state where you just have to stop playing because you can’t stand how slow the game has become and not because you got your closure from your current game, whatever that would be. It’s like being in the close proximity of a black hole: The passage of time slows down relative to real time, but each of us experiences it differently.
Machine specs, play styles, mods, game systems on top of game systems and peoples’ different perceptions and standards make this problem hard to tackle and even harder to speak about without evoking rage. To that end, the silence from the developers’ point of view is partially justified.
However, playing without mods, with a normal/average play style, on medium or smaller maps, can lead the game to a literal standstill even when playing on super-modern hardware specs, before the end game crisis even spawns. As such, this is a justifiable concern and a bug, since it’s within the official specs of the game and within the date range of the stellaris campaign. I’m writing this because I wish to separate cases where people mod the game into performance oblivion, or cases where people play stellaris like an idle game, having a goal to build up 100k pops and 1000 colonies in the year 3000. My effort and focus in this post is for ‘what’s written on the box’, that also forms a formal and perhaps legal agreement between consumers and Paradox, but if you want to play like that, please keep reading.
On paper, the idea that the game introduces jobs instead of tiles is good. It offered more options and the ability to play tall which somehow is a popular? thing if not controversial. However this skyrockets the population numbers as the game goes on, signaling that the game infrastructure, systems and engine were not ready for such a change.
Of course the handling of this issue in the community became the ‘Elephant in the Room’ for paradox as it is dominating most discussions at the expense of everything else - justifiably so and the threads that talked about it had to be finally contained, also justifiably so. I’m stating this as there are many, including me, who feel that this needs to be addressed *urgently*.
2. Microbenchmarking: Inconsistent passage of time and other weird phenomena.
My first reaction to all this was simple. If I can’t play as I like that what are the most performant ways to actually play that could give mt games proper closure? Systems like immigration, trade, mutli-species gene modding, and playing a multi-species empire (hello xeno compatibility) were apparently big bottlenecks to performance – So they were the first to leave the building – Hello machine empires!
The next layer was the galaxy: Playing with many stars and AIs that had the opportunity to expand and grow was bad. Galaxy size choices had to be scaled down. Medium failed, small failed, but as I kept playing and testing tiny failed as well as my machine empire game started slowing down, like as if I was playing on medium.
However, all these explorations started payed off. By removing systems from the performance equation with these isolated ‘microbenchmarks’ I started noticing weird things that were eye opening and demanded explanation and further testing. You might have encountered them yourself and thought them to be glitches or bugs:
a. The passage of time going through a game month was not even: There were fast and slow days and I don’t just mean the 1st and 2nd of each month where the monthly stuff takes place.
b. Sometimes the game would speed up for a period of time that was game months and years making the player very happy but later it would re-lapse into awful performance.
There are many bad things that you could say about the AI, but as you approach the endgame, the biggest performance factor in the game is ‘YOUR STUFF’. So my next thought was to eliminate all AI and species form the galaxy and do more testing and experiments on how I was playing with a mono-species empire, no migration and no trading going on. This lead to a breakthrough.
3. Breakthrough.
By changing how I played, I managed to speed up late game performance 500%+. In fact, on tiny it feels like the game runs almost like day 1 of a new game/galaxy and it seems like there is no upper bound to the number of pops you can have!! You can check the attached autobots save below, load it up, see it and play it for yourself. It has about 70 colonies, and 4.5 – 5k POPs,
I wanted to scale this up and was preparing to try another test game/empire, but then AlkincTeos supplied the performance megathread in the forums with a save on the latest version of the game (with lithoids dlc activated) that was slightly modded and had 18.5k pops and 261 colonies in it. I had to give it a go and as I posted yesterday the results were equally stunning, I would never consider being able to play with 18k POPs on my hardware, let alone it actually playing butter smooth. Here is the original post with the attachments. At the end of the post, you can download and try the new saves.
https://forum.paradoxplaza.com/foru...ance-megathread.1253705/page-19#post-25965104
4. The POPs are innocent: How it actually works (approximately!).
Everyone, including me, stated that the problem was the number of pops and the fact that the game spends too much time processing them. That is wrong. The true problem is vacant jobs. When a planet has no vacant jobs, no processing is taking place, as in:
“When a planet has no vacant jobs, the game engine doesn’t touch a single POP datum. It doesn’t even read them from ram. It’s zero processing”
The pops don’t check for jobs. it’s the other way around:
the jobs check for pops and we all know how this goes:
a. Every month:
b. For each vacant job:
c. Check if even employed pops can fill it – check all pops.
So, if you have a colony with even a single job and 300 pops, that’s 300 checks, including:
i. Pulling all pop data including all traits.
ii. Pulling population rights, and calculating if a pop can be eligible for promotion.
iii. Calculating and comparing weights for both jobs: the job you are looking for and the job the current employed pop candidate you are inspecting is already doing.
Performance is O(c*v*p), c: number of colonies, v: number of vacant jobs, p: number of pops
Now, I really wish this was optimized so I don’t know if the game groups jobs and pops for performance because then the performance model changes. The resettlement window shows pops grouped. I’m not sure if this is just a visual thing or if it has engine significance. Also not sure what, if any, caching is done and if it can be relevant at these large data set numbers – more on that later.
Case study: A colony has 40 vacant jobs in 4 groups (4 different job types) and 200 existing pops working in 20 job groups:
a. No job grouping, no pop grouping: 40x200 = 8000 pair checks – Your CPU is suffering.
b. No job grouping, pop grouping: 40x20 = 800 pair checks – Still a lot!!!
c. Both pop and job grouping: 4x20 = 80 pair checks – Manageable? How about doing it for 3000 colonies?
Note: pop rights and societal strata make the actual numbers smaller – you don’t check if slaves can fill researcher jobs, or at least I hope these optimizations are already implemented! Only the devs know and exploring the source code can reveal that.
Note 2: Colonies with low numbers of pops make this calculation cheaper.
Now there seem to be break points. In my experiments I tried minimizing vacant jobs early but it seems that you must be thorough and only once the number of free jobs is very small the game speeds up. You have to take into account the AI colonies as well here. It may be that:
a. Caching is there and helps up to a specific point then the engine gives up and dies.
b. Data structures that assist in job calculations suffer as they are becoming larger, so if you cross a threshold as free jobs are reduced and they get downsized, things become exponentially faster.
c. other unaccounted engine stuff that I can’t know about.
5. The left hand should know what the right hand does.
I had a discussion about this on Friday and Saturday with a fellow colleague and we came into the same conclusion: that developers in paradox must be implementing stuff in isolation. This is because this design:
1. Was apparently not communicated at all and had no impact analysis done.
2. Is counter to the design of reducing micromanagement.
3. Is ignored by other game systems in terms of their design, including the AI module that expects vacant jobs not to be a bottleneck.
Expanding on 2 above, the player in an effort not to micromanage, enqueues districts and buildings on all his colonies leading into dozens of vacant jobs. This happens especially during the late game, when he doesn’t care so much about resource costs and it’s causing massive lag in an empire with hundreds or thousands of free jobs in total - all working against the engine. The AI for what is worth, does the same, adding to this pile greatly. So you reduce your micro-management and you then spend hours/days at a slow game.
6. The worst and the least offenders.
System Sock level stuff: Choices that change employment across the empire, releasing vacant jobs and creating vast numbers of unemployment. Events that remove pops but leave empty jobs behind. Those traditions that add jobs to buildings? Think twice clicking on them late game – you suddenly have vacant jobs across all colonies – if you don’t have the unemployed for them, you just fall into the black hole (but most empires have all traditions by that time).
Worst: Megastructures and Ecumenopolis: Large pop count, Large batch of vacant jobs released on district completion, that will take decades to fill.
Least: Habitats: Low total population count, with normal districts. The calculation costs are low and they get filled fast.
7. A new resource and a new way to manage your empire: They need to switch the colors around!
Vacant Job handling is a precious resource. You play by making sure you have as little as possible and you always keep around at least 20-30+ unemployed spread around, to instantly fill new jobs. Your computer can only handle X number of them. You grow only a limited number of colonies at a time adding to the rest of your empire that is static and doesn’t eat CPU time each month.
This means that you have to micromanage the resettlement (and you may not like doing this), or use a mod that does it automatically. Careful here, because that mod could be wasting enormous resources to find vacant jobs going through many unemployed in a vast empire and become the new bottleneck.
Funnily enough, the colors of the games’ GUI should have been reversed: red for the vacant jobs and green for the unemployed. The game should have kept a global sum for both in the top bar. These numbers are vastly more important compared to your resources late game.
8. If only we could make all, including the AI play as such… Tall vs Wide?
You can’t control the AI. It will add to this and you can only play wide as in taking over planets and keeping vacant jobs zero. If you stay in your tall empire, you will slowly enter the black hole of processing lag.
Changing AI behaviour to build something -only if he has the unemployed for it- is trivial, but it can only happen in code, not in a mod.
Multiplayer will suffer much less, because players can change how they play. However MP games rarely reach that far and players are not given/do not have the time to play like this. This is much more manageable if your play just with 1 or two other close friends.
With the upcoming federations DLC this is far more relevant. If the pop system is not addressed or AI behavior fixed, why would I participate in a federation and kill game performance by keeping the AI empires around? The only reason I can think is that it will be a quick and peaceful way to annex them, but I can already do that with vassals – or just glass them out.
9. Black Death and other mid/late game Disaster suggestions.
One suggestion that seems to keep coming up is to introduce a mid game disaster to introduce a Black Death plague event that would eradicate pops. You can see how this is wrong and you can emulate it with the saves below. Killing massive amounts of pops in the galaxy would create vast numbers of vacant jobs that would bring the game to a halt. Only if planets were empty of pops it could be cheaper to compute, but it would still trigger going through the vacant jobs across the entire empire. Try loading one of the optimized saves below, add buildings/districts to all planets and console-complete all of them. Your game dies.
Instead, we need a “nanites disaster”: An accident that causes nanites to go around and disassemble infrastructure across the galaxy on a grand scale creating unemployment (and speeding the game up?). Any other disaster that would destroy job providers would be good.
10. Exponential Growth & Stellaris Immortal: It’s Elephants all the way down...
Yet another ‘Elephant in the Room’ was always the fact that pop growth was not based on existing population. In a full planet, we should be getting x% of the total pops per year they saying goes. So if Pop growth was 5% that would be 10 pops in a 200 pop world. For this to work we need long term pop sinks. Perhaps megastructures could have unique infinite jobs that provide some benefit? Game score? But you can see that once you fill a planet up, you can fill the entire galaxy with pops in 5-10 years. Such change requires the galaxy to be much more hostile to pop life in general – or expect pop growth speed to vary greatly from time to time or even entirely stop.
Stellaris Immortal is a cool new mod that tried scaling down the pop numbers, jobs and growth rates. Processing is easier because there are less vacant jobs and the planets have less pops to check when they do so. You can combine the new play style with this mod and play even huge galaxies! It also follows the idea of reducing pop growth by introducing pop sprawl – not yet sure if this is good or bad, but the switch to stop pop growth in the stock game is there for a reason.
It also plays with the idea of pop scale – having less pops that produce more. Pop scale has been repeatedly thrown around as a way to optimize the game and it’s another “elephant in the room” because in Victoria we used to have huge aggregated pop groups – the stellaris engine feels like a step back!
https://steamcommunity.com/sharedfiles/filedetails/?id=1891758612
11. The disaster that is the POP resettlement dialog box.
Granted, that window was never engineered to handle large empires. Once you reach large numbers it’s slow like hell to open in and use it. Bit tt is full of silly problems that are so low effort to solve:
a. If a colony has at least one vacant job, it will have calculate the exact number to display each time.
b. It will always calculate all colonies for all empires as you open it!!
It needs to stop assuming what to calculate and allow you to set sector filters on both sections. It should only start pulling data after you select a sector for each pane. Vacant job counts should be cached as well and not calculated each time.
In the Federation of planets save below, on the optimized save, the window opens and works flawlessly fast! That is because the colonies have no vacant jobs and there are no other empires so it doesn’t check the colonies or pops on them to deduce vacant job numbers. In the same save that includes other empires, the window is deathly slow, despite that our empire is optimized, because other empires’ colonies aren’t, so it has to go in and calculate data and vacant jobs that it will never even display.
12. Solutions & Suggestions.
I’m just throwing ideas here – in no particular order. Some are very easy, some require engine rewrite.
-Limit vacant job processing per empire: Allow only x jobs to be searched for applicants per empire, per sector and put the ones you can’t fill on the back burner with a job_age property and just ignore aging ones.
-Use a statistical model: Check x pops if they would change to vacant jobs per month. Filling empty jobs should be a challenge. Do you know how hard it is to find a good cleaner in a nearly fully employed ecumenopolis?
-Introduce job data cache or increase its size.
-Stop processing colonies if nothing has changed since last month, you worked that out last month!
- Do sector processing in many threads and colonies in each sector in series inside each thread. Isolate sector side effects to make this possible & safe. Use sectors as an engine/computational and organizational tool. We seldom change sectors so those operations can be very slow as they reconfigure the processing engine.
-Demotion timers are evil. They serve no important game function, make the game slower and keep the game slow. Let the game stabilize by filling vacant jobs fast and end processing.
-Make bombardment great again. Remove devastation and allow fleets to destroy buildings easily. Empires should pay if they can’t defend their colonies and creating refugee unemployment is natural for war. Add a cheap resettlement modifier on bombed worlds. Make repairing buildings slow and costly. Right now it feels like our fleets wield water pistols when it comes to bombarding – on all stances – A planet should be devastated in a few minutes if a full fleet is firing at it with fusion weapons. We can do it in a day with fission weapons here on earth now.
- Add MTTH events that destroy buildings and districts with vacant jobs – even if the empire pays the maintenance.
- Add option to disable districts.
- Make the game disable districts and buildings that have vacant jobs if the planet has no unemployed and activate them if the planet has unemployed.
- Add specific pop job rights to make job processing simpler.
- forget job hopping, promote only on building/district complete.
- Make ringworlds and ecumenopolis districts small again in therms of ‘jobs provided’ and reduce their admin cost to fractional numbers or roll the entirety of it to the megastructure/planet – We don’t care about it late game.
13. Evidence Games: Attached saves & their history.
13a: Autobots – Domo arigato, Mr. Roboto
ironman.sav (2.3.3) Vanilla
Machine empire mono species– about 5k pops, 70 colonies tiny. I purged all species out. No trading, no migrations- machines don’t do that.
Originally I played without gateways, I added them in to test what impact they have. It was almost nothing.
13b: Federation of Planets – What a mess!!!
All the rest, (2.5.1) With littlemod
Organic empire, mono species, 19.500 pops, 261 colonies, large map I think, Trading on, Migration on, Purging on.
That save had exactly what I was expecting to find: Large amounts of unfilled jobs, districts and buildings laying around empty. That’s how everyone plays at that stage of the game. It took hours to clean it up:
My first thought was to do it manually – I made a pass and deleted/disabled buildings to minimize vacant jobs: that took me 3 hours. I then realized I need a lifetime to wait for pop demotion at those crazy slow processing times: Rulers need 3 years and that would take a day real time. So I didn’t wait and used grow_pops to fill all vacant jobs, pop levels rose to 19.500 and that took about 1.5+ hours for 261 colonies.
The save didn’t preform after optimization due to other huge empires lying around and having issues. I saved and used play x and kill_country to remove them. Also a better computer might handle them and provide a speed up, more testing is needed, the save with them included is below. After a month of removing them it settled down and it’s very fast about 5-6 times faster!!! I wish to know how much faster it would be if it was a machine empire and if purging would have finished.
Note that colonies do still take some time to process production and consumption, so there is an eventual cap. However this cap looks to be very far away compared to touching most pops each month.
-END-
Now Go, Play & Experiment!
