The lack of decent AA options is one of my pet peeves with modern games btw, you are mostly left with a choice between downscaling and some sort of blur-o-vision shader AA. Maybe someone like @Bad Sector has some idea what it is with modern engines and AA and why that option tends to be lacking these days.
There are basically three types of AA: post-process (e.g. FXAA), sample-based (e.g. MSAA) and temporal (e.g. TAA), each one having their own pros and cons. In general:
- post-process can give you the best performance with the least resources, but they often don't really know much about the actual geometry being rendered. On the other hand they'll work on anything, e.g. textures and shaders
- sample-based will take one or more samples per pixel and they tend to provide a consistent quality over time, but they only work with individual frames and they tend to be the most computationally expensive. MSAA is the most common by far and supported by the hardware but also has its own limitations (later on that). SSAA is the oldest but also the most expensive as it always uses the same number of pixels (in a way rendering at 200% width and height and downscaling is like doing 4x SSAA since it'd be like taking four samples per pixel).
- temporal will use data from multiple previous frames to try and provide high quality antialiasing and also handle aliasing over time (e.g. detail "popping", like specular lighting aliasing as the camera or objects move), however this relies on the assumption that the previous frames are mostly the same and the renderer provides correct motion data to avoid creating ghosting/smearing (which can be easy to screw up). The end result is basically that in good conditions (mostly static scenes) it can produce results that approach SSAA with a high number of samples, but on the other hand in bad conditions (quick motion) it can produce bad or even "wrong" results - thus giving a very inconsistent output (something which is also exacerbated by potential engine issues).
TAA also has two (mostly unrelated) benefits going for it, one being that solving the "perfect" frame over time can also be used to implement some rendering techniques stochastically (i.e. techniques that instead of always being pixel perfect, they only calculate some or most of the samples while randomly ignoring others) that itself allows for faster rendering (or, from another perspective, being able to do things that would otherwise be too slow to do) - this is basically what happens when in some games that are designed with TAA in mind you turn off TAA and get noisy results (commonly seen with transparencies, hair, shadows and now raytracing), it might have been too costly (performance-wise) to get a smoother result.
The other benefit is that as engines are trying to put heavier effects in there (raytracing is a common target now, though TBH i expect in the future this to be less of an issue since current hardware is barely designed for it), they rely on reconstuction techniques like DLSS, FSR 2.0 and Intel's XeSS - or whatever they call it - which use the same input data as TAA to produce results. So if an engine has one it might as well get the other and since if an engine's developers want to implement advanced raytracing effects like realtime global illumination, they'll need DLSS/FSR2/etc to get usable performance in non-enthusiast hardware, so might as well use TAA too and get all the other benefits.
MSAA and FXAA are the other two AA methods you might find, though they're both becoming less common (especially FXAA). MSAA has the main drawback that -aside from being heavy- it only works with polygon edges and so the more a scene relies on shaders to produce detail (even something as mundane as a normal map and/or a specular map would apply here), the more likely it is to not work. One now extreme -but most likely more common as time goes by- case is Unreal Engine 5's Nanite that doesn't use polygons (at least polygons the GPU knows about) at all and so MSAA wouldn't work on it at all. Another drawback, mainly for developers, is that depending on how it is used MSAA requires special shaders for it. If it is the only AA it might be worth it, but as TAA also needs special care, developers may decide that they do not want to bother with MSAA - especially if it doesn't fit their rendering pipeline (see the polygon limitation i mentioned) or rendering budget.
(as a sidenote, pretty much all modern GPUs from Nvidia and AMD that can do MSAA, can have their minimum number of samples per pixel adjusted, thus improving MSAA's visual quality at the cost of worse render performance - but AFAIK for some reason this functionality is not available in Direct3D and since most engines are designed with D3D in mind, they never seem to implement an option for it even when they have OpenGL or Vulkan support despite both of them supporting it)
So anyway, as mentioned above by
soulburner the best approach - if your GPU can handle it - is to render the game at higher-than-normal resolutions (on Linux you can use gamescope or xrandr for that - the former will work on any game with almost any resolution but might have some input lag, the latter will have no input lag but will resize your entire desktop, choose your poison :-P) to get some form of simple SSAA and -IMO- if available also use TAA for that to get temporal aliasing (either because the engine was designed for it so all effects work properly or just to avoid temporal aliasing). Some games might support using a render scale above 100% which should give you pretty much the same result, though with slightly less resource usage and slightly better UI handling, so if the game has that use that option instead.