People definitely did get around, but it's always a question about who these people were and where they went.
Apart from some Nubians fucking around in Egypt, you never had blacks interacting with whites in any major way until the 19th century's scramble for Africa, with some minor exceptions (black slaves in medieval and early modern muslim Arab countries, but they were a small minority; religious conversion efforts by Christian ans Muslim missionaries in African kingdoms in the middle ages), so that's why the whole negro soldiers in the Roman Republic is such an annoying thing.
It's kinda funny, from my calculation Egypt/Nubia interactions actually created a big road block in the east. From what I found, Nubians didn't wanted to migrate to Egypt because they'd be forced to work hard so Pharaoh would have bigger profits, and they are not stupid like Egyptians (white). They even managed to create awesome dog embalming skills, to show a dog mummy and claim it looks better than Pharaoh mummy. So there was quite a few reason against at that time rather technologically advanced Nubia citizens to Egypt. And an Egyptian migrating into Nubia would likely be enslaved and sold somewhere. (I actually never seen any article about Nubians and if they were willing to interbred with someone with different skin color.)
Sahara prevented mass migrations. Then something prevented migrations around west coast. Actually, did they enslaved immigrants and castrated them, or was there different reason? I know that around year 800 - 1000 Kongo area was depopulated by massive epidemic of infectious disease. Population pressure that would otherwise forced them into wars, and move large numbers of migrants into South Africa and europe, which was delayed because of that. But, I don't recall any attempt of mass migration from Africa during Roman era.
With pretty much every major population movement, we have at least a rough idea where they came from (even the Huns who are largely a mystery!)
Actually, I never seen proper materials about central Europe, and some other areas. Either I'm too poor and it's in some academic resources I don't have free access to, or I missed that. But, from what I learned when I wrote algorithms for proper simulation of let say states including epidemic research and population genetics. The only correct and valid resources are properly dated archaeogenetic samples.
Is that dead body with 93+ Avar genes? It's an Avar, not Roman, not czech, not saxon. From what I seen, historians have bad habit of saying, this person used this type of decoration on pots, it was be of this genetic/ethnic/culture. This person used this type of clothing, he must be from this culture.
Thus as a consequence, some of stuff what historians are calling migrations could be simply a propagation of technology, where existing populations learned it themselves.
Also, it has to be kept in mind that most of these people were Caucasian sub-races. Celts and Germans? They're paler and blonder than the Mediterraneans but other than that, they don't look too different from your average Roiman. North Africans? They look like more tanned Romans. Carthaginians are west semitic (Phoenician colony), and look pretty much like people in the Levant... and Syrians even today don't look too different from Europeans.
I actually dislike the idea of calling someone Caucasian. Genetics and population origins of are sharply different between "caucasians".
Really, when it comes to unit models you shouldn't be able to tell much of a difference when it comes to skin tone. Soldiers recruited in Africa should be more tan, yeah, but then when you send a German to Africa and let him serve there for 20 years he's gonna get tanned too.
You might get some noticeable skintone differences when you enter India, but the India that the ancients interacted with is northern India, which has always been the center of the Aryan Indian civilizations, while the dravidian south - where the brown people are - wasn't really in anyone's picture.
Even in the huge multicultural Persian Empire, subject populations were characterized through their hairstyles, clothing, and mannerisms rather than their skin tone or facial structure or anything.
In Roman ethnographic texts, whenever they encounter a people that is not Caucasian, they find their features to be so strange they have to mention them as something strange and foreign, like the descriptions of the Huns (which most Roman authors considered to be barely human).
Well there are differences. People are actually forgetting that heart is straining itself far less by pumping blood in short sturdy 167 tall 80 kg weighting person than with 190 tall 120 kg weighting person. The problem is the shorty could do menial work for 30 years and still be fine, the tall person would die on stroke. That's one difference between populations.
Another difference is in prevalence of malaria resistance between ancient age Mediterranean populations. A rampant malaria is recorded in Rome, and similar areas in late Roman empire. Mutation introduced anti-malaria genes were happening basically in the whole Mediterranean area. Now, of course populations in central and northern europe didn't have any. Anti-malaria resistance come with steep costs, there is decent chance it prevents high intelligence, but there are other rather nasty consequences.
It's actually kinda funny how large differences are between ORIGINAL European populations. Living in isolation helped genetic mutations to create different genetics.
And as for peak skin tanning. Some skin variants basically never tan. I remember I had 3rd degree burns from sunlight during childhood. And when I went outside during summer, I'm white between some brown people. I seen people having weird idea, whiter skin is result of an office job, when they would do field jobs, they would be equally tanned. That's actually not correct, max tan is dependent on genetic, and its especially common with noble genes, or when people have Celt ancestors. When a person could have 3rd degree burns from relatively weak summer sunlight, such person is fucked when it's sold as a white slave to countries like Rome, or Egypt. It's one reason why Celt population never settled in Italy or countries like Spain or Greece. Too much sunlight, especially in summer, and too few forests.