Fhyujk

Consider a population where a low percentage of people (5%) posess a genetic advantage over the rest. In my fictional world, let’s assume this small percent of the population can wield magic while others cannot.

Now over the course of millenia, the count of the advantageous group should increase due to natural selection and eliminate the larger less-advantageous group.

I would like to know how this can be prevented from happening - the possible exceptions that cause this minority to still remain a minority without going extinct.

Infertility, magic talent comes with a severe reduction in fertility if not down right sterility. Now would that not make them die out? Not of the mutation rate among regular humans is high enough.

So your regular humans every now and again have a child with magical ability but it's sterile. So the child will never pass on their highly effective magical genes.

Species that are as complicated as humans typically take many millennia to evolve, but the 2 basic approaches are.

Sexual Selection

If you don't want a gene to reproduce, simply make it undesirable for the population to breed with those of "genetic advantage". The reasons for such could be cultural (anti-magical bigotry), biological (humans at the time see the trait for magic produces ugly people), or purely functional (maybe those with magic cause unintended spells that occasionally slay offspring).

Recessive Trait

Make the gene for Magic be a recessive one. In the same way that blue eyes are more rare than brown ones, magic genes may be rarer that non magic genes.

Sickle cell anaemia and malaria

In a malaria-ridden country, sickle cells provide a marginal advantage to survival because they give immunity to to the disease despite causing some ill-health. The condition allows humans to reach reproductive age.

In malaria-free countries sickle cell is a distinct disadvantage.

A magical ability came about by a genetic mutation. Unfortunately that mutation either reduces fertility or makes for a high death rate before puberty.

The following is a fairly technical explanation but easier ones can be found in more popular-type articles.

Heterozygotes for the sickle cell gene are relatively protected
against malaria, while patients who are homozygous for the sickle cell
gene, suffer from sickle cell disease and are highly prone to the
lethal effects of malaria.
http://www.scientificanimations.com/malaria-loses-sickle-cell-battle/

Well there are a few ways this can be achieved.

They may have worse health overall. I don't know what sort of magic your users have access to, but maybe they're more susceptable to diseases or just live shorter than magicless humans.

Another angle could be to give them lower fertility that magicless humans.

The magicless humans keep the population down. It's not hard to figure out that a guy who has access to magic might be a serious problem for you if he or she decides he or she doesn't like you. The solution? Ensure that there aren't enough of them to be a threat to you.