Prof. du Sautoy asks this very question.

How many times have you heard someone say ”I can’t do maths”? Chances are you’ve said it yourself.

du Sautoy talking to the BBC

In all honesty I find myself thinking the above at least twice a day.

Genes or hard work
I am not an expert in how genes play a role in our intelligence, but for sure they do. That said, no-one is born an expert in mathematics and it takes a lot of hard work. Like everything in life, becoming proficient in mathematics to the level you set yourself is about perseverance and the willingness to struggle with things until you have mastered them.

Link
Can anyone be a maths genius? BBC iWonder