I used to teach at a Steiner school. They have a lot of strange ideas (some are very questionable) but one of their notions is that "mathematics lives in the limbs and fingers" (they were always saying this kind of new-age sounding stuff which I found fascinating and troubling.)
But, I think there may be a drop of truth in this notion. A part of math is a lot like muscle memory, not language based, but kinetic. Beyond words. Or maybe before words.