This is semi-related to Selena's comment - I don't have any number/color association, but I've always had gender and personality associations with all of the numerals and letters of the alphabet. The associations have faded in adulthood, but they used to be strong - as I looked at each character, I saw it as having a complete personality. I remember that 8 hates 5, for instance. Some characters were mean, others nice, some proud, some sneaky, some friends, some enemies, etc. It could be kind of distracting!

I have no idea why but to this day:

0 = male
1 = male
2 = female
3 = female
4 = male
5 = male (favorite #)
6 = male
7 = male
8 = female
9 = male

I guess as very young kids we anthropomorphise everthing. Like our stuffed animals and toy cars. Probably making associations with colors and earlier concepts helps us learn.