It just takes practice, the more math you do the faster you will become. When your friend was young he probably practiced a lot and used flash cards, and memorized his times table and stuff. You did not. As you said you were lazy and used your calculator to much and now your slow. I am like you, I never really practiced much in math, and I don't do much home work(didn't need to). Hence I am very slow with most math. However, I am pretty smart and I grasp concepts easily, so I learn math easily and I can do most problems, I am just slow some times.

I think that may be a trap people fall in, when doing easy low level math you don't practice enough, because it seems easy. Well the practice isn't to learn the math, it is to master it so you can do it very quickly. So even though you understand it all you didn't master the speed factor that only comes from extensive practice.

So you just need a lot of practice. There are a ton of math games on the internet, I would try to find one that isn't to childish and play that. Or just get out some flash cards and actually study like you would for something really difficult.