Well first of all...
Anti-matter wouldn't represent negative numbers and matter wouldn't represent positive numbers.
It's the other way around. Numbers represent the objective thing.

Also, I think this kind of makes sense. But not exactly in the way you described it.
I'm thinking more, that everything can be represented by both it's counter-parts.
It's hard to articulate this.

Maybe more like the universe can be explained by everything being 0 and 1-1.

It's not really what I am trying to convey, I'll have to think about how to say it better.