If I recall correctly, they used the totems in Inception as a mean of verifying whether they were awake.
I think the idea was that they were in another person's dream, and that person would not know how the totem should react.
Since however that person had control of the dream, the totem would react like the person thought it should react, instead of how the totem behaved in the real world.