Im gonna take the Freud approach here because im a terrible person...so take it with a grain of salt..

perhaps these are projections from repressed expression of sexuality. We must always be honest and open because we will be who we are. What we can do is find out why we are the way we are and that is done through effective communication with others. We can see reflections of ourselves in others, and perhaps you are afraid of what he might see in your own shadow. What I am saying is there could be some imbalance in your expectation of his perception of you. If you havent already, it might be good to balance the load of secret sharing.