I understand what you are saying and I share your view of both sides of this coin. Since you dont seem to have any problem with either homosexuals or homophobes. This thread seems to be more about moral perception?

You say that
But logically, that doesn't sound right.

And I understand both these people's point of view and I can empathize with both but like with everything, drastic views are always wrong and both sound ignorant to me.
It sound like you having some trubble with figuring out wich of the two mentalities you should accept as the more moraly accepteble. And stil be able to not denying or condemn the other side? Drastic views might sound wrong but I wouldnt be to quick to say that it have to be the wrong thing to do, even if people often might make wrong choices with such a view.. Also drastic is such a flexible word, that it can be hard to know what drastic implyes to others than oneself.
But since I am just as ignorant, I don't know where to go from here and how to put everything together.
Sound to me like your more curious than ignorant, there is big difference that makes all the difference to the outcome of any dicussion with that curious state of mind. So let's dig into these questions from where we are now.

Do you feel that it is something wrong with you view of understanding both perspectives of homophobes and homosexuals at the same time?

And if that is so. Is it because there something you want to do about it, that makes you wounder about homophobes in particularly?