Why have humans not evolved so that they are all physically attractive?

Just wondering what counters this rather large selection pressure, considering attractiveness is very heritable.