Isn't there something oddly compelling about the way fabric stretches and folds? Obviously with clothing there's a sexual component, but even the billows in a curtain or the folds of a carelessly heaped blanket can entrance us. It's a common subject for artists, and often considered a greater challenge than the human form or landscapes.
The thing about it, though, is that while it seems to hit us at a very basic level, it's not primal. There's nothing in our evolutionary environment or our animal natures that really should respond to fabric, if you think about it. It's a little like hair and a little like water (like an undulating pond or river), maybe. For me, though, there's a very fundamental satisfaction and almost a kind of drama or meaning in the way light strikes folds in fabric. I suppose it's a bit like getting caught up staring at the clouds, so maybe it just meets this minimum threshold of complexity to engage our pattern-seeking natures.
