How am I wrong? I said "most." I don't think a majority of men really care.
Ok, we're acting like it's an all or nothing situation. There are varying degrees of caring.

If most men didn't care, they'd furnish their place with a 1960s red velvet sofa from Aunt Martha's basement that smells like cat pee and unfiltered Marlboros and finish it off with a worn out, orange plaid recliner they picked up for $10 at a garage sale. I can't speak for men, but think most would care a little more than that - even on a budget.
I think there's a sliding scale on this matter, and a lot of men fall somewhere in the mid-range.
Example - When my dad leased his last office space before he retired, within a month he said he couldn't stand sitting in that office with glaring white walls another day, so he had a muted navy plaid wallpaper hung, then hung a large print of a ship at sea, then found a large, faux ship's wheel to hang up - and he was done. He cared.
Example - When my son bought a house 5 years ago, he went shopping for furnishings and bought a black leather sofa, a charcoal gray area rug, and some black, gray, and beige plaid throw pillows. He cared. Otherwise, he would have called up Rent-a-Center and told them to just bring over whatever they have for sale in the back room.