Does anyone else feel like they need to dress fancy in order to carry a fancy handbag?

Okay so I have handbags ranging from Zara and Fossil to Gucci LV and YSL. I find myself gravitating towards the more “down to earth” brands as opposed to the big name brand because on a daily basis I’m a sweatshirt, jeans or sweatpants kinda gal. Don’t even really wear makeup or any fancy jewelry except my wedding ring. I only reach for the fancier bags when going out to a nice dinner every few months. I’d like to get more wear out of them. Where are yall wearing your fancy handbags? Do you feel like an imposter wearing “basic” clothes and carrying fancy bags? Like I don’t feel cohesive. I am always in awe of women who effortlessly wear nice bags with their everyday outfits.