Too many cards in the wallet?

Is there a downside to having a bunch of no-AF fees that you've downgraded that you virtually don't use, or is it better to strategically cancel some? I'm mainly worried about getting to the point where banks are concerned about how much credit they're extending me, even though I never carry a balance on any of my cards and pay off the statement in full every month.