Why wouldn’t Gigi be trusted running the crew Tony had him on?

I realize, no one can (or should) trust anyone on this show, but I was thinking about the scene where Tony tells Junior he isn’t sure who should be running the crew, and Junior tells him “you steer the ship the best you can” and that Gigi isn’t taken seriously by his crew. Why exactly would Eugene, Vito, and Donny have more loyalty to Ralph than Gigi? They all go way back or something?