Participate time-outs when hosting

Usually I only ever manage to host 1 or 3 star raids because of the wait timer but yesterday I hosted 6 Ho-Oh raids and ran into the same problem in 3 of them and got penalized for it. I would be waiting to get through the 400+ queue and get my chance to host the raid. 5 people join, I get 5 friend requests in the game. I accept the friend requests, jump into the raid and invite the friends. Then I got back to Pokegenie click "Raid Invites Sent" to start the raid and it says somebody timed out (even though they have already added me and are in the raid lobby) so Pokegenie drops them and adds a new person. But by that time I'm in the lobby, I can't back out and accept a new friend request then rejoin the lobby and invite them. So even though we have all the trainers we are supposed to, everybody joined and we won the raid I still get penalized by the random 6th player who shouldn't have been in Pokegenie in the first place.

I get that one answer is to just sit in Pokegenie and not do anything in game until it says everybody has joined but that isn't so easy. When you've been waiting 25 minutes for the lobby to fill and the raid timer is counting down and you have local people you're trying to get into the raid it all happens quite fast once you finally reach the front of the queue. It seems like a weird system that I can get the friend requests and the successfully join the raid lobby and then Pokegenie can say "Nope, just kidding" and switch it.