[FL/US] & [QC/CAN] Hypothetical question, could he keep child in Canada?

We've been together for 16 years now, it's a very smooth relationship in general, very peaceful. I'm 5 months pregnant and while I don't think our relationship is in any trouble, we had an argument a little while ago where he threatened to take the child from me and move back to Canada. We are both currently living in the US(FL) and we are both dual-citizens of US/CAN. The child will be born in the US and probably won't have Canadian citizenship for a few months because of paperwork.

Even though things are fine now I find myself dwelling on that comment/threat and I feel like I should arm myself with knowledge about this topic. Because we will definitely be visiting Canada eventually, the three of us together to visit our family there to meet the new child, and I wonder what could happen if he refused to allow the child to re-enter the US with me.

I'm comfortable and happy in the US (it's where we both primarily grew up) -- I have no beef with Canada, I just can't do the cold weather and our entire life is US-based, bank accounts, jobs, taxes, friends, etc.

He is obsessed with Montreal, specifically only that city, nowhere else in Canada. His mom lives there and he spent a few years there as a kid when he was shuffled around from Mom to Dad. He doesn't speak French and probably is incapable of learning it at this stage in his life. In my opinion he is only into the "cool vibes" and he couldn't care less that he doesn't know the language. Most people speak basic English and he thinks that's good enough. But he would be unable to navigate the increasingly French-only system there. There are so many new laws literally preventing service in English. If our child went to school there, they would not be allowed to go to an English school. I spent 4ish years in Montreal (not winters) with him and in that time it went from "cool city" to a hellish place to live. I couldn't wait to leave. My decision to move back to the US was a major emotional deal for him, as he would much prefer to have stayed in Montreal. I couldn't take it there anymore. Luckily he chose to stay with me and moved with me back to the US, I only found out I was pregnant a few weeks after we finally agreed that we would move when our lease was up. So he's didn't move with me just because I was pregnant, we decided quite a bit before that. I could go on and on about why I feel he would be an unsuitable parent for a child living in an aggressively Francophone region.

I would be devastated if he insisted on living there and kept my child there. I don't think he would have the legal right to do that, but I don't know? I would be happy to visit his family in Montreal if I knew he wasn't legally able to keep the child there for any reason. But right now the thought of going to Canada for a visit kind of scares me, even though it was just one comment in an argument that was a long while ago.

This is a real concern of mine because I know a woman who got caught in a cross-border US-CAN custody dispute. She is from FL also, and now she is trapped and has to live in Alberta, and she hates it but has to live there because her former partner and his family have primary custody of the child she brought there with him "for just a visit"

Since this is all just hypothetical, any information is helpful. Even anecdotes or less concrete stuff like that.

ETA: We are not legally married.

Additional question, can I pre-emptively create an agreement through a lawyer that says even if I consent to a visit to Canada, the child will not stay in Canada after the agreed duration of the visit?