You are right.
Most of my arguments with people about this are done online so I try to think of it as I'm not just convincing the person I'm talking to, I'm trying to convince the people that would be reading our conversation also. I find there are some people that I have come across on this topic have not been shown the real facts about it and some are not as hard core in their beliefs as some open borders advocates are. So I normally talk about how there is only so much money to go around to help these people so if it helps more people (pushing that helping more should be the goal) then keeping them where they are rather than brining them to the EU or the West is better as a whole as it helps so many more people rather than the feel good moments of bring in only some lucky few. I push that it is a selfish idea to do that. I feel like it plants the seed in some of the not so hard core about it.