Capital and workers are just the same as goods. If workers or capital move from one country to another then the comparative advantages change. Money is a product. Workers are no different than capital goods.
Also, this is also Macro country level. On a micro level its always better. Since nobody here is claiming to be The Master Planner and control all production of every product, macro level is irrelevant. It is only good for demonstrating the idea. On a micro level it never fails.
1. It is always more profitable for a company to buy the cheapest (of comparable quality) supplies.
2. It is always better for consumers to buy cheaper (of comparable quality) products.
3. It is always better for job seekers to have the highest level and paying jobs available. (Unless they are morons, then its better for them if we had stone age economies but lets just assume average people for now)
All three of these are maximized by completely free exchange of ideas, capital, workers, products.