So a few things with this:
Let's imagine a party stood for election on a platform of overtly tightening down immigration within the EU rules, deporting peoples who do not meet the letter of current EU regulations - as is allowed. Now how many of the same people currently on the Remain side of the Brexit argument, would have labeled that policy and party as far right, toxic, racist, xenophobic etc? My bet would be many. I know this is rather stawman, but it seems disingenuous of you to suggest that now. Please forgive if I've got that wrong, and that you would have actually have supported such a policy.
EU members who have tried to tighten their immigration within the EU rules, have faced criticism and backlash in other ways from the EU. See Italy and Austria. Also see Switzerland under the EEA framework arguing over that.
In any case - Whatever can be done with EU immigration, does nothing about non EU immigration. EU member states in being limited in what they can do for EU migration inevitably then discriminate against non EU immigration requests. It's made hugely expensive and very time consuming and difficult for non EU peoples to apply. That to me is wrong.
Why grant easy access to people simply because they were born in an EU member country rather than a non-EU country. I want individuals to be judged as individuals, via a system that applies the eligibility criteria universally. We cannot do that in the EU. But that is actually a lot less racist/xenophobic than the two-tier system we have now.