It's been a tool of politicians for centuries pointing out that those not like us might be coming here to steal our scarce resources and I think the Tories have always been on the sharp end of that tool.
The thing I don't get is that we've had roughly Eurosceptic governments since the early 1970s, including the Blair one despite what people say, yet we're always top of the list for implementing EU laws, while the likes of France, Spain and Germany know that they can ignore them until at least the third overdue warning and, even then, they can request an extension until people forget about them having to implement it. There's a reason why many of these foreign nationals will happily travel through many of the other wealthy EU countries to get to us and that's because we're the only ones who'll actually follow the spirit of EU-wide benefits and no work restrictions.
It's also a convenient thing for the last few governments to say "we'd love to not implement this but that evil EU makes us". Voting records show that our MEPs toe the party line quite nicely in voting through almost anything the EU Commission want, or abstaining and having the same effect. The other peer EU nations laugh at our bi-polar attitudes of helpful EU-friendly MEPs from the same parties as the anti-EU soundbite national parties.
Regardless of the rhetoric from Tory HQ, it's simple to address the EU immigration issue: simply take the same real-life attitude as the French, Germans, Italians and Spanish to their working conditions and benefits. Once the message gets out that we're not just saying tough things but actually doing them then there's few reasons for those not wanting to work to come here.