I think it could be racist in that scenario, but not necessarily. For me its more about context than just what race the person is. I'm sure most people would have at some point heard a Scottish or Irish person who is perfectly fluent in English yet had no idea what they were saying. In that situation, taking the piss out of them and asking if they are speaking the same language is perfectly fine to me.
If you have some Asian guy who is clearly tripping over himself trying to string a sentence together in English and obviously does not have much of a grasp on the language, and you start making fun of their English for that, that's poor taste for me. But not racist (necessarily). Again this is just my opinion.