This is something you really can’t make people understand. This is the harsh truth of our society that it always humiliates people who are not at speaking English, their mindsets are brewed with such thoughts that if they are good at something they are allowed to humiliate others. 
You should do two things in this case, first, try to ignore such things, and do not bother yourself because of such mindsets. Second, you can try improving your English because it helps in your career too, not to be misunderstood but great positions ask for good & fluent English (speaking as well as writing).